ci: serve drive-gate page over loopback HTTP + retries (robust on win-CI)
windows-latest headless kept flaking on special-scheme pages: data: URLs get re-normalized (re-nav), about:blank + redundant goto destroys the context, and both can carry a CSP that blocks eval(). Serve the test page over a real http://127.0.0.1 instead — none of those quirks, and it adds real-navigation coverage (await a response). Evaluates stay arrow-functions (no eval), listeners are inline-script (no on* attrs), and transient context-destroyed/detached/timeout gets up to 2 retries. A genuinely broken binary fails all 3 attempts.
This commit is contained in:
+76
-58
@@ -14,47 +14,52 @@ It deliberately covers the failure modes that HISTORICALLY shipped green:
|
|||||||
seed must be per-session, not per-readback)
|
seed must be per-session, not per-readback)
|
||||||
- headless navigator tells → navigator.webdriver falsy, languages
|
- headless navigator tells → navigator.webdriver falsy, languages
|
||||||
non-empty, plugins is a real PluginArray
|
non-empty, plugins is a real PluginArray
|
||||||
|
- real HTTP navigation broken → the page is served over http://127.0.0.1
|
||||||
|
and a `response` is awaited (not data:/about:blank)
|
||||||
|
|
||||||
All of this is headless, NO screenshot → GPU-free (can't false-fail on the
|
All of this is headless, NO screenshot → GPU-free (can't false-fail on the
|
||||||
GPU-less hosted runners), and fully offline → safe in public CI. WebGL
|
GPU-less hosted runners). The HTTP server is loopback-only → no external network,
|
||||||
determinism is intentionally NOT checked here (it needs SWGL and can false-fail
|
no proxy, no secrets → safe in public CI. WebGL determinism is intentionally NOT
|
||||||
headless); it lives in the local proxy realness gate.
|
checked here (needs SWGL, false-fails headless); it lives in the local realness
|
||||||
|
gate, along with the faithful cross-origin iframe test (issue #20 — a same-origin
|
||||||
|
in-gate iframe is a weak proxy AND races Juggler's frame tracking).
|
||||||
|
|
||||||
NOT covered here on purpose:
|
Robustness (learned the hard way, across many runner round-trips):
|
||||||
- Cross-origin iframe (issue #20): a same-origin srcdoc/data iframe is a weak
|
- The page is served over real `http://127.0.0.1:<port>/`. A `data:` URL gets
|
||||||
proxy for it AND races Juggler's frame tracking (the frame re-navigates, its
|
re-normalized (re-navigated) by Firefox, `about:blank` + a redundant goto
|
||||||
id changes → "Frame was detached"). The faithful #20 sentinel is
|
intermittently "destroys the execution context by navigation", and both can
|
||||||
`tests/test_cross_origin_iframe.py` (e2e, two localhost origins); wire that
|
carry a CSP that blocks `eval()`. A plain loopback HTTP page has none of that.
|
||||||
as its own gate job rather than a fragile in-gate check.
|
- Every `page.evaluate` is an ARROW FUNCTION (Playwright callFunction, never
|
||||||
|
eval'd) — immune to a page CSP that blocks eval. Listeners are wired in an
|
||||||
Robustness (learned the hard way):
|
inline <script> on the served page, not via inline on* attributes.
|
||||||
- The DOM is built on `about:blank` via `innerHTML`, NOT a `data:` URL. An
|
- Transient "context destroyed / detached / target closed" gets up to 2 logged
|
||||||
unencoded `data:text/html,...` URL gets re-normalized (re-navigated to its
|
retries (the windows-latest headless runner is interaction-flaky); a
|
||||||
percent-encoded form) by Firefox; on the slower windows-latest runner that
|
genuinely broken binary fails ALL attempts → the gate fails.
|
||||||
async re-nav races the evaluates → "execution context destroyed by
|
|
||||||
navigation". `about:blank` is canonical and never re-navigates.
|
|
||||||
- `set_content` is NOT usable — its document.write is rejected on this build
|
|
||||||
("operation is insecure").
|
|
||||||
- A transient "context destroyed / detached / target closed" still gets ONE
|
|
||||||
logged retry; a genuinely broken binary fails BOTH attempts → gate fails.
|
|
||||||
|
|
||||||
Usage: python ci_drive_gate.py /path/to/firefox[.exe | .app/Contents/MacOS/firefox]
|
Usage: python ci_drive_gate.py /path/to/firefox[.exe | .app/Contents/MacOS/firefox]
|
||||||
Exit 0 + "DRIVE GATE OK ..." on success; non-zero with a reason on failure.
|
Exit 0 + "DRIVE GATE OK ..." on success; non-zero with a reason on failure.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import http.server
|
||||||
|
import socketserver
|
||||||
import sys
|
import sys
|
||||||
|
import threading
|
||||||
|
|
||||||
from playwright.sync_api import sync_playwright
|
# Full page served over loopback http. Inline <script> wires the listeners (no
|
||||||
|
# CSP on our own server, so this is fine); reads below still use arrow functions.
|
||||||
# DOM built on about:blank (no data: URL to re-normalize → no spurious nav).
|
HTML = (
|
||||||
# No inline onclick — inline handlers are CSP-sensitive; we wire the listener
|
"<!doctype html><html><head><title>dt</title></head><body>"
|
||||||
# via addEventListener inside the (function, not eval'd) setup call below.
|
|
||||||
BODY = (
|
|
||||||
"<h1 id=x>hello-drive</h1>"
|
"<h1 id=x>hello-drive</h1>"
|
||||||
"<button id=b>go</button>"
|
"<button id=b>go</button>"
|
||||||
"<input id=inp>"
|
"<input id=inp>"
|
||||||
)
|
"<script>"
|
||||||
|
"window.__clicked=0;window.__moves=0;"
|
||||||
|
"document.getElementById('b').addEventListener('click',function(){window.__clicked=1;});"
|
||||||
|
"window.addEventListener('mousemove',function(){window.__moves++;});"
|
||||||
|
"</script>"
|
||||||
|
"</body></html>"
|
||||||
|
).encode()
|
||||||
|
|
||||||
# Identical 2D draw, evaluated twice in one session. The stealth canvas spoof is
|
# Identical 2D draw, evaluated twice in one session. The stealth canvas spoof is
|
||||||
# seeded per-session (see fingerprint-consistency rule), so two identical draws
|
# seeded per-session (see fingerprint-consistency rule), so two identical draws
|
||||||
@@ -67,29 +72,37 @@ CANVAS_DRAW = (
|
|||||||
|
|
||||||
# Substrings of errors that are transient infra/timing, NOT a broken binary.
|
# Substrings of errors that are transient infra/timing, NOT a broken binary.
|
||||||
_TRANSIENT = ("context was destroyed", "frame was detached", "target closed",
|
_TRANSIENT = ("context was destroyed", "frame was detached", "target closed",
|
||||||
"because of a navigation")
|
"because of a navigation", "timeout")
|
||||||
|
|
||||||
|
|
||||||
def _drive(exe: str) -> str:
|
class _Handler(http.server.BaseHTTPRequestHandler):
|
||||||
|
def do_GET(self): # noqa: N802
|
||||||
|
self.send_response(200)
|
||||||
|
self.send_header("Content-Type", "text/html; charset=utf-8")
|
||||||
|
self.send_header("Content-Length", str(len(HTML)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(HTML)
|
||||||
|
|
||||||
|
def log_message(self, *a): # silence the per-request stderr noise
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _start_server():
|
||||||
|
srv = socketserver.TCPServer(("127.0.0.1", 0), _Handler)
|
||||||
|
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||||
|
return srv, srv.server_address[1]
|
||||||
|
|
||||||
|
|
||||||
|
def _drive(exe: str, url: str) -> str:
|
||||||
"""One full drive attempt. Returns the UA on success; raises on failure."""
|
"""One full drive attempt. Returns the UA on success; raises on failure."""
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
|
||||||
with sync_playwright() as p:
|
with sync_playwright() as p:
|
||||||
browser = p.firefox.launch(executable_path=exe, headless=True)
|
browser = p.firefox.launch(executable_path=exe, headless=True)
|
||||||
try:
|
try:
|
||||||
page = browser.new_page()
|
page = browser.new_page()
|
||||||
page.goto("about:blank") # canonical, never re-navigates
|
resp = page.goto(url, wait_until="load")
|
||||||
# Build the DOM + wire click/mousemove listeners in one shot. Passed
|
assert resp and resp.ok, f"navigation to {url} failed: {resp.status if resp else 'no response'}"
|
||||||
# as a FUNCTION (Playwright callFunction, not eval) so a page CSP that
|
|
||||||
# blocks eval()/inline-handlers can't break the gate. All evaluates
|
|
||||||
# below are arrow functions for the same reason.
|
|
||||||
page.evaluate(
|
|
||||||
"(html) => {"
|
|
||||||
" document.body.innerHTML = html;"
|
|
||||||
" document.getElementById('b').addEventListener('click', () => { window.__clicked = 1; });"
|
|
||||||
" window.__moves = 0;"
|
|
||||||
" window.addEventListener('mousemove', () => { window.__moves++; });"
|
|
||||||
"}",
|
|
||||||
BODY,
|
|
||||||
)
|
|
||||||
|
|
||||||
ua = page.evaluate("() => navigator.userAgent")
|
ua = page.evaluate("() => navigator.userAgent")
|
||||||
webdriver = page.evaluate("() => navigator.webdriver")
|
webdriver = page.evaluate("() => navigator.webdriver")
|
||||||
@@ -119,7 +132,7 @@ def _drive(exe: str) -> str:
|
|||||||
assert "Firefox" in ua, f"unexpected UA (binary not driving correctly): {ua!r}"
|
assert "Firefox" in ua, f"unexpected UA (binary not driving correctly): {ua!r}"
|
||||||
assert text == "hello-drive", f"DOM/JS roundtrip failed: {text!r}"
|
assert text == "hello-drive", f"DOM/JS roundtrip failed: {text!r}"
|
||||||
assert not webdriver, f"navigator.webdriver leaked True (stealth regression): {webdriver!r}"
|
assert not webdriver, f"navigator.webdriver leaked True (stealth regression): {webdriver!r}"
|
||||||
assert clicked == 1, "page.click() did not fire onclick — mouse-event synthesis broken (firefox-2 class)"
|
assert clicked == 1, "page.click() did not fire the click listener — mouse-event synthesis broken (firefox-2 class)"
|
||||||
assert moves >= 1, "page.mouse.move() produced no mousemove — jugglerSendMouseEvent regression"
|
assert moves >= 1, "page.mouse.move() produced no mousemove — jugglerSendMouseEvent regression"
|
||||||
assert typed == "ok", f"page.keyboard.type() failed: {typed!r}"
|
assert typed == "ok", f"page.keyboard.type() failed: {typed!r}"
|
||||||
assert canvas_a == canvas_b, "canvas non-deterministic across identical draws (stealth seed broken → bot tell)"
|
assert canvas_a == canvas_b, "canvas non-deterministic across identical draws (stealth seed broken → bot tell)"
|
||||||
@@ -129,21 +142,26 @@ def _drive(exe: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def main(exe: str) -> int:
|
def main(exe: str) -> int:
|
||||||
|
srv, port = _start_server()
|
||||||
|
url = f"http://127.0.0.1:{port}/"
|
||||||
last = None
|
last = None
|
||||||
for attempt in (1, 2):
|
try:
|
||||||
try:
|
for attempt in (1, 2, 3):
|
||||||
ua = _drive(exe)
|
try:
|
||||||
if attempt > 1:
|
ua = _drive(exe, url)
|
||||||
print(f"(note: drive succeeded on retry {attempt} after a transient error)")
|
if attempt > 1:
|
||||||
print(f"DRIVE GATE OK | UA={ua} | click+mousemove+keyboard+canvas-determinism+navsurface=ok")
|
print(f"(note: drive succeeded on attempt {attempt} after a transient error)")
|
||||||
return 0
|
print(f"DRIVE GATE OK | UA={ua} | http+click+mousemove+keyboard+canvas-determinism+navsurface=ok")
|
||||||
except Exception as e: # noqa: BLE001 — gate: any failure must surface
|
return 0
|
||||||
last = e
|
except Exception as e: # noqa: BLE001 — gate: any failure must surface
|
||||||
msg = str(e).lower()
|
last = e
|
||||||
if attempt == 1 and any(t in msg for t in _TRANSIENT):
|
msg = str(e).lower()
|
||||||
print(f"(transient error on attempt 1, retrying once): {e}", file=sys.stderr)
|
if attempt < 3 and any(t in msg for t in _TRANSIENT):
|
||||||
continue
|
print(f"(transient error on attempt {attempt}, retrying): {e}", file=sys.stderr)
|
||||||
break
|
continue
|
||||||
|
break
|
||||||
|
finally:
|
||||||
|
srv.shutdown()
|
||||||
print(f"DRIVE GATE FAILED: {last}", file=sys.stderr)
|
print(f"DRIVE GATE FAILED: {last}", file=sys.stderr)
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user