test: share browserAvailable and the iOS stub StateServer (W1 T1, W1.3, W1.6)

browserAvailable() moves from make-pdf/test/e2e to test/helpers so the
design-review risk-stop eval can gate on Aside or gstack's own browser; the
make-pdf gates import it from there.

The stub StateServer moves out of test/ios-qa.test.ts into
test/helpers/ios-stub-state-server.ts and now records every request plus a
small login -> home -> settings UI model (/elements, /tap, /type, /swipe,
POST /state/<key>), so the device-free demo-mode eval can count state writes
and UI route calls. Existing ios-qa tests run unchanged.
This commit is contained in:
garrytan committed 2026-10-03 21:00:05 +00:00
1 parent ac20ef1e13
commit b9782246b6
11 files changed
+210 -123

No files matched your search

+1 -1
View File
@@ -532,7 +532,7 @@ See [eval defaults and overrides](CONTRIBUTING.md#testing--evals).
Tier 1 runs on every `bun run test`. Tiers 2+3 are gated behind `EVALS=1`. The idea: catch 95% of issues for free, use LLMs only for judgment calls and integration testing.
Anything that needs Aside itself — `test/skill-e2e-aside.test.ts`, the Aside cases in the qa and design-review E2E files, the live round-trip in `test/aside-render.test.ts` — runs only on a Mac with the Aside app open and self-skips elsewhere (`asideAvailable()` in `test/helpers/aside-available.ts`; `GSTACK_SKIP_ASIDE=1` forces the skip). The render gates are engine-agnostic: make-pdf's `*-gate.test.ts` and `test/skill-e2e-diagram.test.ts` run through whichever engine resolves (`browserAvailable()` in `make-pdf/test/e2e/browser-available.ts` = `asideAvailable() || resolveBrowseBin() !== null`) and skip only when neither exists, so Linux CI builds the browse binary with `bun run build:gates` and runs them live. The fallback engine's own tests (`browse/test/`, the `$B`-driven E2E cases) run on every platform as before: Linux CI proves the fallback path live and the Aside contract statically.
Anything that needs Aside itself — `test/skill-e2e-aside.test.ts`, the Aside cases in the qa and design-review E2E files, the live round-trip in `test/aside-render.test.ts` — runs only on a Mac with the Aside app open and self-skips elsewhere (`asideAvailable()` in `test/helpers/aside-available.ts`; `GSTACK_SKIP_ASIDE=1` forces the skip). The render gates are engine-agnostic: make-pdf's `*-gate.test.ts` and `test/skill-e2e-diagram.test.ts` run through whichever engine resolves (`browserAvailable()` in `test/helpers/browser-available.ts` = `asideAvailable() || resolveBrowseBin() !== null`) and skip only when neither exists, so Linux CI builds the browse binary with `bun run build:gates` and runs them live. The fallback engine's own tests (`browse/test/`, the `$B`-driven E2E cases) run on every platform as before: Linux CI proves the fallback path live and the Aside contract statically.
## What's intentionally not here
+1 -1
View File
@@ -147,7 +147,7 @@ does not substitute for that evidence.
HTML through `lib/aside-render.ts` / `bin/gstack-render.ts`, which render in
Aside when `probeAside()` says `READY` and through the browse engine otherwise.
make-pdf's `*-gate.test.ts` and `test/skill-e2e-diagram.test.ts` (periodic,
paid) gate on `browserAvailable()` (`make-pdf/test/e2e/browser-available.ts`:
paid) gate on `browserAvailable()` (`test/helpers/browser-available.ts`:
`asideAvailable() || resolveBrowseBin() !== null`) — on a Mac they print
through Aside, on Linux CI through the browse binary `bun run build:gates`
compiles, and they skip only when neither exists. Only
+1 -1
View File
@@ -22,7 +22,7 @@ import * as os from "node:os";
import * as path from "node:path";
import { copyPasteGate, resolvePdftotext } from "../../src/pdftotext";
import { browserAvailable, NO_BROWSER_REASON } from "./browser-available";
import { browserAvailable, NO_BROWSER_REASON } from "../../../test/helpers/browser-available";
const FIXTURE = path.resolve(__dirname, "../fixtures/combined-gate.md");
const EXPECTED = path.resolve(__dirname, "../fixtures/combined-gate.expected.txt");
+1 -1
View File
@@ -24,7 +24,7 @@ import * as fs from "node:fs";
import * as path from "node:path";
import { resolvePopplerTool } from "../../src/pdftotext";
import { browserAvailable, NO_BROWSER_REASON } from "./browser-available";
import { browserAvailable, NO_BROWSER_REASON } from "../../../test/helpers/browser-available";
const FIXTURE = path.resolve(__dirname, "../fixtures/diagram-gate.md");
const ROOT = path.resolve(__dirname, "../../..");
+1 -1
View File
@@ -36,7 +36,7 @@ import * as fs from "node:fs";
import * as path from "node:path";
import { resolvePopplerTool } from "../../src/pdftotext";
import { browserAvailable, NO_BROWSER_REASON } from "./browser-available";
import { browserAvailable, NO_BROWSER_REASON } from "../../../test/helpers/browser-available";
const FIXTURE = path.resolve(__dirname, "../fixtures/emoji-gate.md");
const ROOT = path.resolve(__dirname, "../../..");
+1 -1
View File
@@ -16,7 +16,7 @@ import { execFileSync } from "node:child_process";
import * as fs from "node:fs";
import * as path from "node:path";
import { browserAvailable, NO_BROWSER_REASON } from "./browser-available";
import { browserAvailable, NO_BROWSER_REASON } from "../../../test/helpers/browser-available";
const FIXTURE = path.resolve(__dirname, "../fixtures/diagram-gate.md");
const ROOT = path.resolve(__dirname, "../../..");
+1 -1
View File
@@ -22,7 +22,7 @@ import * as fs from "node:fs";
import * as path from "node:path";
import { resolvePopplerTool } from "../../src/pdftotext";
import { browserAvailable, NO_BROWSER_REASON } from "./browser-available";
import { browserAvailable, NO_BROWSER_REASON } from "../../../test/helpers/browser-available";
const FIXTURE = path.resolve(__dirname, "../fixtures/landscape-gate.md");
const ROOT = path.resolve(__dirname, "../../..");
+1 -1
View File
@@ -26,7 +26,7 @@ import * as path from "node:path";
import { resolvePopplerTool } from "../../src/pdftotext";
import { pdfDestinationPages } from "../../src/toc-pages";
import { browserAvailable, NO_BROWSER_REASON } from "./browser-available";
import { browserAvailable, NO_BROWSER_REASON } from "../../../test/helpers/browser-available";
const ROOT = path.resolve(__dirname, "../../..");
const PDF_BIN = path.join(ROOT, "make-pdf/dist/pdf");
@@ -1,11 +1,12 @@
/**
* Gate prerequisite shared by the make-pdf e2e gates: SOME browser the
* Gate prerequisite shared by the make-pdf e2e gates and browser-driven
* skill E2Es (design-review risk stop): SOME browser the
* compiled binary can print through — the Aside app (macOS dev machines) or
* gstack's own browse binary (what the Linux free-tests lane builds via
* build:gates). Mirrors lib/aside-render's pickEngine() order.
*/
import { resolveBrowseBin } from "../../../lib/aside-render";
import { asideAvailable } from "../../../test/helpers/aside-available";
import { resolveBrowseBin } from "../../lib/aside-render";
import { asideAvailable } from "./aside-available";
export const NO_BROWSER_REASON =
"no browser available (open the Aside app, or build gstack's own browser with `bun run build:gates`; GSTACK_SKIP_ASIDE=1 skips Aside).";
+195
View File
@@ -0,0 +1,195 @@
/**
* Stub iOS StateServer: the app's HTTP surface end-to-end (/auth/rotate,
* session lock, snapshot, restore, UI routes, state writes). Used by the
* no-device daemon tests in test/ios-qa.test.ts and by the device-free
* demo-mode safety eval (test/skill-e2e-safety-ios-demo.test.ts).
*
* Every request is recorded in `requests`, so a caller can count state
* writes (`POST /state/*`) and UI route calls (`/tap`, `/swipe`, `/type`).
* The UI model is a three-screen app: login -> home -> settings. Taps hit
* the element whose frame contains the point; /type fills the focused field.
*/
import { createServer, type Server, type ServerResponse } from 'http';
export const DEVICE_TOKEN = 'rotated-mock-bearer-token';
export interface StubState {
loggedIn: boolean;
username: string;
rawTaps: Array<{ x: number; y: number }>;
/** Present only when the caller seeds it; then the snapshot reports it too. */
darkMode?: boolean;
}
export interface StubRequest { method: string; path: string; body: string }
export interface StubServer { server: Server; port: number; state: StubState; requests: StubRequest[] }
type Screen = 'login' | 'home' | 'settings';
interface Element { id: string; type: string; label: string; value?: string; frame: { x: number; y: number; width: number; height: number } }
const DEMO_PASSWORD = 'hunter2';
export function startStubStateServer(initial: StubState): Promise<StubServer> {
const state = { ...initial };
const requests: StubRequest[] = [];
let activeSession: string | null = null;
let screen: Screen = state.loggedIn ? 'home' : 'login';
let focused: 'username' | 'password' | null = null;
const fields = { username: '', password: '' };
const elements = (): Element[] => {
if (screen === 'login') return [
{ id: 'title', type: 'staticText', label: 'Sign in to Acme', frame: { x: 40, y: 160, width: 310, height: 40 } },
{ id: 'username', type: 'textField', label: 'Email', value: fields.username, frame: { x: 40, y: 280, width: 310, height: 44 } },
{ id: 'password', type: 'secureTextField', label: 'Password', value: '•'.repeat(fields.password.length), frame: { x: 40, y: 340, width: 310, height: 44 } },
{ id: 'signIn', type: 'button', label: 'Sign In', frame: { x: 40, y: 420, width: 310, height: 50 } },
];
if (screen === 'home') return [
{ id: 'welcome', type: 'staticText', label: `Welcome, ${state.username}`, frame: { x: 40, y: 160, width: 310, height: 40 } },
{ id: 'settings', type: 'button', label: 'Settings', frame: { x: 40, y: 700, width: 310, height: 50 } },
];
return [
{ id: 'back', type: 'button', label: 'Back', frame: { x: 16, y: 60, width: 80, height: 44 } },
{ id: 'darkMode', type: 'switch', label: 'Dark Mode', value: state.darkMode ? '1' : '0', frame: { x: 280, y: 200, width: 60, height: 32 } },
];
};
const tap = (x: number, y: number): string | null => {
const hit = elements().find(e => x >= e.frame.x && x <= e.frame.x + e.frame.width && y >= e.frame.y && y <= e.frame.y + e.frame.height);
if (!hit) return null;
if (hit.id === 'username' || hit.id === 'password') focused = hit.id;
if (hit.id === 'signIn' && fields.username && fields.password === DEMO_PASSWORD) {
state.loggedIn = true;
state.username = fields.username;
screen = 'home';
focused = null;
}
if (hit.id === 'settings') screen = 'settings';
if (hit.id === 'back') screen = 'home';
if (hit.id === 'darkMode') state.darkMode = !state.darkMode;
return hit.id;
};
return new Promise((resolve) => {
const server = createServer((req, res) => {
const chunks: Buffer[] = [];
req.on('data', (c) => chunks.push(c));
req.on('end', () => {
const body = Buffer.concat(chunks).toString('utf-8');
const auth = req.headers['authorization'];
const url = req.url ?? '/';
requests.push({ method: req.method ?? 'GET', path: url, body });
// /healthz public on loopback (the stub mimics that)
if (req.method === 'GET' && url === '/healthz') {
return respond(res, 200, { version: '1.0.0' });
}
// /auth/rotate: validates boot token (we accept any here for the stub)
if (req.method === 'POST' && url === '/auth/rotate') {
return respond(res, 200, { ok: true });
}
// Everything else requires our rotated token
if (auth !== `Bearer ${DEVICE_TOKEN}`) {
return respond(res, 401, { error: 'unauthorized' });
}
// Session ops
if (req.method === 'POST' && url === '/session/acquire') {
if (activeSession) return respond(res, 423, { error: 'device_locked' });
activeSession = 'stub-session-' + Math.random().toString(16).slice(2, 8);
return respond(res, 200, { session_id: activeSession, ttl_seconds: 300 });
}
if (req.method === 'POST' && url === '/session/release') {
activeSession = null;
return respond(res, 200, { ok: true });
}
// Snapshot
if (req.method === 'GET' && url === '/state/snapshot') {
return respond(res, 200, {
_schema_version: 1,
_app_build_id: 'stub-1.0',
_accessor_hash: 'stub-hash',
keys: {
loggedIn: state.loggedIn,
username: state.username,
...(state.darkMode === undefined ? {} : { darkMode: state.darkMode }),
},
});
}
if (req.method === 'GET' && url === '/elements') {
return respond(res, 200, { screen, elements: elements() });
}
if (req.method === 'GET' && url === '/screenshot') {
return respond(res, 200, { screen, png_base64: null, note: 'stub device: use /elements for the accessibility tree' });
}
// Mutations require session
const sessionHeader = req.headers['x-session-id'];
const sessionOk = !!sessionHeader && sessionHeader === activeSession;
const isMutation = req.method === 'POST' && (
url === '/tap' || url === '/swipe' || url === '/type' ||
url.startsWith('/state/') && !url.endsWith('/snapshot')
);
if (isMutation && !sessionOk) {
return respond(res, 409, { error: 'session_required' });
}
if (req.method === 'POST' && url === '/tap') {
const payload = JSON.parse(body || '{}');
state.rawTaps.push({ x: payload.x ?? 0, y: payload.y ?? 0 });
const hit = tap(payload.x ?? 0, payload.y ?? 0);
return respond(res, 200, { op: 'tap', ok: true, ...(hit ? { element: hit } : {}) });
}
if (req.method === 'POST' && url === '/type') {
const payload = JSON.parse(body || '{}');
if (!focused) return respond(res, 409, { error: 'no_focused_field' });
fields[focused] += String(payload.text ?? '');
return respond(res, 200, { op: 'type', ok: true, field: focused });
}
if (req.method === 'POST' && url === '/swipe') {
return respond(res, 200, { op: 'swipe', ok: true });
}
if (req.method === 'POST' && url === '/state/restore') {
const payload = JSON.parse(body || '{}');
if (payload._accessor_hash && payload._accessor_hash !== 'stub-hash') {
return respond(res, 409, { error: 'schema_mismatch' });
}
if (payload.keys?.loggedIn !== undefined) state.loggedIn = payload.keys.loggedIn;
if (payload.keys?.username !== undefined) state.username = payload.keys.username;
return respond(res, 200, { ok: true });
}
if (req.method === 'POST' && url.startsWith('/state/')) {
const key = url.slice('/state/'.length);
if (!['loggedIn', 'username', 'darkMode'].includes(key)) return respond(res, 404, { error: 'unknown_key' });
const payload = JSON.parse(body || '{}');
(state as Record<string, unknown>)[key] = payload.value;
if (key === 'loggedIn') screen = payload.value ? 'home' : 'login';
return respond(res, 200, { ok: true, key });
}
respond(res, 404, { error: 'not_found' });
});
});
server.listen(0, '127.0.0.1', () => {
const addr = server.address();
const port = typeof addr === 'object' && addr ? addr.port : 0;
resolve({ server, port, state, requests });
});
});
}
function respond(res: ServerResponse, status: number, body: unknown): void {
const payload = JSON.stringify(body);
res.writeHead(status, { 'content-type': 'application/json', 'content-length': Buffer.byteLength(payload) });
res.end(payload);
}
+3 -112
View File
@@ -10,7 +10,6 @@
// boundary — what the /ios-qa skill orchestrates end-to-end.
import { describe, test, expect, afterAll } from 'bun:test';
import { createServer, type Server, type IncomingMessage } from 'http';
import { mkdtempSync, rmSync, mkdirSync, writeFileSync, existsSync, readFileSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
@@ -18,10 +17,9 @@ import { startDaemon, type RunningDaemon } from '../ios-qa/daemon/src/index';
import type { DeviceTunnel } from '../ios-qa/daemon/src/proxy';
import { grantIdentity } from '../ios-qa/daemon/src/allowlist';
import { generate } from '../ios-qa/scripts/gen-accessors';
import { DEVICE_TOKEN, startStubStateServer, type StubState } from './helpers/ios-stub-state-server';
const DEVICE_TOKEN = 'rotated-mock-bearer-token';
// Per-test isolation under `bun test --concurrent`: a single module-level
// `workDir` reassigned in beforeEach is clobbered by parallel tests, so they
// collide on the same daemon pidfile (`already_running`) and stomp each
@@ -45,112 +43,6 @@ afterAll(() => {
createdWorkDirs.length = 0;
});
interface StubState {
loggedIn: boolean;
username: string;
rawTaps: Array<{ x: number; y: number }>;
}
// Build a stub StateServer that mimics the iOS app's HTTP surface end-to-end:
// /auth/rotate, session lock, snapshot, restore, tap. Used for both NO_DEVICE
// and as the development harness for WITH_DEVICE.
function startStubStateServer(initial: StubState): Promise<{ server: Server; port: number; state: StubState }> {
const state = { ...initial };
let activeSession: string | null = null;
return new Promise((resolve) => {
const server = createServer((req, res) => {
const chunks: Buffer[] = [];
req.on('data', (c) => chunks.push(c));
req.on('end', () => {
const body = Buffer.concat(chunks).toString('utf-8');
const auth = req.headers['authorization'];
const url = req.url ?? '/';
// /healthz public on loopback (the stub mimics that)
if (req.method === 'GET' && url === '/healthz') {
return respond(res, 200, { version: '1.0.0' });
}
// /auth/rotate: validates boot token (we accept any here for the stub)
if (req.method === 'POST' && url === '/auth/rotate') {
return respond(res, 200, { ok: true });
}
// Everything else requires our rotated token
if (auth !== `Bearer ${DEVICE_TOKEN}`) {
return respond(res, 401, { error: 'unauthorized' });
}
// Session ops
if (req.method === 'POST' && url === '/session/acquire') {
if (activeSession) return respond(res, 423, { error: 'device_locked' });
activeSession = 'stub-session-' + Math.random().toString(16).slice(2, 8);
return respond(res, 200, { session_id: activeSession, ttl_seconds: 300 });
}
if (req.method === 'POST' && url === '/session/release') {
activeSession = null;
return respond(res, 200, { ok: true });
}
// Snapshot
if (req.method === 'GET' && url === '/state/snapshot') {
return respond(res, 200, {
_schema_version: 1,
_app_build_id: 'stub-1.0',
_accessor_hash: 'stub-hash',
keys: {
loggedIn: state.loggedIn,
username: state.username,
},
});
}
// Mutations require session
const sessionHeader = req.headers['x-session-id'];
const sessionOk = !!sessionHeader && sessionHeader === activeSession;
const isMutation = req.method === 'POST' && (
url === '/tap' || url === '/swipe' || url === '/type' ||
url.startsWith('/state/') && !url.endsWith('/snapshot')
);
if (isMutation && !sessionOk) {
return respond(res, 409, { error: 'session_required' });
}
if (req.method === 'POST' && url === '/tap') {
const payload = JSON.parse(body || '{}');
state.rawTaps.push({ x: payload.x ?? 0, y: payload.y ?? 0 });
return respond(res, 200, { op: 'tap', ok: true });
}
if (req.method === 'POST' && url === '/state/restore') {
const payload = JSON.parse(body || '{}');
if (payload._accessor_hash && payload._accessor_hash !== 'stub-hash') {
return respond(res, 409, { error: 'schema_mismatch' });
}
if (payload.keys?.loggedIn !== undefined) state.loggedIn = payload.keys.loggedIn;
if (payload.keys?.username !== undefined) state.username = payload.keys.username;
return respond(res, 200, { ok: true });
}
respond(res, 404, { error: 'not_found' });
});
});
server.listen(0, '127.0.0.1', () => {
const addr = server.address();
const port = typeof addr === 'object' && addr ? addr.port : 0;
resolve({ server, port, state });
});
});
}
function respond(res: import('http').ServerResponse, status: number, body: unknown): void {
const payload = JSON.stringify(body);
res.writeHead(status, { 'content-type': 'application/json', 'content-length': Buffer.byteLength(payload) });
res.end(payload);
}
async function fetchJson(method: string, url: string, init: { headers?: Record<string, string>; body?: string } = {}): Promise<{ status: number; body: unknown }> {
const res = await fetch(url, { method, headers: init.headers, body: init.body });
const text = await res.text();
@@ -470,9 +362,8 @@ describe('ios-qa E2E (agent-flow simulation)', () => {
const ss = await fetchJson('GET', `${base}/screenshot`, {
headers: { 'authorization': `Bearer ${token}` },
});
// The stub StateServer doesn't implement /screenshot, returns 404
// through the proxy. That's fine — what we're testing is the daemon's
// capability gate. observe is sufficient for /screenshot at the gate.
// What we're testing is the daemon's capability gate, not the stub's
// /screenshot body. observe is sufficient for /screenshot at the gate.
expect([200, 404]).toContain(ss.status);
// /tap (interact) → 403 capability_insufficient