Garry Tan
943105f109
* test: delete test-infrastructure dead code (G) - exit-propagation drives the runner's real strict verdict (BunTestOutputClassifier + strictTestExitCode); delete the unused shardRunLooksTruncated predicate. - delete skill-coverage-matrix registry + its gate (nothing reads it; the floor already iterates skillCensus()). - delete touchfiles-facade export-parity tests (Bun fails missing imports at link time) and the duplicated E2E_TIERS tier-value test. - delete brain-cache-spec TRANSPORT_DEFAULT_POLICY, SKILL_RUN_RETENTION_DAYS and the now-unused BrainTrustPolicy type with their literal tests. AUTOPLAN_PREFLIGHT_BUDGET_BYTES stays: skill-preflight-budget enforces it against real resolver output. - delete audit-compliance's JSDoc-comment grep. * test: replace product tests that fake the product with real-boundary tests (F) - design: serve.test.ts drove an inline mirror server; now two tests run the real serve() on an ephemeral port (reload confinement, submit exit 0). - setup-gbrain: rollback + voyage tests execute the template-extracted init blocks (3 sites) instead of drifted local bash copies. - terminal-agent: internalHandler source greps replaced by a behavioral /internal/grant + /internal/revoke auth matrix (no/wrong/valid token). - /health: server-security-surface and the server-auth / security-audit-r2 / sidebar-tabs source greps fold into one liveness-only check on the real body; the L4 sidecar wiring gets a behavioral /pty-inject-scan test. - delete tautologies (browser-manager onDisconnect, memory-command #12), ios swiftui tap fixture self-check, memory-ingest put_page grep, detach source greps, sidebar-agent absence pins, dead-CSS pins + the dead CSS, security-audit-r2 Task 1 + the test-only meta-commands re-export, duplicate generated-SKILL.md checks. - make-pdf coverage-gaps cases move into their owner test files. * test: delete tests of dead eval code (A) - A1: the retired Eng lexical oracle (evaluateEngSeedCoverage, isEngSeedDecisionAUQ), the completion-handoff detector and the retained corpus had no paid caller since v1.87.6; delete their 26 replay files, ~2.6k helper LOC and fixtures, and the dead blocks in 8 mixed files (live hasNativePlanTerminal / batching assertions stay). - A2: dead viewport approvers in autoplan-artifact-permission and their 11 replay files + fixtures; recorder/launcher cases stay. - A3: never-wired oracles and seeders (autoplan-phase-order, eng-finding-fixture, ceo-paired-fixture, design-ui-scope, plan-skill-completion, pty-current-screen, required-reads, transcript-section-logger); plan-seed-submission now decodes through the production createPtyScreen; section manifests name their actual guard. - A4: zero-reference helper exports, plus execGit and invokeAndObserve found by the reachability pass. - 52 fixtures orphaned by the deletions; touchfile and selection-table entries for every deleted path. * test: clean up the paid eval lane (B1-B4, B6, B7) - B1: delete paid files that assert nothing or cannot pass meaningfully: skill-llm-eval-spec and skill-e2e-spec-execute (test.todo), gemini-e2e (+ gemini-session-runner; no gemini CLI in CI), ship-idempotency (red since v1.63), the two opus-4-7 *-sonnet overlay wrappers, conductor-prose (+ its source-evaluation replay), codex-e2e-plan-format; drop their keys, scripts and census rows. - B2: skill-llm-eval grades browse/sections/command-list.md with one union judge that also carries the baseline score pin; regression-vs-baseline deleted (paid run: pass, c4/c4/a4). - B3: memory-pipeline, ios-qa, ios-qa-swift-build and plan-tune-cathedral make no model calls; renamed out of the paid glob so they run on every PR. Swift builds need GSTACK_TEST_SWIFT=1; device stub deleted. - B4: codex-e2e*, outside-voice, aside and ios-device cannot run in the CI image; excluded from the weekly lane with a tracked re-entry condition. - B6: fold opus-47's negative routing controls into skill-routing-e2e journey-negatives (paid run: 3/3 unrouted) and delete the file. - B7: delete the never-green brain-privacy-gate eval; a free gstack-skill-start test now proves consent precedes artifacts egress. * test: retire the finding-count cluster and trim its helpers (C) - C0/C1: the five never-green evals (skill-e2e-autoplan-chain and skill-e2e-plan-{ceo,eng,design,devex}-finding-count) failed on harness and budget, never on skill behavior; delete them, their touchfile/tier ids, AUTOPLAN_CHAIN_BUDGET and the dedicated eighth periodic slice (--slices 7). - C2: delete the helper groups whose only paid consumers were those files (11 modules), trim claude-pty-runner and eng-seeded-coverage to the paid closure, and delete the free replay tests whose assertions exercised only that dead code (89 files, 135 orphaned fixtures). Blocks that used dead code only as input for a live subject keep their assertions: the multiSelect default moved to plan-review-decisions, runner PTY tests use inline caller policies, and the timer-safe budget checks moved to eng-finding-retry-budget. - The eight production-touching files stay except ceo-current-decision-record (its template read only feeds the retired counter). - CARVE_GUARDS.autoplan is behavioral 'none'; TODOS records the lost chain and per-finding cadence coverage with their re-entry tests. * test: fold per-incident replay series into their detector owners (D) Twelve detector families move into one owner test each: 73 incident files become describe blocks in ceo-section-loading-fixture (stale-fill race), model-overlays, coverage-audit-evidence, autoplan-phase-observer, native-auto-decide, outside-voice-evidence, eng-first-review, plan-count-completion, plan-count-file-permission, ceo-mode-option, plan-scope-selection and plan-count-prerequisite. Each block keeps its original code and fixture, so every case still runs; only tests asserting the incident file's own touchfile registration are dropped (41). Touchfile lists that named an incident now name its owner. * test: start the plan-count history PTY on its readiness marker (H) The fake CLI prints a startup marker and the runner waits for it instead of the fixed 8 s startup sleep (8.6 s -> 0.9 s locally). eng-semantic-terminal's sleeping registration cases went with C; plan-count-timeout keeps the fixed wait because it asserts deadline behavior. * test: derive paid touchfiles from each eval's static closure (E) touchfiles.test.ts now checks, per key, that the paid file's static test/helpers and test/fixtures closure (plus fixture paths it names in string literals) is covered, and names the file, path, chain and key to fix when it is not. Free *.test.ts files are no longer touchfiles, so editing a free replay test stops selecting paid evals: 950 entries removed, 653 real closure paths added. The hand-copied inventories go: periodic-fixture-selection, fake-impeccable-touchfiles and 45 per-file selection examples. Selection for the sample edits (plan-eng-review template, claude-pty-runner, plan-count-fixture, gstack-config) loses no case under either profile. CONTRIBUTING documents the rule and its lower bound. * test: skip hollow tier shards and census judges in the paid planner (B5) A paid file is now skipped for a tier lane only when every E2E id it registers is known statically and none has that tier; ids come from the touchfile registrations and literal testName/*IfSelected arguments, so a comment or skill path that quotes another id cannot unschedule it, and computed names keep today's scheduling. --list and the manifest show each skip as "skipped: no E2E_TIERS id has tier <tier>". The weekly gate census drops the LLM judges (--skip-judges); they still run in the periodic census and PR gate lanes. Gate lane 52 -> 42 files, census 41; periodic 77 -> 69. * test: run seven paid evals on the current default capture model (B8) skill-e2e-{auq-matrix,plan-format,qa-bugs,retro,workflow} pinned claude-opus-4-7 and skill-e2e-office-hours plus -brain-writeback pinned claude-sonnet-4-6; none tests a historical model, so they now capture with resolveEvalModel('capture'), and the free harness tests that execute these registrations receive the same resolver. The paid re-pin run passed all of them. skill-e2e-{design,office-hours-phase4,plan-prosons,plan} keep claude-opus-4-7: six of their cases failed on the default model (three timeouts, a missing report file, a format miss and a posture score of 3), so per the plan's fallback they keep their pins with a TODOS entry. The pre-spend estimate and drop threshold are in docs/test-audit-2026-09.md. * test: guard the reduced suite against new test-of-test files - test/test-of-test-ratchet.test.ts records the 228 free tests that import only test/ code and fails on a new one, naming the owner test to extend instead; a stale baseline entry fails with the remove instruction. - test/helpers/resolve-repo-path.ts is the one specifier/literal resolver for the ratchet and the touchfile closure invariant, with its own unit tests. - CONTRIBUTING "Test tiers" describes the paid-failure workflow (fix, then one row in the detector's owner test) and the ratchet; TEST_PORTFOLIO gains the detector -> owner-test table and no longer claims an Autoplan chain eval. - TODOS: automatic exclusion policy for chronically red periodic files (P3), the deferred native-completion table collapse, the unused CEO payment seeder; the PTY readiness item is narrowed to the paid runner. - docs/test-audit-2026-09.md collects the triage, security mapping, inventories, selection proof, behavior-commit decisions and retained false positives. * v1.91.8.0 test: smaller suite, derived paid selection, retired never-green evals Release metadata for the test-reduction branch: VERSION 1.91.8.0 (1.91.7.0 is claimed by #2983), CHANGELOG with the measured before/after table and a contributor section, durations re-recorded on Ubicloud standard-16 (857 files, 0 failures), the agents digest, CONTRIBUTING's after-measurement row, the B8 fallback TODOS entry, and the after metrics, kept-vs-plan notes, B8 run and census estimate in docs/test-audit-2026-09.md. * fix(ubicloud): skip retrieval globs that match nothing instead of reporting a failed pull * test: count issue-numbered Design findings in the UI-scope gate eval
11 KiB
Browser / sidebar / server internals
Scope: this is the internals reference for gstack's own browser engine — the
browsedaemon, GStack Browser, the sidebar extension. That engine is the automatic fallback when the Aside browser is not installed or not running (Linux, Windows, a closed Aside app); Aside is what every skill drives first. The Aside contract lives inscripts/resolvers/aside.tsand BROWSER.md.
Moved verbatim from CLAUDE.md (token-load reduction). These are the
load-bearing invariants for browse/src/server.ts, the Chrome extension,
the sidebar PTY, SSE endpoints, CDP sessions, and the sidebar security
stack. Every rule here is additionally pinned by a CI tripwire test named
in its paragraph.
Sidebar architecture: Before modifying sidepanel.js, background.js,
content.js, terminal-agent.ts, or sidebar-related server endpoints,
read docs/designs/SIDEBAR_MESSAGE_FLOW.md. The sidebar has one primary
surface — the Terminal pane (interactive claude PTY) — with
Activity / Refs / Inspector as debug overlays behind the footer's
debug toggle. The chat queue path was ripped once the PTY proved out;
sidebar-agent.ts and the /sidebar-command / /sidebar-chat /
/sidebar-agent/event endpoints are gone. The doc covers the WS auth
flow, dual-token model, and threat-model boundary — silent failures
here usually trace to not understanding the cross-component flow.
Embedder terminal-agent ownership (v1.42.1.0+, identity-based kill v1.44.0.0+).
buildFetchHandler in browse/src/server.ts accepts ServerConfig.ownsTerminalAgent?: boolean (default true). When true, factory shutdown acquires
acquireAgentStateLock(stateDir) and checks daemon ownership. It removes
terminal-port, terminal-internal-token, and the matching terminal-agent-pid
record only when the recorded agent is absent, already dead, or confirmed stopped
by stopAgentByRecord. Uncertain identity or exit, an unavailable lock, or
successor-owned daemon state leaves those files intact.
Embedders (e.g. the gbrowser phoenix overlay) that pre-launch their own PTY
server must pass false so their discovery files survive gstack teardown cycles.
The flag is the third caller-owned teardown gate in ServerConfig (alongside
xvfb? and proxyBridge?); polarity is inverted (explicit bool vs presence) and
documented in the field's JSDoc. CLI start() always passes true explicitly —
the static-grep test in browse/test/server-embedder-terminal-port.test.ts fails
CI if a refactor drops it. Pre-v1.44 used pkill -f terminal-agent\.ts (regex
match) which would kill sibling gstack sessions on the same host; the new
browse/test/terminal-agent-pid-identity.test.ts static-grep tripwire fails CI
if any source file re-introduces pkill ... terminal-agent or spawnSync('pkill', ...).
WebSocket auth uses Sec-WebSocket-Protocol, not cookies. Browsers
can't set Authorization on a WebSocket upgrade, but they CAN set
Sec-WebSocket-Protocol via new WebSocket(url, [token]). The agent
reads it, validates against validTokens, and MUST echo the protocol
back in the upgrade response — without the echo, Chromium closes the
connection immediately. Set-Cookie: gstack_pty=... is kept as a
fallback for non-browser callers (the cross-port SameSite=Strict
cookie path doesn't survive from a chrome-extension origin).
Cross-pane PTY injection. The toolbar's Cleanup button and the
Inspector's "Send to Code" action both pipe text into the live claude
PTY via window.gstackInjectToTerminal(text), exposed by
sidepanel-terminal.js. No /sidebar-command POST — the live REPL is
the only execution surface in the sidebar now.
/health MUST NOT surface any token — and it no longer does (v1.63+).
The historical headed-mode leak of AUTH_TOKEN is fixed: GET /health is
liveness/status only in every mode. Token bootstrap is POST /extension-token,
which validates the caller's Origin against the pinned extension identity
(the key field in extension/manifest.json pins the extension ID —
GSTACK_EXTENSION_ID in browse/src/server.ts, derivation reproducible via
bun browse/scripts/extension-id.ts) plus a loopback Host. PTY auth still
flows through POST /pty-session only. Don't add any token to /health.
Transport-layer security (v1.6.0.0+). When pair-agent starts an ngrok tunnel,
the daemon binds two HTTP listeners: a local listener (127.0.0.1, full command
surface, never forwarded) and a tunnel listener (locked allowlist: /connect,
/command with a scoped token + 26-command browser-driving allowlist,
/sidebar-chat). ngrok forwards only the tunnel port. Root tokens over the tunnel
return 403. SSE endpoints use a 30-minute HttpOnly gstack_sse cookie minted via
POST /sse-session (never valid against /command). Tunnel-surface rejections go
to ~/.gstack/security/attempts.jsonl via tunnel-denial-log.ts. Before editing
server.ts, sse-session-cookie.ts, or tunnel-denial-log.ts, read
ARCHITECTURE.md —
the module boundary (no imports from token-registry.ts into sse-session-cookie.ts)
is load-bearing for scope isolation.
Unicode sanitization at server egress (v1.38.0.0+). Every server egress that
ships page-content-derived strings MUST go through JSON.stringify(payload, sanitizeReplacer) for object payloads or sanitizeLoneSurrogates(body) for text
bodies. Lone UTF-16 surrogate halves from CDP page content otherwise reach the
Anthropic API as \uD800-style escapes and trigger a 400. Wired at four egress
points today: handleCommandInternal (HTTP + batch via a sanitizing wrapper around
handleCommandInternalImpl) and both SSE producers (/activity/stream,
/inspector/events). Post-stringify regex is a no-op — JSON.stringify has
already escaped the surrogate before regex could match, so the replacer must run
inside the encoding pipeline. Before adding a new SSE/WebSocket writer or HTTP
response in server.ts, read
ARCHITECTURE.md.
browse/test/server-sanitize-surrogates.test.ts pins the wiring with invariant
tests, so bypasses fail CI.
SSE endpoint helper (v1.51.0.0+). New SSE endpoints in server.ts MUST route
through createSseEndpoint(req, config) from browse/src/sse-helpers.ts. The
helper owns the cleanup contract (abort + enqueue-throw + heartbeat-throw, all
idempotent) and bakes in sanitizeLoneSurrogates on every JSON.stringify, so
new subscribers can't accidentally regress either invariant. Inline
ReadableStream wiring leaked subscribers when the TCP connection died without
firing req.signal.abort (Chromium MV3 service-worker suspend, intermediate
proxy half-close). /activity/stream, /inspector/events, and /memory
(SSE-eligible) all route through it. browse/test/sse-helpers.test.ts pins the
cleanup contract.
CDP session lifecycle (v1.51.0.0+). Direct page.context().newCDPSession(page)
calls outside browse/src/cdp-bridge.ts fail CI via the static-grep tripwire in
browse/test/cdp-session-cleanup.test.ts. Use withCdpSession(page, async (s) => {...})
for one-shot CDP work (try/finally detach) or getOrCreateCdpSession(page, cache)
for cached sessions tied to a page's lifetime (close-detach via Map<page, session>).
Three sites migrated: cdp-bridge frame events, write-commands archive capture,
cdp-inspector. The helpers prevent the per-session leak class where successful-path
detach happened but error-path detach was missed.
Setup symlink hardening (v1.38.0.0+). Every link site in setup MUST route
through the _link_or_copy SRC DST helper near the IS_WINDOWS detection. On
Windows without Developer Mode, plain ln -snf produces frozen file copies that
don't refresh on git pull — silent staleness across every host adapter. The
helper preserves ln -snf on Unix and switches to cp -R / cp -f on Windows.
test/setup-windows-fallback.test.ts enforces a static invariant: a single raw
ln call outside the helper body fails CI. Windows users get a one-line note
from _print_windows_copy_note_once reminding them to re-run ./setup after
every git pull.
Sidebar security stack (layered defense against prompt injection):
| Layer | Module | Lives in |
|---|---|---|
| L1-L3 | content-security.ts |
server + read path — datamarking, hidden element strip, ARIA regex, URL blocklist, envelope wrapping |
| L4 | security-classifier.ts (TestSavantAI ONNX) |
security sidecar subprocess only (security-sidecar-entry.ts, driven by security-sidecar-client.ts from server.ts) |
| Canary | security.ts (generate/inject/detect) |
pure utilities — no production injector today (the chat prompt-builder that injected them was ripped) |
| Combiner | security.ts (combineVerdict + THRESHOLDS) |
pure, tested; retains transcript/deberta vote handling for LayerSignal inputs no live layer produces anymore |
History note: an L4b Haiku transcript classifier and an opt-in DeBERTa ensemble
(GSTACK_SECURITY_ENSEMBLE=deberta) existed until the chat-path agent that
invoked them was ripped; both were deleted as dead code (zero production
callers). Do not re-document them as live.
Critical constraint: security-classifier.ts CANNOT be imported from the
compiled browse binary. @huggingface/transformers v4 requires onnxruntime-node
which fails to dlopen from Bun compile's temp extract dir — hence the sidecar
subprocess. Only security.ts (pure-string operations — canary utilities,
verdict combiner, status) is safe for server.ts. See
~/.gstack/projects/garrytan-gstack/ceo-plans/2026-04-19-prompt-injection-guard.md
§"Pre-Impl Gate 1 Outcome" for the original architectural decision.
Thresholds (in security.ts): BLOCK: 0.85, WARN: 0.75, LOG_ONLY: 0.40,
SOLO_CONTENT_BLOCK: 0.92 (label-less content classifiers can't distinguish
"injection" from "phishing aimed at the user", so their solo bar is higher).
The live L4 path applies these in server.ts's sidecar-scan handling; canary
leak always BLOCKs (deterministic).
Env knobs:
GSTACK_SECURITY_OFF=1— emergency kill switch. Classifier stays off even if warmed; the L1-L3 filters keep running.- Classifier model cache:
~/.gstack/models/testsavant-small/(112MB, first run only) - Attack log:
~/.gstack/security/attempts.jsonl— written bytunnel-denial-log.ts(tunnel-surface rejections; rotates at 10MB, 5 generations)
History note (#2557): the cross-process session state
(~/.gstack/security/session-state.json), getStatus(), the /health
security field, and the sidepanel SEC shield were all removed — the state
file lost its only writer when sidebar-agent.ts was ripped, so the shield
reported a permanent 'inactive' or a stale false-green 'protected' from
leftover disk state. The live defenses (L1-L3 filters, L4 sidecar on the
inject-scan path) report through their own call sites, never through
/health. browse/test/extension-token.test.ts pins the removal on the real
/health body and browse/test/pty-inject-scan.test.ts pins the live L4
wiring behaviorally. Do not re-document these as live.