From 55f8151fbe77c5504f1e5a6fa6cee313c539b154 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Tue, 6 Oct 2026 11:07:50 -0700 Subject: [PATCH] v1.91.30.0 fix: honest eval failure reports, Bun floor with safe auto-update, make-pdf and /review on Codex (#3033) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs: approved reliability follow-ups plan (Oct 2026) * D1: score overlay wrong answers as comparison results (contract v4) A completed answer that fails the fixture's output check is now a valid measurement with taskCorrect false and answerError set, in either arm. Only execution, scope and metric failures invalidate a trial. ON correctness still gates the case, so an OFF wrong answer stays in the comparison. OVERLAY_CONTRACT moves to v4; v3 verdicts are not reinterpreted. Replays the 37198445662 sonnet trials (OFF-0 wrong). * C4: score auq-matrix substance with a 3-sample judge panel Each skill keeps one native capture in the same execution; the recommendation-substance grader now runs as a JUDGE_PANEL_SAMPLES panel on that capture and gates on the mean against the unchanged minimum of 4. A failed sample fails the panel without resampling. Approved at the gate 2026-10-04 for censuses after this change. * E1: one Bun version source; setup refuses below the 1.3.3 floor, updater checks before advancing bin/gstack-bun-version.sh holds the security floor (1.3.3) and the tested version (1.4.0, the CI pin and engines.bun). setup refuses Bun below the floor before writing anything, warns between floor and tested, warns and continues on an unparseable version, and prints a 1.4.0 install hint. Prereleases sort below their release (semver). gstack-session-update now fetches, reads the incoming release's floor and checks the bun setup will run before fast-forwarding to that exact revision; below it the update is held with reason bun-too-old and a resume command, leaving the revision and installs untouched. Fixture copies of setup (team-mode, claude-code migration) now include the sourced file; the slow-git lock test sleeps on fetch, the updater's network step. * E3: record the owned Chromium's PID over CDP so the #2709 reap runs on Playwright 1.62 Playwright 1.62's Browser has no process(), so chromiumProcInfo was always null and browse stop never reaped a surviving headless Chromium. The owned headless launch now reads the browser PID from CDP SystemInfo.getProcessInfo (bounded at 2s; a hung or failed call logs once and records nothing). The reaper's start-time and cmdline identity checks are unchanged, so a recycled PID or an externally owned browser is never signalled. * A1: ship docs gate records post-child hashes with gstack-docs-candidate, saves each record once Stored transcripts (37198445662 gate-census-2) show the parent's reads of release files take ~1 s; the parent's own model time dominates (completion 268 s of 392 s, store ~270 s of 387 s), and 45-78 s of it is retyping hash maps, child output and records into several artifacts after the child returns. The child contract is unchanged. * A2: /plan-ceo-review sends the mode handoff line before any other tool call plan-ceo-mode-routing's HOLD SCOPE session spends its 240 s post-selection window on model work: 37176837432 skipped the handoff helper and paraphrased the mode, 37198445662 ran it, then wrote the plan, loaded sections and probed Codex for 114 s before the posture chat. The handoff line is the posture. * B3: accept wrapped unchanged rows in the cropped Edit preview PR run 37176835584 timed out with the floor's owned Edit menu on screen: unchanged diff rows soft-wrap under their gutter with a blank marker column, and a trimmed empty unchanged row keeps only its number. The parser accepted wrapped +/- rows only, so it never verified the owned edit. Both forms are now parsed and still compared with the exact source lines; the captured viewport is replayed with negative controls. * A3: record per-phase headroom diagnosis for docsync, plan-mode-no-op, mode-routing and gate slice 11 * E4: /review Step 1 keeps going when the base fetch fails and reports stale base coverage Under Codex's read-only .git, 'git fetch && merge-base && diff --stat' stopped at the fetch, so the empty output read as 'Nothing to review'. The fetch is now separate, prints BASE_REFRESH fresh or stale with the local base revision, and the review states stale coverage. * D2: reword the format-approach question prompt Both plan-ceo-review-format-approach refusals in census 37195203538 (Fable 5.1 model_refusal_no_fallback, category reasoning_extraction: "duplicating model outputs") were anchored on the session prompt, whose capture instruction asked for the "verbatim" question and "the exact prose you would have shown" in a "format-capture test". The approach case now asks for the same question in the same file as the step's output, without that wording. The other three plan-format cases keep their unchanged prompt (no refusals in 72 trials). * C2: /qa defines its outcome filename fields; QA report templates name the measured timing fields Low judge samples in failing panels named both: qa workflow (37186854666) 'an undefined {user} in the outcome filename', and qa-only workflow (37198445662) 'template asks for Duration / durationMs totals while reporting.md mandates Probe budget and Guarded command time'. The qa union stays under its 1.11 size ratio (93292 of 93294 bytes). * 0.1: session ledger written by the claude -p, Agent SDK, PTY and Codex runners Each armed model session appends one best-effort line to $GSTACK_EVAL_DIR/session-ledger.jsonl: a stable session key, the timeout it armed, monotonic elapsed time, its end reason (completed, session_timeout, observer_timeout, api_error, refusal, aborted, error), one sanitized evidence line and, for stream-JSON runners, a compact liveness summary (longest in-request silence with nothing outstanding, last event kind, tool open at end). Partial messages now stream for every claude -p session and for SDK sessions without a custom query provider (runSharedInteractive opts in); their rows feed only the summary, so transcripts keep their size unless publicStreamDiagnostics is set. Structured refusal and API-error signals win over phrases, and tool results are never read for them. * 0.2: failure_cause beside failure_class, from runner facts and the session ledger One pure failureCauseOf() beside failureClassOf() sets failure_cause (contract > pre_turn_infra > api_error > refusal > provider_stall > session_timeout > observer_timeout > assertion > unknown; no machine-set 'detector') with one bounded evidence line, and failureDetailOf() keeps a Bun matcher's Expected/Received or a judge's failing dimensions in failure_detail. Both record builders (classifyTrialShard, junitCensus) call them and attach the trial's ledger sessions. failure_class, error and TRIAL_ERROR_MAX are computed exactly as before; the new fields are optional, bounded and validated only when present, and a frozen copy of the v1 reader accepts every new record. A provider stall needs partial messages and STALL_WINDOW_MS (120 s, eval-budgets.ts beside HEADROOM_WARN/FAIL) of in-request silence with nothing outstanding; it stays a failure under EVAL_POLICY v1. * E4: fingerprint under a read-only .git via a private object dir owned by review-log Codex's sandbox mounts .git read-only, so gstack-wtree's git add -A could not write blobs and /review lost its start capture. gstack-review-log now creates a private 0700 object directory under TMPDIR, passes it to gstack-wtree (which writes new objects there with the common-dir object store, resolved before exporting, as an alternate, so linked worktrees work), exposes it to the bindReview/checkSharedLibsReuse consumers that read the tree and blobs, and removes it on EXIT/INT/TERM after they finish. gstack-wtree prints one stderr line and exits 1 on a failed write or unwritable TMPDIR instead of printing a partial hash; review-log no longer swallows that line. Standalone gstack-wtree keeps writing to the repository store so git diff between saved snapshots (/ship) still works. * 0.3: red lines name their values, evidence and an after-a-repair command; one published-text sanitizer Each red line in the census report keeps its stored error and adds the machine cause with its evidence, the failure_detail values (Expected/Received, or each failing judge dimension with its mean, sample count and the sample sentence naming it), the slice artifact and shard directory that hold the transcript, a fetch command (eval:pass-rates --run --case ), and 'after a repair:' with a command that selects only that case (E2E ids via --case, judges via their selection and an exact test-name filter). A session that completed and failed a check says so instead of 'assertion (success)'. Report text that reaches a PR comment, issue or committed file goes through scripts/lib/published-text.ts: control and bidi characters stripped, credential-scan patterns redacted, mentions neutralized, HTML and link brackets escaped inline, blocks in a fence longer than any backtick run, length capped. Verdicts and collector verdict objects are unchanged. * E2: one make-pdf resolver for readiness and every $P block on env-var hosts make-pdf exported $P from its readiness block, which Codex and the other env-var hosts lose between blocks. binaryAssignment gains 'make-pdf': the MAKE-PDF SETUP readiness check and the runtime prelude both use it, so every block that runs $P derives the same make-pdf/dist/pdf binary, with MAKE_PDF_BIN (dev builds) winning when it is executable. Claude output keeps its repo-local-first probe; only the false 'subsequent blocks' comment changed. Root plus P=... fits the 400-byte prelude budget; the fixed MAKE_PDF_OVERRIDE segment is the one exemption, named in runtime-root.ts and the conformance test. * 0.1: eval:pass-rates --headroom, --reds and --run report views; 85% headroom alarm in --gate A new pure module, scripts/lib/eval-history.ts, reports per case the slowest armed session against the timeout it armed (censored when it timed out, insufficient below 3 samples, unknown without a session ledger, with the case wall shown separately as an upper bound), the census critical path from the slice jobs' timings, the verdict red ledger per census by failure class and machine cause with shrunk per-case rates, and the all-green probability (an approximation: the product of 1 - p_i, with an interval). --run lists one census's reds with their values and history and downloads only the slice artifacts those reds name, reporting any over 64 MB instead of skipping them. eval-flake-rank.ts only routes flags: --help works offline, an unknown flag or case id exits 2 before any fetch, --case narrows the JSON and the alarms, every view prints its run/branch scope, and --gate also alarms on a session above HEADROOM_FAIL (85%) of its budget. * E5: GSTACK_DESIGN_IMAGE_MODEL override; the live model check always tests the defaults The image_generation tool model gains an env override validated against a conservative gpt-image- pattern; a bad value fails before any request and the error names the variable. Pairing and rejection hints name the tool model actually requested. design/scripts/live-model-check.ts now ignores GSTACK_DESIGN_MODEL and GSTACK_DESIGN_IMAGE_MODEL (and says so), so an override cannot mask a retirement of a default, and its exit 2 (no key) says nothing was checked. * C3: /deslop-shared-libs names a concrete risk for each recommendation, never 'none' Both shared-libs-opportunity-judgment reds (37193478719 t3, 37195203538 t1) failed adoption_and_risk because the top recommendation said 'Main risk: none found inside the repo' / 'none in behavior'. The skill asked to 'name the main risk or uncertainty' without saying what counts. * B4: --case selects loop-registered cases; an unknown id exits before any process caseSelection() resolves the one paid file that registers a case and how a trial selects it: by its literal Bun test name, by the id a registration loop names its test at runtime (coverage audits, TPA, journeys), or by running the whole file when it registers no other case (plan-mode-no-op, whose test names are not the id, so the old name pattern ran nothing). A whole-file case runs one file shard per trial and is judged by shard status, as CI judges it. --list prints the selection. Every E2E_TIERS id now resolves except carve-section-loading, which fifteen files register (CI runs each as its own shard) and which still fails with the owning files named. * 0.3: after-a-repair runs the shard file when --case cannot resolve a case A case that several files register (carve-section-loading) cannot be selected by --case; its red line now prints the bun test command for the shard file that went red, so every printed command selects exactly one case. * C1: pre-register workflow-judge control bundles for qa, qa-only, review and setup-deploy Positive (production bundle), negative (headings only) and one targeted execution-blocking defect per judge, one 3-sample panel each, built with the eval's own reader, prompt builder, options and floors. * F: cost_known: a trial whose harness captured no billing says so; totals never hide it cost_usd stays a finite number. Trial and eval records gain an optional cost_known boolean (absent means known, so old records still parse; the validator accepts only a boolean). Session-ledger rows record whether the session's terminal result carried total_cost_usd; PTY and Codex sessions never do, and SDK captures without a terminal result do not. A trial is cost-known only when it has eval records and every record and session captured billing, so JUnit-only PTY cases (125 of 298 trials in run 37198445662 recorded 0) now read cost unknown. The census headline prints the known sum plus the unknown count, never a bare total over unknown costs, and no total when every cost is unknown; the history import keeps the flag. PTY transcripts are not priced: test/helpers/pricing.ts estimates exclude cache writes, so they cannot give the billed cost. * B1: replay corpora for the five repeat-offender cases shared-libs-plan-callers, plan-ceo-section-loading, auto-decide-preserved, plan-design-review-plan-mode and ship-docsync-late-result each get an inventory of every wave census in scope (all artifacts still available) and minimal, redacted detector or actor inputs replayed through the real detector in the free suite: census passes, every distinct census red, and derived negative controls labelled as derived. Fixed false reds stay as expected passes with their history; the two genuine auto-decide product failures (37176837432, 37179171083) stay refused. A contract test bounds each entry at 64 KB, scans it for HIGH credentials, requires a pass and a refusal per case, and reports entries whose series identity is stale. Building the auto-decide corpus exposed a latent native detector misread: a bold 'Mode application and rationale.' heading was parsed as a withdrawn Mode field (37186854666). A field status now ends at its sentence. * B2: read approved stale-fill decisions from more valid ledger records The structured ledger reader is authoritative before the prose fallback, but it missed two valid approved records in passing censuses (37186854666: a camelCase writeProfile subject; 37195203538: a slash-joined 'D1 / STALE-FILL' row id), which then rested on the prose detector that caused both historical reds. Both forms are accepted; an unresolved row, an unrelated decision, the copied fixture plan and non-write camelCase words stay refused. * C1: record workflow-judge control results (negatives separated; qa and review miss a removed step) * C2: record every low workflow-judge sample in the wave censuses with its disposition * 0.3: published lines cannot break a fixed fence evals.yml wraps collector-outcomes failures in a fixed three-backtick fence for the PR comment, so those lines (and eval:pass-rates output meant for the tracking issue) go through sanitizeFixedFenceLine: cleaned like every published line, with backticks turned into quotes. Inline published text also neutralizes backticks. * A1: fit the post-child record wording under ship's 1.428 parity size cap * E4: inline the Step 1 base refresh to fit review's 1.199 parity size cap * B2: inventory of gate and periodic phrase-only detector checks docs/evals/detector-inventory.md lists every pass/fail decision in gate and periodic E2E cases that rests on phrases in model free text, with its structured source (if any), whether the contract is user-visible, and a disposition: already structured-first, convert candidate, phrase witness required, or no structured source. Convert candidates are ordered by census red history. * 0.2: failureCauseOf takes the stall window instead of importing it; the PTY ledger never throws eval-store.ts imported STALL_WINDOW_MS from eval-budgets.ts, which free tests replace with mock.module stubs (review-finalization-budget, office-hours, qa-fix-loop, qa-only), so loading eval-store failed there. failureCauseOf is now dependency-free: the record builders pass STALL_WINDOW_MS. A PTY session whose handle cannot show its text (test doubles) records no panel instead of throwing from runPtySession's cleanup. The 37186854666 replay also runs through classifyTrialShard. * A2/auto-decide: the CEO handoff chat repeats the helper's printed line verbatim Claude Code 2.1.284 collapses the helper's Bash output ('Ran 1 shell command'), so the 'Auto-decided review mode → …' line reaches the user only through chat. In lane D's auto-decide-preserved corpus every passing capture's chat starts with that line, and the reds paraphrased it ('HOLD SCOPE was auto-selected…', 'Mode decision done…', "I'm defaulting to HOLD SCOPE…"). The template now says verbatim and why. * C2: record the judge panels rerun on the final ship and review wording * integrate: touchfiles and module-size ratchet entries for the wave's new modules GLOBAL_TOUCHFILES gains session-ledger.ts, published-text.ts and auq-substance-panel.ts; bin/gstack-bun-version.sh joins every entry that lists setup or the updater; auq-matrix stays a rule case with its C4 evidence beside E2E_KINDS; eval-history, published-text and session-ledger are registered new modules. * E1: BROWSE SETUP install hint pins the tested Bun (1.4.0) * E2: quote "$P" in make-pdf commands so install paths with spaces work * E1: /gstack-upgrade checks the incoming Bun floor before pulling gstack_bun_incoming_hold (bin/gstack-bun-version.sh) is now the one pre-advance check: the auto-updater calls it before its fast-forward and /gstack-upgrade's git path calls it after fetching. Below the incoming release's floor the upgrade stops with BUN_TOO_OLD and the checkout, VERSION and installs stay untouched. Vendored installs were already safe: setup refuses before writing and the swap restores the previous install. * 0.3: only main posts or closes tracking issues; reports carry alarms, red ledger, headroom and all-green A branch-dispatched periodic or marathon run writes its report to the step summary and a report artifact; the upsert and close steps run only on refs/heads/main. The periodic body now keeps every pass-rate alarm line (the old grep kept only the header), adds the red ledger and session headroom read with an explicit --branch, the all-green estimate and the triage guide link, and routes report text through scripts/lib/published-text.ts. The marathon lane gains the matching close-on-green step. Migrated: evals-workflow-wiring pinned report-summary.md in the upsert step; the body is now written by 'Write the census report'. * C3: prosons neutral-posture detector ignores a negated "taste call" Both plan-review-prosons-neutral-neg reds (37179171083 t3, 37193478719 t3) recommended A and closed with "a coverage call, not a taste call"; the old /taste call/i matched the negation. The detector moves to a helper so the two census captures replay free as negatives, with positive controls. * B2: credit the CEO mode handoff only when the chat shows its line Claude Code 2.1.284 collapses the gstack-ceo-mode-handoff Bash result, so the printed AUTO_DECIDE line is user-visible only when a later chat message of the session begins with it. The native detector now requires that witness; the helper's tool result locates it but no longer replaces it, and a generic mode statement cannot stand in once a handoff ran. Census 37182865432 is relabelled a genuine miss (expected: refused): the chat said "I'm defaulting to HOLD SCOPE…" and "Mode decided: HOLD SCOPE". Corpus replay: 5 credited, 3 refused. ceo-mode-handoff tests migrated to the witness rule (deliberate behavior change). * test: baseline the wave's approved detector and harness replay tests B1 corpora, B3, C3, C4 and D2 replay captures through paid-case detectors, actors or fixtures (test code by design); each entry carries its reason. * E6: live Codex multi-block periodic case in the host-run Codex job test/codex-e2e-multiblock-live.test.ts runs ./setup --host codex once from the job's checkout into a fresh HOME and CODEX_HOME (auth.json copied from the CI login home), asserts codex debug prompt-input lists the gstack router as a real file, seeds one learning, and runs /learn's Show recent block through codex exec: that later block must root itself through its prelude, run the installed gstack-learnings-search and print the seeded learning; no block may report a missing install or an unresolved gstack helper path. The file joins CODEX_CI_FILES so the planner gives it to the host-run Codex job. paid-test-durations.json carries a 180 s packing estimate (refreshed by --write-durations after the first census) so the plan keeps every slice within max-parallel. Oracles live in test/helpers/codex-multiblock.ts with free replays in test/codex-multiblock.test.ts. Registration (E2E_TOUCHFILES, E2E_TIERS periodic, E2E_KINDS rule) is the integrator's. * E5: design live-model check as a periodic case test/skill-e2e-design-model-smoke.test.ts runs design/scripts/live-model-check.ts (default models only) under a temp HOME. Without OPENAI_API_KEY the case is a Bun skip that the census lists as SKIPPED with no coverage credit; with a key, exit 0 passes and any other exit fails with the script output (exit 2 under a present key fails as 'nothing was checked'). paid-test-durations.json carries a 90 s packing estimate. Registration (E2E_TOUCHFILES, E2E_TIERS periodic, E2E_KINDS rule) is the integrator's. * A1: docsync parent points children at a saved interface file The /ship docsync fixture saved its observation interface only in the parent prompt and told the parent to copy it into every child prompt, about 30-40 s of parent generation per session (lane S, 37198445662). docsSessionOptions now writes it to /fixture-interface.md (0600) and the parent tells each child to Read that file first. Fault-actor prompts, which prepare already writes to a file, are unchanged. Fixture-only. * A3: a JUnit case's recorded wall is capped by its shard's wall junitCensus summed a case's JUnit test times as duration_ms; for plan-mode-no-op's concurrent PTY tests that is 346 s inside a 176 s shard (the "432-613 s" history). The case wall headroom shows is now min(sum, shard elapsed), a true upper bound. Session headroom already used the ledger's per-session clocks. * A3: refresh paid-test durations from census 37198445662 Recorded 2026-09-29 estimates under-packed gate slice 11 (17 files estimated 1078 s ran 1353 s; plan-review-report 76 s -> 198 s, plan-ceo-plan-mode 37 s -> 112 s, safety-codex-consult 83 s -> 150 s). Refreshed both tiers with --report --write-durations from the latest census's slice walls. Old plan re-costed with observed walls: worst slice ~677 s (slice 11); refreshed plan: every slice estimated <= 539 s. * C2: Browser Quick gets a feasible budget; QA loop names its directory and stop path Judge rationales on the qa-only panels (2 of the last 3) cited the 30 s Browser Quick budget: homepage + 5 targets, each a checkpoint write and a guarded command, left ~5 s per probe. Quick is now 180 s (method text: 3 minutes); the qa-only-no-fix paid check expects the same native deadline. PROBE_DIR points at its source line, a stopped loop goes to the final report steps (annotations, materialize, Markdown), and /review and /ship say that after smoke expiry only plan checks rerun in revalidation. * C2: name the flows judges found implicit in review resolvers Structured outside review: MISSING COVERAGE asks no fix question and is never reported clean (was "preserve the existing user decision flow"). /review's HIGH-impact plan gate: spawned or non-interactive runs report REQUIREMENTS MISSING and continue, writing no TODOs. Value cards: the byte clamp applies to card lines; JSON records you write keep full values. * C1: a free test catches a removed workflow step, which the judges missed C1 controls showed the qa workflow judge passing /qa without Phase 10 and the review judge passing /review without Step 3. The new test pins every step/phase heading of qa, qa-only, review, ship, plan-ceo-review, plan-eng-review and setup-deploy (derived from the templates' step headings) and names any that goes missing or out of order; new step headings must be pinned. The judge rubric is unchanged (EVAL_POLICY). * size: fit the wave's wording under ship, review and qa parity caps Measured after all lane merges: ship -79, review -210, qa -68 bytes over. Trimmed the added wording first (post-expiry rule, MISSING COVERAGE, plan-gate default, value-card JSON, stop line) and folded /review's redundant INFORMATIONAL sentence and findings bullet. ship 268,036 (room 8), review 130,118 (room 1). qa needed 11 more bytes: cap 1.11 -> 1.1102 with the measured 93,305 / 84,049 = 1.11013 and its reason beside it. * docs: troubleshooting sections for the wave's new messages Bun floor (setup, auto-update, /gstack-upgrade), /review stale base and fingerprint temp-dir errors, the design image model override, the unrecorded Chromium PID, and a contributor section for census report lines (pass-rates flags, failure causes, headroom, cost unknown, overlay v4, stale corpora, auq substance). A free test checks every troubleshooting.md#anchor printed by setup or bin scripts exists. * TODOS: reliability follow-ups deferrals; close the Bun floor, make-pdf $P and #2709 reap * 0.4: 'A census went red' guide; docs for causes, session ledger, headroom and new messages docs/evals/census-red.md walks one red line to a diagnosis: prerequisites, the report fields, eval:pass-rates --run/--reds/--headroom with expected output, what unknown/insufficient/unrecorded data means, free replay, and the single 'after a repair' paid run allowed by TESTING_INTERNALS. Linked from CONTRIBUTING, TESTING_INTERNALS and both census reports. TESTING_INTERNALS documents the failure_cause glossary (one row per FAILURE_CAUSES value, diagnostic only under EVAL_POLICY v1), failure_detail, sessions[], cost_known, the session ledger, the pass-rates views and the 85% headroom alarm, main-only tracking issues, overlay contract v4, auq-matrix's 3-sample substance panel and the E5/E6 periodic cases. troubleshooting.md gains #bun-too-old and an anchor per new user-facing message from lanes P, D and F. README names GSTACK_DESIGN_IMAGE_MODEL. test/census-red-docs.test.ts pins the glossary to FAILURE_CAUSES, parses every guide command and checks the links. * v1.91.20.0: CHANGELOG and version for the reliability follow-ups * regen: SKILL.md and sections for all hosts, digest, context-budget fixture bun run gen:skill-docs --host all; goldens and llms.txt re-rendered unchanged; bun test/helpers/capture-context-budget.ts. * integrate: register codex-multiblock-live and design-model-smoke (periodic, rule) * 0.4: census report and eval:pass-rates views link the census-red guide * test: cookie approval fixture reverses the Bun install-hint pin (E1) The historical approved cookie-workflow prompt carried BUN_VERSION=1.3.10; the regenerated install hint now pins 1.4.0, so the fixture's reversal list gains that edit like the earlier waves' entries. * test: release floor counts the two new periodic cases (E5, E6) * fix: /freeze, /guard, /unfreeze and /investigate work on Codex and other env-var hosts Templates quoted "$HOME/.claude/skills/gstack/...", which the env-var hosts' rewrites turned into "$HOME/.agents/skills/gstack/..." on Codex (a path a global install never has; same class on the other hosts), and setup's runtime roots had no freeze/bin. Quoted runtime paths now use {{RUNTIME_ROOT}}: $GSTACK_ROOT on env-var hosts (the fence gets the runtime prelude), the unchanged $HOME path on Claude. _link_runtime_dists links freeze/bin and the careful/bin helper it sources into every env-var host's root. Also fixes the same guess in /benchmark-models, /retro and /diagram. test/env-host-runtime-assets.test.ts fails on the previous code: no fence guesses a $HOME//skills/gstack path, every $GSTACK_ROOT path a fence uses is in the host's runtime root, and the installed Codex /freeze and /unfreeze fences run from a fresh shell. Template-reading tests now read the Claude render or expect the placeholder (deliberate change). * CHANGELOG/TODOS: lane W's workflow, cases and guide; the env-var host freeze fix * E6: a block that reads a SKILL.md is not a missing install The first live codex-multiblock-live run passed its real check (the /learn block found the install and printed the seeded learning) but the oracle flagged a sed of the learn SKILL.md, whose prelude source quotes the 'gstack: no install found (tried $_r)' text. Only an emitted line that starts with the message and names a real root counts now; free replay negative added. * test: qa-only fixture keeps its caller-set 30 s deadline The qa-only-no-fix fixture tells the model to start the guard with 30 s (a caller limit), so its deadline assertion's default stays 30 000 ms; the product's Browser Quick default (180 s) does not apply to it. Reverts the helper and its test to their base. * C2: keep PROBE_DIR 'owned' (qa-functional-prompt pins it); qa measured 93,311 * test: paid file count and periodic floor include the two new periodic cases * test: ship docs freshness pin checks meaning after A1's wording trim The snapshot still hashes selected release paths (--select narrows them), generated outputs and docs; the pin now checks that instead of the pre-trim sentence. * E3: reap the recorded Chromium only after the daemon has exited With the PID now recorded (E3), browse stop reaped Chromium while the daemon was still shutting down; the daemon read that as a crash, exited 1 and left its state file (chromium-profile-isolation's headless stop test failed 3/3 locally, passes on base). The CLI waits up to 5 s for the daemon to exit before reaping a survivor. * regen: qa and qa-only exploratory sections (PROBE_DIR wording) * B1: batching counter reads a target declared mid-line; replay corpus GSTA-23's census 37228573062 failed plan-eng-multi-finding-batching with reviewCount=0 after 975 s although D1-D10 were asked one per finding, each with its saved decision-ledger record. The report wrote 'Report destination requested by the user. Review target (fixed): `PLAN.md` ...' and the counter read target fields only at a line start, so no brief inherited the plan identity. A target field may now open any sentence of its line; a mid-sentence mention or another file still gives no identity. Floor (3), budget and kind unchanged. Corpus: 9 earlier passing censuses, the red (now credited at D5), and a derived batching refusal; 975 s noted in the headroom diagnosis. * B1: shared-libs corpus labels its git path and covers the supported shape Every census entry was recorded in CI's eval image (git 2.43), where gstack-safe-git falls back because --no-lazy-fetch is rejected; each entry now says so (git_path: fallback). Two synthetic supported-path entries (HEAD and a history citation in the brief, as git >= 2.44 produces) keep the 37193478719 pass and the 37195203538 refusal, and a test asserts both shapes are covered and the actor reads no git-path-specific text. * v1.91.21.0: move off v1.91.20.0 (claimed by #3031) * test: settle nav-guard commands before asserting; make two new tests Windows-safe navigation-guard: Bun's expect(promise).rejects slows each Playwright round-trip to ~1s while it waits (50 page.evaluate: 43ms plain vs 46.6s inside the matcher), so the click's 5s timeout expired before the guard saw the link-local request (CI: 'click: Timeout 5000ms exceeded ... waiting for scheduled navigations'). Settle the command first, then assert on the refusal; a source check keeps it so. env-host-runtime-assets: PROBE_FILES now matches either path separator. detector-corpus ship-docsync-late-result: JSON-escape the substitution so a Windows temp path's backslashes do not break JSON.parse. * fix: CI run/job ids are not phone numbers; context-recovery fixture gets the full runtime bin pii.phone.e164 redacted the run id in the published eval report's fetch command (--run 37235771700), breaking the command it prints. Digit-only spans in a run/job flag, gh run argument, actions/runs/ or /job/ URL segment, or run_id/job_id key are now exempt; the same digits elsewhere still report. context-recovery-artifacts: the fixture's ./bin/ lacked gstack-skill-start and gstack-context-recovery (the preamble's runtime scripts since v1.71), so the agent spent its 10 turns hunting for them and hit error_max_turns with no final message (run 37235771700, 11 turns). Link the whole runtime bin; diagnostic rerun passed in 6 turns. * test: replay run 37237194905's late-result capture through the docs authority detector The only failure was call 20's write to a mistyped shard root (3GanyY for 3GanyU), which the model disclosed; the detector is correct and the A1 post-snapshot --out under the fixture home stays permitted. Named red recorded in TODOS. * test: keep docsync-authority in the Windows profile The replay test's literal helper path tripped the Windows curator's shebang-spawn pattern, dropping the whole file from the Windows profile (test-free-shards invariant). The snapshot --out case is already covered by docsync-command-grammar, so it is removed here, and the Linux-path replay skips on win32. * fix: gstack-docs-candidate --help works; docs gate red lines name the denying tool failure PR run 37239016858 ship-docsync-completion: the parent ran 'gstack-docs-candidate --help' (the helper's usage error points at --help, which failed). That undeclared command made docsToolFailures non-empty, which denied the atomic-replacement proof of the child's legitimate native Edit, so the red line blamed the Edit's temp file. A replay of the capture proves the atomic replace once --help is declared. - the helper prints its usage for --help/-h and exits 0 - the fixture interface and command grammar admit the helper's bare --help - the atomic-proof denial names its check and the tool failures behind it (deny labels renumbered to their lines) - replay with negative controls (undeclared command, temp left behind, renamed elsewhere); diagnostic rerun of ship-docsync-completion passed * test: retro-base-branch fixture uses main and a shallow report path PR run 37241493377: the agent wrote retro-output.md one directory above the requested /retro-base/retro-output.md (it dropped the retro-base segment), so the report check found nothing. The fixture also ran plain git init, which makes master on CI's git while the prompt says the repo is on main; the agent logged that mismatch as a learning. Init on main and use the temp dir itself as the repo. Diagnostic rerun passed (18 turns, 203s). * test: qa-quick sets a caller probe limit and drains past its session budget C2 raised Browser Quick's native budget to 180 s. qa-quick is the only gate case on the native budget, and its wall went 194 s (main periodic, pre-C2) -> 254 s and 274 s (this PR's runs) -> 300 s session timeout (run 37242691913): the agent now probes every navigation target with a checkpoint, then needs ~80 s to report. Like qa-only-no-fix, the eval now sets a caller probe limit (30 s), scopes the fixture, and gives bun's outer timeout the session drain margin instead of racing the 300 s session. The product's 180 s budget and the session budget are unchanged. Diagnostic rerun passed in 195 s. * review/ship: use main's post-expiry smoke sentence; v1.91.24.0 GSTA-23's measurement on the review workflow judge: main's 'Post-expiry smoke rechecks are not-run.' passed 10/10; the C2 wording passed 6/10 and the '..., even in 4c.' variant failed in PR run 37248650879. qa-probe-gates pins main's sentence before step 4c. * test: record the plan-design corpus test in the test-of-test baseline #3031's ratchet now counts tests that slice a paid test file's source; the B1 plan-design-review-plan-mode replay corpus reads test/skill-e2e-design.test.ts like its sibling corpora already listed. * fix: --case no longer deadlocks importing the running CLI runCaseDiagnosis loaded runPaidShards with a dynamic import of scripts/test-paid-shards.ts; when that file is the entry it is still in its top-level await, so the import never resolved and `--case` hung after its first line (on main since #3031, found by the per-case measurement runs). The CLI now passes its runner in; library callers keep the lazy import. * test: name the in-flight file and the native error when bun dies before its summary (#3037) * Bun 1.4.2 everywhere CI and setup pin the tested version Bun 1.4.0 on Windows aborts about 3% of free-suite shards with native bad-handle errors (main too; #3037); the same head on 1.4.2 had none in 180 shard runs. Every current 1.4.0 pin moves together, as bun-version-drift requires: setup-bun in every workflow (CSO orchestration included), the eval image (sha256 from bun-v1.4.2's SHASUMS256.txt, re-hashed locally), GitLab, the Ubicloud free-suite setup, gstack-bun-version.sh's tested version, engines.bun and @types/bun, the install hint in generated skills, the Dia and cookie native-qualification pins, and docs and tests that state the current pin. Recorded 1.4.0 observations, fixtures, CHANGELOG history and the CSO images' oven/bun:1.3.10 base stay. * fix: --case runs a single-case file whole A session option naming the case (testName: 'office-hours-section-loading') made caseSelection pick test-name mode, whose filter matched no Bun test, so every trial would run hollow. A file that registers only this case now runs whole before any name match is tried. * fix(office-hours): reviewer returns a hash-bound receipt instead of echoing its verdict The spec reviewer now Writes its verdict once, seals it with gstack-office-hours-review seal (validates the saved JSON and prints OFFICE_HOURS_VERDICT round=N sha256= path=), and returns only that line. check --receipt recomputes the hash and fails closed on a missing, malformed, stale-round, wrong-path or mismatched receipt. The saved file stays the sole findings inventory; validators bind the receipt to the saved bytes and the judge reads those bytes. * fix(office-hours): only blocking findings send a design back for another review round Review findings carry severity blocking|minor (verdict schema v2). PASS when no blocking findings remain; minor findings are recorded in Reviewer Concerns and the Spec Review table, never a reason for another round. CONVERGENCE needs a persisting blocking prior. A persisting or unverified blocking prior cannot be relabeled minor. The 3-round cap is unchanged. * fix(office-hours): rounds 2 and 3 are delta re-reviews of the captured design diff prepare saves each reviewed design as round-N.design.md and gives later reviewers the exact line diff since the previous round. A new later-round blocking finding must cite (changed_text) an excerpt of a changed line; seal, check and the validators verify it against the snapshots and fail closed when a citation or snapshot is missing. Other new observations are minor. Severity, anti-downgrade and CONVERGENCE rules are unchanged. * test(office-hours): preservation judge accepts COMPLETED with recorded minor findings * fix(office-hours): parent fixes only blocking findings; callers may set --max-rounds Between rounds the parent edits for blocking findings only; minor findings stay recorded. gstack-office-hours-review takes --max-rounds 1|2|3 (default 3) on every command; reaching it stops at MAX_ITERATIONS. The marathon fixture sets 2 rounds so round 1 (full) and round 2 (delta) run live. * docs(changelog): office-hours review loop changes in v1.91.28.0 * fix(cso): helper JSON keeps run URLs only in the validated workflow slot The shared redactor now treats GitHub Actions run ids as CI metadata (c245d1bc2), so main's #3041 invariant that the same run digits stay redacted outside qualification.workflow held only through the phone false positive. Make it explicit in the CSO sanitizer. * test(plan-ceo): split-overflow progress detector accepts the skill's candidate wording Collection never ended when the model wrote 'E1 Slack:' / 'E1-SLACK:' or labels like 'Include in this quarter' / 'Defer with flip trigger', so the session ran the whole review (930-1464 s) and failed on its extra findings or the case deadline (6 of 21 runs). The candidate check now accepts a platform-bound E-id and the action is the bucket's leading verb, as the skill names them (Include/Defer/Cut/Hold). The semantic judge, call floor and include/defer/cut validation are unchanged. Replays the captured CI 37374526437, t2, t9 and t15 menus with negative controls. * test(plan-ceo): split-overflow candidate identity is the one E1-E5 token The skill fixes no header or question wording, and shape-matching kept missing new forms (header 'D1.1 Slack' with 'SCOPE-E1:', 'D1.1 E1 Slack', 'E1 Slack DM bot ...:', a misspelled 'Mattermst'). Identity is now exactly one E1-E5 token across the header and the question's first line, with no other candidate's platform named. Options rules, judge, call floor and include/defer/cut validation are unchanged. Replays all 31 measured runs; the quoted-row controls now quote a second ledger id. * v1.91.30.0: move past #3044's claimed 1.91.29.0 * test: register the split-overflow wording replay as a free fixture * fix(skills): resolve judge-cited ambiguities in /qa and /plan-eng-review - tasks JSONL keeps the raw branch as GIT_BRANCH instead of overwriting gstack-slug's sanitized BRANCH used for artifact names - the Codex outside-voice block runs for ready or unverified, matching its routing rule - plan-eng startup names the preamble's own setup prompts; the cross-project question waits for Step 0 - /qa 8c: a fix that applies but leaves the finding reproducing gets one revised fix, else it is reverted; browser verification names its before screenshot; the exploratory loop drops its duplicate 3-step summary * fix(plan-eng): keep the startup steps in their pinned order * fix(plan-eng): explicit spawned sessions follow the spawned rule at the scope gate; one announcement rule A missing tool still sends the prose menu whatever the session type, but a session the caller or host explicitly marks spawned selects the menu's Recommendation without prose or BLOCKED. The pre-load announcement sentence now states Step 0's single check. * eval policy v3: judge panels gate on the per-dimension median of 3 samples A judge dimension now passes when at least 2 of its 3 samples meet the unchanged threshold (judgePanelMedian); any other sample count or a non-numeric sample fails closed. Thresholds, sample count, judge model and prompts are unchanged; panel logs show every sample and the mean, and name the median as the gate. Applies to the skill-llm-eval panels, the workflow-judge cache, the auq-matrix substance panel and the calibration replicas. EVAL_POLICY 2 -> 3 and HARNESS_VERSION 1 -> 2 (verdicts change), so every case starts a new pass-rate series. Replays the captured panels from jobs 112032744416 and 112049856586 as replayed evidence. --- .github/docker/Dockerfile.ci | 4 +- .github/scripts/dia-launch-driver.mjs | 6 +- .github/scripts/qualify-dia-macos.ts | 6 +- .../scripts/run-dia-native-qualification.ts | 4 +- .github/workflows/arm-setup-smoke.yml | 8 +- .github/workflows/cso-runtime-images.yml | 8 +- .github/workflows/cso-runtime-promote.yml | 2 +- .../workflows/cso-runtime-qualification.yml | 2 +- .github/workflows/cso-scanner-images.yml | 8 +- .github/workflows/evals-marathon.yml | 64 +- .github/workflows/evals-periodic.yml | 77 +- .github/workflows/evals.yml | 6 +- .github/workflows/free-tests.yml | 16 +- .github/workflows/make-pdf-gate.yml | 2 +- .github/workflows/native-qualification.yml | 6 +- .github/workflows/platform-qualification.yml | 6 +- .github/workflows/quality-gate.yml | 2 +- .github/workflows/skill-docs.yml | 2 +- .github/workflows/test-health.yml | 2 +- .github/workflows/version-gate.yml | 2 +- .github/workflows/windows-free-tests.yml | 8 +- .github/workflows/windows-setup-e2e.yml | 2 +- .gitlab-ci.yml | 2 +- AGENTS.md | 4 +- CHANGELOG.md | 71 + CONTRIBUTING.md | 27 +- README.md | 10 +- TODOS.md | 98 +- VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- benchmark-models/SKILL.md.tmpl | 2 +- bin/gstack-bun-version.sh | 75 + bin/gstack-docs-candidate | 9 +- bin/gstack-office-hours-review | 61 +- bin/gstack-review-log | 17 +- bin/gstack-session-update | 40 +- bin/gstack-wtree | 47 +- browse/src/browser-manager.ts | 43 +- browse/src/cli.ts | 5 + browse/test/cookie-import-native-job.test.ts | 2 +- .../cookie-import-native-qualification.ts | 2 +- browse/test/dia-launch-comparison.test.ts | 4 +- browse/test/headless-gpu-and-reap.test.ts | 94 +- browse/test/navigation-guard.test.ts | 29 +- bun.lock | 6 +- design/scripts/live-model-check.ts | 29 +- design/src/models.ts | 50 +- design/test/image-gen-pairing.test.ts | 76 + deslop-shared-libs/SKILL.md | 4 +- deslop-shared-libs/SKILL.md.tmpl | 4 +- diagram/SKILL.md.tmpl | 2 +- docs/OVERLAY_BENCHMARK_CONTRACT.md | 18 +- docs/TESTING_INTERNALS.md | 120 +- docs/designs/RELIABILITY_FOLLOWUPS_2026_10.md | 112 + docs/evals/census-red.md | 134 + docs/evals/detector-inventory.md | 320 + docs/evals/headroom-diagnosis-2026-10.md | 98 + docs/evals/judge-controls-2026-10.md | 83 + docs/evals/judge-controls/results.json | 365 ++ docs/evals/judge-low-samples-2026-10.md | 66 + docs/troubleshooting.md | 180 + freeze/SKILL.md.tmpl | 2 +- gstack-upgrade/SKILL.md | 3 + gstack-upgrade/SKILL.md.tmpl | 3 + guard/SKILL.md.tmpl | 2 +- hosts/codex.ts | 2 +- hosts/define-host.ts | 2 +- hosts/opencode.ts | 2 +- investigate/SKILL.md.tmpl | 6 +- lib/cso/process.ts | 5 +- lib/office-hours-review.ts | 172 +- lib/redact-patterns.ts | 26 +- make-pdf/SKILL.md | 40 +- make-pdf/SKILL.md.tmpl | 26 +- office-hours/sections/design-and-handoff.md | 43 +- open-gstack-browser/SKILL.md | 2 +- package.json | 6 +- pair-agent/SKILL.md | 2 +- plan-ceo-review/SKILL.md | 2 +- plan-ceo-review/SKILL.md.tmpl | 2 +- plan-ceo-review/sections/review-sections.md | 8 +- .../sections/review-sections.md | 6 +- plan-devex-review/sections/review-sections.md | 6 +- plan-eng-review/SKILL.md | 8 +- plan-eng-review/SKILL.md.tmpl | 8 +- plan-eng-review/sections/review-sections.md | 10 +- qa-only/SKILL.md | 2 +- qa-only/sections/exploratory.md | 3 - qa/SKILL.md | 7 +- qa/SKILL.md.tmpl | 5 +- qa/sections/browser-verify.md | 3 +- qa/sections/browser-verify.md.tmpl | 3 +- qa/sections/exploratory.md | 3 - retro/SKILL.md.tmpl | 2 +- review/SKILL.md | 10 +- review/SKILL.md.tmpl | 10 +- review/sections/adversarial.md | 2 +- review/sections/plan-completion.md | 6 +- scripts/eval-flake-rank.ts | 65 +- scripts/harness-version.json | 77 +- scripts/judge-calibration-configs.ts | 16 +- scripts/judge-controls.ts | 123 + scripts/lib/eval-history.ts | 354 + scripts/lib/paid-cases.ts | 1 + scripts/lib/paid-report.ts | 200 +- scripts/lib/paid-types.ts | 7 +- scripts/lib/published-text.ts | 41 + scripts/paid-test-durations.json | 278 +- scripts/resolvers/browse.ts | 2 +- scripts/resolvers/index.ts | 4 + scripts/resolvers/make-pdf.ts | 39 +- scripts/resolvers/outside-voice-steps.ts | 4 +- scripts/resolvers/plan-gates.ts | 6 +- scripts/resolvers/qa.ts | 3 - scripts/resolvers/runtime-root.ts | 37 +- scripts/resolvers/spec-review.ts | 43 +- scripts/resolvers/tasks-section.ts | 6 +- scripts/resolvers/test-value.ts | 2 +- scripts/test-free-shards.ts | 9 + scripts/test-paid-shards.ts | 41 +- scripts/typecheck-test-baseline.json | 1 - scripts/ubicloud/setup-free-suite.sh | 2 +- setup | 27 +- setup-browser-cookies/SKILL.md | 2 +- ship/sections/adversarial.md | 2 +- ship/sections/documentation.md | 13 +- ship/sections/documentation.md.tmpl | 13 +- ship/sections/test-coverage.md | 2 +- test-audit/SKILL.md | 2 +- test/arm-setup-smoke-workflow.test.ts | 4 +- test/auq-substance-panel.test.ts | 49 + test/bun-subprocess-fd-lifetime.test.ts | 2 +- test/bun-version-drift.test.ts | 88 + test/census-red-docs.test.ts | 45 + test/ceo-mode-handoff.test.ts | 32 +- test/ceo-split-collection.test.ts | 38 +- test/ceo-stale-fill-decision.test.ts | 31 + test/ci-paid-coordination.test.ts | 2 +- test/codex-e2e-multiblock-live.test.ts | 116 + test/codex-multiblock.test.ts | 66 + test/cookie-workflow-judge-input.test.ts | 6 +- test/cost-known.test.ts | 95 + test/detector-corpora.test.ts | 77 + ...ector-corpus-auto-decide-preserved.test.ts | 79 + ...or-corpus-plan-ceo-section-loading.test.ts | 66 + ...orpus-plan-design-review-plan-mode.test.ts | 109 + ...us-plan-eng-multi-finding-batching.test.ts | 72 + ...or-corpus-shared-libs-plan-callers.test.ts | 67 + ...or-corpus-ship-docsync-late-result.test.ts | 49 + test/docsync-authority.test.ts | 16 + test/docsync-command-grammar.test.ts | 14 +- test/docsync-nested-writes.test.ts | 30 + test/eng-finding-retry-budget.test.ts | 2 +- test/env-host-fence-conformance.test.ts | 70 +- test/env-host-runtime-assets.test.ts | 123 + test/eval-history.test.ts | 209 + test/evals-tracking-issue.test.ts | 193 + test/evals-workflow-wiring.test.ts | 6 +- test/fixtures/ceo-split-wording-variants.json | 5771 +++++++++++++++++ test/fixtures/context-budget.json | 18 +- .../37174266054-t1-pass.json | 224 + .../37176837432-t1-fail.json | 321 + .../37179171083-t1-fail.json | 287 + .../37182865432-t1-fail.json | 246 + .../37186854666-t1-pass.json | 236 + .../37193478719-t1-pass.json | 212 + .../37195203538-t1-pass.json | 217 + .../37198445662-t1-pass.json | 323 + .../auto-decide-preserved/inventory.json | 82 + .../37174266054-t1-pass.json | 20 + .../37179171083-t1-pass.json | 20 + ...32-t1-fail-derived-unrelated-decision.json | 21 + .../37182865432-t1-fail.json | 20 + .../37186854666-t1-pass.json | 20 + .../37193478719-t1-fail.json | 20 + .../37195203538-t1-pass.json | 20 + .../37198445662-t1-pass.json | 20 + .../plan-ceo-section-loading/inventory.json | 85 + .../37174266054-t1-pass.json | 22 + .../37179171083-t1-derived-unrated-fail.json | 23 + .../37179171083-t1-derived-unsaved-fail.json | 23 + .../37179171083-t1-fail.json | 22 + .../37193478719-t1-pass.json | 22 + .../37195203538-t1-pass.json | 22 + .../37198445662-t1-pass.json | 22 + .../inventory.json | 77 + .../37162480720-t1-pass.json | 111 + .../37174266054-t1-pass.json | 116 + .../37176837432-t1-pass.json | 111 + .../37179171083-t1-pass.json | 122 + .../37182865432-t1-pass.json | 116 + .../37186854666-t1-pass.json | 122 + .../37193478719-t1-pass.json | 121 + .../37195203538-t1-pass.json | 136 + .../37198445662-t1-derived-refuse.json | 57 + .../37198445662-t1-pass.json | 116 + .../37228573062-t1-fail.json | 165 + .../inventory.json | 92 + .../37174266054-t1-derived-refuse.json | 43 + .../37174266054-t1-fail.json | 43 + .../37176837432-t1-pass.json | 43 + .../37179171083-t1-pass.json | 43 + .../37182865432-t1-pass.json | 43 + .../37186854666-t1-derived-refuse.json | 43 + .../37186854666-t1-fail.json | 43 + .../37193478719-t1-pass.json | 43 + .../37193478719-t1-supported-git-pass.json | 44 + .../37195203538-t1-derived-refuse.json | 43 + .../37195203538-t1-fail.json | 43 + ...03538-t1-supported-git-derived-refuse.json | 43 + .../37198445662-t1-pass.json | 39 + .../shared-libs-plan-callers/inventory.json | 74 + .../37174266054-t1-pass.json | 94 + .../37176837432-t1-pass.json | 94 + .../37179171083-t1-pass.json | 94 + .../37182865432-t1-pass.json | 94 + .../37186854666-t1-pass.json | 94 + .../37193478719-t1-pass.json | 94 + .../37195203538-t1-pass.json | 94 + .../37196811165-t1-derived-current-claim.json | 95 + ...6811165-t1-derived-repair-before-stop.json | 95 + .../37196811165-t1-fail.json | 94 + .../37198445662-t1-derived-published.json | 99 + .../37198445662-t1-pass.json | 94 + .../ship-docsync-late-result/inventory.json | 423 ++ .../37237194905-late-result-calls.json | 136 + .../37239016858-completion-native.json | 393 ++ .../eval-history/wave-reds-8-censuses.json | 1 + .../judge-panel-replay-2026-10-05.json | 893 +++ test/fixtures/module-size-ratchet.json | 3 + .../overlay-v4-replay-37198445662.json | 396 ++ .../plan-floor-edit-wrap-37176835584.json | 24 + .../prosons-neutral-neg-captures.json | 20 + test/fixtures/trial-cause/replays.json | 1 + .../thinking-captures-37198445662.json | 1 + test/fixtures/trial-record-v1-reader.ts | 34 + test/fixtures/workflow-required-steps.json | 214 + test/freeze-owned-lifecycle.test.ts | 3 +- test/gen-skill-docs.test.ts | 9 +- test/gstack-docs-candidate.test.ts | 30 +- test/gstack-retro-metrics.test.ts | 3 +- test/harness-version.test.ts | 2 +- test/helpers/agent-sdk-runner.ts | 36 +- test/helpers/auq-substance-panel.ts | 19 + test/helpers/carve-guards.ts | 2 +- test/helpers/ceo-split-question-policy.ts | 37 +- test/helpers/ceo-stale-fill-decision.ts | 11 +- test/helpers/codex-eval.ts | 8 +- test/helpers/codex-multiblock.ts | 71 + test/helpers/docsync-observer.ts | 71 +- test/helpers/eng-seeded-coverage.ts | 6 +- test/helpers/eval-budgets.ts | 17 + test/helpers/eval-store.test.ts | 32 + test/helpers/eval-store.ts | 196 + test/helpers/free-fixtures-data.ts | 18 + test/helpers/llm-judge.ts | 29 +- test/helpers/manual-judge-review-fixture.ts | 2 + test/helpers/native-auto-decide.ts | 24 +- test/helpers/office-hours-completion.ts | 85 +- test/helpers/overlay-attempt.ts | 19 +- test/helpers/overlay-case-policy.ts | 7 +- test/helpers/overlay-case.ts | 4 +- test/helpers/periodic-exclude-data.ts | 21 +- test/helpers/plan-count-file-permission.ts | 19 +- test/helpers/prosons-posture.ts | 8 + test/helpers/pty/launch.ts | 23 +- test/helpers/pty/runners/counting.ts | 2 +- test/helpers/pty/runners/floor.ts | 2 +- test/helpers/pty/runners/observation.ts | 2 +- test/helpers/pty/screen.ts | 7 + test/helpers/pty/session.ts | 23 +- test/helpers/session-ledger.ts | 213 + test/helpers/session-runner.ts | 30 +- test/helpers/shared-libs-eval-fixture.ts | 13 +- test/helpers/touchfiles-data.ts | 66 +- test/helpers/workflow-judge-cache.ts | 15 +- test/investigate-freeze-path.test.ts | 5 +- test/judge-calibration.test.ts | 9 +- test/judge-panel-median.test.ts | 62 + test/llm-judge-recommendation.test.ts | 6 +- test/office-hours-completion.test.ts | 173 +- test/office-hours-review.test.ts | 218 +- test/outside-voice-preflight.test.ts | 6 + test/overlay-measurement.test.ts | 67 +- test/paid-report-census.test.ts | 120 +- test/paid-shards.test.ts | 92 +- test/periodic-exclude-policy.test.ts | 3 +- test/plan-floor-edit-wrap.test.ts | 61 + test/plan-format-approach-prompt.test.ts | 21 + test/prosons-neutral-posture.test.ts | 33 + test/published-text.test.ts | 88 + test/qa-probe-gates.test.ts | 39 + test/redact-context-exemptions.test.ts | 23 + test/review-base-refresh.test.ts | 69 + test/review-start-evidence.test.ts | 116 +- test/review-workflow-clarity.test.ts | 3 + test/session-runner-tools.test.ts | 28 +- test/session-update-autostash.test.ts | 2 +- test/session-update-stages.test.ts | 71 +- test/setup-bun-floor.test.ts | 140 + test/setup-claude-code-migration.test.ts | 2 +- test/shared-libs-rendering.test.ts | 9 + test/ship-control-flow.test.ts | 3 + test/skill-ceo-section-ordering.test.ts | 19 + test/skill-e2e-auq-matrix.test.ts | 23 +- test/skill-e2e-design-model-smoke.test.ts | 41 + ...l-e2e-office-hours-section-loading.test.ts | 15 +- test/skill-e2e-plan-format.test.ts | 17 +- test/skill-e2e-plan-prosons.test.ts | 2 +- test/skill-e2e-retro.test.ts | 7 +- test/skill-e2e-session-intelligence.test.ts | 17 +- test/skill-llm-eval.test.ts | 34 +- test/team-mode.test.ts | 2 +- test/test-free-shards.test.ts | 11 + test/test-of-test-ratchet.test.ts | 10 + test/test-value-bar.test.ts | 6 + test/trial-cause.test.ts | 278 + test/troubleshooting-anchors.test.ts | 9 + test/upgrade-setup-recovery.test.ts | 69 + test/workflow-judge-cache.test.ts | 24 +- test/workflow-required-steps.test.ts | 72 + unfreeze/SKILL.md.tmpl | 2 +- 322 files changed, 22550 insertions(+), 1051 deletions(-) create mode 100644 bin/gstack-bun-version.sh create mode 100644 docs/designs/RELIABILITY_FOLLOWUPS_2026_10.md create mode 100644 docs/evals/census-red.md create mode 100644 docs/evals/detector-inventory.md create mode 100644 docs/evals/headroom-diagnosis-2026-10.md create mode 100644 docs/evals/judge-controls-2026-10.md create mode 100644 docs/evals/judge-controls/results.json create mode 100644 docs/evals/judge-low-samples-2026-10.md create mode 100644 scripts/judge-controls.ts create mode 100644 scripts/lib/eval-history.ts create mode 100644 scripts/lib/published-text.ts create mode 100644 test/auq-substance-panel.test.ts create mode 100644 test/census-red-docs.test.ts create mode 100644 test/codex-e2e-multiblock-live.test.ts create mode 100644 test/codex-multiblock.test.ts create mode 100644 test/cost-known.test.ts create mode 100644 test/detector-corpora.test.ts create mode 100644 test/detector-corpus-auto-decide-preserved.test.ts create mode 100644 test/detector-corpus-plan-ceo-section-loading.test.ts create mode 100644 test/detector-corpus-plan-design-review-plan-mode.test.ts create mode 100644 test/detector-corpus-plan-eng-multi-finding-batching.test.ts create mode 100644 test/detector-corpus-shared-libs-plan-callers.test.ts create mode 100644 test/detector-corpus-ship-docsync-late-result.test.ts create mode 100644 test/env-host-runtime-assets.test.ts create mode 100644 test/eval-history.test.ts create mode 100644 test/evals-tracking-issue.test.ts create mode 100644 test/fixtures/ceo-split-wording-variants.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/37174266054-t1-pass.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/37176837432-t1-fail.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/37179171083-t1-fail.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/37182865432-t1-fail.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/37186854666-t1-pass.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/37193478719-t1-pass.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/37195203538-t1-pass.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/37198445662-t1-pass.json create mode 100644 test/fixtures/detector-corpora/auto-decide-preserved/inventory.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/37174266054-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/37179171083-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/37182865432-t1-fail-derived-unrelated-decision.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/37182865432-t1-fail.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/37186854666-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/37193478719-t1-fail.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/37195203538-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/37198445662-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-ceo-section-loading/inventory.json create mode 100644 test/fixtures/detector-corpora/plan-design-review-plan-mode/37174266054-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-design-review-plan-mode/37179171083-t1-derived-unrated-fail.json create mode 100644 test/fixtures/detector-corpora/plan-design-review-plan-mode/37179171083-t1-derived-unsaved-fail.json create mode 100644 test/fixtures/detector-corpora/plan-design-review-plan-mode/37179171083-t1-fail.json create mode 100644 test/fixtures/detector-corpora/plan-design-review-plan-mode/37193478719-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-design-review-plan-mode/37195203538-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-design-review-plan-mode/37198445662-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-design-review-plan-mode/inventory.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37162480720-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37174266054-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37176837432-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37179171083-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37182865432-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37186854666-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37193478719-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37195203538-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37198445662-t1-derived-refuse.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37198445662-t1-pass.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/37228573062-t1-fail.json create mode 100644 test/fixtures/detector-corpora/plan-eng-multi-finding-batching/inventory.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37174266054-t1-derived-refuse.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37174266054-t1-fail.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37176837432-t1-pass.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37179171083-t1-pass.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37182865432-t1-pass.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37186854666-t1-derived-refuse.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37186854666-t1-fail.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37193478719-t1-pass.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37193478719-t1-supported-git-pass.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37195203538-t1-derived-refuse.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37195203538-t1-fail.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37195203538-t1-supported-git-derived-refuse.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/37198445662-t1-pass.json create mode 100644 test/fixtures/detector-corpora/shared-libs-plan-callers/inventory.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37174266054-t1-pass.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37176837432-t1-pass.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37179171083-t1-pass.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37182865432-t1-pass.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37186854666-t1-pass.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37193478719-t1-pass.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37195203538-t1-pass.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37196811165-t1-derived-current-claim.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37196811165-t1-derived-repair-before-stop.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37196811165-t1-fail.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37198445662-t1-derived-published.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/37198445662-t1-pass.json create mode 100644 test/fixtures/detector-corpora/ship-docsync-late-result/inventory.json create mode 100644 test/fixtures/docsync-replay/37237194905-late-result-calls.json create mode 100644 test/fixtures/docsync-replay/37239016858-completion-native.json create mode 100644 test/fixtures/eval-history/wave-reds-8-censuses.json create mode 100644 test/fixtures/judge-panel-replay-2026-10-05.json create mode 100644 test/fixtures/overlay-v4-replay-37198445662.json create mode 100644 test/fixtures/plan-floor-edit-wrap-37176835584.json create mode 100644 test/fixtures/prosons-neutral-neg-captures.json create mode 100644 test/fixtures/trial-cause/replays.json create mode 100644 test/fixtures/trial-cause/thinking-captures-37198445662.json create mode 100644 test/fixtures/trial-record-v1-reader.ts create mode 100644 test/fixtures/workflow-required-steps.json create mode 100644 test/helpers/auq-substance-panel.ts create mode 100644 test/helpers/codex-multiblock.ts create mode 100644 test/helpers/prosons-posture.ts create mode 100644 test/helpers/session-ledger.ts create mode 100644 test/judge-panel-median.test.ts create mode 100644 test/plan-floor-edit-wrap.test.ts create mode 100644 test/plan-format-approach-prompt.test.ts create mode 100644 test/prosons-neutral-posture.test.ts create mode 100644 test/published-text.test.ts create mode 100644 test/review-base-refresh.test.ts create mode 100644 test/setup-bun-floor.test.ts create mode 100644 test/skill-e2e-design-model-smoke.test.ts create mode 100644 test/trial-cause.test.ts create mode 100644 test/workflow-required-steps.test.ts diff --git a/.github/docker/Dockerfile.ci b/.github/docker/Dockerfile.ci index b0befb3a0..97b5b6b55 100644 --- a/.github/docker/Dockerfile.ci +++ b/.github/docker/Dockerfile.ci @@ -103,11 +103,11 @@ RUN curl --retry 5 --retry-delay 5 --retry-connrefused -fsSL "https://nodejs.org # Hashes come from the release's SHASUMS256.txt. The image builds linux/amd64 # only (the Node.js tarball above is linux-x64); an architecture without a # verified archive fails here. test/ci-image-cli-pin.test.ts pins this shape. -ARG BUN_VERSION=1.4.0 +ARG BUN_VERSION=1.4.2 ARG TARGETARCH ENV BUN_INSTALL="/usr/local" RUN case "${TARGETARCH:-amd64}" in \ - amd64) bun_target=linux-x64; bun_sha256=2d03fb5fb83ac8b567aca0a281b2ce1a1a19d488f56c2968d88c3f25e92fe452 ;; \ + amd64) bun_target=linux-x64; bun_sha256=36368faef7527875d5ffa52e53cd48021741f2a83eb6208a8dd64068d422a913 ;; \ *) echo "Dockerfile.ci: no verified Bun ${BUN_VERSION} archive for ${TARGETARCH}" >&2; exit 1 ;; \ esac \ && curl --retry 5 --retry-delay 5 --retry-connrefused -fsSL \ diff --git a/.github/scripts/dia-launch-driver.mjs b/.github/scripts/dia-launch-driver.mjs index 028076148..ef4538f3b 100644 --- a/.github/scripts/dia-launch-driver.mjs +++ b/.github/scripts/dia-launch-driver.mjs @@ -46,7 +46,7 @@ export function compareDiaLaunchReceipts(left, right) { || qualification.isolation?.sharedRegisteredHome !== true || qualification.artifact?.signatureVerified !== true || qualification.artifact?.gatekeeperNotarized !== true || qualification.artifact?.macosCompatibility?.compatible !== true || qualification.platform?.os !== 'darwin' || qualification.platform?.architecture !== 'arm64' - || qualification.platform?.bun !== '1.4.0' || qualification.platform?.playwright !== '1.62.1' + || qualification.platform?.bun !== '1.4.2' || qualification.platform?.playwright !== '1.62.1' || !/^\d+(?:\.\d+){1,2}$/.test(qualification.artifact.macosCompatibility.hostVersion) || qualification.artifact?.architectures?.includes('arm64') !== true || ['serviceStopped', 'userDomainStopped', 'userProcessesStopped', 'accountRemoved', 'groupRemoved', 'stagingRemoved'].some(key => receipt.launcherCleanup?.[key] !== true) @@ -63,7 +63,7 @@ export function compareDiaLaunchReceipts(left, right) { || policy?.mockKeychainFlag !== false || policy?.passwordStoreFlag !== false || policy?.firstRunSuppressed !== false || policy?.headlessFlag !== true || policy?.expectedProfile !== true || policy?.detached !== true || policy?.shellDisabled !== true || policy?.stdioCount !== 5 || policy?.extraPipeDescriptors !== true || policy?.profileArgumentCount !== 1 - || result.driver?.runtime !== runtime || result.driver?.version !== (runtime === 'bun' ? '1.4.0' : '24.18.0') + || result.driver?.runtime !== runtime || result.driver?.version !== (runtime === 'bun' ? '1.4.2' : '24.18.0') || result.driver?.os !== 'darwin' || result.driver?.architecture !== 'arm64' || result.driver?.playwright !== '1.62.1' || result.driver?.release !== qualification.platform.release || result.driver?.executableSha256 !== config.executableSha256 || result.driver?.driverSha256 !== config.driverSha256 @@ -176,7 +176,7 @@ async function main() { const account = readFreshAccountConfiguration(process.argv[2], 'comparison-driver'); const config = account.launchComparison; const runtimeVersion = process.versions.bun ?? process.versions.node; - if ((config.runtime === 'bun' ? process.versions.bun !== '1.4.0' : Boolean(process.versions.bun) || runtimeVersion !== '24.18.0') + if ((config.runtime === 'bun' ? process.versions.bun !== '1.4.2' : Boolean(process.versions.bun) || runtimeVersion !== '24.18.0') || process.arch !== 'arm64' || require('playwright/package.json').version !== '1.62.1') throw new Error('invalid_driver_runtime'); if (await sha256(process.execPath) !== config.executableSha256 || await sha256(import.meta.filename) !== config.driverSha256 || await sha256(path.join(import.meta.dirname, 'qualify-dia-macos.ts')) !== config.helpersSha256) throw new Error('driver_inputs_changed'); diff --git a/.github/scripts/qualify-dia-macos.ts b/.github/scripts/qualify-dia-macos.ts index cdaaa3124..b5fc3b9e3 100644 --- a/.github/scripts/qualify-dia-macos.ts +++ b/.github/scripts/qualify-dia-macos.ts @@ -179,7 +179,7 @@ export async function runDiaLaunchComparison(account: FreshAccount, purpose: 'co 'argvSha256', 'environmentSha256', 'rootCount', 'launchAttempts', 'rootsBeforeCleanup', 'stderrBeforeCleanup', 'stderrAfterCleanup', 'rootsAfterCleanup', 'driver']; if (!response || !safeDiaComparisonResponse(response) || Object.keys(response).some(key => !keys.includes(key)) || response.protocol !== 1 || response.purpose !== purpose || response.samplingEnabled !== false || response.driver?.runtime !== config.runtime - || response.driver?.version !== (config.runtime === 'bun' ? '1.4.0' : '24.18.0') || response.driver?.architecture !== 'arm64' + || response.driver?.version !== (config.runtime === 'bun' ? '1.4.2' : '24.18.0') || response.driver?.architecture !== 'arm64' || response.driver?.os !== 'darwin' || response.driver?.playwright !== '1.62.1' || response.driver?.executableSha256 !== config.executableSha256 || response.driver?.driverSha256 !== config.driverSha256 || response.driver?.helpersSha256 !== config.helpersSha256 || typeof response.ready !== 'boolean' @@ -931,7 +931,7 @@ export async function qualifyDia(isolation: { root: string; configFile: string } validateQualificationHost(process.env); const account = readFreshAccountConfiguration(isolation.configFile); if (account.guiReadiness || !account.destinationExecutable || !account.destinationSha256) throw new Error('browser_qualification_authority_required'); - if (Bun.version !== '1.4.0' || require('playwright/package.json').version !== '1.62.1') throw new Error('pinned_runtimes_required'); + if (Bun.version !== '1.4.2' || require('playwright/package.json').version !== '1.62.1') throw new Error('pinned_runtimes_required'); const runnerTemp = realpathSync(process.env.RUNNER_TEMP!); const output = path.join(runnerTemp, 'dia-native-qualification.json'); if (existsSync(output)) throw new Error('fresh_receipt_path_required'); @@ -1383,7 +1383,7 @@ if (import.meta.main) { if (process.argv[2] !== '--fresh-account' || !process.argv[3]) throw new Error('fresh_account_configuration_required'); const account = readFreshAccountConfiguration(process.argv[3]); if (account.guiReadiness) throw new Error('browser_qualification_authority_required'); - if (Bun.version !== '1.4.0' || require('playwright/package.json').version !== '1.62.1') throw new Error('pinned_runtimes_required'); + if (Bun.version !== '1.4.2' || require('playwright/package.json').version !== '1.62.1') throw new Error('pinned_runtimes_required'); const originalHome = account.home; const originalHomeEnvironment = process.env.HOME; const runnerTemp = realpathSync(process.env.RUNNER_TEMP!); diff --git a/.github/scripts/run-dia-native-qualification.ts b/.github/scripts/run-dia-native-qualification.ts index f93c76d62..95236b1f9 100644 --- a/.github/scripts/run-dia-native-qualification.ts +++ b/.github/scripts/run-dia-native-qualification.ts @@ -406,7 +406,7 @@ async function freshWorker(configFile: string) { if (await digest(account.bun) !== account.bunSha256 || await digest(account.destinationExecutable) !== account.destinationSha256) throw new Error('staged_executable_changed'); receipt.reason = 'fresh_dependency_install'; run(account.bun, ['install', '--frozen-lockfile', '--ignore-scripts'], 180_000); - if (Bun.version !== '1.4.0' || require(path.join(account.snapshot, 'node_modules/playwright/package.json')).version !== '1.62.1') throw new Error('pinned_runtime_mismatch'); + if (Bun.version !== '1.4.2' || require(path.join(account.snapshot, 'node_modules/playwright/package.json')).version !== '1.62.1') throw new Error('pinned_runtime_mismatch'); mkdirSync(probe, { mode: 0o700 }); receipt.reason = 'background_keychain_preflight'; snapshot = captureUserKeychains(env, [account.home, account.temporary]); @@ -538,7 +538,7 @@ export async function runFreshAccountQualification(comparisonRuntime?: 'bun' | ' validateQualificationHost(process.env); if (comparisonRuntime !== undefined && !['bun', 'node'].includes(comparisonRuntime)) throw new Error('invalid_comparison_runtime'); if (guiReadinessOnly && comparisonRuntime !== undefined) throw new Error('conflicting_diagnostic_modes'); - if (process.getuid?.() === 0 || Bun.version !== '1.4.0') throw new Error('run_as_unprivileged_pinned_ci_runner'); + if (process.getuid?.() === 0 || Bun.version !== '1.4.2') throw new Error('run_as_unprivileged_pinned_ci_runner'); const outputRoot = realpathSync(process.env.RUNNER_TEMP!); const output = path.join(outputRoot, 'dia-native-qualification.json'); if (existsSync(output)) throw new Error('fresh_output_required'); diff --git a/.github/workflows/arm-setup-smoke.yml b/.github/workflows/arm-setup-smoke.yml index 0c5ec0284..014e9c513 100644 --- a/.github/workflows/arm-setup-smoke.yml +++ b/.github/workflows/arm-setup-smoke.yml @@ -30,13 +30,13 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Seal the checked-out source and native toolchain shell: bash run: | set -euo pipefail test "$(uname -m)" = aarch64 - test "$(bun --version)" = 1.4.0 + test "$(bun --version)" = 1.4.2 test "$(bun -p 'process.arch')" = arm64 test "$(git rev-parse HEAD)" = "$GITHUB_SHA" mkdir -p "$RUNNER_TEMP/arm-input" "$RUNNER_TEMP/arm-evidence" @@ -76,8 +76,8 @@ jobs: apt-get install -y --no-install-recommends ca-certificates git nodejs npm fonts-noto-color-emoji install -m 0755 /input/bun /usr/local/bin/bun ln -s bun /usr/local/bin/bunx - test "$(bun --version)" = 1.4.0 - test "$(bunx --version)" = 1.4.0 + test "$(bun --version)" = 1.4.2 + test "$(bunx --version)" = 1.4.2 test "$(bun -p 'process.arch')" = arm64 test "$(node -p 'process.arch')" = arm64 node --version diff --git a/.github/workflows/cso-runtime-images.yml b/.github/workflows/cso-runtime-images.yml index e9c3259f0..2e83097e8 100644 --- a/.github/workflows/cso-runtime-images.yml +++ b/.github/workflows/cso-runtime-images.yml @@ -36,7 +36,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Validate committed image digests and complete platform matrix id: inputs run: | @@ -69,7 +69,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e - name: Require native Docker and verify reviewed source manifests env: @@ -144,7 +144,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Require the declared native architecture and Docker daemon env: CSO_ARCH: ${{ matrix.arch }} @@ -308,7 +308,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: cso-staged-${{ matrix.stack }}-${{ matrix.arch }} diff --git a/.github/workflows/cso-runtime-promote.yml b/.github/workflows/cso-runtime-promote.yml index 5a012ec19..63972ccdd 100644 --- a/.github/workflows/cso-runtime-promote.yml +++ b/.github/workflows/cso-runtime-promote.yml @@ -41,7 +41,7 @@ jobs: persist-credentials: true - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Authenticate the completed qualification run env: GH_TOKEN: ${{ github.token }} diff --git a/.github/workflows/cso-runtime-qualification.yml b/.github/workflows/cso-runtime-qualification.yml index 3f90d4ee8..d86c22b23 100644 --- a/.github/workflows/cso-runtime-qualification.yml +++ b/.github/workflows/cso-runtime-qualification.yml @@ -67,7 +67,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Require the configured private-evaluator actor env: CSO_QUALIFICATION_ACTOR: ${{ vars.CSO_QUALIFICATION_ACTOR }} diff --git a/.github/workflows/cso-scanner-images.yml b/.github/workflows/cso-scanner-images.yml index 11613d586..90804b9e2 100644 --- a/.github/workflows/cso-scanner-images.yml +++ b/.github/workflows/cso-scanner-images.yml @@ -30,7 +30,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Prove scanner normalization and fail-closed behavior run: | set -eu @@ -64,7 +64,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Require native Docker and the declared architecture env: CSO_ARCH: ${{ matrix.arch }} @@ -289,7 +289,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: pattern: cso-scanner-profile-* @@ -329,7 +329,7 @@ jobs: persist-credentials: true - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: cso-scanner-catalog-proposal diff --git a/.github/workflows/evals-marathon.yml b/.github/workflows/evals-marathon.yml index e80e23e8d..565c32d77 100644 --- a/.github/workflows/evals-marathon.yml +++ b/.github/workflows/evals-marathon.yml @@ -94,7 +94,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 # One marathon file per runner: a 1-second budget never packs two # recorded files together. @@ -268,7 +268,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: @@ -289,25 +289,47 @@ jobs: # PIPESTATUS[0], NOT $?: the default step shell has no pipefail. echo "exit=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT" - # One tracking issue for the whole lane (never one per week). - - name: Upsert tracking issue on failure + # A red marathon run writes its report to the step summary and the + # marathon-report artifact on every ref. Only main upserts the lane's ONE + # tracking issue (never one per week) and closes it on the next green + # run; a branch-dispatched run never touches main's issue. + - name: Write the marathon report if: always() && (steps.reconcile.outputs.exit != '0' || needs.eval-slices.result != 'success') + run: | + set -euo pipefail + { + echo "Marathon report (non-blocking lane): branch ${GITHUB_REF_NAME} @ ${GITHUB_SHA}, event ${GITHUB_EVENT_NAME}, run ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" + echo + echo "A census went red: triage with ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/blob/${GITHUB_SHA}/docs/evals/census-red.md" + echo + echo "- reconciliation exit: ${{ steps.reconcile.outputs.exit }}" + echo "- marathon slices job: ${{ needs.eval-slices.result }}" + echo + { tail -c 6000 /tmp/report.txt 2>/dev/null || echo "(no reconciliation output)"; } \ + | bun --no-install -e 'import { publishedFence } from "./scripts/lib/published-text"; + const lines = (await Bun.stdin.text()).split("\n").filter(Boolean); + console.log(publishedFence(lines.length ? lines : ["(none)"]).join("\n"));' + } > /tmp/marathon-report.md + cat /tmp/marathon-report.md >> "$GITHUB_STEP_SUMMARY" + + - name: Upload the marathon report + if: always() && (steps.reconcile.outputs.exit != '0' || needs.eval-slices.result != 'success') + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: marathon-report-a${{ github.run_attempt }} + path: /tmp/marathon-report.md + if-no-files-found: ignore + retention-days: 90 + + - name: Upsert tracking issue on failure + if: always() && github.ref == 'refs/heads/main' && (steps.reconcile.outputs.exit != '0' || needs.eval-slices.result != 'success') env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | set -euo pipefail TITLE="Weekly marathon evals: red lane needs triage" - BODY_FILE=/tmp/issue-body.md - { - echo "Automated weekly marathon report (non-blocking lane) — run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" - echo - echo "- reconciliation exit: ${{ steps.reconcile.outputs.exit }}" - echo "- marathon slices job: ${{ needs.eval-slices.result }}" - echo - echo '```' - tail -c 6000 /tmp/report.txt 2>/dev/null || echo "(no reconciliation output)" - echo '```' - } > "$BODY_FILE" + BODY_FILE=/tmp/marathon-report.md + [ -s "$BODY_FILE" ] || echo "Marathon report missing; run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" > "$BODY_FILE" EXISTING=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "in:title \"$TITLE\"" --json number --jq '.[0].number // empty') if [ -n "$EXISTING" ]; then gh issue comment "$EXISTING" --repo "$GITHUB_REPOSITORY" --body-file "$BODY_FILE" @@ -316,6 +338,18 @@ jobs: gh issue create --repo "$GITHUB_REPOSITORY" --title "$TITLE" --body-file "$BODY_FILE" fi + - name: Close the tracking issue on a green run + if: always() && github.ref == 'refs/heads/main' && steps.reconcile.outputs.exit == '0' && needs.eval-slices.result == 'success' + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + TITLE="Weekly marathon evals: red lane needs triage" + EXISTING=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "in:title \"$TITLE\"" --json number --jq '.[0].number // empty') + if [ -n "$EXISTING" ]; then + gh issue close "$EXISTING" --repo "$GITHUB_REPOSITORY" --comment "Green marathon run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" + fi + - name: Fail the workflow when reconciliation failed if: always() && (steps.reconcile.outputs.exit != '0' || needs.eval-slices.result != 'success') run: exit 1 diff --git a/.github/workflows/evals-periodic.yml b/.github/workflows/evals-periodic.yml index 2fec19a62..af95cb699 100644 --- a/.github/workflows/evals-periodic.yml +++ b/.github/workflows/evals-periodic.yml @@ -135,7 +135,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Emit run manifest (ALL periodic tests minus reasoned excludes) env: @@ -587,7 +587,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: @@ -675,7 +675,7 @@ jobs: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | set +e - bun run eval:pass-rates --gate --runs 10 > /tmp/pass-rates.txt 2>&1 + bun run eval:pass-rates --gate --runs 10 --branch "$GITHUB_REF_NAME" > /tmp/pass-rates.txt 2>&1 echo "exit=$?" >> "$GITHUB_OUTPUT" cat /tmp/pass-rates.txt @@ -700,28 +700,42 @@ jobs: echo "redispatch=false" >> "$GITHUB_OUTPUT" fi - # A red weekly lane nobody must action is waste — upsert ONE tracking - # issue (never a new issue per week) with the headline and failure block - # of both lanes, and close it on the next green run. - - name: Upsert tracking issue on failure - if: always() && github.ref == 'refs/heads/main' && (steps.reconcile.outputs.exit != '0' || steps.gate-reconcile.outputs.exit != '0' || steps.pass-rates.outputs.exit != '0' || needs.eval-slices.result != 'success' || !contains(fromJSON('["success","skipped"]'), needs.eval-codex-slices.result) || needs.gate-census.result != 'success') + # A red census writes one report on every ref: the step summary and the + # census-report artifact. Only main (scheduled weekly, or dispatched on + # main) upserts ONE tracking issue (never a new issue per week) and + # closes it on the next green run; a branch-dispatched census never + # comments on or closes main's issue. Every report-derived line goes + # through the one published-text sanitizer (scripts/lib/published-text.ts). + - name: Write the census report + if: always() && (steps.reconcile.outputs.exit != '0' || steps.gate-reconcile.outputs.exit != '0' || steps.pass-rates.outputs.exit != '0' || needs.eval-slices.result != 'success' || !contains(fromJSON('["success","skipped"]'), needs.eval-codex-slices.result) || needs.gate-census.result != 'success') env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} REDISPATCH: ${{ steps.verdict.outputs.redispatch }} REDISPATCH_OF: ${{ inputs.redispatch_of }} run: | set -euo pipefail - TITLE="Weekly periodic evals: red lane needs triage" - BODY_FILE=/tmp/issue-body.md + published() { # $1 fence|line: stdin lines through the published-text sanitizer + bun --no-install -e 'import { publishedFence, sanitizePublishedText } from "./scripts/lib/published-text"; + const lines = (await Bun.stdin.text()).split("\n").filter(Boolean); + console.log(process.argv[1] === "line" ? sanitizePublishedText(lines.join(" ")) : publishedFence(lines.length ? lines : ["(none)"]).join("\n"));' "$1" + } RUN_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" + GUIDE_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/blob/${GITHUB_SHA}/docs/evals/census-red.md" + bun run eval:pass-rates --reds --runs 10 --branch "$GITHUB_REF_NAME" > /tmp/pass-rates-reds.txt 2>&1 \ + || echo "history unavailable: eval:pass-rates --reds exited $?" >> /tmp/pass-rates-reds.txt + bun run eval:pass-rates --headroom --runs 10 --branch "$GITHUB_REF_NAME" > /tmp/pass-rates-headroom.txt 2>&1 \ + || echo "history unavailable: eval:pass-rates --headroom exited $?" >> /tmp/pass-rates-headroom.txt { - echo "Automated weekly report — run: ${RUN_URL}" + echo "Periodic census report: branch ${GITHUB_REF_NAME} @ ${GITHUB_SHA}, event ${GITHUB_EVENT_NAME}, run ${RUN_URL}" + echo + echo "A census went red: triage with ${GUIDE_URL}" if [ -n "$REDISPATCH_OF" ]; then echo; echo "This run is the one INFRA/INCOMPLETE re-dispatch of run ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${REDISPATCH_OF}; both runs are reported."; fi if [ "$REDISPATCH" = "true" ]; then echo; echo "Every red verdict is machine-classified INFRA/INCOMPLETE: re-dispatching once as a new run (EVAL_POLICY.infraRedispatch). This run stays red and reported."; fi echo echo "- periodic reconciliation exit: ${{ steps.reconcile.outputs.exit }} (slices job: ${{ needs.eval-slices.result }}, Codex slices job: ${{ needs.eval-codex-slices.result }})" echo "- gate census reconciliation exit: ${{ steps.gate-reconcile.outputs.exit }} (census job: ${{ needs.gate-census.result }})" echo "- pass-rate history gate exit: ${{ steps.pass-rates.outputs.exit }}" + echo "- $({ grep -E '^all-green probability' /tmp/pass-rates-reds.txt || echo 'all-green probability: unavailable (see the red ledger)'; } | published line)" echo echo "### Periodic lane" cat /tmp/paid-report/report-summary.md 2>/dev/null || echo "(no periodic report summary)" @@ -730,23 +744,44 @@ jobs: cat /tmp/gate-census-report/report-summary.md 2>/dev/null || echo "(no gate census report summary)" echo echo "### Pass-rate history (ACTION REQUIRED)" - echo '```' - { grep -E 'ACTION REQUIRED|history unavailable' /tmp/pass-rates.txt || echo "(no pass-rate alarms)"; } | sed 's/@/@\xe2\x80\x8b/g' | head -c 6000 - echo '```' + { awk '/^ACTION REQUIRED/ { on = 1; print; next } on && /^ \[/ { print; next } { on = 0 } /history unavailable/ { print }' /tmp/pass-rates.txt 2>/dev/null \ + | grep . || echo "(no pass-rate alarms)"; } | head -c 6000 | published fence + echo + echo "### Red ledger: bun run eval:pass-rates --reds --runs 10 --branch ${GITHUB_REF_NAME}" + { head -n 1 /tmp/pass-rates-reds.txt; tail -n +2 /tmp/pass-rates-reds.txt | tail -n 60; } | cut -c1-400 | published fence + echo + echo "### Session headroom: bun run eval:pass-rates --headroom --runs 10 --branch ${GITHUB_REF_NAME}" + head -n 40 /tmp/pass-rates-headroom.txt | cut -c1-400 | published fence echo echo "
Full reconciliation output" echo - echo '```' - tail -c 6000 /tmp/report.txt 2>/dev/null | sed 's/@/@\xe2\x80\x8b/g' || echo "(no reconciliation output)" - echo '```' + { tail -c 6000 /tmp/report.txt 2>/dev/null || echo "(no reconciliation output)"; } | published fence echo - echo '```' - tail -c 6000 /tmp/gate-report.txt 2>/dev/null | sed 's/@/@\xe2\x80\x8b/g' || echo "(no gate census reconciliation output)" - echo '```' + { tail -c 6000 /tmp/gate-report.txt 2>/dev/null || echo "(no gate census reconciliation output)"; } | published fence echo "
" echo echo "Policy: EVAL_POLICY and CASE_QUARANTINE in test/helpers/periodic-exclude-data.ts; history: \`bun run eval:pass-rates\`." - } > "$BODY_FILE" + } > /tmp/census-report.md + cat /tmp/census-report.md >> "$GITHUB_STEP_SUMMARY" + + - name: Upload the census report + if: always() && (steps.reconcile.outputs.exit != '0' || steps.gate-reconcile.outputs.exit != '0' || steps.pass-rates.outputs.exit != '0' || needs.eval-slices.result != 'success' || !contains(fromJSON('["success","skipped"]'), needs.eval-codex-slices.result) || needs.gate-census.result != 'success') + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: census-report-a${{ github.run_attempt }} + path: /tmp/census-report.md + if-no-files-found: ignore + retention-days: 90 + + - name: Upsert tracking issue on failure + if: always() && github.ref == 'refs/heads/main' && (steps.reconcile.outputs.exit != '0' || steps.gate-reconcile.outputs.exit != '0' || steps.pass-rates.outputs.exit != '0' || needs.eval-slices.result != 'success' || !contains(fromJSON('["success","skipped"]'), needs.eval-codex-slices.result) || needs.gate-census.result != 'success') + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + TITLE="Weekly periodic evals: red lane needs triage" + BODY_FILE=/tmp/census-report.md + [ -s "$BODY_FILE" ] || echo "Periodic census report missing; run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" > "$BODY_FILE" EXISTING=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "in:title \"$TITLE\"" --json number --jq '.[0].number // empty') if [ -n "$EXISTING" ]; then gh issue comment "$EXISTING" --repo "$GITHUB_REPOSITORY" --body-file "$BODY_FILE" diff --git a/.github/workflows/evals.yml b/.github/workflows/evals.yml index 6f196636b..9263e39ab 100644 --- a/.github/workflows/evals.yml +++ b/.github/workflows/evals.yml @@ -191,7 +191,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Restore this PR's verified judge and E2E results id: store @@ -267,7 +267,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 # DX-1: a dispatch from scripts/eval-bg.ts names the revision it means # to test; a different checkout fails here before any paid work. @@ -526,7 +526,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: diff --git a/.github/workflows/free-tests.yml b/.github/workflows/free-tests.yml index 66bd911b6..33e04fae5 100644 --- a/.github/workflows/free-tests.yml +++ b/.github/workflows/free-tests.yml @@ -52,7 +52,7 @@ jobs: fetch-depth: 0 - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - id: plan name: Inventory and balance the complete free suite # Warns in the job summary when more than 5 files are unseeded. @@ -77,7 +77,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install helper dependencies run: bun install --frozen-lockfile --ignore-scripts - name: Build and sign the native macOS startup boundary @@ -112,7 +112,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install dependencies run: bun install --frozen-lockfile --ignore-scripts - name: Build browser prerequisites @@ -144,7 +144,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install helper dependencies run: bun install --frozen-lockfile --ignore-scripts - name: Build the native Windows startup boundary from Git Bash @@ -172,7 +172,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Require local Docker containment prerequisites # A missing daemon is a failed gate, never a skipped integration test. run: | @@ -202,7 +202,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install dependencies run: bun install --frozen-lockfile --ignore-scripts - name: Typecheck product code (zero errors) @@ -227,7 +227,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: @@ -406,7 +406,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: free-plan diff --git a/.github/workflows/make-pdf-gate.yml b/.github/workflows/make-pdf-gate.yml index 75c5992f1..23ae86b66 100644 --- a/.github/workflows/make-pdf-gate.yml +++ b/.github/workflows/make-pdf-gate.yml @@ -53,7 +53,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install dependencies run: bun install --frozen-lockfile diff --git a/.github/workflows/native-qualification.yml b/.github/workflows/native-qualification.yml index c1739c9db..23c63ef36 100644 --- a/.github/workflows/native-qualification.yml +++ b/.github/workflows/native-qualification.yml @@ -41,7 +41,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 with: node-version: 24.18.0 @@ -70,7 +70,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 with: node-version: 24.18.0 @@ -117,7 +117,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 if: inputs.mode == 'dia-launch-comparison' with: diff --git a/.github/workflows/platform-qualification.yml b/.github/workflows/platform-qualification.yml index e3e1782fb..5b8aa4d15 100644 --- a/.github/workflows/platform-qualification.yml +++ b/.github/workflows/platform-qualification.yml @@ -31,7 +31,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install dependencies run: bun install --frozen-lockfile - name: Install Chromium @@ -56,7 +56,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install dependencies run: bun install --frozen-lockfile - name: Install gitleaks @@ -78,7 +78,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install dependencies run: bun install --frozen-lockfile - name: Run Swift build invariants diff --git a/.github/workflows/quality-gate.yml b/.github/workflows/quality-gate.yml index 806c81bd8..2fa5475fa 100644 --- a/.github/workflows/quality-gate.yml +++ b/.github/workflows/quality-gate.yml @@ -44,7 +44,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install frozen dependencies run: bun install --frozen-lockfile --ignore-scripts diff --git a/.github/workflows/skill-docs.yml b/.github/workflows/skill-docs.yml index 9a8124633..b9585bdfb 100644 --- a/.github/workflows/skill-docs.yml +++ b/.github/workflows/skill-docs.yml @@ -25,7 +25,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - run: bun install --frozen-lockfile # One generation pass for ALL 10 hosts. gen-skill-docs --host all # hard-fails on any per-host generation error (scripts/gen-skill-docs.ts diff --git a/.github/workflows/test-health.yml b/.github/workflows/test-health.yml index f09423cd3..5bb711963 100644 --- a/.github/workflows/test-health.yml +++ b/.github/workflows/test-health.yml @@ -38,7 +38,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Install dependencies run: bun install --frozen-lockfile diff --git a/.github/workflows/version-gate.yml b/.github/workflows/version-gate.yml index 3468fbd5f..05aad51f7 100644 --- a/.github/workflows/version-gate.yml +++ b/.github/workflows/version-gate.yml @@ -29,7 +29,7 @@ jobs: - name: Setup Bun uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - name: Read versions id: versions diff --git a/.github/workflows/windows-free-tests.yml b/.github/workflows/windows-free-tests.yml index f579fec47..0b8adec1d 100644 --- a/.github/workflows/windows-free-tests.yml +++ b/.github/workflows/windows-free-tests.yml @@ -51,7 +51,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - id: plan name: Curate and balance the Windows-safe free suite # Six shards: ~85 s of Linux-measured work each, ~2.5 min on a @@ -82,7 +82,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 with: @@ -190,7 +190,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: windows-plan @@ -213,7 +213,7 @@ jobs: persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 with: node-version: 24.18.0 diff --git a/.github/workflows/windows-setup-e2e.yml b/.github/workflows/windows-setup-e2e.yml index 906598346..1216f6624 100644 --- a/.github/workflows/windows-setup-e2e.yml +++ b/.github/workflows/windows-setup-e2e.yml @@ -45,7 +45,7 @@ jobs: - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: - bun-version: 1.4.0 + bun-version: 1.4.2 # Same lockfile-keyed install cache as windows-free-tests.yml (install # was 45s of a 64s job, all network). diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4f9345feb..eb7735ac0 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -6,7 +6,7 @@ stages: - check variables: - BUN_VERSION: "1.4.0" + BUN_VERSION: "1.4.2" .setup-bun: &setup-bun - apt-get update -qq && apt-get install -qq -y curl jq git diff --git a/AGENTS.md b/AGENTS.md index d2d545325..6c86be265 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -213,8 +213,8 @@ When fixing failures or preparing `/ship`, follow this order: do not satisfy coverage; preserve every attempt. Paid evals never retry. Each case's kind (`E2E_KINDS`) fixes its trials before the run: `rule` one trial; `behavior` a panel of 3 independent trials, PASS at >= 2 with no contract - violation; `judge` 3 samples on one output, gated on the mean against the - unchanged threshold. Never add trials, samples or dispatches after seeing a + violation; `judge` 3 samples on one output, each dimension gated on its + median (at least 2 of 3 samples) against the unchanged threshold. Never add trials, samples or dispatches after seeing a result, never change a kind to change a verdict without pass-rate evidence, and report every trial. Quarantine follows `CASE_QUARANTINE`'s entry and exit rules only (`EVAL_POLICY`, `docs/TESTING_INTERNALS.md`). A census whose every diff --git a/CHANGELOG.md b/CHANGELOG.md index 9da7ab543..8977338b3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,76 @@ # Changelog +## [1.91.30.0] - 2026-10-05 + +**A failed eval now says what happened, with the evidence and the next command.** +**Bun is checked before anything installs, and make-pdf and /review work in Codex.** + +This release follows up on the severe fix wave. Its eight census runs had 21 red verdicts, and 27 of the 28 failed trials were labelled `assertion`, including timeouts and provider refusals, so the reports misdescribed their own failures. Now every failed trial records a machine-checked cause beside the unchanged pass/fail class, and each red line in the census report carries the expected and received values (or the judge's failing scores), the artifact holding the transcript, and a command that reruns only that case. For everyone using gstack: setup refuses a Bun too old to keep a project's `.env` away from gstack's tools, auto-update checks that before moving your checkout, and make-pdf and /review work inside Codex. + +### The numbers that matter + +Each row is a regression test on this release that fails on v1.91.19.0, or a measurement from the 2026-10-03/04 census artifacts named in the row. + +| Check | v1.91.19.0 | v1.91.30.0 | +|---|---|---| +| make-pdf blocks on Codex that use `$P` without finding it | 7 of 8 | 0 | +| /review with a read-only `.git` (Codex's sandbox) | prints nothing, "Nothing to review" | reviews against the local base, says `stale at ` | +| `./setup` with Bun 1.3.2 | installs; compiled tools read the project's `.env` | refused before writing anything | +| Census cost line when billing was not captured | `$0.00` | known sum plus "N of M trial(s) cost unknown" | +| plan-mode-no-op recorded wall (census 37151477069) | 346 s (its test times summed) | 176 s (the shard's wall) | +| Gate census slowest slice, replanned with measured walls (census 37198445662) | ~677 s | <=539 s planned | + +The first three rows are the ones users feel. The last three are why the next census reports are believable: costs, times and causes now mean what they say. + +### What changes for you + +- **Failure reports explain themselves.** A red census line keeps its error and adds `cause : ` (refusal, API error, provider stall, session or observer timeout, assertion), Expected/Received or `dimension mean < threshold (n samples) "rationale"`, `evidence: ` and `after a repair: `. Model-written text is sanitized before it reaches an issue or PR comment. +- **Bun floor and safe auto-update.** Setup refuses Bun older than 1.3.3 and warns below 1.4.0, the tested version. Team-mode auto-update and `/gstack-upgrade` read the incoming release's Bun requirement first and hold the update, leaving your install untouched, when your Bun is too old. The first update run by an older auto-updater cannot do this check. +- **make-pdf on Codex** and the other env-var hosts finds its binary in every block, and `MAKE_PDF_BIN` works there too. Install paths with spaces work. +- **/review in Codex's sandbox.** The working-tree fingerprint writes its git objects to a private temp directory, and a base that cannot be fetched is reported as stale instead of ending the review. +- **/freeze, /guard, /unfreeze and /investigate's edit boundary work on Codex** and the other env-var hosts. Their scripts used a path a global install does not have, and setup never installed `freeze/bin`; both are fixed, and /benchmark-models, /retro and /diagram find their helpers the same way. +- **`browse stop` reaps a surviving headless Chromium** again on Playwright 1.62, only for browsers gstack launched. +- **/ship's documentation gate** records post-audit file hashes with its helper instead of having the model retype them. +- **/plan-ceo-review** shows its mode line (`Mode: ...` or `Auto-decided review mode → ...`) word for word right after the mode is chosen. +- **/qa** names where its report filename parts come from and labels timing as probe budget and guarded command time. **/deslop-shared-libs** names a concrete risk for every recommendation. +- **/office-hours spec review stops when nothing blocking is left.** Each finding is blocking or minor; minor findings are recorded in Reviewer Concerns and never force another round, and the parent fixes only blocking ones between rounds. Rounds 2 and 3 re-review the exact design diff the helper captured since the last round. The reviewer returns a one-line sha256 receipt for its saved verdict instead of echoing it. Callers can lower the 3-round cap with `--max-rounds`. +- **`GSTACK_DESIGN_IMAGE_MODEL`** switches the design tool's image model (default `gpt-image-2`); a value that is not a gpt-image model name is refused before any request. + +### Behavior changes you may notice + +- **LLM judges pass a dimension when 2 of their 3 samples meet its threshold** (EVAL_POLICY v3, the per-dimension median; it was the mean, so one low sample beside two passing ones failed the panel). Thresholds, sample count, judge model and prompts are unchanged; panel logs still show every sample and the mean. Policy v3 and HARNESS_VERSION 2 start a new pass-rate series for every case. +- **Office-hours review verdicts are schema version 2** (`severity` and `changed_text` on every finding). An in-progress review saved by an older version is refused; start a fresh review directory. +- **Bun below 1.3.3 stops setup** with `gstack needs Bun 1.3.3 or newer`; nothing is installed. Fix: `bun upgrade`, then `./setup`. +- **Auto-update can say `update held (bun-too-old: ...)`** and `/gstack-upgrade` can stop with `BUN_TOO_OLD`; your current version keeps working. Fix: `bun upgrade`. +- **/review in a read-only checkout reports `Base coverage: stale at `** instead of stopping with "Nothing to review". +- **Browser Quick QA has a 3-minute budget** (was 30 seconds, too short for a homepage and five pages with their checkpoints). +- **/review's plan-completion question is not asked in spawned or non-interactive runs;** they report REQUIREMENTS MISSING and continue without recording anything. +- **`live-model-check.ts` ignores `GSTACK_DESIGN_MODEL`** and always checks the default models. +- **A census dispatched on a branch no longer comments on or closes main's tracking issue** (periodic and marathon). Its report goes to the run's step summary and a `census-report` (or `marathon-report`) artifact; main posts as before. +- For contributors: `bun run eval:pass-rates` rejects unknown flags and case ids (exit 2) and `--gate` also fails when a session ran above 85% of its armed budget; every `claude -p` eval session streams partial messages (only a liveness summary is kept) and writes `session-ledger.jsonl`; overlay comparisons count an overlay-off wrong answer as a measured result (contract v4); the AUQ matrix scores recommendation substance with a 3-sample judge panel. + +### Itemized changes + +#### Added +- `bun run eval:pass-rates --headroom` (slowest armed session per case against its budget), `--reds` (verdict reds by class and cause, with the all-green probability and its interval) and `--run ` (one census's reds, downloading only the slices they name); `--help` works offline. +- The weekly tracking issue shows every pass-rate alarm line (it showed only the header), the red ledger by class and cause, session headroom, the all-green probability, and a link to the new guide `docs/evals/census-red.md` (from a red census line to a diagnosis with free commands). The marathon issue closes itself on the next green main run. +- Periodic cases `codex-multiblock-live` (installs gstack for Codex into a fresh `CODEX_HOME` and runs a later /learn block that must find gstack on its own) and `design-model-smoke` (the design tool's default OpenAI models; reported as skipped without `OPENAI_API_KEY`, never as a pass). +- `docs/troubleshooting.md` sections for the Bun floor, held updates, stale /review bases, fingerprint temp-dir errors, the image model override, an unrecorded Chromium PID, and census report lines. + +#### Fixed +- `bun run scripts/test-paid-shards.ts --case ` selects cases registered in a loop and whole-file cases such as plan-mode-no-op; it used to select nothing. +- The plan-review floor test approves its own plan-file edit when the diff preview soft-wraps unchanged lines. +- The /plan-ceo-review approach-menu eval no longer asks for "verbatim ... exact prose" output, which tripped the provider's output-extraction refusal. +- `gstack-docs-candidate --help` prints its usage and exits 0; its usage error pointed at a `--help` that failed. The docs gate fixtures declare it, and an "atomic replacement unproven" line now names the check and the tool failure that denied it. +- The prosons neutral-posture check no longer reads "a coverage call, not a taste call" as the neutral dodge. + +#### For contributors +- Session ledger (`test/helpers/session-ledger.ts`), a pure failure-cause classifier beside `failureClassOf`, and one sanitizer for published text (`scripts/lib/published-text.ts`). Verdicts, `failure_class` and `error` are unchanged under EVAL_POLICY v1 (main and branch report code agree on all 15 stored census lanes). +- Replay corpora for five repeat-offender detectors in the free suite, and an audit of phrase-only detectors (`docs/evals/detector-inventory.md`). The CEO auto-decide detector credits the mode handoff only when the chat shows its line. +- Judge calibration controls (`docs/evals/judge-controls-2026-10.md`). The workflow judges passed bundles with a required step removed, so `test/workflow-required-steps.test.ts` now checks every step and phase heading of the judged workflow skills. +- Paid shard durations refreshed from census 37198445662; a JUnit case's recorded wall is capped by its shard's wall. +- The docsync fixture saves its observation interface to a file its children read instead of having the parent copy it into each child prompt. + ## [1.91.29.0] - 2026-10-05 **Native runtime qualification can pass for Node, Bun, Python and Rails.** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8947eab8e..4cdcd74fa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -182,9 +182,9 @@ the new defaults. | I want to… | Command | Cost / time | Needs | |---|---|---|---| -| Check an ordinary edit quickly | `bun run test:quick` | Free, about a minute | Bun 1.4.0 | +| Check an ordinary edit quickly | `bun run test:quick` | Free, about a minute | Bun 1.4.2 | | Run one free test file while repairing | `bun test ` | Free, seconds | Never bare `bun test` for the suite | -| Run full free acceptance before publishing | `bun run test` | Free, a few minutes | Bun 1.4.0 | +| Run full free acceptance before publishing | `bun run test` | Free, a few minutes | Bun 1.4.2 | | Run the full free suite from a small machine | `bun run test:ubicloud` | Free suite on a billed 16-vCPU VM, about 5 minutes | `UBICLOUD_API_KEY` | | Run the curated Windows-safe subset | `bun run test:windows` (CI: six `windows-latest` jobs) | Free | Windows, Git Bash | | Refresh the Windows duration seed | `gh workflow run windows-free-tests.yml --ref -f record_durations=true`, then commit the `free-test-durations-windows` artifact | Free CI runners | Pushed branch, `gh` with workflow rights | @@ -207,7 +207,7 @@ Old command names are listed under [Retired commands](#retired-commands). ### Setup -Development and tests require Bun 1.4.0 or newer; CI pins and tests 1.4.0. +Development and tests require Bun 1.4.2 or newer; CI pins and tests 1.4.2. Earlier Linux versions can close unrelated live file descriptors during subprocess garbage collection, causing intermittent browser and HTTP fixture failures ([upstream diagnosis](https://github.com/oven-sh/bun/issues/34785#issuecomment-5020318035)). @@ -284,9 +284,10 @@ constants in `EVAL_POLICY` (`test/helpers/periodic-exclude-data.ts`): PASS at 2 or more with no contract violation (`expectContract()`). Use it only when a live model choice decides the verdict and an occasional deviation is acceptable product behavior; the one-line reason goes in `BEHAVIOR_WHY`. -- `judge`: an LLM judge scoring a fixed input; 3 samples of the same prompt, - gated on the per-dimension mean (booleans on a majority) against the - unchanged threshold. An erroring sample fails the panel and is never resampled. +- `judge`: an LLM judge scoring a fixed input; exactly 3 samples of the same + prompt, each dimension gated on its median, at least 2 of 3 samples, against + the unchanged threshold (booleans on a majority; the mean is reported only). + An erroring sample fails the panel and is never resampled. A timed-out, crashed or infrastructure-failed trial counts as a failed trial and is reported with its class; a missing trial makes the case INCOMPLETE, which @@ -296,6 +297,13 @@ this policy. Quarantine (`CASE_QUARANTINE`) and history are described in `docs/TESTING_INTERNALS.md`; `bun run eval:pass-rates --case ` shows a case's per-trial pass rate with its Wilson interval. +**A census went red?** Follow [docs/evals/census-red.md](docs/evals/census-red.md): +read the red line, inspect its evidence with `bun run eval:pass-rates --run `, +check `--reds` and `--headroom`, repair with a free regression test, and only +then spend one paid run with the line's `after a repair:` command. A red census +on `main` is reported on the weekly tracking issue; a census dispatched on a +branch writes the same report to its run summary and `census-report` artifact. + CI enables verified first-attempt reuse for 16 workflow quality judges for 24 hours within the same PR. The cookie workflow's custom input and the other 11 quality cases stay fresh. PR-profile E2E shards that run once (no retry, so the @@ -477,6 +485,9 @@ bun run eval:list # list all eval runs (turns, duration, cost per run bun run eval:compare # compare two runs — shows per-test deltas + Takeaway commentary bun run eval:summary # aggregate stats + per-test efficiency averages across runs bun run eval:pass-rates # per-case trial pass rates + Wilson intervals from recent weekly runs (--case, --runs, --dir, --backfill, --json, --gate) +bun run eval:pass-rates --reds # verdict reds per census by failure class and cause, all-green probability +bun run eval:pass-rates --headroom # slowest session per case vs its armed budget (alarm above 85% in --gate) +bun run eval:pass-rates --run # one census's reds with their values and fetched transcript evidence ``` **Detached runs for agents and long suites.** When an agent (or you, for a run @@ -534,8 +545,8 @@ Override the judge model per run with `GSTACK_EVAL_MODEL_JUDGE`: - **Actionability** — Can the agent execute tasks using only the information in the doc? Each dimension is scored 1-5 by a panel of 3 samples of the same prompt, drawn -concurrently; each dimension's panel mean must meet that judge's threshold (≥ 4 -for most dimensions; see each case). An erroring sample fails the panel. There's also a regression test that compares generated docs against the hand-maintained baseline from `origin/main` — generated must score equal or higher. +concurrently; each dimension's panel median (at least 2 of 3 samples) must meet +that judge's threshold (≥ 4 for most dimensions; see each case). An erroring sample fails the panel. There's also a regression test that compares generated docs against the hand-maintained baseline from `origin/main` — generated must score equal or higher. Needs `ANTHROPIC_API_KEY` in `.env`. The judge files run in every paid lane (`bun run eval:bg:pr` selects the ones your diff touches). diff --git a/README.md b/README.md index 2a6d9b854..d35450885 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ Fork it. Improve it. Make it yours. And if you want to hate on free open source `./setup` took 31 seconds after the clone on a 4-vCPU Linux cloud machine (v1.91.13.0, clean HOME, including the binary build and the Chromium download). Slower networks and laptops take longer. -**Requirements:** [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Git](https://git-scm.com/), [Bun](https://bun.sh/) v1.0+, [Node.js](https://nodejs.org/) (Windows only). **Recommended on macOS:** the [Aside](https://aside.com) browser (macOS 15+) — browser skills, `/make-pdf`, and `/diagram` drive it first, with your real logged-in sessions. Without it, `./setup` builds gstack's own bundled browser and the same skills use that. `/cso` additionally needs a Bun release with all four `--no-compile-autoload-*` build flags plus a native toolchain: a static-capable C compiler on Linux, Xcode command-line tools on macOS, or Visual Studio 2022 Build Tools with Desktop development with C++ on Windows. If those are absent, setup installs everything else, removes stale CSO helpers, and `/cso` reports `not assessed` with the prerequisite. +**Requirements:** [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Git](https://git-scm.com/), [Bun](https://bun.sh/) v1.4.2+ (the tested version; setup refuses Bun older than 1.3.3, which would let compiled tools read a project's `.env`, and warns in between), [Node.js](https://nodejs.org/) (Windows only). **Recommended on macOS:** the [Aside](https://aside.com) browser (macOS 15+) — browser skills, `/make-pdf`, and `/diagram` drive it first, with your real logged-in sessions. Without it, `./setup` builds gstack's own bundled browser and the same skills use that. `/cso` additionally needs a Bun release with all four `--no-compile-autoload-*` build flags plus a native toolchain: a static-capable C compiler on Linux, Xcode command-line tools on macOS, or Visual Studio 2022 Build Tools with Desktop development with C++ on Windows. If those are absent, setup installs everything else, removes stale CSO helpers, and `/cso` reports `not assessed` with the prerequisite. When qualified CSO runtime images are published, setup gives each automatic preload a 30-second window plus a bounded setup allowance for the declared catalog. For slower registries, set an integer such as `GSTACK_CSO_IMAGE_PULL_TIMEOUT_SECONDS=120` (accepted range: 5–300 seconds). One image timing out does not consume the remaining images' windows; setup reports partial progress and a later run resumes from exact digests already present in local Docker. The complete preload is capped at one hour. @@ -186,8 +186,12 @@ for capture, judge, and benchmark model selection. The design binary (`$D`) uses OpenAI's `gpt-5.5` by default, with image generation through `gpt-image-2`. Set `GSTACK_DESIGN_MODEL=` to use another gpt-5-class model for both image generation and screenshot analysis; -if OpenAI rejects it, the error names `GSTACK_DESIGN_MODEL`. Check a key -against the defaults with `bun run design/scripts/live-model-check.ts`. +if OpenAI rejects it, the error names `GSTACK_DESIGN_MODEL`. Set +`GSTACK_DESIGN_IMAGE_MODEL=` to change the image tool model; a +value that is not a gpt-image model name is refused before any request. Check a +key against the defaults with `bun run design/scripts/live-model-check.ts`, +which always tests the default models and ignores both overrides; the weekly +periodic census runs the same check. **Want to add support for another agent?** See [docs/ADDING_A_HOST.md](docs/ADDING_A_HOST.md). Rendering a new agent is one TypeScript config file; installing it also needs a diff --git a/TODOS.md b/TODOS.md index 9e81b6dac..ca00db47d 100644 --- a/TODOS.md +++ b/TODOS.md @@ -2,6 +2,96 @@ ## NEXT PRIORITY +### P2/P3: reliability follow-ups deferrals (filed 2026-10-04, v1.91.30.0) + +Deferred from the approved reliability follow-ups plan (docs/designs/RELIABILITY_FOLLOWUPS_2026_10.md), each with its reason. + +- **Free-suite git hang behind shard neighbors (B5): Bun spawnSync loses the + child's exit (oven-sh/bun#34069, open upstream)** — CI shard 1 (PR runs + 37242691914 and earlier) timed out `execFileSync`/`spawnSync` git calls in + redact-prepush-rebase-force-push, render-sections-provenance and + shared-libs-stage-actor. A watcher on Ubicloud caught it: every "hung" git is a + zombie (exited, never reaped) while the `bun test` main thread runs, matching + the upstream report. CI's shard-1 file list reproduces it in about 3 of 60 + runs (none of 20 without pty-workspace-trust, not significant). Next step: + move the git calls in shared fixture helpers to async spawn with a hard + timeout, or take a Bun release that fixes #34069. **Effort:** M. **Priority:** P2. +- **Judge rubric anchors (C1, needs Garry: EVAL_POLICY)** — the pre-registered + control panels (`docs/evals/judge-controls-2026-10.md`, 36 calls) showed + every workflow judge separates gross breakage, but the qa workflow judge + passed a /qa bundle with Phase 10 Report removed (no sample noticed) and the + review judge passed /review with Step 3 removed (two of three samples noticed + it and rated it minor); no positive sample scored 5 on any dimension. + Missing-step coverage now lives in `test/workflow-required-steps.test.ts`. + Proposal: anchor completeness/actionability 4 and 5 to named deliverables + and inputs, as a pre-registered EVAL_POLICY change. **Effort:** M. **Priority:** P2. +- **auto-decide-preserved: the CEO handoff line must reach the chat verbatim** + — the native detector now credits the helper's AUTO_DECIDE line only when a + chat message begins with it (Claude Code collapses the Bash result). One + diagnostic run on the pre-verbatim wording paraphrased it; one on the + verbatim wording passed. Watch the next censuses; a red here is a template + miss, not a detector false red. **Effort:** S. **Priority:** P2. +- **Named red: ship-docsync-late-result on PR run 37237194905** — the model + wrote its report to a mistyped shard root (`…-3GanyY` for `…-3GanyU`) at + call 20, disclosed it, then wrote the real report at the right path. The + authority detector is correct (the file was created outside the fixture); + `test/docsync-authority.test.ts` replays the capture. If any docsync case + writes to a near-miss shard path again, give the model a short stable + fixture root (env var or fixed short symlink) instead of random tokens to + copy, without loosening the authority check. **Effort:** S. **Priority:** P2. +- **PTY sessions are cost-unknown** — `test/helpers/pricing.ts` excludes + cache-write charges, so PTY trials stay `cost_known: false`. Price them from + native transcript usage once pricing covers cache writes. **Effort:** S. **Priority:** P3. +- **Custom SDK query providers stream no partial messages** — overlay-case, + the ship hook/skip actors, workflow-boundaries and the setup-gbrain sandbox + record raw events, so their ledger rows carry no stall evidence (never + `provider_stall`). Let them drop `stream_event` rows and opt in to stream + liveness. **Effort:** S. **Priority:** P3. +- **Name PTY sessions by case** at their call sites so multi-case shards + attribute ledger rows (single-case shards already do). **Effort:** S. **Priority:** P3. +- **Red lines link the native-captures artifact** — they name the slice + artifact; `native-captures-` embeds the job id, which the + report job does not know. **Effort:** S. **Priority:** P3. +- **EVAL_POLICY v2: provider_stall as INFRA** — decide with two censuses of + `eval:pass-rates --reds` cause data (labels only today). **Effort:** S. **Priority:** P3 (Garry). +- **Detector convert candidates (B2)** — `docs/evals/detector-inventory.md` + lists 38 phrase-only checks with a structured source (top: S5 screen-first + auto-decision, PTY plan_ready/silent_write from journal tool calls, setup + selectors keyed on wording, the bws READY probe), and periodic "surface" + checks that match the injected SKILL.md body (context-restore-*, + scrape-prototype-path). **Effort:** M. **Priority:** P3. +- **Retain the owned transcript tail in floor captures** — PR run 37176835584's + public transcript never showed the pending Edit, so the B3 parser fix may not + have been sufficient there. **Effort:** S. **Priority:** P3. +- **CEO decision procedure latency** (~60 s per question: grid, save, + Read-back, ask) bounds plan-ceo-mode-routing's 240 s post-selection window. + Named, not cut. **Effort:** M. **Priority:** P3. +- **Value-card JSON wording** — two of three qa-only judge samples still say + "written JSON keeps full values" names no file. Name the JSON per mode (ship + Step 7, /test-audit's sidecar) and drop the clause where no JSON carries + cards. **Effort:** S. **Priority:** P3. +- **Windows check of E4's private object directory** — + `GIT_ALTERNATE_OBJECT_DIRECTORIES` uses `;` on Git for Windows (only matters + when alternates are already set) and the `cygpath -m` conversion is + unverified there. **Effort:** S. **Priority:** P3. +- **Fresh /ship marathon case** — the marathon lane runs a fresh + /plan-eng-review; a fresh /ship journey case is new harness work. + **Effort:** M. **Priority:** P3. +- **Refresh the E5/E6 duration seeds** — `codex-multiblock-live` (180 s) and + `design-model-smoke` (90 s) are estimates in `scripts/paid-test-durations.json`; + refresh them with `--write-durations` from their first census, and close + "Periodic paid smoke for the default design model pairing" once it has run + there. **Effort:** S. **Priority:** P3. +- **Env-var hosts: optional runtime files outside the runtime root** — the + Chrome extension lookup in /open-gstack-browser and /connect-chrome, and the + prose pointers to `$GSTACK_ROOT/docs/...` and `$GSTACK_ROOT/scripts/...` + (question-split guide, jargon list, question registry) name files the + Codex/Factory runtime roots do not link. Executed fences are covered by + `test/env-host-runtime-assets.test.ts`; these reads are not. **Effort:** S. **Priority:** P3. +- **Product A/B against the stock host agent** — compare gstack end to end + with the stock Claude Code/Codex agent on product outcomes (the overlay + harness covers model-level deltas only). Its own project. **Effort:** L. **Priority:** P3. + ### P1–P3: test/eval/CI audit wave follow-ups (filed 2026-10-04) From the approved 2026-10 test/eval/CI audit plan; the audit PR files these. @@ -35,7 +125,7 @@ From the approved 2026-10 test/eval/CI audit plan; the audit PR files these. Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10.md), each with its reason. -- **Enforce the Bun floor at setup (#2815)** — `engines.bun` is now `>=1.4.0` +- ~~**Enforce the Bun floor at setup (#2815)**~~ — done in v1.91.30.0: setup refuses Bun below 1.3.3 and warns below 1.4.2; auto-update and /gstack-upgrade check the incoming floor before moving the checkout. Original note: — `engines.bun` is now `>=1.4.0` (the CI pin), but Bun does not enforce `engines` and setup never checks the version. Bun older than 1.3.3 silently ignores `--no-compile-autoload-dotenv` and `--no-env-file` (verified with 1.3.2), so D0's protection against a @@ -49,12 +139,12 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 - **C8 router wording behind a routing panel** — disabled skills now leave the router, but the "When in doubt, invoke the skill" wording change (Tier 3) was not run through a routing eval panel. **Effort:** S. **Priority:** P3. -- **Cross-block variables on fresh-shell hosts: make-pdf `$P`** — make-pdf +- ~~**Cross-block variables on fresh-shell hosts: make-pdf `$P`**~~ — done in v1.91.30.0: env-var hosts re-derive `$P` in every block (MAKE_PDF_BIN honored). Original note: — make-pdf "exports" `$P` for later blocks, which Codex and the other env-var hosts lose between blocks (same class as `$B`/`$D`, which the shared prelude now re-derives). Add `P` to the runtime prelude or resolve it per block. **Effort:** S. **Priority:** P2. -- **#2709's recorded-Chromium reap never runs on Playwright 1.62** — Playwright +- ~~**#2709's recorded-Chromium reap never runs on Playwright 1.62**~~ — done in v1.91.30.0: the owned headless launch records the browser PID over CDP `SystemInfo.getProcessInfo`. Original note: — Playwright 1.62's `Browser` has no `process()`, so `chromiumProcInfo` is always null and the cleanup is a no-op (verified by probe). Record the PID another way (CDP `SystemInfo.getProcessInfo` or the launch server). **Effort:** S. **Priority:** P2. @@ -201,7 +291,7 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 36898545245, 36787269090, 36641820398, 36633323521) across skills and checks (a two-`(recommended)` option count; office-hours substance). It is kind `rule` but gates on one judge sample. Decide under EVAL_POLICY whether it is - a `judge` case (3 samples, mean) and supply pass-rate evidence before + a `judge` case (3 samples, median) and supply pass-rate evidence before changing the kind. Effort S. **Priority:** P2. - **qa/SKILL.md workflow judge dips below its actionability floor** — census 37186854666 (b24b6d8) scored actionability 3.67 < 4 (clarity 3, completeness diff --git a/VERSION b/VERSION index ad61f5dd7..4609f2bd2 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.29.0 +1.91.30.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 09b43a5fd..aa4dfe875 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.29.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.30.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/benchmark-models/SKILL.md.tmpl b/benchmark-models/SKILL.md.tmpl index 27a0413cd..2d804237f 100644 --- a/benchmark-models/SKILL.md.tmpl +++ b/benchmark-models/SKILL.md.tmpl @@ -37,7 +37,7 @@ Different from `/benchmark` — that skill measures web page performance (Core W ## Step 0: Locate the binary ```bash -BIN="$HOME/.claude/skills/gstack/bin/gstack-model-benchmark" +BIN="{{RUNTIME_ROOT}}/bin/gstack-model-benchmark" [ -x "$BIN" ] || BIN=".claude/skills/gstack/bin/gstack-model-benchmark" [ -x "$BIN" ] || { echo "ERROR: gstack-model-benchmark not found. Run ./setup in the gstack install dir." >&2; exit 1; } echo "BIN: $BIN" diff --git a/bin/gstack-bun-version.sh b/bin/gstack-bun-version.sh new file mode 100644 index 000000000..b7b8ccc5a --- /dev/null +++ b/bin/gstack-bun-version.sh @@ -0,0 +1,75 @@ +# shellcheck shell=bash +# gstack-bun-version.sh — the one source for the Bun versions gstack needs. +# Sourced, never executed: ./setup refuses or warns from it before writing +# anything, and bin/gstack-session-update and /gstack-upgrade read the +# INCOMING release's floor (gstack_bun_incoming_hold) before advancing a +# live checkout. test/bun-version-drift.test.ts +# keeps it equal to package.json engines.bun, every CI pin, setup's install +# hint and README. Bash 3.2 builtins only. +# +# GSTACK_BUN_FLOOR is the security floor. Older Bun accepts +# --no-compile-autoload-dotenv/--no-compile-autoload-bunfig but ignores them, +# so gstack's compiled tools would read a project's .env. Verified by a +# compiled probe (a binary built with all four no-autoload flags, run beside a +# .env): Bun 1.3.2 loads it, 1.3.3 and 1.4.0 do not. +# GSTACK_BUN_TESTED is the supported minimum: the CI pin and engines.bun. +GSTACK_BUN_FLOOR="1.3.3" +GSTACK_BUN_TESTED="1.4.2" + +# _gstack_bun_parse — sets _gb_core ("MAJOR MINOR PATCH" as decimal +# integers) and _gb_pre (prerelease, may be empty); returns 1 when malformed. +# A leading "v" and "+build" metadata are ignored. +_gstack_bun_parse() { + _gb_v="${1#v}"; _gb_v="${_gb_v%%+*}"; _gb_pre="" + case "$_gb_v" in + *-*) _gb_pre="${_gb_v#*-}"; _gb_v="${_gb_v%%-*}"; [ -n "$_gb_pre" ] || return 1 ;; + esac + case "$_gb_v" in *.*.*) ;; *) return 1 ;; esac + _gb_maj="${_gb_v%%.*}"; _gb_rest="${_gb_v#*.}"; _gb_min="${_gb_rest%%.*}"; _gb_pat="${_gb_rest#*.}" + case "$_gb_maj" in ''|*[!0-9]*) return 1 ;; esac + case "$_gb_min" in ''|*[!0-9]*) return 1 ;; esac + case "$_gb_pat" in ''|*[!0-9]*) return 1 ;; esac + case "$_gb_pre" in *[!0-9A-Za-z.-]*) return 1 ;; esac + _gb_core="$((10#$_gb_maj)) $((10#$_gb_min)) $((10#$_gb_pat))" +} + +# _gstack_bun_below — true when version sorts below the +# release. Prerelease rule (semver): 1.3.3-canary.1 is below 1.3.3 and above +# 1.3.2, so a prerelease of the floor itself is refused. +_gstack_bun_below() { + _gstack_bun_parse "$2" || return 1 + set -- "$1" $_gb_core + _gstack_bun_parse "$1" || return 1 + set -- "$@" $_gb_core + [ "$5" -ne "$2" ] && { [ "$5" -lt "$2" ]; return; } + [ "$6" -ne "$3" ] && { [ "$6" -lt "$3" ]; return; } + [ "$7" -ne "$4" ] && { [ "$7" -lt "$4" ]; return; } + [ -n "$_gb_pre" ] +} + +# gstack_bun_status [floor] — prints too-old, untested, ok or +# malformed. The optional floor lets the auto-updater apply the incoming +# release's floor with this release's comparison rule. +gstack_bun_status() { + if ! _gstack_bun_parse "$1"; then echo malformed + elif _gstack_bun_below "$1" "${2:-$GSTACK_BUN_FLOOR}"; then echo too-old + elif _gstack_bun_below "$1" "$GSTACK_BUN_TESTED"; then echo untested + else echo ok + fi +} + +# gstack_bun_incoming_hold — the pre-advance check shared by +# the auto-updater and /gstack-upgrade. Reads the floor from 's copy of +# this file and checks the `bun` setup will run (first on PATH). Below that +# floor it prints the held reason and returns 0; otherwise it prints nothing +# and returns 1 (no bun, no floor in , or an unparseable version, which +# setup only warns on). The comparison rule is this release's. +gstack_bun_incoming_hold() { + command -v bun >/dev/null 2>&1 || return 1 + _gb_floor=$(git -C "$1" show "$2:bin/gstack-bun-version.sh" 2>/dev/null | sed -n 's/^GSTACK_BUN_FLOOR="\([0-9A-Za-z.+-]*\)".*/\1/p' | head -1) + [ -n "$_gb_floor" ] || return 1 + _gb_found=$(bun --version 2>/dev/null | head -1) + [ "$(gstack_bun_status "$_gb_found" "$_gb_floor")" = too-old ] || return 1 + _gb_ver=$(git -C "$1" show "$2:VERSION" 2>/dev/null | head -1) + echo "bun-too-old: found Bun $_gb_found at $(command -v bun); gstack ${_gb_ver:-update} needs $_gb_floor or newer" +} diff --git a/bin/gstack-docs-candidate b/bin/gstack-docs-candidate index 79dd2fdcb..880f48cdf 100755 --- a/bin/gstack-docs-candidate +++ b/bin/gstack-docs-candidate @@ -83,8 +83,15 @@ function one(args: string[], name: string): string { return values[0]!; } +const USAGE = `usage: gstack-docs-candidate snapshot --out --audit-id --mode edit|read-only --base [--select ]... [--docs ]... + gstack-docs-candidate compare `; + function main(argv: string[]): void { const [command, ...args] = argv; + if ((command === '--help' || command === '-h') && !args.length) { + console.log(USAGE); + return; + } if (command === 'snapshot') { const known = new Set(['--out', '--audit-id', '--mode', '--base', '--select', '--docs']); for (let i = 0; i < args.length; i += 2) if (!known.has(args[i]!)) throw new Error(`unknown argument: ${args[i]}`); @@ -131,7 +138,7 @@ function main(argv: string[]): void { })); return; } - throw new Error('usage: gstack-docs-candidate snapshot|compare (see --help in the file header)'); + throw new Error(USAGE); } try { main(process.argv.slice(2)); } diff --git a/bin/gstack-office-hours-review b/bin/gstack-office-hours-review index 125c08d1b..4716e1d14 100755 --- a/bin/gstack-office-hours-review +++ b/bin/gstack-office-hours-review @@ -2,30 +2,54 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { atomicWriteSync } from '../lib/fs-atomic.ts'; -import { assessOfficeHoursReviews, loadOfficeHoursReviews, renderOfficeHoursReview, renderOfficeHoursReviewerPrompt, replaceOfficeHoursReviewBlock } from '../lib/office-hours-review.ts'; +import { assessOfficeHoursReviews, loadOfficeHoursReviews, officeHoursDesignChanges, officeHoursMaxRounds, officeHoursSnapshotChanges, officeHoursVerdictReceipt, parseOfficeHoursVerdictReceipt, renderOfficeHoursReview, renderOfficeHoursReviewerPrompt, replaceOfficeHoursReviewBlock, verifyOfficeHoursCitations, verifyOfficeHoursVerdictReceipt } from '../lib/office-hours-review.ts'; + +const shellQuote = value => `'${value.replaceAll("'", `'\\''`)}'`; function main(args) { const command = args.shift(); - if (!['check', 'prepare', 'finalize'].includes(command)) throw new Error('usage: gstack-office-hours-review check | prepare --design --out-dir [round.json...] | finalize --design [--report ] [--unreviewed ] '); + if (!['check', 'prepare', 'seal', 'finalize'].includes(command)) throw new Error('usage: gstack-office-hours-review check --receipt | prepare --design --out-dir [round.json...] | seal --design --verdict [preceding round.json...] | finalize --design [--report ] [--unreviewed ] (each also takes --max-rounds 1|2|3, default 3)'); const options = {}; const files = []; while (args.length) { const value = args.shift(); if (value.startsWith('--')) { - const allowed = command === 'prepare' ? ['--design', '--out-dir'] : command === 'finalize' ? ['--design', '--report', '--unreviewed'] : []; + const allowed = [...{ check: ['--receipt'], prepare: ['--design', '--out-dir'], seal: ['--design', '--verdict'], finalize: ['--design', '--report', '--unreviewed'] }[command], '--max-rounds']; if (!allowed.includes(value) || options[value] !== undefined || !args.length) throw new Error(`invalid option: ${value}`); options[value] = args.shift(); } else files.push(value); } - const rounds = loadOfficeHoursReviews(files); + const maxRounds = officeHoursMaxRounds(options['--max-rounds']); + const rounds = loadOfficeHoursReviews(files, maxRounds); if (command === 'check') { - console.log(JSON.stringify(assessOfficeHoursReviews(rounds))); + // The receipt binds the reviewer's response to the last round's exact bytes. + if (!options['--receipt'] || !files.length) throw new Error('check requires --receipt with the reviewer response and every completed round'); + const last = files.at(-1); + const named = parseOfficeHoursVerdictReceipt(options['--receipt']).path; + const same = fs.existsSync(named) && fs.realpathSync(named) === fs.realpathSync(last); + verifyOfficeHoursVerdictReceipt(options['--receipt'], { round: files.length, verdictPath: same ? named : last, bytes: fs.readFileSync(last) }); + console.log(JSON.stringify(assessOfficeHoursReviews(rounds, maxRounds))); return; } if (!options['--design']) throw new Error(`${command} requires --design`); const designPath = path.resolve(options['--design']); if (rounds.some(review => path.resolve(review.document) !== designPath)) throw new Error('artifact document does not match --design'); + if (command === 'seal') { + if (!options['--verdict']) throw new Error('seal requires --verdict'); + const verdictPath = path.resolve(options['--verdict']); + const round = rounds.length + 1; + if (path.basename(verdictPath) !== `round-${round}.json`) throw new Error(`verdict must be round-${round}.json after the supplied history`); + const bytes = fs.readFileSync(verdictPath); + let value; + try { value = JSON.parse(bytes.toString('utf8')); } + catch (error) { throw new Error(`saved verdict is invalid JSON: ${error instanceof Error ? error.message : String(error)}`); } + const review = assessOfficeHoursReviews([...rounds, value], maxRounds).rounds.at(-1); + if (path.resolve(review.document) !== designPath) throw new Error('verdict document does not match --design'); + if (round > 1) verifyOfficeHoursCitations(review, officeHoursSnapshotChanges(verdictPath, round).changed); + console.log(officeHoursVerdictReceipt(round, verdictPath, bytes)); + return; + } if (command === 'prepare') { if (!options['--out-dir']) throw new Error('prepare requires --out-dir'); if (!fs.statSync(designPath).isFile()) throw new Error('design must be an existing file'); @@ -34,17 +58,30 @@ function main(args) { const round = rounds.length + 1; const promptPath = path.join(outDir, `round-${round}.prompt.md`); const verdictPath = path.join(outDir, `round-${round}.json`); - const prompt = renderOfficeHoursReviewerPrompt({ document: designPath, verdictPath, previous: rounds.at(-1) }); + // Each round records the exact design it reviewed; later rounds see the diff since then. + const snapshotPath = path.join(outDir, `round-${round}.design.md`); + const design = fs.readFileSync(designPath, 'utf8'); + let changes; + if (round > 1 && assessOfficeHoursReviews(rounds, maxRounds).stop !== 'CONTINUE') throw new Error('cannot prepare another round after a terminal review'); + if (round > 1) { + const before = path.join(outDir, `round-${round - 1}.design.md`); + if (!fs.existsSync(before)) throw new Error(`out-dir lacks round-${round - 1}.design.md; prepare every round in the same review directory`); + changes = officeHoursDesignChanges(fs.readFileSync(before, 'utf8'), design).diff; + } + const prompt = renderOfficeHoursReviewerPrompt({ document: designPath, verdictPath, previous: rounds.at(-1), changes, maxRounds }); const inputs = new Set([fs.realpathSync(designPath), ...files.map(file => fs.realpathSync(file))]); - for (const destination of [promptPath, verdictPath]) { + for (const destination of [promptPath, verdictPath, snapshotPath]) { if (fs.existsSync(destination) && inputs.has(fs.realpathSync(destination))) throw new Error('output destination overlaps input evidence'); } if (fs.lstatSync(verdictPath, { throwIfNoEntry: false })) throw new Error('next verdict already exists; check it before preparing another attempt'); + const captured = fs.lstatSync(snapshotPath, { throwIfNoEntry: false }); + if (captured && (!captured.isFile() || fs.readFileSync(snapshotPath, 'utf8') !== design)) throw new Error(`the design changed after round ${round} was prepared`); const existing = fs.lstatSync(promptPath, { throwIfNoEntry: false }); - if (existing) { - if (!existing.isFile() || fs.readFileSync(promptPath, 'utf8') !== prompt) throw new Error('existing review prompt differs from the complete saved evidence'); - } else atomicWriteSync(promptPath, prompt, { mode: 0o600 }); - const dispatch = `Read ${JSON.stringify(promptPath)} and execute the independent spec review as instructed.\nDocument: ${designPath}\nPrompt: ${promptPath}\nVerdict: ${verdictPath}`; + if (existing && (!existing.isFile() || fs.readFileSync(promptPath, 'utf8') !== prompt)) throw new Error('existing review prompt differs from the complete saved evidence'); + if (!captured) atomicWriteSync(snapshotPath, design, { mode: 0o600 }); + if (!existing) atomicWriteSync(promptPath, prompt, { mode: 0o600 }); + const seal = [shellQuote(fs.realpathSync(process.argv[1])), 'seal', ...(maxRounds === 3 ? [] : [`--max-rounds ${maxRounds}`]), '--design', shellQuote(designPath), '--verdict', shellQuote(verdictPath), ...files.map(file => shellQuote(fs.realpathSync(file)))].join(' '); + const dispatch = `Read ${JSON.stringify(promptPath)} and execute the independent spec review as instructed.\nDocument: ${designPath}\nPrompt: ${promptPath}\nVerdict: ${verdictPath}\nSeal: ${seal}`; console.log(JSON.stringify({ round, promptPath, verdictPath, dispatch })); return; } @@ -55,7 +92,7 @@ function main(args) { if (inputs.has(fs.realpathSync(destination))) throw new Error('output destination overlaps a review artifact'); } if (reportPath && fs.realpathSync(reportPath) === fs.realpathSync(designPath)) throw new Error('design and report must be different files'); - const rendered = renderOfficeHoursReview(rounds, options['--unreviewed']); + const rendered = renderOfficeHoursReview(rounds, options['--unreviewed'], maxRounds); // Validate/read/render both destinations before publishing either. Each rename // is atomic; any publication error fails the command and final verification. const design = fs.readFileSync(designPath, 'utf8'); diff --git a/bin/gstack-review-log b/bin/gstack-review-log index c1bdab264..b2b309523 100755 --- a/bin/gstack-review-log +++ b/bin/gstack-review-log @@ -40,7 +40,22 @@ WTREE="" DIRTY="" if [ -n "$COMMIT_FULL" ]; then TREE=$(git rev-parse 'HEAD^{tree}' 2>/dev/null || true) - WTREE=$("$SCRIPT_DIR/gstack-wtree" 2>/dev/null || true) + # E4: the fingerprint's new objects go to a private 0700 directory this + # operation owns (a read-only .git still fingerprints), and stay readable + # until the consumers below (bindReview / checkSharedLibsReuse read the tree + # and blobs) finish; removed on EXIT/INT/TERM. gstack-wtree's one-line + # diagnostic reaches stderr; stdout stays the hash. + if WTREE_OBJECTS=$(mktemp -d "${TMPDIR:-/tmp}/gstack-wtree-objects-XXXXXX" 2>/dev/null); then + case "$(uname -s)" in + MINGW*|MSYS*|CYGWIN*) command -v cygpath >/dev/null 2>&1 && WTREE_OBJECTS="$(cygpath -m "$WTREE_OBJECTS")" ;; + esac + trap 'rm -rf "$WTREE_OBJECTS"' EXIT + trap 'exit 130' INT TERM + WTREE=$(GSTACK_WTREE_OBJECT_DIR="$WTREE_OBJECTS" "$SCRIPT_DIR/gstack-wtree" || true) + export GIT_ALTERNATE_OBJECT_DIRECTORIES="$WTREE_OBJECTS${GIT_ALTERNATE_OBJECT_DIRECTORIES:+:$GIT_ALTERNATE_OBJECT_DIRECTORIES}" + else + echo "gstack-review-log: cannot create a private object directory under ${TMPDIR:-/tmp}; no working-tree fingerprint. Set TMPDIR to a writable directory." >&2 + fi if [ -n "$(git status --porcelain -uno 2>/dev/null | head -1)" ]; then DIRTY="true" else diff --git a/bin/gstack-session-update b/bin/gstack-session-update index 0fa33bb35..259ec2aa9 100755 --- a/bin/gstack-session-update +++ b/bin/gstack-session-update @@ -5,6 +5,11 @@ # The entire update runs in background (forked). The hook itself exits # immediately so session startup is never delayed. # +# Before the live checkout moves, the updater fetches, reads the incoming +# release's Bun floor (bin/gstack-bun-version.sh) and checks the `bun` setup +# will run; below it the update is held with reason bun-too-old and nothing +# changes (E1). Updaters older than this check cannot protect their first pull. +# # Stages (B11): pull, setup and migrations are tracked separately. The # "just upgraded" marker is written only when every stage succeeded. A failed # stage is persisted in $STATE_DIR/session-update-pending and retried even @@ -25,6 +30,9 @@ STATE_DIR="$(gstack_state_root; printf x)"; STATE_DIR="${STATE_DIR%x}" # Egress receipt helpers (_receipted_git): fail-open — an update pull must # never block a session over a receipt hiccup. . "$(cd "$(dirname "$0")" && pwd)/gstack-egress-lib.sh" +# Bun floor comparison (E1): the incoming release's floor is applied before +# the live checkout moves. Missing helper fails open, like every error here. +. "$(cd "$(dirname "$0")" && pwd)/gstack-bun-version.sh" 2>/dev/null THROTTLE_FILE="$STATE_DIR/.last-session-update" LOCK_DIR="$STATE_DIR/.setup-lock" LOG_FILE="$STATE_DIR/analytics/session-update.log" @@ -63,7 +71,12 @@ if [ -f "$PENDING_FILE" ]; then case "$_P_REASON" in bun*) _P_FIX="install Bun (https://bun.sh), then $_P_FIX" ;; esac echo "gstack auto-update: ${_P_STAGES} did not finish (${_P_REASON:-unknown reason}); installed skills may be out of date. gstack retries automatically. Fix now: $_P_FIX" elif [ -n "$_P_PULL" ]; then - echo "gstack auto-update: git pull did not finish ($_P_PULL); gstack is not updating. Fix now: cd $GSTACK_DIR && git pull --ff-only && ./setup" + case "$_P_PULL" in + bun-too-old*) + echo "gstack auto-update: update held ($_P_PULL); nothing was installed or changed. Fix now: bun upgrade && cd $GSTACK_DIR && git pull --ff-only && ./setup" ;; + *) + echo "gstack auto-update: git pull did not finish ($_P_PULL); gstack is not updating. Fix now: cd $GSTACK_DIR && git pull --ff-only && ./setup" ;; + esac fi fi @@ -158,7 +171,7 @@ fi # heartbeat. trap 'kill "$HB_PID" 2>/dev/null; [ "$(cat "$LOCK_DIR/pid" 2>/dev/null)" = "$MYPID" ] && rm -rf "$LOCK_DIR" 2>/dev/null' EXIT - # ── Pull latest ── + # ── Fetch, check the incoming release's Bun floor, then fast-forward (E1) ── OLD_HEAD=$(git -C "$GSTACK_DIR" rev-parse HEAD 2>/dev/null) UPDATE_URL=$(git -C "$GSTACK_DIR" remote get-url origin 2>/dev/null || echo "") UPDATE_HOST="${UPDATE_URL#*://}"; UPDATE_HOST="${UPDATE_HOST#*@}"; UPDATE_HOST="${UPDATE_HOST%%[/:]*}" @@ -170,8 +183,22 @@ fi # Capture stderr: the log must carry WHY a pull failed, never just the code. PULL_ERR_FILE=$(mktemp "${TMPDIR:-/tmp}/gstack-session-pull-XXXXXX" 2>/dev/null || echo "") GSTACK_HOME="$STATE_DIR" _receipted_git open session-update "${UPDATE_HOST:-unknown}" gstack-self-update-pull "auto_upgrade=true" \ - bash -c 'git -C "$1" pull --ff-only --autostash -q 2>"${2:-/dev/null}"' _ "$GSTACK_DIR" "$PULL_ERR_FILE" + bash -c 'git -C "$1" fetch -q 2>"${2:-/dev/null}"' _ "$GSTACK_DIR" "$PULL_ERR_FILE" PULL_EXIT=$? + # The fast-forward targets the exact revision whose requirement was read. + # Setup runs `bun` from this PATH, so that is the binary checked; a + # malformed version passes here because setup only warns on it. + INCOMING="" BUN_HELD="" + if [ "$PULL_EXIT" -eq 0 ]; then + INCOMING=$(git -C "$GSTACK_DIR" rev-parse --verify '@{upstream}^{commit}' 2>"${PULL_ERR_FILE:-/dev/null}") || PULL_EXIT=1 + fi + if [ "$PULL_EXIT" -eq 0 ] && [ "$INCOMING" != "$OLD_HEAD" ]; then + BUN_HELD=$(gstack_bun_incoming_hold "$GSTACK_DIR" "$INCOMING" 2>/dev/null) || BUN_HELD="" + fi + if [ "$PULL_EXIT" -eq 0 ] && [ -z "$BUN_HELD" ]; then + git -C "$GSTACK_DIR" merge --ff-only --autostash -q "$INCOMING" >/dev/null 2>"${PULL_ERR_FILE:-/dev/null}" + PULL_EXIT=$? + fi NEW_HEAD=$(git -C "$GSTACK_DIR" rev-parse HEAD 2>/dev/null) # Heartbeat: pull done — keep the TTL clock fresh for the setup step. @@ -185,6 +212,13 @@ fi case "$P_FAILURES" in ''|*[!0-9]*) P_FAILURES=0 ;; esac case "$P_NEXT" in ''|*[!0-9]*) P_NEXT=0 ;; esac + if [ -n "$BUN_HELD" ]; then + log_entry "HELD $BUN_HELD incoming=$INCOMING" + P_PULL="$BUN_HELD" + pending_save + rm -f "$PULL_ERR_FILE" 2>/dev/null + exit 0 + fi if [ "$PULL_EXIT" -ne 0 ]; then PULL_REASON=$(head -c 300 "$PULL_ERR_FILE" 2>/dev/null | tr '\n' ' ' | tr -s ' ') log_entry "PULL_FAILED exit=$PULL_EXIT reason=${PULL_REASON:-unknown}" diff --git a/bin/gstack-wtree b/bin/gstack-wtree index 94265fa7d..b2c1c258c 100755 --- a/bin/gstack-wtree +++ b/bin/gstack-wtree @@ -21,11 +21,17 @@ # produce the identical write-tree hash. Fallback: `read-tree HEAD` when the # index copy is unavailable (fresh repo, exotic index). # -# The real repo index is never touched. Staged blobs land in the object store -# as unreachable objects and get gc'd like stash churn (note: this means the +# The real repo index is never touched. Run standalone, staged blobs land in +# the object store as unreachable objects and get gc'd like stash churn (the # CONTENT of untracked, non-ignored files enters .git/objects until gc — the -# same property `git stash -u` has). Exit 1 outside a git repo or in a repo -# with no commits — callers treat that as "no fingerprint". +# same property `git stash -u` has), so `git diff ` between saved +# snapshots works. With GSTACK_WTREE_OBJECT_DIR set (gstack-review-log, E4), +# new objects go to that private directory instead, with the repository's +# object store as an alternate: the fingerprint works under a read-only .git +# (Codex's sandbox), and the caller owns the directory's lifetime because its +# consumers read the tree afterwards. Exit 1 outside a git repo, in a repo with +# no commits, or when a write fails (one stderr line names the cause) — callers +# treat that as "no fingerprint"; a partial hash is never printed. set -euo pipefail TOP=$(git rev-parse --show-toplevel 2>/dev/null) || exit 1 @@ -33,14 +39,34 @@ TOP=$(git rev-parse --show-toplevel 2>/dev/null) || exit 1 # var set, `git rev-parse --git-path index` returns the temp index itself and # the stat-cache seed silently self-copies into a dead fast path. REAL_INDEX=$(git -C "$TOP" rev-parse --git-path index 2>/dev/null || true) -TMPIDX=$(mktemp "${TMPDIR:-/tmp}/gstack-wtree-XXXXXX") +# Same for the object store: the common dir's objects (linked worktrees share +# it), resolved to an absolute path before GIT_OBJECT_DIRECTORY is exported. +REAL_OBJECTS=$(git -C "$TOP" rev-parse --git-path objects 2>/dev/null || true) +OBJ_DIR="${GSTACK_WTREE_OBJECT_DIR:-}" +if [ -n "$OBJ_DIR" ] && [ ! -d "$OBJ_DIR" ]; then + echo "gstack-wtree: private object directory $OBJ_DIR does not exist" >&2 + exit 1 +fi +TMPIDX=$(mktemp "${TMPDIR:-/tmp}/gstack-wtree-XXXXXX" 2>/dev/null) || { + echo "gstack-wtree: cannot create a temp index under ${TMPDIR:-/tmp}; set TMPDIR to a writable directory" >&2 + exit 1 +} trap 'rm -f "$TMPIDX"' EXIT +trap 'exit 130' INT TERM export GIT_INDEX_FILE="$TMPIDX" # Resolve relative --git-path output against the repo root. case "$REAL_INDEX" in ""|/*) ;; *) REAL_INDEX="$TOP/$REAL_INDEX" ;; esac +case "$REAL_OBJECTS" in + ""|/*) ;; + *) REAL_OBJECTS="$TOP/$REAL_OBJECTS" ;; +esac +if [ -n "$OBJ_DIR" ]; then + export GIT_OBJECT_DIRECTORY="$OBJ_DIR" + export GIT_ALTERNATE_OBJECT_DIRECTORIES="$REAL_OBJECTS${GIT_ALTERNATE_OBJECT_DIRECTORIES:+:$GIT_ALTERNATE_OBJECT_DIRECTORIES}" +fi if [ -n "$REAL_INDEX" ] && [ -f "$REAL_INDEX" ] && cp "$REAL_INDEX" "$TMPIDX" 2>/dev/null; then # Carry the real index's mtime onto the copy. Git's racy-git protection # re-hashes any entry whose cached mtime is not older than the index file @@ -58,5 +84,12 @@ if [ -n "$REAL_INDEX" ] && [ -f "$REAL_INDEX" ] && cp "$REAL_INDEX" "$TMPIDX" 2> else git -C "$TOP" read-tree HEAD 2>/dev/null || exit 1 fi -git -C "$TOP" add -A 2>/dev/null || exit 1 -git -C "$TOP" write-tree 2>/dev/null +if ! ERR=$(git -C "$TOP" add -A 2>&1 >/dev/null); then + echo "gstack-wtree: cannot stage the working tree: $(printf '%s' "$ERR" | head -1)" >&2 + exit 1 +fi +if ! TREE=$(git -C "$TOP" write-tree 2>&1) || ! printf '%s' "$TREE" | grep -Eqx '[0-9a-f]{40}|[0-9a-f]{64}'; then + echo "gstack-wtree: cannot write the tree: $(printf '%s' "$TREE" | head -1)" >&2 + exit 1 +fi +printf '%s\n' "$TREE" diff --git a/browse/src/browser-manager.ts b/browse/src/browser-manager.ts index 2ae2221d8..c976511ed 100644 --- a/browse/src/browser-manager.ts +++ b/browse/src/browser-manager.ts @@ -182,6 +182,41 @@ function launchedProcess(browser: Browser | null | undefined): ChildProcess | nu return typeof withProcess?.process === 'function' ? withProcess.process() : null; } +/** + * #2709: PID of the Chromium browser process gstack launched, read over CDP + * (`SystemInfo.getProcessInfo`, type "browser") because Playwright 1.62's + * Browser has no `process()`. Call it only for a launch gstack owns; a + * connected browser belongs to someone else and must never be reaped. Bounded: + * a hung or failed CDP call logs once and returns null, which means no reap. + */ +export async function launchedChromiumPid(browser: Browser, timeoutMs = 2_000): Promise { + let timer: ReturnType | undefined; + const query = async (): Promise => { + const session = await browser.newBrowserCDPSession(); + try { + const { processInfo } = await session.send('SystemInfo.getProcessInfo'); + const pid = processInfo.find((p) => p.type === 'browser')?.id; + return Number.isInteger(pid) && pid! > 0 ? pid! : null; + } finally { + await session.detach().catch(() => {}); + } + }; + try { + return await Promise.race([ + query(), + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`timed out after ${timeoutMs}ms`)), timeoutMs); + }), + ]); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + console.warn(`[browse] Could not record the Chromium PID (CDP SystemInfo.getProcessInfo: ${msg}); browse stop cannot reap a surviving Chromium.`); + return null; + } finally { + clearTimeout(timer); + } +} + /** * Resolve why the underlying Chromium ChildProcess is going away. * @@ -669,12 +704,10 @@ export class BrowserManager { }); // #2709: record the child's identity so the CLI can reap a survivor after - // daemon shutdown. `.process()` exists here — we launched this browser. + // daemon shutdown. We launched this browser, so its PID is ours to record. { - const proc = launchedProcess(this.browser); - this.chromiumProcInfo = proc?.pid - ? { pid: proc.pid, startTime: readPidStartTime(proc.pid) } - : null; + const pid = await launchedChromiumPid(this.browser); + this.chromiumProcInfo = pid ? { pid, startTime: readPidStartTime(pid) } : null; } this.context = await this.browser.newContext(contextOptions); diff --git a/browse/src/cli.ts b/browse/src/cli.ts index 0bc17876e..e7b5b1f91 100644 --- a/browse/src/cli.ts +++ b/browse/src/cli.ts @@ -2080,7 +2080,12 @@ Refs: After 'snapshot', use @e1, @e2... as selectors: // Playwright — but on macOS 26 the GPU process can survive that close and // spin at ~800% CPU forever. The state snapshot read above still carries // the launched child's identity; reap a verified survivor. + // Reap only after the daemon has finished its own shutdown: killing Chromium + // while the daemon is still closing reads as a crash, and the daemon exits(1) + // without removing its state file. if (command === 'stop') { + const deadline = Date.now() + 5_000; + while (Date.now() < deadline && isProcessAlive(state.pid)) await Bun.sleep(100); await reapRecordedChromium(state); } diff --git a/browse/test/cookie-import-native-job.test.ts b/browse/test/cookie-import-native-job.test.ts index de5c726ca..6b0431008 100644 --- a/browse/test/cookie-import-native-job.test.ts +++ b/browse/test/cookie-import-native-job.test.ts @@ -343,7 +343,7 @@ const request: NativeCookieRequest = { profile: 'Default', domains: ['example.test'], deadline: 25_000, - qualifiedBunVersions: ['1.4.0'], + qualifiedBunVersions: ['1.4.2'], }; function kernelContract(mode: string) { diff --git a/browse/test/cookie-import-native-qualification.ts b/browse/test/cookie-import-native-qualification.ts index 950fcc39b..8c703c9fe 100644 --- a/browse/test/cookie-import-native-qualification.ts +++ b/browse/test/cookie-import-native-qualification.ts @@ -24,7 +24,7 @@ function incomplete(reason: string, diagnostic?: NativeCookieDiagnostic): never console.log(JSON.stringify({ ...receipt, artifactDirectory: output })); process.exit(2); } -if (Bun.version !== '1.4.0') incomplete('bun_1_4_0_required'); +if (Bun.version !== '1.4.2') incomplete('bun_1_4_2_required'); let emptyJob: NativeCookieJob | undefined; try { emptyJob = await createNativeCookieJob(); diff --git a/browse/test/dia-launch-comparison.test.ts b/browse/test/dia-launch-comparison.test.ts index 871473c2a..cab823ae6 100644 --- a/browse/test/dia-launch-comparison.test.ts +++ b/browse/test/dia-launch-comparison.test.ts @@ -98,7 +98,7 @@ describe('diagnostic-only Dia runtime comparison', () => { const arm = (runtime: 'bun' | 'node', ready: boolean) => { const sha = 'a'.repeat(64); const config = { mode: 'launch-only', runtime, qualificationCredit: false, executableSha256: runtime === 'bun' ? sha : 'b'.repeat(64), driverSha256: sha, helpersSha256: sha }; - const driver = { runtime, version: runtime === 'bun' ? '1.4.0' : '24.18.0', architecture: 'arm64', os: 'darwin', release: '24.6.0', playwright: '1.62.1', + const driver = { runtime, version: runtime === 'bun' ? '1.4.2' : '24.18.0', architecture: 'arm64', os: 'darwin', release: '24.6.0', playwright: '1.62.1', executableSha256: config.executableSha256, driverSha256: sha, helpersSha256: sha }; const result = (purpose: string, success: boolean) => ({ protocol: 1, purpose, ready: success, launchReturned: success, protocolResponded: success, samplingEnabled: false, rootCount: 1, supervisor: { closed: true, exitCode: 0 }, cleanup: { confirmed: true, childClosed: true, groupAbsent: true, launchSettled: true, @@ -112,7 +112,7 @@ describe('diagnostic-only Dia runtime comparison', () => { qualification: { launchComparison: { qualificationCredit: false, source: result('source', ready) }, counts: { pass: 0, fail: 0, skip: 0 }, keychainStage: 'completed', isolation: { registeredIdentity: true, sharedRegisteredHome: true }, cleanup: { ownedBrowsersStopped: true, sourceProfileRemoved: true, keychainRestored: true, mountDetached: true, fixtureRemoved: true }, - platform: { os: 'darwin', architecture: 'arm64', bun: '1.4.0', playwright: '1.62.1', release: '24.6.0' }, + platform: { os: 'darwin', architecture: 'arm64', bun: '1.4.2', playwright: '1.62.1', release: '24.6.0' }, artifact: { signatureVerified: true, gatekeeperNotarized: true, macosCompatibility: { compatible: true, hostVersion: '15.7.9' }, architectures: ['arm64'], sha256: sha, executableSha256: sha, version: '1.49.1', bundleId: 'company.thebrowser.dia', team: 'S6N382Y83G' } } }; }; diff --git a/browse/test/headless-gpu-and-reap.test.ts b/browse/test/headless-gpu-and-reap.test.ts index cef5776e1..af12ada21 100644 --- a/browse/test/headless-gpu-and-reap.test.ts +++ b/browse/test/headless-gpu-and-reap.test.ts @@ -14,10 +14,11 @@ */ import { describe, expect, test } from 'bun:test'; import { spawn } from 'node:child_process'; -import { headlessGpuArgs } from '../src/browser-manager'; +import { BrowserManager, headlessGpuArgs, launchedChromiumPid } from '../src/browser-manager'; import { reapRecordedChromium } from '../src/cli'; import { readPidStartTime } from '../src/xvfb'; import { isProcessAlive } from '../src/error-handling'; +import { chromium, type Browser } from 'playwright'; describe('headlessGpuArgs (#2709)', () => { test('darwin gets the validated four-flag set', () => { @@ -105,6 +106,83 @@ describe.skipIf(process.platform !== 'linux')('reapRecordedChromium identity gat }); }); +// E3: Playwright 1.62's Browser has no process(), so the recorded identity was +// always null and the reap above never ran. The PID now comes from CDP +// SystemInfo.getProcessInfo, only for a launch gstack owns. +describe.skipIf(process.platform !== 'linux')('Chromium PID via CDP on the pinned Playwright (#2709, E3)', () => { + const closeQuietly = (b: { close(): Promise } | null | undefined) => + Promise.race([b?.close().catch(() => {}), new Promise((r) => setTimeout(r, 3_000))]); + + test('an owned headless launch records the browser PID and start time, and the reap kills it', async () => { + const bm = new BrowserManager(); + await bm.launch(); + try { + const info = bm.getChromiumProcInfo(); + expect(info, 'headless launch recorded no Chromium identity').not.toBeNull(); + expect(isProcessAlive(info!.pid)).toBe(true); + expect(info!.startTime).toBe(readPidStartTime(info!.pid)); + expect(info!.startTime).not.toBe(''); + await reapRecordedChromium({ chromiumPid: info!.pid, chromiumStartTime: info!.startTime }); + expect(isProcessAlive(info!.pid)).toBe(false); + } finally { + await closeQuietly(bm); + } + }, 60_000); + + test('an externally owned Chromium is never reaped: its PID is not recorded and a stale identity never matches', async () => { + const external = await chromium.launch({ headless: true, chromiumSandbox: false }); + const bm = new BrowserManager(); + await bm.launch(); + try { + const externalPid = await launchedChromiumPid(external); + expect(externalPid).not.toBeNull(); + const ours = bm.getChromiumProcInfo(); + expect(ours?.pid).not.toBe(externalPid); + await reapRecordedChromium({ chromiumPid: ours!.pid, chromiumStartTime: ours!.startTime }); + expect(isProcessAlive(externalPid!)).toBe(true); + // A recycled PID: a live Chromium at the recorded PID with another start time. + await reapRecordedChromium({ chromiumPid: externalPid!, chromiumStartTime: 'Mon Jan 1 00:00:00 1990' }); + expect(isProcessAlive(externalPid!)).toBe(true); + } finally { + await closeQuietly(bm); + await closeQuietly(external); + } + }, 60_000); +}); + +describe('launchedChromiumPid bounded CDP failure (E3)', () => { + const stub = (newBrowserCDPSession: () => Promise) => ({ newBrowserCDPSession }) as unknown as Browser; + + test('a hung CDP session returns null within the bound and logs once', async () => { + const warn = console.warn; + const lines: string[] = []; + console.warn = (...args: unknown[]) => { lines.push(args.join(' ')); }; + try { + const t0 = Date.now(); + expect(await launchedChromiumPid(stub(() => new Promise(() => {})), 100)).toBeNull(); + expect(Date.now() - t0).toBeLessThan(2_000); + } finally { + console.warn = warn; + } + expect(lines).toEqual([ + '[browse] Could not record the Chromium PID (CDP SystemInfo.getProcessInfo: timed out after 100ms); browse stop cannot reap a surviving Chromium.', + ]); + }); + + test('a failing or browser-less answer returns null', async () => { + const warn = console.warn; + console.warn = () => {}; + try { + expect(await launchedChromiumPid(stub(async () => { throw new Error('Target closed'); }))).toBeNull(); + const session = (processInfo: unknown[]) => async () => ({ send: async () => ({ processInfo }), detach: async () => {} }); + expect(await launchedChromiumPid(stub(session([{ type: 'GPU', id: 4242 }])))).toBeNull(); + expect(await launchedChromiumPid(stub(session([{ type: 'browser', id: 4242 }])))).toBe(4242); + } finally { + console.warn = warn; + } + }); +}); + // ─── Source pins: the #2709 pieces stay WIRED ──────────────────────────── // The unit tests above prove headlessGpuArgs and reapRecordedChromium behave; // these pins prove the daemon persists the child's identity, the CLI reaps on @@ -176,10 +254,12 @@ describe('stop-reap wiring pins (#2709)', () => { ); // 5. Post-graceful-stop: the daemon closed Chromium via Playwright, but a - // surviving GPU process must still be reaped after sendCommand('stop'). + // surviving GPU process must still be reaped after sendCommand('stop') — + // only once the daemon has exited (an earlier kill reads as a crash and + // the daemon exits without removing its state file). const postStop = between('await sendCommand(state, command, commandArgs);', "if (command === 'focus')"); expect(postStop).toMatch( - /if \(command === 'stop'\) \{\s*\n\s*await reapRecordedChromium\(state\);/, + /if \(command === 'stop'\) \{[\s\S]*while \(.*isProcessAlive\(state\.pid\)\)[^\n]*\n\s*await reapRecordedChromium\(state\);/, ); }); @@ -195,4 +275,12 @@ describe('stop-reap wiring pins (#2709)', () => { const headless = src.slice(src.indexOf(' async launch() {'), src.indexOf(' async launchHeaded(')); expect(headless).toContain('launchArgs.push(...headlessGpuArgs(process.platform, process.env));'); }); + + test('launchedChromiumPid is called only by the owned headless launch()', () => { + const src = read('browser-manager.ts'); + const calls = src.match(/await launchedChromiumPid\(/g) ?? []; + expect(calls.length).toBe(1); + const headless = src.slice(src.indexOf(' async launch() {'), src.indexOf(' async launchHeaded(')); + expect(headless).toContain('const pid = await launchedChromiumPid(this.browser);'); + }); }); diff --git a/browse/test/navigation-guard.test.ts b/browse/test/navigation-guard.test.ts index 45da751ca..4a28cb918 100644 --- a/browse/test/navigation-guard.test.ts +++ b/browse/test/navigation-guard.test.ts @@ -4,6 +4,7 @@ * server.ts runs write commands: through BrowserManager.failIfNavigationBlocked. */ import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { readFileSync } from 'node:fs'; import { BrowserManager } from '../src/browser-manager'; import { handleWriteCommand } from '../src/write-commands'; @@ -48,19 +49,31 @@ function run(command: string, args: string[]): Promise { return bm.failIfNavigationBlocked(session.getPage(), handleWriteCommand(command, args, session, bm)); } +// While Bun's promise matchers (rejects/resolves) wait, each Playwright round-trip takes +// about a second, which pushed the 5 s click timeout past the guard's block under load. +// Settle the command first, then assert on the refusal. +async function refusal(command: string, args: string[]): Promise { + try { + await run(command, args); + } catch (err) { + return err instanceof Error ? err.message : String(err); + } + throw new Error(`${command} ${args.join(' ')} was not refused`); +} + describe('D2: navigation guard', () => { test('a redirect to 169.254.0.0/16 fails the goto and leaves the tab blank', async () => { - await expect(run('goto', [`${base}/to-link-local`])).rejects.toThrow(/169\.254\.170\.2 is a cloud metadata or link-local address/); + expect(await refusal('goto', [`${base}/to-link-local`])).toMatch(/169\.254\.170\.2 is a cloud metadata or link-local address/); expect(bm.getActiveSession().getPage().url()).toBe('about:blank'); }, 30_000); test('a redirect to an IPv4-mapped IPv6 link-local address is refused too', async () => { - await expect(run('goto', [`${base}/to-mapped`])).rejects.toThrow(/cloud metadata or link-local/); + expect(await refusal('goto', [`${base}/to-mapped`])).toMatch(/cloud metadata or link-local/); expect(bm.getActiveSession().getPage().url()).toBe('about:blank'); }, 30_000); test('a blocked target that fails at once still leaves the tab blank for the next command', async () => { - await expect(run('goto', [`${base}/to-fast-fail`])).rejects.toThrow(/cloud metadata or link-local/); + expect(await refusal('goto', [`${base}/to-fast-fail`])).toMatch(/cloud metadata or link-local/); expect(bm.getActiveSession().getPage().url()).toBe('about:blank'); const result = await run('goto', [`${base}/safe`]); expect(result).toContain('Navigated to'); @@ -69,13 +82,13 @@ describe('D2: navigation guard', () => { test('clicking a link to a link-local address is refused', async () => { await run('goto', [`${base}/link`]); - await expect(run('click', ['#go'])).rejects.toThrow(/cloud metadata or link-local/); + expect(await refusal('click', ['#go'])).toMatch(/cloud metadata or link-local/); expect(bm.getActiveSession().getPage().url()).toBe('about:blank'); }, 30_000); test('a script-driven navigation to a link-local address is refused', async () => { await run('goto', [`${base}/script`]); - await expect(run('click', ['#go'])).rejects.toThrow(/cloud metadata or link-local/); + expect(await refusal('click', ['#go'])).toMatch(/cloud metadata or link-local/); expect(bm.getActiveSession().getPage().url()).toBe('about:blank'); }, 30_000); @@ -85,3 +98,9 @@ describe('D2: navigation guard', () => { expect(bm.getActiveSession().getPage().url()).toBe(`${base}/safe`); }, 30_000); }); + +test('live browser commands settle before the refusal is asserted', () => { + const source = readFileSync(import.meta.path, 'utf8'); + expect(source).not.toMatch(new RegExp(['expect\\(', 'run\\('].join(''))); + expect(source).not.toMatch(new RegExp(['\\.', 'rejects'].join(''))); +}); diff --git a/bun.lock b/bun.lock index 42dd194ba..8cf207df2 100644 --- a/bun.lock +++ b/bun.lock @@ -17,7 +17,7 @@ "devDependencies": { "@anthropic-ai/claude-agent-sdk": "0.2.117", "@anthropic-ai/sdk": "^0.78.0", - "@types/bun": "1.4.0", + "@types/bun": "1.4.2", "prettier": "3.9.9", "typescript": "7.0.2", "xterm": "^5.3.0", @@ -176,7 +176,7 @@ "@protobufjs/utf8": ["@protobufjs/utf8@1.1.2", "", {}, "sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug=="], - "@types/bun": ["@types/bun@1.4.0", "", { "dependencies": { "bun-types": "1.4.0" } }, "sha512-K+lZULY23vRgK/CfTjFIV+tyifaNdSMlPh9j+6mQ/cLfpOznLyAuzgV/JQysyECpkBQLVMSyvjlr2fBUSA9wFQ=="], + "@types/bun": ["@types/bun@1.4.2", "", { "dependencies": { "bun-types": "1.4.2" } }, "sha512-GimotNn7+ZV0uVArItBbriZsR1oNf0+WTzPkdcFrzShI7k2norL0uzEaJT8T33dWr7O/c9ZDuAFQrctKCi72oQ=="], "@types/node": ["@types/node@26.4.0", "", { "dependencies": { "undici-types": "~8.3.0" } }, "sha512-faiGnoIrLH/V8cibOMEAZ8pMw6oXqSukl29ra4mN8GdaB2ZewzeaLj+INpV5N+Z1eKWzY+IzaIZH2EIR6YZRNQ=="], @@ -234,7 +234,7 @@ "browser-split": ["browser-split@0.0.1", "", {}, "sha512-JhvgRb2ihQhsljNda3BI8/UcRHVzrVwo3Q+P8vDtSiyobXuFpuZ9mq+MbRGMnC22CjW3RrfXdg6j6ITX8M+7Ow=="], - "bun-types": ["bun-types@1.4.0", "", { "dependencies": { "@types/node": "*" } }, "sha512-iIKw23BspnQQYd3prITOBxeUsxBHnwzX6YJfGMuNOZzeNcMmVqzIIVGRm1l69ogaPQmb4wB6BN8mA5bE9YuC5Q=="], + "bun-types": ["bun-types@1.4.2", "", { "dependencies": { "@types/node": "*" } }, "sha512-bxV1FgK7yBIzjRe5zBozIM4Bem11ZJcCXSrjWRG3YWLt8yFDePu4cLjpebO8OvPeIE9trbyPF4fuj3Cia4Fj3w=="], "bytes": ["bytes@3.1.2", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="], diff --git a/design/scripts/live-model-check.ts b/design/scripts/live-model-check.ts index e887afc06..111a18587 100644 --- a/design/scripts/live-model-check.ts +++ b/design/scripts/live-model-check.ts @@ -1,31 +1,39 @@ /** - * Opt-in live check of the design binary's OpenAI defaults (#2807). Not part - * of any test lane: it spends real tokens (one low-quality 1024x1024 image, - * one short vision answer). + * Live check of the design binary's OpenAI defaults (#2807, E5). It spends + * real tokens (one low-quality 1024x1024 image, one short vision answer). * * OPENAI_API_KEY=... bun run design/scripts/live-model-check.ts * * Sends exactly the request bodies the design commands send (built by - * design/src/models.ts), honoring GSTACK_DESIGN_MODEL when set. Prints the - * models requested and returned, HTTP status, token usage and PASS/FAIL per - * call — never the key, and any key-shaped text in an error body is redacted. - * Exit 0 both pass, 1 any fail, 2 OPENAI_API_KEY unset. + * design/src/models.ts) with the DEFAULT models: GSTACK_DESIGN_MODEL and + * GSTACK_DESIGN_IMAGE_MODEL are ignored (and named when set), so an override + * cannot mask a retirement of a default. Prints the models requested and + * returned, HTTP status, token usage and PASS/FAIL per call — never the key, + * and any key-shaped text in an error body is redacted. + * Exit 0 both pass, 1 any fail, 2 OPENAI_API_KEY unset: a skip that checked + * nothing, which a caller must record as skipped with no coverage credit. */ import { + DESIGN_IMAGE_MODEL_ENV, DESIGN_MODEL_ENV, - IMAGE_TOOL_MODEL, designModel, imageRequestBody, + imageToolModel, visionRequestBody, } from "../src/models"; import { receiptedFetch } from "../src/receipted-fetch"; const TINY_PNG = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkAAIAAAoAAv/lxKUAAAAASUVORK5CYII="; +const ignored = [DESIGN_MODEL_ENV, DESIGN_IMAGE_MODEL_ENV].filter((name) => process.env[name]?.trim()); +for (const name of ignored) delete process.env[name]; +console.log(`testing defaults: image=${designModel("image")} tool=${imageToolModel()} vision=${designModel("vision")}` + + (ignored.length ? ` (ignored: ${ignored.join(", ")})` : "")); + const key = process.env.OPENAI_API_KEY?.trim(); if (!key) { - console.log("SKIP: OPENAI_API_KEY is not set"); + console.log("SKIP: OPENAI_API_KEY is not set; the design model defaults were not checked (no coverage)"); process.exit(2); } @@ -56,7 +64,6 @@ async function post(payloadClass: string, url: string, body: string, timeoutMs: const report = (fields: Record) => console.log(Object.entries(fields).map(([k, v]) => `${k}=${typeof v === "string" ? v : JSON.stringify(v)}`).join(" ")); -console.log(`${DESIGN_MODEL_ENV}=${process.env[DESIGN_MODEL_ENV]?.trim() || "(unset, defaults)"}`); let failed = false; let image = TINY_PNG; @@ -78,7 +85,7 @@ let image = TINY_PNG; report({ call: "image", result: pass ? "PASS" : "FAIL", status: r.status, ms: r.ms, requested_model: designModel("image"), returned_model: r.json?.model ?? null, - tool_model: IMAGE_TOOL_MODEL, image_bytes: pass ? Buffer.from(item.result, "base64").length : 0, + tool_model: imageToolModel(), image_bytes: pass ? Buffer.from(item.result, "base64").length : 0, ...(pass ? {} : { error: redact(r.json?.error?.message ?? r.text) }), }); failed ||= !pass; diff --git a/design/src/models.ts b/design/src/models.ts index 081709eb8..5fed94daf 100644 --- a/design/src/models.ts +++ b/design/src/models.ts @@ -8,6 +8,9 @@ * analysis — check, diff, memory, design-to-code, evolve's analysis step. * * `GSTACK_DESIGN_MODEL` (named like `GSTACK_CODEX_MODEL`) replaces both. + * `GSTACK_DESIGN_IMAGE_MODEL` replaces the image_generation tool's model; it + * must look like a gpt-image model name (IMAGE_TOOL_MODEL_PATTERN) and a bad + * value fails before any request, naming the variable. * * Defaults (checked against OpenAI's model and deprecation pages, 2026-10): * gpt-5.5 lists Chat Completions, image input and the Responses @@ -24,7 +27,8 @@ * The limit is a cap, not a charge — only generated tokens are billed. * * `bun run design/scripts/live-model-check.ts` probes these defaults with a - * real key (opt-in, not in any test lane). + * real key. It always tests the defaults (overrides are ignored), so an + * override cannot mask a retirement. */ export const DESIGN_MODEL_ENV = "GSTACK_DESIGN_MODEL"; @@ -36,9 +40,29 @@ export const DEFAULT_DESIGN_MODELS: Readonly> = { vision: "gpt-5.5", }; -/** The image model the hosted image_generation tool runs; never omitted (the API default is gpt-image-1). */ +/** The image model the hosted image_generation tool runs by default; never omitted (the API default is gpt-image-1). */ export const IMAGE_TOOL_MODEL = "gpt-image-2"; +export const DESIGN_IMAGE_MODEL_ENV = "GSTACK_DESIGN_IMAGE_MODEL"; + +/** Conservative shape for an image tool model override: gpt-image-[-variant]. */ +export const IMAGE_TOOL_MODEL_PATTERN = /^gpt-image-[0-9][a-z0-9.-]{0,31}$/; + +/** + * The image_generation tool's model: `GSTACK_DESIGN_IMAGE_MODEL` when set, + * else IMAGE_TOOL_MODEL. Throws, naming the variable, when the override does + * not match IMAGE_TOOL_MODEL_PATTERN. + */ +export function imageToolModel(env: Record = process.env): string { + const override = env[DESIGN_IMAGE_MODEL_ENV]?.trim(); + if (!override) return IMAGE_TOOL_MODEL; + if (!IMAGE_TOOL_MODEL_PATTERN.test(override)) { + throw new Error(`${DESIGN_IMAGE_MODEL_ENV}=${JSON.stringify(override.slice(0, 80))} is not a gpt-image model name ` + + `(expected something like ${IMAGE_TOOL_MODEL}); fix it or unset it to use ${IMAGE_TOOL_MODEL}`); + } + return override; +} + /** Reasoning allowance added to every vision call's visible-answer budget. */ export const REASONING_HEADROOM_TOKENS = 8192; @@ -47,10 +71,10 @@ export function designModel(call: DesignCall, env: Record = {}, ): string { const model = designModel("image"); - const problem = imagePairingProblem(model); + const toolModel = imageToolModel(); + const problem = imagePairingProblem(model, toolModel); if (problem) throw new Error(problem); return JSON.stringify({ model, input, ...extra, - tools: [{ type: "image_generation", model: IMAGE_TOOL_MODEL, size: tool.size, quality: tool.quality }], + tools: [{ type: "image_generation", model: toolModel, size: tool.size, quality: tool.quality }], }); } @@ -100,8 +125,11 @@ export function modelRejectionHint(status: number, body: string, call: DesignCal if (!/\bmodel\b|unsupported (?:parameter|value)|not supported/i.test(body)) return ""; const model = designModel(call); const source = process.env[DESIGN_MODEL_ENV]?.trim() ? `${DESIGN_MODEL_ENV}=${model}` : `the default model ${model}`; - const what = call === "image" - ? `image-generation (Responses API, tool model ${IMAGE_TOOL_MODEL})` - : "vision (Chat Completions)"; - return ` — OpenAI rejected ${source} for this ${what} call; set ${DESIGN_MODEL_ENV} to a model your account can use for it`; + if (call !== "image") { + return ` — OpenAI rejected ${source} for this vision (Chat Completions) call; set ${DESIGN_MODEL_ENV} to a model your account can use for it`; + } + const toolOverride = process.env[DESIGN_IMAGE_MODEL_ENV]?.trim(); + const tool = toolOverride ? `${DESIGN_IMAGE_MODEL_ENV}=${toolOverride}` : `the default tool model ${IMAGE_TOOL_MODEL}`; + return ` — OpenAI rejected ${source} or ${tool} for this image-generation (Responses API) call; ` + + `set ${DESIGN_MODEL_ENV} or ${DESIGN_IMAGE_MODEL_ENV} to models your account can use for it`; } diff --git a/design/test/image-gen-pairing.test.ts b/design/test/image-gen-pairing.test.ts index 8e72ded15..dba74642a 100644 --- a/design/test/image-gen-pairing.test.ts +++ b/design/test/image-gen-pairing.test.ts @@ -1,13 +1,17 @@ import { afterEach, describe, expect, test } from "bun:test"; +import { spawnSync } from "child_process"; import fs from "fs"; +import os from "os"; import path from "path"; import { DEFAULT_DESIGN_MODELS, + DESIGN_IMAGE_MODEL_ENV, DESIGN_MODEL_ENV, IMAGE_TOOL_MODEL, imagePairingProblem, imageRequestBody, + imageToolModel, } from "../src/models"; // Tripwire for the image_generation orchestrator/tool pairing. @@ -33,9 +37,12 @@ const sources = fs .map((f) => ({ rel: `src/${f}`, body: fs.readFileSync(path.join(DESIGN_SRC, f), "utf-8") })); const savedOverride = process.env[DESIGN_MODEL_ENV]; +const savedImageOverride = process.env[DESIGN_IMAGE_MODEL_ENV]; afterEach(() => { if (savedOverride === undefined) delete process.env[DESIGN_MODEL_ENV]; else process.env[DESIGN_MODEL_ENV] = savedOverride; + if (savedImageOverride === undefined) delete process.env[DESIGN_IMAGE_MODEL_ENV]; + else process.env[DESIGN_IMAGE_MODEL_ENV] = savedImageOverride; }); describe("design image-generation tool/orchestrator pairing (#1771, #2807)", () => { @@ -71,3 +78,72 @@ describe("design image-generation tool/orchestrator pairing (#1771, #2807)", () }); } }); + +// E5: users switch the tool model without editing code; a bad value fails +// before any request, naming the variable; the live check always tests the +// defaults, and a missing key is a skip, never a pass. +describe("GSTACK_DESIGN_IMAGE_MODEL override (E5)", () => { + const body = () => JSON.parse(imageRequestBody("x", { size: "1024x1024", quality: "low" })); + + test("a gpt-image model name replaces the tool model; blank means the default", () => { + delete process.env[DESIGN_MODEL_ENV]; + for (const model of ["gpt-image-1.5", "gpt-image-2-mini", "gpt-image-3"]) { + process.env[DESIGN_IMAGE_MODEL_ENV] = ` ${model} `; + expect(body().tools[0].model).toBe(model); + } + process.env[DESIGN_IMAGE_MODEL_ENV] = " "; + expect(body().tools[0].model).toBe(IMAGE_TOOL_MODEL); + expect(imageToolModel({})).toBe(IMAGE_TOOL_MODEL); + }); + + test("anything else is refused before a request, naming the variable", () => { + for (const bad of ["dall-e-3", "gpt-5.5", "GPT-IMAGE-2", "gpt-image-2 --x", 'gpt-image-2"}', "gpt-image-", `gpt-image-2${"a".repeat(40)}`]) { + process.env[DESIGN_IMAGE_MODEL_ENV] = bad; + expect(() => body()).toThrow(`${DESIGN_IMAGE_MODEL_ENV}=`); + expect(() => imageToolModel()).toThrow(`unset it to use ${IMAGE_TOOL_MODEL}`); + } + }); + + test("the pairing refusal names the tool model actually requested", () => { + process.env[DESIGN_MODEL_ENV] = "gpt-4o"; + process.env[DESIGN_IMAGE_MODEL_ENV] = "gpt-image-1.5"; + expect(() => body()).toThrow("cannot drive the gpt-image-1.5 image_generation tool"); + }); +}); + +describe("live-model-check always tests the defaults (E5)", () => { + const script = path.join(import.meta.dir, "..", "scripts", "live-model-check.ts"); + const run = (env: Record) => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "design-live-check-")); + try { + return spawnSync(process.execPath, ["run", script], { + encoding: "utf8", timeout: 60_000, + env: { PATH: process.env.PATH ?? "/usr/bin:/bin", HOME: home, GSTACK_HOME: home, ...env }, + }); + } finally { + fs.rmSync(home, { recursive: true, force: true }); + } + }; + const overrides = { [DESIGN_MODEL_ENV]: "gpt-4.1", [DESIGN_IMAGE_MODEL_ENV]: "gpt-image-1" }; + + test("no key: exit 2 with a no-coverage SKIP line, after naming the ignored overrides", () => { + const r = run(overrides); + expect(r.status).toBe(2); + expect(r.stdout.trim().split("\n")).toEqual([ + `testing defaults: image=${DEFAULT_DESIGN_MODELS.image} tool=${IMAGE_TOOL_MODEL} vision=${DEFAULT_DESIGN_MODELS.vision} (ignored: ${DESIGN_MODEL_ENV}, ${DESIGN_IMAGE_MODEL_ENV})`, + "SKIP: OPENAI_API_KEY is not set; the design model defaults were not checked (no coverage)", + ]); + }); + + test("with a key, the requests carry the defaults even when overrides are set", () => { + // An unreachable proxy fails both calls locally: no request leaves the machine. + const r = run({ ...overrides, OPENAI_API_KEY: "sk-test-not-a-real-key", HTTPS_PROXY: "http://127.0.0.1:9", https_proxy: "http://127.0.0.1:9" }); + expect(r.status).toBe(1); + const image = r.stdout.split("\n").find((l) => l.startsWith("call=image")) ?? ""; + const vision = r.stdout.split("\n").find((l) => l.startsWith("call=vision")) ?? ""; + expect(image).toContain(`requested_model=${DEFAULT_DESIGN_MODELS.image}`); + expect(image).toContain(`tool_model=${IMAGE_TOOL_MODEL}`); + expect(vision).toContain(`requested_model=${DEFAULT_DESIGN_MODELS.vision}`); + expect(r.stdout).not.toContain("sk-test-not-a-real-key"); + }); +}); diff --git a/deslop-shared-libs/SKILL.md b/deslop-shared-libs/SKILL.md index 9a5fbead3..64a7b68e2 100644 --- a/deslop-shared-libs/SKILL.md +++ b/deslop-shared-libs/SKILL.md @@ -204,7 +204,9 @@ Keep explanations short and plain spoken. Report: do not hide test or integration costs in a single optimistic number. 3. Up to three ranked recommendations. Link both verified callers, describe the smallest helper and destination, sketch migration and compatibility tests, - explain why it is useful now, and name the main risk or uncertainty. Briefly + explain why it is useful now, and name the main risk or uncertainty: a concrete + behavior that could differ after migration, or the shared-failure blast radius, + never "none". Briefly explain why the other candidates rank lower and how recency affected the choice. Separately identify work covered by existing PRs. If no worthwhile opportunity survives validation, say so and state the evidence limits. diff --git a/deslop-shared-libs/SKILL.md.tmpl b/deslop-shared-libs/SKILL.md.tmpl index 29cf64c5d..b87f791da 100644 --- a/deslop-shared-libs/SKILL.md.tmpl +++ b/deslop-shared-libs/SKILL.md.tmpl @@ -171,7 +171,9 @@ Keep explanations short and plain spoken. Report: do not hide test or integration costs in a single optimistic number. 3. Up to three ranked recommendations. Link both verified callers, describe the smallest helper and destination, sketch migration and compatibility tests, - explain why it is useful now, and name the main risk or uncertainty. Briefly + explain why it is useful now, and name the main risk or uncertainty: a concrete + behavior that could differ after migration, or the shared-failure blast radius, + never "none". Briefly explain why the other candidates rank lower and how recency affected the choice. Separately identify work covered by existing PRs. If no worthwhile opportunity survives validation, say so and state the evidence limits. diff --git a/diagram/SKILL.md.tmpl b/diagram/SKILL.md.tmpl index d36f93856..e6d3dc795 100644 --- a/diagram/SKILL.md.tmpl +++ b/diagram/SKILL.md.tmpl @@ -73,7 +73,7 @@ gstack versions never clobber each other. ```bash BUNDLE="" -for c in "$HOME/.claude/skills/gstack/lib/diagram-render/dist/diagram-render.html" \ +for c in "{{RUNTIME_ROOT}}/lib/diagram-render/dist/diagram-render.html" \ "$(git rev-parse --show-toplevel 2>/dev/null)/lib/diagram-render/dist/diagram-render.html"; do [ -f "$c" ] && BUNDLE="$c" && break done diff --git a/docs/OVERLAY_BENCHMARK_CONTRACT.md b/docs/OVERLAY_BENCHMARK_CONTRACT.md index 1c15e1fe5..d6f3cc38e 100644 --- a/docs/OVERLAY_BENCHMARK_CONTRACT.md +++ b/docs/OVERLAY_BENCHMARK_CONTRACT.md @@ -1,4 +1,4 @@ -# Overlay benchmark behavior contract v2 +# Overlay benchmark behavior contract v4 The paid overlay fixtures verify supported behavior, with comparative efficacy reported separately as research evidence. A passing case does **not** establish @@ -7,7 +7,7 @@ blocking efficacy contract; historical failures retain their original verdicts. The contract is defined by `OVERLAY_CONTRACT` in `test/helpers/overlay-case-policy.ts`. Every new measurement and aggregate record -identifies version 3. Eval entry names include `contract-v3` so automated result +identifies version 4. Eval entry names include `contract-v4` so automated result matching cannot silently compare the old and new contracts by the same name. Selection IDs for the six retained paid cases stay unchanged. @@ -20,9 +20,17 @@ OFF does not. Each retained fixture runs ten samples per arm, separately for Every planned sample in both arms must execute successfully, produce exactly one successful native terminal result without an error flag, preserve workspace scope, and yield a finite metric within the fixture's bounds. Missing reasoning -metadata, failed output validation, oracle process errors, partial sampling, -deadlines, cleanup failures, or recording failures keep the case failed. Missing -observations are never replaced with zero or discarded to obtain a passing arm. +metadata, oracle process errors, partial sampling, deadlines, cleanup failures, +or recording failures keep the case failed. Missing observations are never +replaced with zero or discarded to obtain a passing arm. + +A completed answer that fails the fixture's output check is a measured +correctness result, not an invalid measurement, in either arm: its record has +`measurementPassed: true`, `taskCorrect: false` and the check's message in +`answerError`. Only ON correctness gates the case, so an OFF wrong answer stays +in the comparison and an ON wrong answer fails the case. Version 3 counted an +OFF wrong answer as an invalid measurement (census 37198445662); v3 verdicts +are not reinterpreted. | Fixture family | Exact task and scope contract | Additional ON requirement | Measurement | | --- | --- | --- | --- | diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index bf72850a1..3cd498af8 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -96,9 +96,13 @@ against the previous finalized run (in-flight `_partial` files are never used as a baseline, so a run can't compare against itself). The periodic overlay fixtures use a versioned behavior gate with efficacy -reported separately. See [Overlay benchmark contract v2](OVERLAY_BENCHMARK_CONTRACT.md) +reported separately. See [Overlay benchmark contract v4](OVERLAY_BENCHMARK_CONTRACT.md) for exact correctness requirements, retired fanout cases, immutable evidence, -and the limits of a passing result. +and the limits of a passing result. Under v4 a completed wrong answer is a +measured correctness result in either arm (`taskCorrect: false` with +`answerError`); only an overlay-ON wrong answer fails the case, so an +overlay-OFF wrong answer stays in the comparison. v3 verdicts keep their +original meaning. ## Coverage ownership @@ -379,7 +383,22 @@ report and tracking issue, and nothing requires it. evals-periodic.yml runs ALL periodic-tier files weekly (the coverage contract) minus the reasoned exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking required per entry; removal re-activates the file), plus a weekly -`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. The CI +`EVALS_ALL` gate census. A red census writes one report on every ref, to the +step summary and the `census-report-a` artifact: the run's branch, sha +and event, both lanes' summaries, every pass-rate alarm line, the red ledger +and session headroom (`eval:pass-rates --reds` / `--headroom --runs 10 --branch +`), the all-green estimate and a link to the red-census guide, with +report-derived text passed through `scripts/lib/published-text.ts`. Only a run +on `refs/heads/main` comments on (or opens) the one tracking issue and closes it +on the next green run; the marathon lane does the same with its own issue and +`marathon-report-a` artifact. `test/evals-tracking-issue.test.ts` +pins the main-only guard and runs both report scripts. The host-run Codex job +also carries the live Codex multi-block case (`codex-multiblock-live`: `./setup +--host codex` from the job's checkout into a fresh `CODEX_HOME`, the router in +`codex debug prompt-input`, and a later /learn block rooted by its own +prelude), and the periodic lane runs the design default-model smoke +(`design-model-smoke`), a visible SKIPPED with no coverage credit when +`OPENAI_API_KEY` is absent. The CI image pins the claude CLI to an exact version (`.github/docker/Dockerfile.ci`, enforced by `test/ci-image-cli-pin.test.ts` — bumps ride PRs that run the PTY gate), and every eval-store run records `claude --version`, resolved once in @@ -387,6 +406,7 @@ the runner parent and handed to shard children as `GSTACK_CLAUDE_CLI_VERSION` (never spawned on a test thread), so a TUI-drift flake hunt is a grep, not archaeology. + **Eval verdict policy** (`EVAL_POLICY` version 2 in `test/helpers/periodic-exclude-data.ts`; version 1 pre-registered 2026-09-29, version 2 approved 2026-10-04 with the decision memo in @@ -407,12 +427,20 @@ and the kind fixes its trials before the run: - `judge`: an LLM judge scoring a fixed input. `judgePanel()` (`test/helpers/llm-judge.ts`) draws 3 samples of the same prompt concurrently inside the unchanged `JUDGE_MS`; numeric dimensions gate on the per-dimension - mean against the unchanged threshold (no dimension compensates for another), - booleans on a strict majority. A sample that errors (refusal, truncation, + median of exactly 3 samples, so at least 2 of 3 must meet the unchanged + threshold (`judgePanelMedian()`; no dimension compensates for another, and + any other sample count fails closed), booleans on a strict majority. Panel + logs show every sample and the mean for information; the median is the gate + (EVAL_POLICY v3, approved 2026-10-06). A sample that errors (refusal, truncation, non-JSON, a malformed field) fails the panel and is never resampled; a refusal counts as an unscored panel only when every sample refused. `callJudge`'s 429 backoff happens before any model output and is transport, not a verdict retry. The workflow-judge cache stores whole panels only. + The `rule` case auq-matrix uses the same panel inside its one execution: each + skill's single native capture has its recommendation substance scored by 3 + judge samples, gated on their median against the unchanged minimum of 4, and a + failed sample fails the panel without resampling (approved 2026-10-04 for + censuses after the change). `panelVerdict()` (`test/helpers/eval-store.ts`) is the single verdict function the report, `collector-outcomes.json`, the PR comment and pass-rates @@ -430,6 +458,59 @@ reported. Changing any `EVAL_POLICY` constant after seeing census results needs Garry's re-approval, a `version` bump and a fresh census; `test/periodic-exclude-policy.test.ts` pins the approved values. + +**Failure causes and details** (diagnostic only; verdicts unchanged under v1). +Every failed trial record keeps `failure_class` (`assertion`, `contract`, +`timeout`, `infra`: the field verdicts, INFRA re-dispatch and pass rates read) +and its 300-character `error` exactly as before. `failureCauseOf()` +(`test/helpers/eval-store.ts`, called by both record builders, including the +JUnit census for PTY and JUnit-only cases) adds an optional `failure_cause` +with a one-line `failure_cause_evidence`. It never changes `failure_class`. +The highest-precedence cause with evidence wins: + +| `failure_cause` | Recorded when | First step | +|---|---|---| +| `contract` | `expectContract()` stamped a contract violation | fix the product; a violation fails a behavior panel outright | +| `pre_turn_infra` | the trial failed before the first model turn (`failure_class: 'infra'`) | check the runner and API status; an all-INFRA census is re-dispatched once | +| `api_error` | a session ended on an API error (for PTY, the `API Error:` panel on an idle turn) or the runner's exit reason was `error_api` | read the evidence line; a provider outage is a named red | +| `refusal` | the structured stop reason was a refusal (phrases only as a last resort; quoted refusal text in a tool result does not count) | check whether the fixture prompt invites the refusal | +| `provider_stall` | a session that streamed partial messages saw no event of any kind, thinking deltas included, for `STALL_WINDOW_MS` (120 s) while a model request was in flight and no tool, permission prompt, hook or subagent was outstanding; sessions without partial messages are never stalls | read the session ledger row; still a failed trial under v1 | +| `session_timeout` | a runner's armed session timeout fired | `eval:pass-rates --headroom`; cut work, never raise the budget | +| `observer_timeout` | the test's observer gave up waiting for an outcome (`outcome=timeout`) | read the transcript tail: a missed detector or a product that never answered | +| `assertion` | a check failed after the session ran (`session completed; check failed` when it ended normally) | the Expected/Received values or judge dimensions | +| `unknown` | the trial failed with no evidence for any cause above | inspect the shard directory | + +A detector, harness or product fault is a human diagnosis, never a cause. +Readers before this field accept the new records (a frozen v1 reader runs over +them in `test/eval-store.test.ts`); an unknown cause from a newer gstack is +rejected with `unknown failure_cause "" (written by a newer gstack; update +this checkout to read it)`. + +`failure_detail` carries the values `error` cannot hold: a failed Bun matcher's +`{expected, received}`, or for a judge one `{dimension, mean, threshold, +samples, rationale}` entry per failing dimension (since EVAL_POLICY v3 `mean` +holds the panel's gating median), each part capped at 200 +characters. `sessions[]` (at most 32) summarizes the trial's session-ledger +rows. `cost_known: false` marks a record or trial whose harness captured no +billing: PTY and Codex sessions never bill, and an SDK session bills only with a +terminal `total_cost_usd`. An absent field means known. Census and marathon +cost lines print the known sum and `N of M trial(s) cost unknown`, never a bare +total over unknowns. + + +**Session ledger.** Every runner (session-runner, the Agent SDK runner +including `runSharedInteractive`, the PTY session and `runRecordedCodexEval`) +appends one best-effort line per armed session to +`$GSTACK_EVAL_DIR/session-ledger.jsonl` (`test/helpers/session-ledger.ts`): +a stable key `: