docs: approved reliability follow-ups plan (Oct 2026)

docs/designs/RELIABILITY_FOLLOWUPS_2026_10.md

@@ -0,0 +1,112 @@
+<!-- /autoplan restore point: "/home/user/.gstack/projects/garrytan-gstack/garrytan-reliability-followups-oct4-autoplan-restore-20261004-150446.md" -->
+# gstack reliability follow-ups (October 4, 2026): headroom, honest eval verdicts, last host gaps
+
+## Implementation plan
+
+### Context
+
+Base: origin/main `2db0b3adc` (v1.91.19.0, #3025). The severe fix wave left follow-ups in TODOS.md. Its eight periodic censuses on the wave branch (37174266054 … 37198445662) are the evidence base for this plan:
+
+- 0 of 8 censuses were green; 21 verdict reds in total (2-3 per census).
+- By cause: 10 detectors, scripted actors or the CEO mode handoff (each repaired one at a time); 5 judge low samples; 3 provider-shaped events (one stream stall, two refusals); 2 fixture/product; 1 headroom.
+- 27 of 28 failed trials are labelled `assertion`, including observer timeouts, so the reports themselves misdescribe what happened.
+- At the measured rate (about 1.2% red per verdict across ~193 verdicts) an all-green census is about 7% likely. The lever is the per-verdict red rate, and the biggest single source is brittle detectors and scripted actors, not provider events.
+- Slowest cases: plan-ceo-mode-routing up to 687 s and plan-mode-no-op 613 s; ship-docsync-completion at 88% and store at 79% of budget; census wall time 14-15 minutes against the ~10-minute target. review-army-perf (102-140 s of 300 s) and plan-design-review-plan-mode (142-240 s, no timeouts) are not headroom problems.
+- In census 37198445662, 20 of the 21 workflow judges that score actionability sat at exactly 4.00 and none scored 5: text edits can remove low samples but cannot lift a mean that the rubric caps at 4.
+
+### Release promises
+
+- **P0. Census reds become rarer and honest.** The PR reports, per census, verdict reds by machine class and the expected all-green probability, for the 8 pre-PR censuses and every census on this PR. Every red class that recurred has a structural fix (replay corpus, structured signal or speedup), not a one-off spelling patch.
+- **P1. The slowest cases get headroom.** Cases the headroom report flags above 75% of their armed budget are brought under it by doing less work; no budget is raised. The weekly `eval:pass-rates --gate` alarm (not a free check against a committed file, which could never fail on a code change) flags any case above 85%. The census wall target is 10 minutes, so the slowest gate slice must fall to about 450 s (today 711 s).
+- **P2. Labels say what happened.** Every trial record keeps `failure_class` (`assertion | contract | timeout | infra`, which verdicts read) and its 300-character `error` field exactly as today, and gains optional, bounded fields: `failure_cause` (`contract`, `pre_turn_infra`, `api_error`, `refusal`, `provider_stall`, `session_timeout`, `observer_timeout`, `assertion`, `unknown`), `failure_cause_evidence`, and `failure_detail` (assertion `{expected, received}` or judge `{dimension, mean, samples, rationale}`, each part capped). A machine cannot tell a detector miss from a product miss, so there is no `detector` cause; that is a diagnosis written by a person. Verdict outputs (`panelVerdict`, `collector-outcomes.json`, lane exit codes, INFRA re-dispatch eligibility) are identical under EVAL_POLICY v1, proven by running main's and the branch's verdict code over the 8 baseline censuses' artifacts and by running a frozen copy of the v1 reader over new records. This PR is judged under v1.
+- **P2b. A red points to its evidence and the next step.** Each red line in the census report and tracking issue shows the case and trial, the observed failure with its Expected/Received values (or, for a judge, the dimension, mean, samples and lowest-scoring rationale), the artifact that holds the transcript, and the next action: inspect and replay free first; a paid run only "after a repair", selecting just that case. Target: diagnosis from the report line alone in 5 minutes per red (today 15-20). Every model- or fixture-written string that reaches a public issue, PR comment or committed file passes one sanitizer: control characters stripped, mentions neutralized, a code fence longer than any backtick run, HTML escaped, length capped, and the repo's credential-scan patterns redacted (adversarial test: @mention, triple backtick, `</details>`, image link, `ghp_`/`sk-` token, 10 KB string).
+- **P3. Every low judge sample has a cause.** Each low workflow-judge sample in recent censuses is traced to a skill-text fix or recorded as a density-only complaint; judges are first checked on deliberately bad control bundles to learn whether they can discriminate at all.
+- **P4. The wave's small product gaps close** (Bun floor, make-pdf `$P`, the #2709 reap, /review's fingerprint under Codex's read-only `.git`, design model smoke, live Codex periodic case).
+
+### Workstream 0 (first, free): evidence
+
+- **0.1 Session ledger, headroom report and red ledger.** Runners (SDK, PTY, Codex) append one line per armed session to a small session ledger file in the trial's artifact dir (same pattern as `contract-violations.jsonl`): stable session key, the exact timeout value it armed, monotonic elapsed, end reason, and a compact liveness summary (longest in-request silence, last event kind, tool open at end). The record builders (including `junitCensus` for PTY and JUnit-only cases) attach it to trial records. A new module (not `eval-flake-rank.ts`) reports per case: elapsed vs armed budget per session (a case's headroom is its slowest session; timed-out samples are marked censored), the lane critical path from runner slice timings, and verdict reds by machine cause with per-case rates. Past censuses carry no session clocks, so history shows the case wall as a labelled upper bound and session headroom starts with the first census after this lands; below a minimum sample count a case is `insufficient`, never a pass. Reached as `bun run eval:pass-rates --headroom | --reds | --run <id>` with offline `--help`, unknown-flag and unknown-case rejection, `--case` applied to JSON and alarms, and the printed run/branch scope; `--run` downloads only the slices its red lines name. The all-green probability is Π(1-p_i) over per-case shrunk rates with an interval, labelled an approximation (independence is not proven).
+- **0.2 Failure cause classifier (policy-neutral).** One pure, dependency-free `failureCauseOf()` beside `failureClassOf()` (not in `e2e-helpers.ts`, which imports `bun:test` and selection machinery), called by both record builders from runner observations and the session ledger; precedence contract > pre_turn_infra > api_error > refusal > provider_stall > session_timeout > observer_timeout > assertion > unknown; it never changes `failure_class`, and conflicting evidence yields the higher-precedence cause with both recorded: a PTY idle turn showing an `API Error:` panel records `api_error` with the panel line (the PTY runner's `observe()` already stops on it; other runners match); refusals are detected from the structured stop reason first, phrases last; a stream stall is recorded only when the session streams partial messages and no event of any kind, thinking deltas included, arrives for 120 s (pre-registered; the longest in-message silence across 50 captured messages on Claude Code 2.1.284 was 9.6 s) while a model request is in flight and no tool call, permission prompt, hook or subagent is outstanding. Sessions without partial messages are never classified as stalls. Partial messages are enabled for this, but only the compact liveness summary is kept (full diagnostics would roughly quadruple transcript size). `panelVerdict` and lane verdicts are unchanged. Replays: 37195203538 (refusals), 37174266054 and 37176837432 (timeouts), and 37186854666 (it ran without partial messages, so it must come out `session_timeout`); the `provider_stall` positive is a synthetic fixture with an injected silence; controls: long-thinking, hung-tool, permission-pending, hook-running and subagent-running transcripts are not `provider_stall`; a tool result quoting refusal text is not `refusal`; one fixture per runner (SDK, PTY, Codex, JUnit).
+
+- **0.3 Report and tracking issue.** Expected/Received values (today `sanitizeTrialError` keeps only the first line, so 16 of 28 wave failures show nothing useful) and judge scores and rationale go into `failure_detail`, not `error`; every red names its artifact; `rerun:` becomes "after a repair:" and selects only that case; the weekly issue includes the full pass-rate alarm lines (today its grep keeps only the header), the red ledger and the all-green probability (a documented formula with a command anyone can rerun); branch-dispatched censuses no longer post to or close main's tracking issue (both steps gated on `refs/heads/main`, same in evals-marathon.yml), and the periodic workflow's concurrency group is scoped by branch so a branch census no longer cancels main's; branch and run identity are passed explicitly instead of `git rev-parse --abbrev-ref HEAD`.
+- **0.4 "A census went red" guide.** A short page linked from the report, the issue and CONTRIBUTING: prerequisites, the exact inspect and replay commands, expected output, what unknown data means, and when a paid run is allowed (only after a repair, per TESTING_INTERNALS). Durable evidence (the baseline red ledger, judge diagnoses) is committed under `docs/evals/`; scratch stays in `.context/`.
+
+### Workstream A: headroom on the cases the report flags (P1)
+
+- **A1. ship-docsync completion (88%) and store (79%).** Diagnose from the stored transcripts first. If the parent's reading of release-file contents dominates, the parent stops reading them and recomputes hashes with `bin/gstack-docs-candidate` (cheap, no model reading), comparing them with the child's claimed hashes; a mismatch stays a hard failure. Any change to the child's result contract updates the parser, child, actor and parent together, defines the hash algorithm and file set, and keeps negatives for stale results, missing files and unreported changes.
+- **A2. plan-mode-no-op (432-613 s; likely the 711 s critical-path gate slice) and plan-ceo-mode-routing (687 s)**: time-boxed to the stored transcripts, report a per-phase breakdown (preamble, Step 0, model latency, tool time) first, then cut work in the skill or fixture; if model latency dominates, it becomes a named item rather than a cut that weakens the workflow.
+- **A3.** Any other case the 0.1 report puts above 75%. review-army-perf and plan-design-review-plan-mode get no work unless the report flags them.
+- **A4. Fresh critical journey.** The marathon lane runs the existing fresh /plan-eng-review case (plan-eng-review-artifact-full) on the PR head; time to the first question and to the written test plan are reported in the PR (non-blocking lane). A fresh /ship marathon case would be new harness work and goes to TODOS.
+- Before/after numbers come from named CI censuses, never local runs.
+
+### Workstream B: robust detectors and actors (the largest red class)
+
+- **B1. Replay corpora for repeat offenders.** shared-libs-plan-callers, plan-ceo-section-loading, auto-decide-preserved, plan-design-review-plan-mode and ship-docsync-late-result each get the stored passing and failing captures still downloadable (inventory first; artifacts expire after 30-90 days, and expired ones are listed as unavailable) replayed through their detector or actor in the free suite (≥1 pass and ≥1 fail each). Entries are minimal, redacted excerpts (the detector's input slice, size-capped, credential-scanned in CI), pinned to the series identity they came from, with stale entries reported for refresh. Each corpus entry records its original census outcome, the expected detector/actor result and a one-line justification, so a fixed false red is an expected pass with its history kept, and genuine negatives stay refused.
+- **B2. Structured signals before phrases.** Where a skill already writes a structured record (decision ledgers, review logs, handoff helper output, `VERDICT:` lines), valid current structured evidence (right session, after the action, not copied fixture input) is authoritative; phrase detectors run only when no structured evidence exists. Where the contract is user-visible output (a question shown, text rendered), the behavioral witness stays required and the record only locates it. Audit every periodic and gate detector for phrase-only checks and convert the ones with a structured source.
+- **B3. Floor runner TTY preview** (TODOS P2): accept collapsed context rows and gutterless wraps in `currentEditPreview`, still compared against exact source lines; replay the screen from PR run 37176835584.
+- **B4. `--case` selects loop-registered cases** (coverage audits, TPA, plan-mode-no-op) and an unknown id exits non-zero before any process starts.
+- **B5. Pre-push hook hang behind shard neighbors**: reproduce with the failing plan under xvfb and diagnose first; fix if the cause is in scope, otherwise a named open item.
+
+### Workstream C: judges (P3)
+
+- **C1. Calibration controls first, only for judges that produced low samples** (qa, qa-only, review, plan-ceo modes, plan-eng sections, setup-deploy as applicable). Score compression at 4 is a hypothesis, not a fact (the schema allows 5). Pre-register in `docs/evals/` the positive, negative and targeted-defect control bundles, the expected outcome for each, and one 3-sample panel per judge, reusing `scripts/judge-calibration-configs.ts`. If a judge cannot separate the negative controls, list its cases in the PR as giving no coverage and bring a rubric proposal to Garry as a pre-registered EVAL_POLICY change.
+- **C2. Trace low samples.** Collect every low workflow-judge sample's rationale in the last N censuses into `.context/`; fix skill text only where a rationale names a concrete, execution-relevant gap (and tie it to an observable failure where one exists); density-only complaints are recorded, not chased. Regenerate, run parity, prompt-size and removed-clause checks, then one 3-sample panel per changed judge.
+- **C3. Behavior cases in drift.** plan-review-prosons-neutral-neg (flagged ACTION REQUIRED by `eval:pass-rates`) and shared-libs-opportunity-judgment: diagnose from stored transcripts and fix the cause, or propose CASE_QUARANTINE entries only if the diagnosis is detector/harness/model-latency and the entry rule is met.
+- **C4. auq-matrix.** Its recommendation-substance check gates a `rule` case on one judge sample and was red in 6 of the last 24 censuses. Gate question below: keep capture and judging in one execution and score that one capture with a 3-sample panel (mean, unchanged threshold), recording the evidence in `E2E_KINDS`.
+
+### Workstream D: provider events and the overlay control arm
+
+- **D1. Overlay comparison control arm (gate question below).** Today `fixture.verify()`/`measurementsValid` turns a control-arm wrong answer into an invalid measurement that fails the comparison. Proposed: a completed wrong answer in either arm stays a measured correctness result in the comparison report, and only predefined execution or measurement failures invalidate a trial, applied symmetrically; the overlay contract moves to v4. Because this changes what a gate scores after the 37198445662 red was seen, it needs Garry's approval and applies only to censuses after it.
+- **D2. plan-ceo-review-format-approach refusals.** Check whether its capture prompt ("verbatim … exact prose you would have shown") triggers the refusal classifier; reword the fixture if so.
+
+### Workstream E: small product gaps (P4)
+
+- **E1. Bun versions from one source.** Supported minimum stays 1.4.0 (`engines.bun`, the tested CI version); 1.3.3 is the security floor (below it D0's no-autoload flags are silently ignored, verified by a compiled probe, not a version comparison). Extend `test/bun-version-drift.test.ts` so setup, `engines`, the install hint (now 1.4.0 instead of 1.3.10), README and CI agree. Setup refuses below 1.3.3 and warns between 1.3.3 and 1.4.0 ("untested, upgrade with `bun upgrade`"), with a troubleshooting anchor; prerelease strings are compared by a defined rule; `--help` and `--status` keep working without Bun. The auto-updater fetches, reads the incoming release's requirement and checks the `bun` that setup will use before advancing the live checkout, leaving the live revision and installs untouched on rejection with a stable `bun-too-old` reason and resume command. The first upgrade run by an older updater cannot be protected; the CHANGELOG says so.
+- **E2. make-pdf `$P`** joins `$B`/`$D` in the shared runtime prelude for env-var hosts through one make-pdf resolver used for both readiness and every block, preserving the `MAKE_PDF_BIN` override and the real `make-pdf/dist/pdf` name (paths with spaces tested).
+- **E3. #2709 reap on Playwright 1.62:** record the Chromium PID via CDP `SystemInfo.getProcessInfo` (or the launch server) only for launches gstack owns; the reaper keeps checking recorded start time and command identity before signaling; tests cover the pinned launcher, bounded CDP failure, a recycled PID and an externally owned browser that is never reaped.
+- **E4. /review fingerprint under Codex's sandbox:** `bin/gstack-wtree` writes new objects into a private object directory (`mktemp -d`, 0700, under `$TMPDIR`, absolute common-dir objects as alternates resolved before exporting, linked worktrees supported) whose lifetime is owned by the review-log operation, because `sharedLibsSnapshotCoverage()` later reads the fingerprint's tree and blobs; it is removed (EXIT/INT/TERM) only after that consumer finishes, and a failed write exits 1 instead of printing a partial hash; a failed `git fetch` reports stale-base coverage naming the base revision used; `gstack-review-log` keeps the fingerprint helper's one-line stderr diagnostic (stdout stays hash-only), and /review Step 1.3's `git fetch &&` chain no longer stops at "Nothing to review" under a read-only `.git`; tested through `--start` → `--finish` → shared-libs reuse under a read-only `.git`, with dirty tracked files, untracked files and a linked worktree.
+- **E5. Design model smoke (periodic).** The defaults already pin gpt-image-2 (`design/src/models.ts:40`); register `design/scripts/live-model-check.ts` as a periodic case so a retirement or entitlement change shows up before users hit it. A missing key (exit 2) is a visible skip with no coverage credit. The image tool model gains an env override (`GSTACK_DESIGN_IMAGE_MODEL`, validated against a conservative pattern, named in errors) so users can switch without editing code; the periodic check always tests the default, so an override cannot mask a retirement.
+- **E6. Live Codex multi-block check as a periodic case** in the host-run Codex job, reusing the job's checkout and running setup once so it does not become the new critical path.
+
+### Gate questions for Garry (decided before Workstream 0 lands)
+
+EVAL_POLICY v1 lets an all-INFRA census be re-dispatched once, and counts only errors before the first model turn as INFRA. Should `provider_stall` (with the streaming evidence in 0.2) join INFRA under a v2 for future censuses? This PR is judged under v1 either way. Refusals stay a failure class of their own in both options (a refusal is model output users also see). Recommendation from the CEO review: labels only now (v1 unchanged); revisit v2 with the pooled provider-event base rate from 0.1 attached.
+
+Also for approval: D1 (overlay control-arm scoring, applies to later censuses only) and C4 (auq-matrix scores its single capture with a 3-sample judge panel).
+
+### Delivery
+
+One PR, a commit per item. Lanes (each commits sources only; one integrator owns `touchfiles-data.ts`, the ratchet file, generated SKILL.md, VERSION, CHANGELOG): F (session ledger, classifier, report, issue/workflow), P (A1, A2 headroom fixes), S (B1-B5 detectors and harness), D (C1-C4 judges, D1-D2), W (E1-E6 product); F, P, S and D start together, W's E1 waits for F's workflow edits. Freeze harness edits before the first paid PR-lane push, because every push re-runs the full gate. Patch version from the queue.
+
+### Validation
+
+- Each fix has a free regression test that fails on main; detector and classifier changes replay captured transcripts with negative controls.
+- Paid runs are pre-registered at the gate as an inventory: one diagnostic run per concrete repair of a `rule` case (never on unchanged inputs), a 3-trial panel for a changed `behavior` case, one 3-sample panel per changed judge and per C1 calibration set, the new E5/E6 cases once each to prove wiring, then the PR lane once on the frozen head (full gate, ~106 cases and 24 judges, because `engines.bun` and new files force full fallback), one periodic census on the frozen head, and the marathon run. Diagnostic runs never form a verdict.
+- Free suite once on Ubicloud at the end; PR gate lane; one periodic census on the frozen head.
+- Each workstream names the docs it updates (TESTING_INTERNALS, troubleshooting, the new red-census guide, CONTRIBUTING, AGENTS.md, README, TODOS.md). The 75%/85% thresholds and the stall window are named constants in one module; per-case exemptions go through a list with reason and expiry, never a raised budget.
+- **Done when** PR CI is green and the census on the frozen head has every red either machine-classified INFRA under the approved policy or listed as a named red with diagnosis in the PR and TODOS.md. A repair made after that census needs a fresh census on the new head (TESTING_INTERNALS:407-411). "Diagnosis complete" is not "release accepted": under v1, an INFRA-only census may be re-dispatched once; any other red left unresolved needs Garry's explicit exception before merge. The PR body carries the red-class table, the all-green probability before/after, and a "diagnosable from the report line alone" column.
+
+### Not in scope
+
+Comparing gstack end to end against the stock host agent on product outcomes (the overlay harness covers model-level deltas; a full product A/B is its own project, filed to TODOS). Severe-wave items not taken here: C8 router panel, #2670 bash half, G2 TS twin, Codex telemetry/notice follow-ups, gate-outcome telemetry, design health subcommand, cmd.exe autoplan hook, /gstack-upgrade redesign, marketplace, placeholder convention, #2750, ios-qa tokens, #931, #1713, Windows SAC Node lane, navigation pre-send blocking, gbrain batch lookup, memory follow-ups, #2971 non-privacy pieces, redaction allowlist #2598.
+
+## Review record
+
+Autoplan run started 2026-10-04 (GSTA-21). Source: TODOS.md on main 2db0b3adc.
+
+### CEO phase, 2026-10-04
+
+Voices: Codex gpt-6-astra (completed, 8 findings, recommends staged releases), Claude native blind (hash 3163a6b1 matched, 2 critical: stall rule ambiguity, PR judged under its own new policy), Claude primary (8-census data, 15 amendments). Consensus: confirmed on all six dimensions that the draft optimized stale premises (A2/A3 not slow; real slow cases missing), that B as drafted would mislabel thinking or tool hangs and let the PR judge itself under v2, that A5's data source cannot support per-case headroom, and that judge margin cannot come from text alone. Disagreement (taste): Codex wanted separate releases; Claude voices kept one PR (owner rule). User challenge candidates were reframed as the gate's policy question (labels-only vs stall-as-INFRA) because all voices agree the change needs Garry's approval regardless. Rewritten plan above adopts AM-1..AM-15 plus native F1/F3/F4/F7/F10/F12/F13 and Codex 2/3/4/5/6/8.
+
+### DX phase, 2026-10-04
+
+Voices: Codex gpt-6-astra (completed, 10 findings), Claude native blind (hash a661b072 matched; 2 critical: failure-class schema clash, non-deterministic free 85% check), Claude primary (DX 4.3 → 7.1; time from red census to diagnosis 15-20 min per red → target 5). Consensus confirmed on all six dimensions: the report line must carry evidence and the next action (not `rerun:`), the new classes must not touch the policy field or verdicts, the headroom check needs a defined clock, statistic and committed snapshot, the CLI needs help/validation, a two-minute triage guide, and E1 needs an upgrade recovery path. Taste: T-DX-1 Bun floor 1.3.3 (security) with 1.4.0 as tested version, chosen over a hard 1.4.0 refuse (Claude primary; avoids stranding git installs at SETUP_FAILED; Codex wanted one minimum, which 1.3.3 from one source satisfies); T-DX-2 separate `failure_cause` field (all three voices); T-DX-3 85% alarm also in `eval:pass-rates --gate` (accepted).
+
+### Eng phase (last), 2026-10-04
+
+Voices: Codex gpt-6-astra (completed, 16 findings, recommends splitting), Claude native blind (hash 43124ba1 matched; high: error-field compat, unsanitized public text, headroom data gap, D1 is a post-hoc gate change), Claude primary (feasibility verified: thinking deltas present on Claude Code 2.1.284 with 9.6 s max in-message silence; past censuses carry no session clocks; full-fallback paid selection; census critical path 711 s; 16 amendments). Consensus confirmed on all six dimensions. Accepted: session ledger + pure classifier beside failureClassOf; bounded new fields with `error` untouched and a frozen-v1-reader test; one sanitizer for published text; no `detector` cause; synthetic stall positive; E4 object lifetime owned by review-log; E1 checks before advancing the checkout with 1.4.0 kept as supported minimum; D1 and C4 to the gate; main-only issue steps and branch-scoped concurrency; structured-first detectors keep behavioral witnesses for user-visible contracts; pre-registered paid inventory. Trimmed for proportion: the committed-snapshot 85% free check (could never fail on code), the fresh /ship marathon case, and the separate auq-matrix judge case; C1 runs only for judges with low samples. Disagreement (taste): Codex recommended splitting into several PRs; kept one PR per Garry's standing rule, with dependency-ordered lanes. Test plan: ~/.gstack/projects/garrytan-gstack/garrytan-reliability-followups-oct4-test-plan-20261004.md.
+
+### Final approval gate, 2026-10-04
+
+Garry approved the plan as one PR. Decisions: (1) provider stalls are labels only (`provider_stall` cause, still a failure; EVAL_POLICY v1 unchanged); (2) D1 approved: a completed wrong answer in the overlay-OFF arm counts as a comparison result, not a failure, for censuses after this change (overlay contract v4); (3) C4 approved: auq-matrix scores its one capture with a 3-sample judge panel, mean against the unchanged threshold, for censuses after this change. Status: APPROVED.
+