v1.91.24.0 test: audit wave — PR evals keep finished work, honest ceilings, EVAL_POLICY v2, faster free suite (#3031)

* chore(scripts): retire hollow and legacy paid package scripts behind one-release stubs (W8a, DX-4, CEO-31)

test:codex*, test:evals*, test:e2e*, test:gate, test:periodic, eval:bg,
eval:bg:all, eval:flake-rank, eval:watch and test:audit now print their
replacement and exit 1 via scripts/retired-command.ts. CONTRIBUTING.md
carries the same table; test/retired-commands.test.ts pins both. Eval CLI
empty-state hints point at eval:bg:pr.

* chore: delete dead update-readme-throughput, eval-watch, orphan fixtures and QA_FUNCTIONAL_INPUTS (W8b, CEO-24)

Retired with named behavior:
- update-readme-throughput.ts + readme-throughput test: replaced a README
  anchor that no longer exists from a JSON file that was never produced;
  the script was a no-op. Guarantee withdrawn (nothing to protect).
- eval-watch.ts + its dashboard tests: rendered ~/.gstack-dev/e2e-live.json
  and _partial-e2e.json, written only by unsharded runs, so it showed
  nothing for every supported (sharded) entry point. Guarantee withdrawn;
  live progress is the gstack-detach log or gh run watch. Manual-review
  rendering stays covered by eval-list and eval-summary tests.
- Orphan fixtures (autoplan-pending-artifact-ae.json, autoplan-existing-app/,
  autoplan-sign-in-ui/, parity baselines v1.47.0.0/v1.57.7.0/v1.69.1.0):
  no test, touchfile or script reads them; superseded baselines stay in
  git history.
- QA_FUNCTIONAL_INPUTS: exported, never imported.

* refactor(test-free): delete dead WORKER_HOSTILE list and buildShardArgs parallel mode

Neither had a production caller: full-suite mode runs shard processes, so the
worker quarantine was inert and --parallel was never passed (W8b).

* docs(test-value): narrow prompt-byte contract to machine-read tokens (D2)

Define prompt-byte contract as machine-read tokens (markers, field names,
command lines, enum values, file paths, security-boundary strings) and add a
sentence-pin glossary entry: convert to structural checks or delete when a
behavioral eval covers the behavior.

* test(ratchet): count free tests that slice a paid test's source

A free test that binds readFileSync(<paid test file>) and cuts it with
.slice() tests the paid harness's code out of context even when it imports a
lib/ symbol. Record the 15 existing files with reasons; new ones fail with fix
text. Self-test covers positive and negative shapes.

* test(browse): un-quarantine compare-board; all 16 board tests pass headless (W8d, CEO-24)

The GSTACK_COMPARE_BOARD_TESTS opt-in hid a headless mismatch that no
longer reproduces: all 16 tests pass on Bun 1.4.0 with the bundled
headless Chromium, solo, repeated, and beside other in-process browser
files. Protected behavior (the comparison-board DOM contract the
plan-design-review/office-hours/design-consultation feedback loop polls)
is checked on every free run again.

* test: replace silent existsSync early returns with explicit skips or assertions (W8d)

- browse/test/build.test.ts: server-node.mjs checks skip only when the
  bundle is absent and GSTACK_EXPECT_BINARIES is unset; the free CI lane
  and test:ubicloud build it and set the flag, so a dropped build step
  fails with the build command instead of passing with no assertions.
- gstack-update-check: VERSION is tracked; the guard could only hide a bug.
- gen-skill-docs host leakage/frontmatter checks assert the host tree
  exists (the module-level render always produces it).
- cso-docker staged-runtime test uses test.skipIf so reports show it as
  skipped, not passed.

* test(ship): convert ship-workflow-clarity sentence pins to structural checks

Add test/helpers/prompt-structure.ts (between, expectTokens, expectAbsent,
expectOrdered, expectMentions) with fix text on failure. ship-workflow-clarity
now pins headings, step order and machine-read tokens (commands, fields, enum
and status strings); safety rules are meaning-level keyword checks. Wording is
judged by the 'ship/SKILL.md workflow' judge in skill-llm-eval.
Sentence pins: 90 -> 0.

* ci: add platform-qualification lane for opt-in ML, gitleaks and Swift tests (W8d, CEO-24)

security-bench + security-live-playwright L4 (SECURITY_BENCH), the real
gitleaks 8.30.1 memory-ingest cases (GSTACK_TEST_GITLEAKS) and the
DebugBridge Swift build invariants (GSTACK_TEST_SWIFT) self-skip in every
lane today. No scheduled lane has their prerequisites, so per CEO-24 they
move to an owned dispatch + quarterly workflow instead of being deleted.
Each job was run locally (ML 10 pass, gitleaks 130 pass; Swift needs
macOS). test/platform-qualification-workflow.test.ts pins gate variable,
job and file together.

* fix(scripts): capture-baseline and preflight-agent-sdk no longer act on --help or typos (W8e)

capture-baseline parses its flags strictly: --help prints usage, an unknown
or valueless flag exits 2, and only a valid invocation writes a fixture
(--help used to write test/fixtures/parity-baseline-current.json).
preflight-agent-sdk gains --help, rejects unknown flags, and runs the live
SDK query (API spend) only with --live.

* test(evals): register the three keyless periodic paid files with case ids (W1b)

skill-e2e-auq-consistency, skill-e2e-auq-verbose-vs-carved-ab and
codex-e2e-recommendation-substance now carry touchfiles, a periodic tier and
kind rule (one trial, unchanged), so editing one selects its own case instead
of restoring the full PR gate. The verbose A/B case also registers the vendored
pre-cut fixture it reads, and review-army-migration-safety registers the SQL
fixture it reads.

* feat(evals): narrow PR fallback for free-only inputs (W1a, CEO-13/20, ENG-13)

The PR profile now exempts the duration seeds, every free-lane workflow
(explicit list; inventory test fails on an unclassified workflow, .yaml
and workflow_call included), scripts/ubicloud/** and fixtures declared in
the explicit FREE_FIXTURES map. Each remaining unknown file carries a label
and fix (needs touchfile entry / add to FREE_ONLY_PR_FILES / real unknown
dependency). evals.yml itself still restores the full gate.

* ci: delete ci-image.yml; the eval workflows already build and push the same content-hash tag (W8f)

evals.yml, evals-periodic.yml and evals-marathon.yml each compute the
identical hashFiles() tag and build/push it (with :buildcache and :latest)
when missing, so ci-image's push trigger and weekly cron were no-ops
(30-day median 0.5 min, 36/36 success). ci-image-tag-binding now pins the
three eval workflows.

* ci(evals): key periodic/marathon concurrency on ref+event; branch periodic dispatch skips the gate census (W2a, ENG-9, CEO-04)

* refactor(paid): drop ~160 unused imports; leaf paid-types/paid-select modules break the paid-* import cycle (W8c)

* ci(quality-gate): per-PR concurrency, free runner, drop unscoped bun audit (W2a, W8f)

- Concurrency keys on pull_request.number || run_id, so back-to-back main
  pushes no longer cancel each other's credential scan.
- ubuntu-24.04 instead of a billed 8-vCPU Ubicloud runner for a ~1 min job.
- bun audit --audit-level=critical ran on every PR regardless of diff, so a
  newly published advisory turned unrelated PRs red, with its own ignore
  policy. dependency-review (lockfile PRs, fail >= high) and the weekly OSV
  sweep remain the dependency gates.

* chore(test-free): refresh the free duration seed from the audit's Ubicloud recording

Adds the 19 files merged after the last refresh, including the 115 s
claude-overlay-setup that packed as ~1 s and doubled a shard (W6a). The seed
is re-recorded on the final tree at the end of this wave.

* ci: move tiny ubicloud-standard-2 jobs to free ubuntu-24.04 runners (W8f)

actionlint, skill-docs, dependency-review, version-gate and pr-title-sync
are sub-minute jobs; GitHub-hosted runners are free for this public repo.

* feat(test-free): pessimistic unseeded packing, seed growth ratchet and CI health summary

- Unseeded files pack at the seed's p99 instead of p75 (W6a).
- --ci-plan warns in the job summary past 5 unseeded files; --ci-verify
  lists shards that outran their prediction by 1.5x (and 30 s) plus every
  flaky pass absorbed by the serial retry (W6a, W7a job summary).
- test/free-seed-ratchet.test.ts fails when a seeded file over 60 s is not
  on scripts/free-test-seed-allowlist.json, when an entry is stale, and when
  the allowlist grows versus its merge-base copy unless the new entry is an
  annotated exception with a todo (CEO-10, DX-7). free-plan runs it with
  full history so the shrink check is real in the required lane.
- Every message names the fix command and docs anchor (DX-6).

* feat(evals): state PR coverage mode, counts, reuse and fallback files in the summary and comment (CEO-13)

* test: make the tracked-tree skill-doc freshness check local-only; tidy skill-docs header (W8f)

Every free CI shard regenerates the tracked SKILL.md files before running
tests, so the --dry-run freshness assertion compared the regenerated tree
with itself and could not fail there. It now skips under GitHub Actions;
skill-docs.yml's git diff --exit-code is CI's freshness gate. Removes a
stale concurrency comment in skill-docs.yml.

* ci(free-tests): install zsh so the bash+zsh portability arms run (W6f)

~42 zsh-arm tests (aside-probe-shell, qa-browser-preservation,
generated-docs-fences) skipped in CI because zsh is not in the runner image.
All 141 tests in those files pass with zsh present.

* feat(evals): executors write negative receipts at the execution boundary (CEO-25, ENG-6)

A completed single-trial rule/judge shard that did not pass now writes its
negative receipt beside the pass receipts in the slice artifact, so a
cancelled run's FAIL still blocks an older PASS when its report never runs.
Receipts merge negatives first and order by (run attempt, completedAt); a
same-second FAIL/PASS tie resolves to FAIL. The runner call site is a
one-line hunk in scripts/test-paid-shards.ts.

* docs(cso): state that contained runtime/scanner execution is unavailable today

The scanner catalog is empty while its build inputs await review and the
runtime catalog holds no qualified profile until the private evaluator
exists, so /cso runs static assessment only. AGENTS.md, README and
docs/cso-release-qualification.md now say so; docs/cso-scanners.md
already did.

* test(helpers): expectMentions splits on sentence ends only

Clauses joined by ';' or ':' are one rule; splitting there made a reworded
rule fail for punctuation.

* test(plan-ceo): convert skill-ceo-section-ordering sentence pins

Safety and authority rules (never/do not/stop/only/without/wait/approval) are
meaning-level expectMentions checks; other sentence pins are dropped. Headings,
labels, option rows, placeholders and machine-read lines (auto-decide
annotation, Mode:/approved decisions format, STATUS/SOURCE fields,
CROSS-MODEL marker, gstack-ceo-mode-handoff) stay as tokens; ordering checks
unchanged. Behavior is covered by the periodic PTY case in
skill-e2e-plan-ceo-review-section-loading.test.ts.
Sentence pins: 210 -> 23.

* test(helpers): expectMentions treats wrapped lines as one sentence

Templates hard-wrap prose; only sentence ends and blank lines separate rules.

* test(gen-skill-docs): convert sentence pins to meaning checks or drop them

Safety/authority sentences become expectMentions keyword checks; other English
sentences are dropped (wording is judged by the per-skill workflow judges in
skill-llm-eval and the E2E cases in touchfiles-data). Commands, config keys,
markers (PLAN MODE EXCEPTION, NO UNRESOLVED DECISIONS, Gate outcome), field
names and alternation-based meaning regexes stay. Removed the now-empty
'contains opt-in AskUserQuestion text' test (only pinned a sentence; the
resolver's AskUserQuestion wiring is covered by the codex second-opinion
tests in the same describe). Lane E's existsSync hunks are untouched.
Sentence pins: 216 -> 65 (remaining lines carry machine tokens).

* ci(windows): shard the Windows lane via --ci-plan/--ci-run/--ci-verify; move native campaigns to native-qualification.yml

- windows-free-tests.yml: a Linux plan job curates and packs the Windows-safe
  files into 6 shards, one strict serial shard per windows-latest job, and a
  Linux aggregate verifies exact coverage and every result (8.1 min on one
  runner today). --windows-only now combines with the CI modes; the plan binds
  the curated file set (W6e).
- Every Windows plan and --list run prints the curated count with its top
  exclusion reasons in the job summary; a free ratchet fails when the curated
  subset drops below its floor, with the fix (W6e).
- The four Dia/cookie diagnostic dispatch campaigns move to the dispatch-only
  native-qualification.yml with one exclusive `mode` choice input, so their
  red dispatch runs no longer read as a red PR lane (W8f). Their workflow pins
  move to test/windows-native-workflows.test.ts; the boolean-exclusivity
  guard is replaced by the choice input.

* feat(paid): per-slice CI job ceilings and an in-process slice deadline with not_run/hung outcomes and checkpoints (W2c, ENG-2)

* fix(pass-rates): segment trial history strictly by EVAL_POLICY version

Reader-compatibility floor for EVAL_POLICY v2 (ENG-14/CEO-35/DX-12). Only
trials under the reader's own policy version feed labels and alarms; older
policy trials stay visible but are never scored or used as a drift baseline;
trials from a newer policy are ignored with a printed count and fix text.
Reverting the later v2 writer commits leaves a reader that cannot pool v2
records into v1 series (the pre-PR reader at 2db0b3a does: it reports a
mixed fixture as FAILING with drift alarms).

* docs: one contributor command table; retired commands, seed sources and current workflows (W8g, DX-5, DX-13, CEO-12)

- CONTRIBUTING.md gains 'Which command do I run?' (task -> command ->
  cost/time -> needs, including branch validation via gh workflow run);
  CLAUDE.md, AGENTS.md and TESTING_INTERNALS.md link to it instead of
  repeating command lists. Tier tables drop retired commands and stale
  per-run dollar estimates.
- Retired names (test:evals, test:e2e, eval:bg, eval:flake-rank, eval:watch,
  ...) are gone from CLAUDE.md, CONTRIBUTING.md, TESTING_INTERNALS.md,
  ARCHITECTURE.md and USING_GBRAIN_WITH_GSTACK.md.
- The flake-ledger claim now says what happens: CI uploads flake-ledger-*
  artifacts; eval:pass-rates reads only the local ledger.
- TESTING_INTERNALS states that only test:ubicloud --record-durations
  produces an acceptable duration seed (DX-13).
- CI section: quality gate no longer runs bun audit; dependency-review's
  real trigger; the platform-qualification lane.
- docs/test-audit-2026-09.md gets a corrections section for its stale claims.
- TODOS.md: audit follow-ups (post-merge metrics, stub removal, name reclaim,
  diff-aware planner); eval-watch shard-awareness and the image freshness
  tripwire closed as won't-do.

* feat(evals): recover receipts of cancelled PR runs (W1c, CEO-14/34, ENG-4/5/7)

A new recover-receipts job runs base-ref code with actions: read, restores
the PR's store, merges the receipts of completed cancelled evals.yml runs
for this PR (provenance-checked, at most 5, oldest first, negatives first),
reconstructs completed panel negatives with panelVerdict() (partial panels
get no credit) and keeps a recovery.json checkpoint that never advances past
an unresolved run. plan-slices keeps contents: read, takes the store as an
artifact, turns reuse off when recovery degraded or a run is unresolved, and
carries the whole store to slices-report so nothing is lost. Any gh problem
degrades to no recovered receipts with a named reason.

* feat(evals): classify this wave's free-only tools and workflows for PR selection

platform-qualification.yml, retired-command.ts, test-health-report.ts,
bump-harness-version.ts and eval-bg.ts never run inside a paid case.

* feat(evals): debounce PR push bursts before planning paid slices (W1d, CEO-15, ENG-11, DX-7)

A tiny ubuntu-24.04 job waits 90 s only on synchronize when another run for
the PR started in the last 15 minutes; cancel-in-progress supersedes, and a
run that outlives the wait reports "superseded by <sha>" and plans nothing.
Labels: evals-no-debounce skips the wait, evals-fresh disables receipt reuse.

* perf(test): move the real-process browser settlement cases out of test-free-shards.test.ts (W6b)

The 19 owned-browser settlement modes were 85 s of the file's 95 s and could
not be packed apart from the fast runner unit cases. They move to a shared
fixture with two files (lifecycle 47 s, ownership 39 s); the unit file drops
to 10 s. Same modes, same assertions.

* chore(evals): delete the finished validation_phase dispatch and its inline planner (W8f)

Retirement (CEO-24): the protected behavior was a one-off cookie-repair
validation subset (eight dependent cases) and a quality/behavior split of
the census; that validation is complete, so the guarantee is withdrawn.
Branch validation dispatches evals.yml with evals_all.

* perf(test): inject the office-hours record grace in non-cooperative deadline tests

* perf(test): shrink aside-render process slack via GSTACK_RENDER_SLACK_MS in timeout tests

* perf(test): make the memorable lock give-up configurable for lock-contention tests

* perf(test): check install-timeout knob normalization through its own setup lines

* perf(test): inject a 2 s storage-reset deadline in the late isolated dispatch test

* perf(test): shorten the slow gbrain fake to 1 s, still past the 300 ms probe timeout

* ci(free-tests): run only cso-docker-integration in the free CSO Docker job (F16)

cso-node-lifecycle-integration and cso-stack-cold-integration need
GSTACK_CSO_TEST_STACK images and always skipped there; they stay in the CSO
runtime workflows through test:cso:docker.

* test(gen-skill-docs,plan-ceo): split mixed sentence pins into tokens and rules

Sentences that wrap a backticked or bold token keep the token (expectTokens)
and, when they state a safety rule, a keyword check; short safety sentences
(25-39 chars) get the same treatment.

* test(plan-ceo): convert ceo-workflow-clarity.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: periodic PTY case skill-e2e-plan-ceo-review-section-loading and the plan-ceo workflow judges.
Sentence pins: 13 -> 1.

* test(plan-eng): convert eng-review-routing.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: plan-eng E2E cases (skill-e2e-plan-eng-*) and the plan-eng workflow judge.
Sentence pins: 28 -> 6.

* test(plan-eng): convert eng-scope-entry-ap.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: plan-eng E2E cases (skill-e2e-plan-eng-*) and the plan-eng workflow judge.
Sentence pins: 16 -> 5.

* test(plan): convert plan-scope-selection.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: plan-review E2E cases and workflow judges in skill-llm-eval.
Sentence pins: 19 -> 2.

* test(qa): convert qa-lazy-sections.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge.
Sentence pins: 71 -> 11.

* test(qa): convert qa-browser-preservation.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge.
Sentence pins: 25 -> 11.

* test(qa): convert qa-probe-gates.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge.
Sentence pins: 20 -> 5.

* test(qa): convert qa-checkpoint-evidence.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge.
Sentence pins: 7 -> 5.

* test(qa): convert qa-evidence.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge.
Sentence pins: 4 -> 3.

* test(design): convert design-consultation-contract.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: design-consultation E2E/judge cases in touchfiles-data.
Sentence pins: 44 -> 11.

* test(outside-voice): convert run-in-background-guidance.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: outside-voice E2E cases and plan-review workflow judges.
Sentence pins: 35 -> 5.

* test(spec): convert spec-template-invariants.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: spec E2E/judge cases in touchfiles-data.
Sentence pins: 36 -> 21.

* perf(evals): 420 s census slice budget; case-shard skill-e2e-qa-workflow; seed its case walls (W5c)

* feat(evals): evals.yml dispatch contract for eval:bg (CEO-17/29, DX-1, ENG-8/12)

workflow_dispatch gains base_ref, base_sha, expected_sha, nonce and
pr_receipts. The nonce appears in run-name so the dispatcher resolves its
own run; evals_all=false runs the PR profile against base_sha/base_ref; a
checkout that differs from expected_sha fails before any paid work with a
named fix; pr_receipts reads that PR's receipts read-only through the
recover-receipts artifact path (never the cache) and never saves.

* ci(windows): pack the Windows lane with Windows-measured durations (W6e)

The first sharded run (37221451711) finished in 6.0 min: the Linux seed has
browse/test/cookie-import-native-job.test.ts at ~0 s (its cases skip on
Linux) but it takes 122 s on Windows, so one shard ran 242 s against an 84 s
prediction. The plan now reads scripts/free-test-durations-windows.json; a
record_durations dispatch times every Windows-safe file alone and uploads it.

* test(free): settlement case files import the runner rule they exercise

Keeps the split files out of the test-of-test ratchet: the retry-eligibility
rule is a direct runner import passed to the shared fixture.

* fix(marathon): no hollow plan-decision-classification shard; office-hours marathon scenario scoped to the pilot (W2f)

* test(free): satisfy the typecheck:test ratchet in the seed ratchet test

* refactor(ci-history): one shared GitHub history reader with the CI flake-ledger reader

Extract pass-rates' gh fetch layer into scripts/lib/ci-history.ts (CEO-09) and
add the W7a reader: paged run listings that fail whole on any page error,
per-run artifact downloads (cached, size-capped, data only), flake-ledger
parsing, the pre-registered flaky-file rule (> 5% of main free-tests runs over
a window of at least 20 runs, widened to the newest 20 runs when the window is
thinner), a job-summary table, and named unavailable reasons with next steps.

* feat(test-health): test:health report of the audit success metrics

scripts/test-health-report.ts (CEO-09/06/22/32, DX-8/9/12/13) prints every
success metric with its counting rule from CI history: PR full-fallback rate,
reused PR trial records, PR-lane spend per merged PR and per validated
revision, cancelled slice-minutes per week, push-to-verdict p50/p95, census
expected failed trials per run with raw failure rate per kind/class and
P(lane red) under the recorded panels vs single-trial aggregation, INFRA
re-dispatches, quarantine evidence (insufficient evidence distinct from
healthy), free flaky-pass rate, free/Windows wall, free serial time and the
net-complexity line. An unreadable input prints 'unavailable: <reason>; next:
<step>', never 0. --enforce fails on a flaky file (W7a) or > 5 unseeded free
files (CEO-10); unavailable checks never fail and keep the issue open.

* test(review): convert review-workflow-clarity.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval.
Sentence pins: 85 -> 17.

* test(review): convert review-entry-and-design-clarity-au.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval.
Sentence pins: 15 -> 4.

* test(review): convert review-enum-lifecycle.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval.
Sentence pins: 8 -> 0.

* test(ship): convert ship-control-flow.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 74 -> 9.

* test(ship): convert ship-skip-requeue.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 28 -> 3.

* test(ship): convert ship-plan-completion-invariants.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 28 -> 12.

* test(ship): convert ship-reentry-gates.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 16 -> 4.

* test(ship): convert ship-document-release-dispatch.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 11 -> 4.

* test(ship): convert ship-apple-gate.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 7 -> 0.

* test(ship): convert ship-skip-actor.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 6 -> 1.

* test(ship): convert ship-template-redaction.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 5 -> 4.

* test(ship): convert ship-publication-gates.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 5 -> 2.

* test(ship): convert ship-review-loop.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 3 -> 1.

* test(ship): convert ship-version-sync.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 5 -> 2.

* test(ship): convert ship-pr-liveness-policy.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge.
Sentence pins: 4 -> 0.

* test(plan-review): convert plan-review-cases.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: plan-review E2E cases and workflow judges in skill-llm-eval.
Sentence pins: 58 -> 15.

* test(autoplan): convert autoplan-phase-order.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge.
Sentence pins: 42 -> 11.

* test(autoplan): convert autoplan-phase-handoff.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge.
Sentence pins: 20 -> 4.

* test(autoplan): convert autoplan-snapshot.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge.
Sentence pins: 21 -> 5.

* test(outside-voice): convert outside-voice-preflight.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: outside-voice E2E cases and plan-review workflow judges.
Sentence pins: 26 -> 9.

* test(ask-format): convert resolver-ask-user-format.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: AUQ E2E cases (skill-e2e-auq-*); prompt-injection rule kept as a meaning check.
Sentence pins: 32 -> 15.

* test(skills): convert binding-template-drift.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: workflow judges in skill-llm-eval.
Sentence pins: 19 -> 13.

* test(skills): convert carve-plan-fixture.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: workflow judges in skill-llm-eval.
Sentence pins: 20 -> 6.

* test(land-and-deploy): convert land-and-deploy-postfail.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge.
Sentence pins: 21 -> 14.

* test(land-and-deploy): convert land-and-deploy-ci-gate.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge.
Sentence pins: 5 -> 4.

* test(land-and-deploy): convert land-and-deploy-flow.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge.
Sentence pins: 5 -> 3.

* test(office-hours): convert office-hours-completion.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: office-hours E2E cases (skill-e2e-office-hours*).
Sentence pins: 16 -> 1.

* test(plan-ceo): restore the stale closing-route guard

The negative check for 'Closing hooks' / 'return to section' guards a retired
route token, not a sentence; the first conversion pass dropped it.

* ci(test-health): weekly test-health workflow with an honest tracking issue

New test-health.yml (CEO-18/DX-9): Monday 10:00 UTC after the periodic census
plus dispatch, free ubuntu-24.04 runner, permissions contents:read +
actions:read + issues:write only. Runs bun run test:health --since-days 7
--enforce; an enforced failure upserts one tracking issue, a run with
unavailable inputs keeps it open and lists them, and it closes only when every
enforced check has evidence and passes. Only main runs touch the issue.

* feat(evals): an edited paid test file runs its own gate cases in the PR lane (DX-11)

A gate case outside the audited fast profile is selected when its single
owning paid test file changed and the runner can select it by name (its
Bun test is named by the case id). The manifest records directCases and
validation accepts only recorded, name-addressable ones; the runner and
planner read the file->cases map from test-pr-profile. Cases the runner
cannot address stay deferred with the fix. An inventory test covers every
owned gate case. Touches one hunk each in test-paid-shards.ts and
lib/paid-plan.ts, and pr-shared-input-selection.test.ts's ship-skip pin.

* ci(osv): tracking issue for scheduled reds and a 14-day suppression expiry test

F11/W8f: a failed main OSV scan now upserts one tracking issue and a clean
scan closes it (scheduled reds notified no one during the August backlog).
A free test fails 14 days before any .osv-scanner.toml ignoreUntil (all four
expire 2026-11-30), with the fix: upgrade and delete, or re-justify and move
the date.

* test(ship,review): keep guards against retired fail-open instructions

'skip silently' on reviewer failure and 'do not block /ship on subagent
failure' were harmful instructions, not prose style; keep them absent with the
shortest distinctive phrase.

* feat(eval-select): default to the pr profile

DX-10: eval:select previews the PR lane's selection by default, matching
test:pr and eval:bg:pr; --profile full (or EVALS_PROFILE=full) keeps the plain
touchfile selection.

* test(evals): type-clean the recovery gh fake (typecheck:test ratchet)

* chore(windows): record the Windows duration seed (run 37221963236)

573 Windows-safe files timed alone on windows-latest; cookie-import-native-job
is 102 s there. Six shards now predict 154 s each.

* feat(evals): eval:bg dispatches the pushed ref to CI or runs locally under a 1.5x-planned cap (W5d, CEO-17/29, DX-1/2/3/7, ENG-12/19)

* feat(runners): --help on both shard runners via the shared flag loop; paid runner header describes today's planner/executor/report (W8e)

* test(windows): drop the stale Playwright exclusion and force-include probe-proven files (W6e)

A temporary windows-latest probe (run 37221995077) ran each of the 97 files
excluded for 'launches Playwright browser' or 'raw /tmp/ path' alone: all 17
Playwright files pass (Chromium is installed and Bun drives it there), and 41
of the /tmp files pass with executed cases. The Playwright pattern is removed
and the 41 files are force-included with the probe as their reason; the 36
real failures stay excluded. The curated subset grows from 573 to 631 files
and the ratchet floor follows. Windows curation moves to
scripts/lib/windows-curation.ts (the runner drops to 2000 lines).

* test(free): one install fixture cloned from a read-only seed (W6c, ENG-1)

Merge test/helpers/setup-codex-scope-fixture.ts into install-fixture.ts.
The checkout is built once per test file into a read-only seed and cloned
per fixture with copy-on-write where the filesystem has it (full copy
otherwise), restoring each file's own mode; clones never hard-link the
seed because put() and tests write in place. Callers register
afterAll(cleanupSeed); Bun's runner fires no exit event, so seeds of
killed runs are swept by pid. test/install-fixture.test.ts is the ENG-1
tripwire: put/setVersion/chmod in one clone leave the seed and a sibling
unchanged, and clones share no inodes with the read-only seed.

* test(free): split claude-overlay-setup by describe (W6b)

The 115 s serial file becomes default/lifecycle/installs files (5/4/3
tests, unchanged bodies) sharing test/helpers/claude-overlay-fixture.ts,
so the shard packer can balance them.

* test(cso): convert cso-contracts.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval).
Sentence pins: 16 -> 8.

* test(cso): convert cso-preserved.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval).
Sentence pins: 10 -> 3.

* test(cso): convert cso-eval.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval).
Sentence pins: 7 -> 4.

* test(browser): convert aside-driver.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: Aside/browse E2E cases; the installer ban stays a meaning check.
Sentence pins: 27 -> 9.

* test(browser): convert third-party-actions.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: Aside/browse E2E cases; the installer ban stays a meaning check.
Sentence pins: 17 -> 2.

* test(skills): convert skill-validation.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: workflow judges in skill-llm-eval.
Sentence pins: 19 -> 6.

* test(skills): convert workflow-excerpt.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: workflow judges in skill-llm-eval.
Sentence pins: 37 -> 10.

* test(skills): convert question-tuning-registry-path.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: workflow judges in skill-llm-eval.
Sentence pins: 4 -> 0.

* test(skills): convert host-conformance.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: workflow judges in skill-llm-eval.
Sentence pins: 4 -> 4.

* test(memory): convert memory-ingest-landing.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: memory-ingest free tests that run the binary.
Sentence pins: 11 -> 9.

* test(windows): exclude the skip-only claude-overlay setup files from the Windows subset

Their real-setup cases are describe.skipIf(win32); after the split their bin/
spawns live in the shared helper, so the source scan stopped seeing them.

* test(skills): convert agents-digest.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 1 -> 0.

* test(skills): convert brain-preflight.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 2 -> 1.

* test(skills): convert codex-skill-isolation.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 1 -> 0.

* test(skills): convert office-hours-review.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 2 -> 1.

* test(skills): convert onboarding-moved-literals.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 2 -> 2.

* test(skills): convert open-gstack-browser-liveness.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 3 -> 2.

* test(skills): convert plan-floor-review.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 3 -> 2.

* test(skills): convert plan-review-report-recording.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 4 -> 2.

* test(skills): convert regression-1539-review-self-verify.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 4 -> 0.

* test(skills): convert skill-fixture.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 6 -> 3.

* test(skills): convert template-context-parity.test.ts sentence pins

Safety and authority sentences become expectMentions keyword checks (reword
freely, keep the rule); other English sentence pins are dropped. Headings,
labels, placeholders, commands, field names, enum/status tokens and ordering
checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone.
Sentence pins: 1 -> 0.

* test: restore pins on renderer output and paid-fixture inputs

cso-contracts (the report renderer's required empty phrase), memory-ingest
(binary output), carve-plan-fixture, ship-skip-actor and cso-eval (inputs the
paid harness builds) assert product or fixture output, not skill prose, so the
value-bar narrowing does not apply to them.

* feat(eval-policy): v2 series identity, HARNESS_VERSION guard and identity backtest

W4a (CEO-27/ENG-3/DX-6), not yet wired to the writer: caseSeriesIdentitiesV2
keys a case by the bytes it owns (paid test file, fixtures, its skills' prompt
files including the generated SKILL.md) plus HARNESS_VERSION, and records the
full consumed-input fingerprint as provenance. scripts/harness-version.json
pins the blob of every shared-harness file (GLOBAL_TOUCHFILES plus the session
runners, PTY harness, e2e gate and LLM judge), one line per file;
test/harness-version.test.ts fails on drift with both fixes, and
scripts/bump-harness-version.ts records the decision: --bump (new series for
every case) or --non-behavioral (compatibility entry, no reset).
eval-trial-series --backtest replays identities over weekly main revisions
and recorded trials (ENG-3/DX-12).

* feat(pass-rates): weekly history is main's scheduled runs plus main dispatches

W4b: pass-rates reads evals-periodic history from main only, keeping scheduled
runs and main dispatches and never a branch dispatch, so back-to-back branch
validation runs neither feed pass rates nor count as weeks for quarantine
expiry or drift. --branch <name> still reads one branch's runs for inspection.

* docs(test-audit): EVAL_POLICY v2 decision memo with the identity backtest

D1 memo (CEO-33, ENG-3, DX-12, CEO-27): P(green) arithmetic, false-blocking
targets, the v2 identity rule and harness guard, weekly history, W4c evidence
(the named 'stochastic' cases pass 26/27 to 34/34, so they stay rule), the
backtest over 29 weekly main revisions and 14,898 recorded trials, what it
means for quarantine reachability, alternatives (gating vs trended, adaptive
panels not adopted, buy vs build), and the rollback proof on the pre-PR reader.

* feat(eval-policy): EVAL_POLICY v2 writer: case-owned series identity

The v2 writer, last commit of the W4 block (DX-12/ENG-14/ENG-21): the report
job's stamp now writes caseSeriesIdentitiesV2 (case-owned bytes plus
HARNESS_VERSION) as series_identity, the full consumed-input fingerprint as
series_fingerprint and harness_version as provenance, and EVAL_POLICY.version
is 2 (approved 2026-10-04, D1). The v1 touchfile identity is removed.
Rollback = revert this commit (and the W4 commits above the reader-compat
floor); readers then ignore the v2 records with a printed count.

* test: restore pins on paid-harness prompt inputs

review-enum-lifecycle and plan-review-report-recording assert the prompt a
paid case's harness builds (opts.prompt / call.prompt), a fixture contract,
not skill prose.

* test(security): check injected safety lines on meaning, not capitals

content-security, the canary instruction and the arm judge prompt still must
carry their never-follow / never-output / exact-"none" rules; the checks are
case-insensitive keyword checks so emphasis can change. The UNTRUSTED envelope
markers and "none" value stay exact tokens. (Hidden element stripping fails
locally only because Playwright Chromium is not installed; same at base.)

* test(plan-ceo): keep the outside-voice scope-summary rule as a meaning check

The first codemod pass left an empty if body (TS1313).

* chore(windows): re-record the Windows duration seed for the 631-file subset (run 37222668939)

* ci(free-tests): summarize every shard's flake ledger in the aggregate job (W7a)

The aggregate downloads flake-ledger-* (one subdirectory per shard) and
appends lane C's `scripts/test-health-report.ts flake-summary` output to the
job summary before strict verification. The runner's own flaky table is
dropped so the ledger reader stays the single source; --ci-verify keeps the
shard-overrun warning.

* feat(evals): free test files anywhere and the stored-zip helper never restore the full PR gate (W1a)

The CEO-23 replay found design/test free tests forcing fallback (36485436963).

* feat(evals): replay recorded PR pushes through today's selector (CEO-23, ENG-18)

scripts/replay-pr-selection.ts reports per push the recorded vs replayed
mode and case count, dropped gate cases, retired failed cases and every
miss (a failed case that depends on the diff but is no longer selected).

* refactor(evals): one case-sharded carve-section-loading file with a case id and touchfiles per skill (W5a)

* test(free): hold qa-deadline's full-pipe case on a FIFO, not a paused Bun stream

Bun gives a child 'pipe' stdio a unix socketpair, so whether the 2 MB fill
actually blocks depends on the parent's reader and the host's socket buffer
sizes. CI (run 36761671811, both attempts) saw exit 0 after ~130 ms; with
net.core.{w,r}mem_default=4 MiB locally the old case fails 3/3 the same way.
A FIFO the test never reads is bounded by the kernel pipe buffer, so the
receipt write genuinely blocks and the 5 s settlement deadline must fire.
The 7 s bound now starts once the preload has queued the write.

* test(free): event-driven waits in pty-output-wake

close-releases-observers: the 500 ms wall race becomes an ordering check
(the waiter is released while only SIGINT has been sent, before close's
escalation finishes), and the Bun.spawn spy fakes only the PTY launch so a
shard neighbor's in-process spawn cannot add a stray SIGTERM (CI runs
36015568981, 36036582721, 36074636848).

continuous-redraws: the fixture spinner stops once the observer has made
five coalesced reads, so the run reaches its work deadline with no redraw
in flight. Streaming right up to the deadline raced the helper's screen
drain and failed as 'PTY screen parse failed; viewport is incomplete'
(5 CI runs 09-30..10-03; 4/40 locally under 8 busy loops, 0/40 after).

* test(free): session-runner long-capture handshake waits for the collected Read event

The fake CLI's init line arms the runner's work-phase timer, so the worker's
handshake (work-ready + a >200 s timer) could pass before the runner's read
loop had collected the Read event printed just before work-ready. Advancing
the clock then killed the run with the Read lost: readSection false (CI run
37194511124). The worker now also waits for the runner's own progress line
for that Read. Forcing the observed ordering (work-ready 200 ms before the
Read line) fails the old test deterministically and passes the new one.

* test(free): budget-override restores the CI variables it rewrites

Its provenance cases set and then deleted CI, GITHUB_ACTIONS, GITHUB_REF_NAME,
GITHUB_SHA and the GitLab CI_* names without restoring them, so in a CI shard
every later file ran with CI unset. qa-only-cleanup's real browse daemons
then launched Chromium without --no-sandbox, the sandbox failed on the
runner, and both real-Chromium cases timed out with 'QA cleanup fixture did
not become ready' (shard 12 attempt 1 of six runs on 10-04, retry alone
green). Retained startup log: 'Chromium sandboxing failed!', child env with
no CI. Reproduced 3/3 with CI=true, userns denied and budget-override ahead
of qa-only-cleanup in one process; 0/3 after.

* test(free): pr-shared-input-selection names its base and package history

Every selection now passes EVALS_BASE and packageVersionOnly explicitly, so
no case reads the checkout's git state. Before, 12 calls ran
detectBaseBranch and the cumulative case diffed package.json against
merge-base(origin/main, HEAD) — the ambient input that turned main red in
runs 36572856858 and 36594314027. A PATH git spy counts 14 git calls in the
old file and 0 now.

* feat(evals): derive PR-lane dependents for unmapped files instead of restoring the full gate

scripts/pr-dependencies.ts derives, from the checked-out tree, which paid
cases consume each repository file: test import closures, repo paths and
path.join segments in code/shell/skill text (whole-line comments and
selection data excluded), bin script names, compiled binaries by outfile
path (scripts/build.sh), package scripts, template placeholders through
RESOLVERS, and host configs by host id. The global touchfiles' import
closure is global.

PR selection: an unmapped file with derived consumers puts the run in the
new 'dependents' mode (every dependent gate case and judge, no profile cut);
a tracked file under a derivable directory that nothing reaches is exempt;
periodic/marathon workflows, .gitignore and tsconfig.test.json never run
the PR gate. PR_FULL_GATE_FILES is the fail-closed list (deps, CI image,
setup, build, evals.yml and its actions, test preload, tsconfig.json,
.gitattributes, the skill generator and preamble resolvers, the eval
policy). Untracked or underivable files restore the full gate with fix
text. The summary and PR comment list derived dependents.

* feat(evals): replay misses count derived dependents; report wave vs ordinary pushes (CEO-23)

* test: drop FREE_FIXTURES entries for fixtures deleted in W8b

* test: re-shape the census attribution fixture to qa-workflow and carve case shards

* test: census fixture case order

* ci: 420 s PR-lane slices with per-slice ceilings; periodic tracking issue only on main

evals.yml joins the census lanes at the 420 s slice budget (W5c/CEO-15) and
indexes each eval slice's job timeout from .plan.sliceCiTimeoutMinutes with
the slice-deadline clock as its first step (W2c/ENG-2). evals-periodic.yml's
tracking-issue upsert and close run only on refs/heads/main, so a branch
dispatch whose gate census is skipped (CEO-04) never opens or closes #2755.

* docs: integrate lane docs for the audit wave (eval-bg backends, PR fallback, policy v2, ceilings, evidence)

- CLAUDE.md: 'Running evals as an agent' describes scripts/eval-bg.ts's
  dispatch and local backends, forcing flags, the sentinel and status;
  eval:select's PR-profile default; prompt-structure helper and value-bar
  terms. instruction-facts now requires scripts/eval-bg.ts in that section.
- CONTRIBUTING.md: decision table gains test:health, eval:select default,
  eval:bg backends and the include_gate_census / evals_all dispatch lines;
  420 s runners and per-runner ceilings; flake ledger, OSV tracking issue
  and expiry test, weekly test health; prompt-structure guidance.
- AGENTS.md: build commands name the eval:bg backend choice and test:health.
- docs/TESTING_INTERNALS.md: anchors the fix texts point to
  (#pr-paid-lane-fallback, #pass-rate-policy-versions, #harness-version,
  #flake-ledger, #duration-seed), receipts/recovery/debounce, per-slice
  ceilings, the slice deadline with not_run/hung and checkpoints, the eval-bg
  cap formula in place of the detach floors, EVAL_POLICY v2 history, carve
  case shards via --case.
- docs/test-audit-2026-10.md: CEO-23 replay (165/168 -> 164/168
  full-fallback, 0 misses), ENG-17 slice table (re-measured on this tree),
  D2 prose-pin metric.
- TODOS.md: progress note on 'Let pass-rate history decide the rest'.
- test/testing-internals-anchors.test.ts: every TESTING_INTERNALS.md#anchor
  named in code must match a heading.

* test(free): hold qa-deadline's blocked-output-pipe receipts on a FIFO

The five 'guard %s receipts survive a blocked output pipe' cases filled a
paused Bun child pipe with 2 MiB. That pipe is a unix socketpair, so the
fill only blocks while the host's socket buffers are smaller than the write:
with net.core.{w,r}mem_default=4 MiB the old cases fail 13 of 15 case runs
(the guard finishes before the test resumes the stream). The held stream is
now a FIFO the test drains only after resume(), bounded by the kernel pipe
buffer; Windows keeps the paused pipe. Bounds and assertions are unchanged.

* test(free): qa-deadline waits for a written pid, not just the pid file

DRIVER and LEAF write their pid with writeFileSync, so under load ready()
can see the file before its digits and fail 'expected > 0, received 0'
(seen once in 150 loaded runs). After any early failure the finally
SIGKILLs the CLI, which orphans the detached command group; a leaf whose pid
file is not yet readable then keeps the output stream open and the case
hangs to its 15 s timeout (2/80 old-file runs and 3/310 FIFO-only runs
under 8 busy loops; the instrumented hang showed the CLI already SIGKILLed
by the finally). ready() now polls within the same 3 s bound until the file
holds a positive pid.

* fix(evals): a deleted file selects the dependents of the live files that still reference it

PR selection treated every path absent from the head tree as an unknown
dependency, so any PR that deleted a file forced the full gate. A deleted
path is now placed like a live one, against the head tree: a matching
touchfile selects its cases; live code, skill text or a full-gate input that
still references it (path, extensionless import target, relative import,
bin name, ancestor directory, compiled-binary outfile; whole-line comments
and data files excluded) contributes its dependents, a full-gate referencer
restores the full gate with a 'deleted but still referenced' fix, and an
unreferenced deleted path has no dependents. Deleted PR_FULL_GATE_FILES
entries still force the full gate. Tests that used absent paths as stand-ins
for new unmapped files now use tracked root files (or an explicit tracked
set); the CEO-23 replay counts misses through those references too.

* docs: integrate free-suite lane docs (seed anchor, Windows lane, native qualification, install fixture, flake rules)

- docs/TESTING_INTERNALS.md: '### Free suite duration seed' (the anchor the
  seed messages link) with DX-13, the 99th-percentile packing, --ci-plan
  warning, --ci-verify overruns and the CEO-10 seed ratchet; the Windows
  6-job lane, its Windows-measured seed and refresh dispatch, curation and
  WINDOWS_CURATED_FLOOR; native-qualification.yml modes; the merged
  install fixture; test-only timing knobs; the flake-ledger job summary,
  Windows ledger names and three free-suite flake rules; zsh in the
  Ubicloud mirror.
- scripts/test-health-report.ts links #free-suite-duration-seed, so the
  two seed messages share one heading (the anchor test caught the split).
- CONTRIBUTING.md: Windows seed refresh and native qualification rows;
  seed-source pointer. AGENTS.md: Windows six-job lane.
- docs/cso-release-qualification.md: the free CSO Docker job runs only
  cso-docker-integration.test.ts.
- docs/test-audit-2026-10.md: free-suite evidence lines.
- TODOS.md: CEO-11b progress note on the pre-push hook hang.

* test: final duration seeds on the integrated tree; retire the emptied cookie-validation-phases file

Linux seed re-recorded on Ubicloud standard-16 and the Windows seed from
dispatch run 37227140871, both at c903310. The seed allowlist drops the three
entries that are no longer over the ceiling. cookie-validation-phases lost its
validation-phase pins with W8f and its Windows/Dia pins moved to
windows-native-workflows with native-qualification.yml, so the file is
deleted and the curated Windows floor rises to 638. The Windows recorder writes
the seed at its default path so GSTACK_FREE_TEST_DURATIONS never leaks into
timed tests.

* test: re-express stale PR-lane pins against the new evals.yml contract

- ci-eval-cache: the profile/fresh expressions now include evals_all=false
  dispatches and pr_receipts/evals-fresh; the PR store is restored only by
  recover-receipts (still scoped to repository_id + PR, PR events only),
  plan-slices takes it as an artifact and never touches the cache, saves
  stay PR-only, EVALS_CACHE_PR binds pr_receipts dispatches, and the report
  verdict artifact carries report receipts.
- ci-paid-coordination: plan-slices needs only the two ubuntu coordination
  jobs (never the image); the fork/Dependabot clause is unchanged on
  plan-slices and also gates debounce and recover-receipts; only recovery
  holds actions: read.
- qa-supervision-selection: auq-consistency and auq-verbose-vs-carved-ab
  (W1b) reach claude-pty-runner and pty-screen through auq-sdk-capture,
  exactly as auq-format-gate does.
- paid-retry-supervision: the full-validation plan uses a tracked full-gate
  input (setup); an absent path is now a deletion placed by its references.

* ci(windows): record the Windows seed at the default path

* chore: HARNESS_VERSION non-behavioral decision for the audit-wave integration

* v1.91.20.0 release metadata: test, eval and CI audit wave

* test(compare-board): wait for the post-submit state the feedback POST settles

The two post-submit assertions read the board once, but the board disables
the submit button and shows the message only after its feedback POST
resolves. On a loaded Ubicloud shard the read beat the round trip (FLAKY-PASS
in the final free-suite run). Delaying the POST by 400 ms reproduces both
failures at HEAD; polling for the settled state passes.

* test(plan-eng plan-mode): run the seeded STOP-gate smoke in its own fixture project

The seeded plan proposes a custom test runner, so the review's Step 0 reads
the working tree's test setup. In the gstack checkout that is gstack's own
sharded runner, and the case timed out at 300 s in census 37228573062 after
this wave reshaped the runner. It now runs in a four-file Bun project with
git history, with the product-notice suppression auto-decide-preserved's
standalone fixture already uses. Local pass on Bun 1.4.0 + Claude Code
2.1.284: asked in 121 s, seed tokens observed.

* test(plan-mode evidence): pin the seeded eng smoke's isolated fixture cwd and env

* ci(image): Git >= 2.44 so paid shared-libs cases run the supported safe-git path

bin/gstack-safe-git prefixes every /deslop-shared-libs Git read with
--no-lazy-fetch, which noble's git 2.43 rejects (exit 129). The eval image
therefore ran every paid shared-libs case on the unsupported-Git fallback:
local history unavailable, branch HEAD unknown. shared-libs-read-only failed
on exactly that in PR run 37230639146 ("Current branch HEAD: unknown. The
safe-git probe failed (unknown option: --no-lazy-fetch, git 2.43.0)") and in
2 of 74 earlier PR runs. The image installs current Git from the Ubuntu Git
Maintainers PPA and fails its build if --no-lazy-fetch is missing; the
deliberate unsupported-Git case keeps its fake Git.

* eval: pool matching branch census trials into main's pass-rate series (D1 option b)

Approved 2026-10-05. eval:pass-rates also reads completed branch census runs
in the weekly window. A branch trial counts only toward a series a main run
also has (same case-owned bytes, HARNESS_VERSION, model and CLI), so a branch
extends main's history but never starts or becomes a case's current series;
weeks for quarantine expiry still count main runs only. The D1 backtest put
this at 67 of 210 cases with enough trials for the entry rule, against 0 of
195 on main alone.

* C2: /qa defines its outcome filename fields; QA report templates name the measured timing fields

Low judge samples in failing panels named both: qa workflow (37186854666)
'an undefined {user} in the outcome filename', and qa-only workflow
(37198445662) 'template asks for Duration / durationMs totals while
reporting.md mandates Probe budget and Guarded command time'. The qa
union stays under its 1.11 size ratio (93292 of 93294 bytes).

(cherry picked from commit e14c0c1de1)

* C2: Browser Quick gets a feasible budget; QA loop names its directory and stop path

Judge rationales on the qa-only panels (2 of the last 3) cited the 30 s
Browser Quick budget: homepage + 5 targets, each a checkpoint write and a
guarded command, left ~5 s per probe. Quick is now 180 s (method text: 3
minutes); the qa-only-no-fix paid check expects the same native deadline.
PROBE_DIR points at its source line, a stopped loop goes to the final
report steps (annotations, materialize, Markdown), and /review and /ship
say that after smoke expiry only plan checks rerun in revalidation.

(cherry picked from commit 78d4a2a343)

* qa: fit the C2 judge fixes under the qa and review size caps

Brings GSTA-21's qa.ts trims (3bd625d, ce73af0) for the cherry-picked C2
wording: PROBE_DIR keeps 'owned', the stop line points at §4, and the
post-expiry rule is one sentence, so /review fits its cap. qa measures
93,303 / 84,049 = 1.11010 against the same 1.1102 cap GSTA-21 set. The
qa-only workflow judge passed 10 of 10 local trials on this wording (7 of 10
before the C2 fixes).

* test: qa-only fixture keeps its caller-set 30 s deadline

The qa-only-no-fix fixture tells the model to start the guard with 30 s
(a caller limit), so its deadline assertion's default stays 30 000 ms; the
product's Browser Quick default (180 s) does not apply to it. Reverts the
helper and its test to their base.

(cherry picked from commit e36e622add)

* test: qa-quick sets a caller probe limit and drains past its session budget

C2 raised Browser Quick's native budget to 180 s. qa-quick is the only gate case
on the native budget, and its wall went 194 s (main periodic, pre-C2) -> 254 s and
274 s (this PR's runs) -> 300 s session timeout (run 37242691913): the agent now
probes every navigation target with a checkpoint, then needs ~80 s to report.
Like qa-only-no-fix, the eval now sets a caller probe limit (30 s), scopes the
fixture, and gives bun's outer timeout the session drain margin instead of racing
the 300 s session. The product's 180 s budget and the session budget are unchanged.
Diagnostic rerun passed in 195 s.

(cherry picked from commit f8dc64031e)

* chore: restore the audit wave's package scripts after the main merge (version 1.91.23)

* qa: keep main's one-sentence post-expiry rule for /review and /ship

The C2 variants ("step 4c reruns plan checks only" and "even in 4c") read
to the review workflow judge as conflicting with Step 5.8's repeat-pass
reruns: 6 of 10 local trials passed with the longer sentence and PR run
37248650879 failed on the shorter one. Main's "Post-expiry smoke rechecks are
not-run." passed 10 of 10. qa (9/10), qa-only (10/10) and
ship-managed-hook-refresh (10/10) were measured on the same tree.

* v1.91.24.0: move off the slot #3036 claims
This commit is contained in:
Garry Tan authored and GitHub committed 2026-10-05 01:29:54 +00:00
1 parent f97aefe654
commit 55fbe77195
288 files changed
+13383 -10108

No files matched your search

+1 -1
View File
@@ -93,7 +93,7 @@ gstack/
├── docs/designs/ # Design documents (incl. IMPECCABLE_INTEROP.md = the design detector / catalog / open DESIGN.md record, and fork-port-residual-2026-09/ evaluation evidence)
├── setup-deploy/ # /setup-deploy skill (one-time deploy config)
├── .github/ # CI workflows + shared composite actions (.github/actions/) + Docker image (claude CLI pinned)
│ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, and 8 more (windows, periodic evals, release gates, ci-image)
│ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, plus free-tests, Windows, periodic and marathon evals, CSO release, platform qualification and smaller gates
│ └── docker/ # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium)
├── contrib/ # Contributor-only tools (never installed for users)
│ └── add-host/ # /gstack-contrib-add-host skill
+284 -70
View File
@@ -184,10 +184,9 @@ This does not change the separate CI machine count. Full-suite shards are packed
DURATIONS (LPT, `packShardsByDuration`) when the committed seed
`scripts/free-test-durations.json` exists — refresh it with
`bun run test:ubicloud --record-durations`, which times each file in its own
child on a VM with the CI lane's environment and copies the seed back (CI never
records; a seed recorded where browser or display tests skip underestimates
them). Missing seed → silent hash-shard fallback; corrupt seed → one warning +
fallback; unknown files get 75th-percentile pessimism, and both full-suite and
child on a VM with the CI lane's environment and copies the seed back; see
[Free suite duration seed](#free-suite-duration-seed). Missing seed → silent hash-shard fallback; corrupt seed → one warning +
fallback; unknown files get 99th-percentile pessimism, and both full-suite and
`--ci-plan` runs name them on stderr so a new slow file cannot silently become
the long pole. Packed
shards get duration-aware walls (`max(base, predicted × 3, files × 5s)`). The
@@ -257,15 +256,17 @@ never retry; each case's kind fixes its trials before the run (see "Eval verdict
policy" below). Files in `CASE_SHARDED_FILES` run one registered case per
process (`<file>#<case id>`, an exact `--test-name-pattern`, exactly one executed
case), so a long file of short cases spreads across runners and each case gets
its own SDK semaphore.
its own SDK semaphore. `test/carve-section-loading.test.ts` is one of them: it registers
one periodic case per carved skill (`carve-section-loading-<skill>`, each with
its own touchfiles and history). Run one case with
`bun run scripts/test-paid-shards.ts --tier periodic --case carve-section-loading-<skill>`.
Trial telemetry rides the store: every recorded test carries its 1-based
`attempt` plus, on an isolated trial shard, its `case_id`, `kind`, `trial`,
`panel` and `policy_version`, and each lane's report uploads one
`trial-outcomes` JSONL line per trial. `bun run eval:pass-rates`
(`eval:flake-rank` is an alias) turns that history into per-case pass rates
(see "Pass-rate history" below; the free lane's flake ledger is folded in from
`flakeLedgerPath()` — override with `GSTACK_FLAKE_LEDGER`, the same env var the
CI free lane sets before uploading the ledger as the `flake-ledger` artifact). Census integrity is
turns that history into per-case pass rates
(see "Pass-rate history" below; free-suite flakes are a separate record, see
[Flake ledger](#flake-ledger)). Census integrity is
enforced from the free suite: every `E2E_TOUCHFILES` / `LLM_JUDGE_TOUCHFILES`
key must name a living paid test (`test/touchfiles.test.ts`'s reverse
invariant), and `git show <sha>:path` fixtures are banned — vendor the bytes
@@ -291,7 +292,8 @@ are not collector results and do not establish that an unfinished test passed.
retains the broad census; no case IDs or tier assignments are removed. The plan
records both E2E and judge selections and lists deferred coverage. Executors
receive those selections separately and the report checks actual executed case
counts. Unknown source dependencies restore the broad gate; unmapped prompts
counts. Unknown source dependencies restore the broad gate (see
[PR paid lane fallback](#pr-paid-lane-fallback)); unmapped prompts
without registered coverage require full validation. Known broad-only prompt
changes are explicitly deferred, not counted as PR passes. Weekly/manual runs of
`evals-periodic.yml` execute both complete censuses fresh; manual `evals.yml`
@@ -342,10 +344,22 @@ into the fullest slice whose estimated wall on J FIFO workers stays within S
seconds, else a new slice; a shard with no recorded wall weighs the whole budget
(its own runner), a shard longer than the budget runs alone, and overlays keep
one final one-at-a-time slice. The manifest's `plan` records each slice estimate
and `ciTimeoutMinutes` (every slice's supervised worst case plus 20 minutes
setup); CI derives the matrix (`[range(1; .sliceCount + 1)]`) and job timeout
from it, and executors refuse an `EVALS_JOBS` other than the planned J and run
their shards longest first. `--slices K` keeps the supervised round-robin
and its own CI job ceiling in `sliceCiTimeoutMinutes`: max(2 × slice budget, the
slice's longest shard supervised wall, the serialized overlay envelope) + 20
minutes setup (`CI_SETUP_ALLOWANCE_MINUTES`); `ciTimeoutMinutes` is the largest of
them. CI derives the matrix (`[range(1; .sliceCount + 1)]`) from the manifest and
indexes each slice's ceiling as its job timeout
(`fromJSON(needs.plan-slices.outputs.slice_timeouts)[matrix.slice]`). Executors
refuse an `EVALS_JOBS` other than the planned J and run their shards longest first.
Each executor's first step records its job start (`GSTACK_SLICE_JOB_STARTED_AT`).
The slice deadline is that start + the slice ceiling − a 5-minute upload reserve:
after it the executor starts no more shards. A shard that never started reports
`not_run` (INFRA, `never-started`); a shard still running is killed and reports
`hung` (TIMEOUT, `timed-out`). The executor checkpoints `slice-N.json` after every
shard start and finish, so a job killed before its final write reports from the
checkpoint (`slice N/M ended before its final result …: TIMEOUT`), never as a
missing slice. `--slices K` keeps the supervised round-robin
baseline re-packed by recorded times for local runs. Durations are recorded per
tier (a file's gate and periodic cases differ); refresh one tier from a
downloaded report directory with `--report <dir> --write-durations`. Under `EVALS_ALL` the hollow-shard guard marks exit-0 shards with
@@ -373,8 +387,10 @@ the runner parent and handed to shard children as `GSTACK_CLAUDE_CLI_VERSION`
(never spawned on a test thread), so a TUI-drift flake hunt is a grep, not
archaeology.
**Eval verdict policy** (`EVAL_POLICY` version 1 in
`test/helpers/periodic-exclude-data.ts`, pre-registered 2026-09-29). Paid evals
**Eval verdict policy** (`EVAL_POLICY` version 2 in
`test/helpers/periodic-exclude-data.ts`; version 1 pre-registered 2026-09-29,
version 2 approved 2026-10-04 with the decision memo in
[test-audit-2026-10.md](test-audit-2026-10.md)). Paid evals
never retry. Each live case has exactly one kind in `E2E_KINDS`
(`test/helpers/touchfiles-data.ts`; `test/eval-kinds.test.ts` enforces coverage),
and the kind fixes its trials before the run:
@@ -429,14 +445,19 @@ least 97% over at least 10 trials) without being removed, when an entry is 8
weekly runs old, or when a tier is over its cap.
**Pass-rate history** (`bun run eval:pass-rates`, `scripts/eval-flake-rank.ts`).
It reads the `trial-outcomes` artifact of the last N completed
`evals-periodic.yml` runs on the current branch and `main` (flags: `--case`,
`--runs N`, `--branch`, `--dir`, `--backfill`, `--json`, `--gate`) and prints
It reads the `trial-outcomes` artifact of the last N completed weekly
`evals-periodic.yml` runs, meaning scheduled runs on `main` plus `main`
dispatches, plus completed branch census runs in the same window (flags:
`--case`, `--runs N`, `--branch` to inspect one branch, `--dir`, `--backfill`,
`--json`, `--gate`). A branch trial counts only toward a series `main` has also
run (same case-owned bytes, `HARNESS_VERSION`, model and CLI); a branch never
starts or becomes a case's current series, and weeks for quarantine expiry
count `main` runs only. It prints
per-case per-trial pass rates with 95% Wilson intervals. A series is one case
under one input identity, the hash of its own touchfiles minus
`GLOBAL_TOUCHFILES` (harness edits do not restart it), per model, Claude CLI
version and policy version; a change starts a new series and older ones stay
visible. Labels: INCONCLUSIVE below 10 trials, BROKEN when the latest run is
under one input identity ([Harness version](#harness-version)), per model,
Claude CLI version and policy version
([Pass-rate policy versions](#pass-rate-policy-versions)); a change starts a
new series and older ones stay visible. Labels: INCONCLUSIVE below 10 trials, BROKEN when the latest run is
0/n after a prior interval at or above 95%, FLAKY when failures leave the
interval straddling 95%, FAILING when the whole interval is below it, PASSING
otherwise. `--backfill` imports legacy slice artifacts as pre-policy trials
@@ -530,19 +551,20 @@ its source and policy identifier. Custom drivers must resolve each job instead
of passing their ordinary 1800-second default as an explicit cap;
their outer controller/detach wall must also cover the allocated work and cleanup.
The paid census counts are printed by `--list` for each tier.
`eval:bg:pr` and `eval:bg:periodic` have 92820/67380-second outer caps, above their recomputed floors (PR fallback 78,425 s, periodic 33,821 s including the trial shards); the PR
wrapper covers a full-gate fallback at its default two workers. The broad gate
wrapper reserves 49320 seconds (floor 21,725 s), and release reserves 116700 seconds for both
tiers; free tests recompute each floor from the live shard census, case shards
included. Legacy monolithic
`eval:bg`/`eval:bg:all` retain their shorter 5400/7200-second caps; use the
sharded periodic path for complete coverage.
`eval:bg:<pr|gate|periodic|release>` runs `scripts/eval-bg.ts`. On the local
backend its outer cap is ceil(1.5 × planned serial seconds / `EVALS_JOBS`) + 20
minutes, at most 4 hours, computed from `scripts/paid-test-durations.json` for
the lane's current plan (`--timeout SECS` overrides). On the dispatch backend the
CI job ceilings above bound the run, and the local follower stops after 4 hours.
`test/eval-detach-timeout-floor.test.ts` pins the cap formula, the 4-hour
ceiling, the backend choice, the sentinel on both backends, the conclusion
mapping and dispatch dedupe.
CI plans with `--slice-budget 540 --jobs 2` for the PR gate, the periodic census
CI plans with `--slice-budget 420 --jobs 2` for the PR gate, the periodic census
and the weekly gate census (the gate census also `--skip-judges`), and
`--slice-budget 1 --jobs 1` (one file per runner) for marathon. The live plans
must fit their workflow's `max-parallel` so every slice starts at once, and
`ciTimeoutMinutes` must stay within 360; `test/evals-workflow-wiring.test.ts`
must fit their workflow's `max-parallel` so every slice starts at once, and every
per-slice ceiling must stay within 360 minutes; `test/evals-workflow-wiring.test.ts`
recomputes both from the complete census. Reconciliation rejects missing,
duplicated or misplaced registered work, absent budget records, case shards that
did not execute exactly their case, and reused results outside the PR profile.
@@ -590,6 +612,35 @@ against a temp `GSTACK_INSTALL_DIR` / `GSTACK_SKILLS_DIR`, and
`freeze/bin/check-freeze.sh` with JSON payloads on stdin (including the
`GSTACK_HOME` state-root parity against `bin/gstack-paths`).
### Pass-rate policy versions
Readers score only trials recorded under their own `EVAL_POLICY.version`.
Trials from an older policy are shown display-only. Trials from a newer policy
are ignored with a printed count ("ignored N trial(s) recorded under a policy
newer than vX"). When you see that line, run pass-rates from a checkout at or
after the commit that bumped the version. The supported rollback of a policy
change is reverting its writer commit; the reader-compatibility commit before it
stays.
### Harness version
Under `EVAL_POLICY` v2 a case's series identity is the bytes the case owns (its
paid test file, its fixtures and its skills' prompt files, including the
generated SKILL.md) plus `HARNESS_VERSION`, keyed with model, Claude CLI version
and policy version (`scripts/eval-trial-series.ts`). Each trial also records the
full consumed-input fingerprint as provenance. `scripts/harness-version.json`
pins every shared-harness file (`GLOBAL_TOUCHFILES` plus the session runners, the
PTY harness, the e2e gate and the LLM judge) by blob hash. When
`test/harness-version.test.ts` fails, run one of:
```bash
bun run scripts/bump-harness-version.ts --non-behavioral "<why no verdict can change>" # no series reset
bun run scripts/bump-harness-version.ts --bump "<what behavior changed>" # new series for every case
```
On a merge conflict in the manifest, take the higher version, then rerun the
script.
## Ubicloud VMs (`bun run test:ubicloud`)
`bun run test:ubicloud [test:free args]` runs the free suite on an ephemeral
@@ -605,8 +656,8 @@ checkout to it: tracked files, untracked files that are not ignored, and
`.git`, so uncommitted edits are tested. It then runs
`scripts/ubicloud/setup-free-suite.sh`, which mirrors the `free-suite` CI job
(same Bun pin, Playwright Chromium with its setuid sandbox helper, Xvfb,
poppler, emoji fonts, generated host outputs, gate binaries, and the CSO
helper), and runs `xvfb-run -a bun run test:free` with `GSTACK_EXPECT_BINARIES=1`
poppler, emoji fonts, zsh for the bash+zsh portability arms (#2669), generated
host outputs, gate binaries, and the CSO helper), and runs `xvfb-run -a bun run test:free` with `GSTACK_EXPECT_BINARIES=1`
and `GSTACK_FREE_RETRY_FLAKY=1`. Shard logs are copied to
`.context/ubicloud/<timestamp>/`, and the VM is destroyed on every exit path.
The exit status is the suite's.
@@ -646,17 +697,19 @@ serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so
a broken tree can't masquerade as flaky). The required CI free lane and the
Windows lane set the retry knob too, appending every flaky pass to the JSONL ledger it uploads
(`GSTACK_FLAKE_LEDGER`) — a flaky pass never reds the lane, but it never
disappears either.
disappears either ([Flake ledger](#flake-ledger)).
## Test selection and tiers
Moved verbatim from CLAUDE.md (#2096 size limit).
**Diff-based test selection:** `test:evals` and `test:e2e` auto-select tests based
on `git diff` against the base branch. Each test declares its file dependencies in
`test/helpers/touchfiles.ts`. Changes to global touchfiles (session-runner, eval-store,
touchfiles.ts itself) trigger all tests. Use `EVALS_ALL=1` or the `:all` script
variants to force all tests. Run `eval:select` to preview which tests would run.
**Diff-based test selection:** the sharded paid runner (`test:pr`, `eval:bg:pr`,
`test:gate:sharded`) selects tests based on `git diff` against the base branch.
Each test declares its file dependencies in `test/helpers/touchfiles.ts`. Changes
to global touchfiles (session-runner, eval-store, touchfiles.ts itself) trigger
all tests. Use `EVALS_ALL=1` to force all tests. Run `eval:select` to preview
which tests would run; it applies the PR profile by default, and
`--profile full` shows the plain touchfile selection. Commands by task: [CONTRIBUTING.md](../CONTRIBUTING.md#which-command-do-i-run).
**Two-tier system:** Tests are classified as `gate` or `periodic` in `E2E_TIERS`
(in `test/helpers/touchfiles.ts` — a facade over `touchfiles-data.ts` +
@@ -680,14 +733,61 @@ in `bun test`): a `skill-e2e-*` file named in a touchfiles dep list whose
suite. Files not named in any dep list are reported, not enforced — keep both
in sync.
### PR paid lane fallback
PR selection (`scripts/test-pr-profile.ts`) classifies every changed file. A file
mapped by a paid case's touchfiles selects that case. These files are exempt and
select nothing: docs, any non-paid `*.test.ts(x)`, `FREE_ONLY_PR_FILES` (the
free and paid duration seeds, `scripts/ubicloud/**`, every workflow that runs no
paid evals, and free-only tools), and the fixtures in `FREE_FIXTURES`
(`test/helpers/free-fixtures-data.ts`, each fixture with its free consumers).
`PAID_WORKFLOW_FILES` (`evals.yml`, `evals-periodic.yml`, `evals-marathon.yml`)
and any other unknown file restore the full gate (`prCoverage.mode =
full-fallback`). The job summary and PR comment list each file that caused a
fallback with one of three labels:
- **needs touchfile entry**: a paid test, helper or fixture no case maps; add it
to the owning case in `E2E_TOUCHFILES` (`test/helpers/touchfiles-data.ts`).
- **add to FREE_ONLY_PR_FILES**: a file only free lanes read; add it to the list
in `scripts/test-pr-profile.ts` (or a fixture to `FREE_FIXTURES`).
- **real unknown dependency**: shared code (bin/, lib/, resolvers, hosts, setup)
whose dependents are not mapped; the full gate is the correct result.
`test/free-fixtures.test.ts` holds the inventory: each `test/fixtures/**` path
belongs to a paid touchfile or `FREE_FIXTURES`, and each workflow to the paid or
free-only list. Editing a paid test file runs its own gate cases on the PR when
their Bun test is named by the case id; the file's other cases stay deferred with
a reason that says to name the test by its case id.
### PR receipts, recovery and debounce
Passes and failures of single-trial shards are written as receipts at the
execution boundary, so a cancelled run's finished work survives. The
`recover-receipts` job runs `scripts/recover-receipts.ts` from the base ref with
`actions: read`: it merges the receipts of at most 5 completed, cancelled
`evals.yml` runs of the same PR (oldest first, negatives before passes, so a newer
FAIL blocks an older PASS) and hands the store to `plan-slices`, which keeps
`contents: read` only. Its `recovery.json` checkpoint never advances past an
unresolved run. `plan-slices` turns reuse off when recovery degraded, the
checkpoint is missing or a cancelled run is still unresolved; recovery itself
never fails the run, and the job summary names the reason (rate limit with its
reset time, `gh auth status`, the 60-second budget). The whole store is saved by
`slices-report`.
On a `synchronize` push, when another run for the same PR started in the last 15
minutes, the debounce job waits 90 seconds before planning; a run that outlives
the wait reports "superseded by <sha>". PR labels: `evals-no-debounce` skips the
wait and `evals-fresh` turns receipt reuse off. Labels apply from the next push,
because adding a label does not start `evals.yml`.
## Free suite runner, judge reuse and engine skips
Moved verbatim from CLAUDE.md (#2096 size limit).
`bun run test` routes through `scripts/test-free-shards.ts` (N concurrent
shard processes, serial within each, packed by recorded per-file durations
when `scripts/free-test-durations.json` exists — refresh occasionally with
`bun run test:free --record-durations`; strict-output classification per
when `scripts/free-test-durations.json` exists — refresh it with
`bun run test:ubicloud --record-durations`; strict-output classification per
shard: a shard without bun's terminal summary line FAILS — silent truncation
cannot report green). `TREE_MUTATING` lists the files that still run in their
own trailing serial shard (today only `test/bootstrap-retention.test.ts`, for
@@ -736,6 +836,105 @@ Fix a writer with `usePrivateStateRoot()` (`test/helpers/private-state-root.ts`)
or a child HOME/GSTACK_HOME, and resolve product state paths at write time,
never at import.
### Flake ledger
`GSTACK_FREE_RETRY_FLAKY=1` reruns a failing free test file once, serially. A
clean retry turns the shard green as a flaky pass and appends an entry to that
shard's flake ledger, uploaded as `flake-ledger-<shard>` (90-day retention). The
Monday "Weekly test health" workflow (`bun run test:health --enforce`) reads
main's ledgers through `scripts/lib/ci-history.ts` and fails when a file flakes
in more than 5% of main free-tests runs over a window of at least 20 runs; a
window thinner than 20 runs in the last 7 days widens to the newest 20 runs, and
fewer than 20 runs is report-only. `bun run test:health flake-summary <dir>`
prints a run's ledgers as a table. A flaky file is fixed at source; the retry
never masks it. `bun run eval:pass-rates` lists only the local ledger
(`flakeLedgerPath()`, overridden by `GSTACK_FLAKE_LEDGER`).
The aggregate job of `free-tests.yml` downloads every `flake-ledger-*` artifact
of the run and appends `bun run scripts/test-health-report.ts flake-summary <dir>`
to its job summary. The Windows lane uploads `flake-ledger-windows-<shard>`.
Free-suite flake rules:
- Bun gives a child `stdio: 'pipe'` a unix socketpair. A test that must hold a
write blocked uses a FIFO the test never reads (`test/qa-evidence.test.ts`,
`test/qa-deadline.test.ts`), not a paused child stream: socket capacity follows
`net.core.{w,r}mem_default` and the parent's reader.
- A test that rewrites CI provenance variables (`CI`, `GITHUB_*`, `CI_*`)
restores them. Free shards run many files in one process, and a later browse
daemon without `CI=true` launches Chromium sandboxed.
- Handshakes with a runner wait for the runner's own observable effect (its
progress line, a coalesced read count), not for a fixture side file or a timer.
### Free suite duration seed
`scripts/free-test-durations.json` drives shard packing; files missing from it
pack at the seed's 99th percentile. Acceptable seeds come from
`bun run test:ubicloud --record-durations` (each file timed in its own child on a
VM with the CI lane's environment) or a CI recording. A 4-core laptop or
small-sandbox recording is not acceptable: browser and display tests skip or slow
down there, so the seed misstates them.
- `--ci-plan` warns in the job summary, with the refresh command, when more than 5
files are unseeded; it never fails a PR. `--ci-verify` lists shards that ran
longer than 1.5× their prediction and at least 30 seconds over.
- The weekly test-health run fails its enforced `unseeded-free-files` check
("stale seed: N unseeded file(s) > 5") when more than 5 free test files are
missing from the seed, so contributors without Ubicloud access learn within a
week that it needs a refresh.
- `test/free-seed-ratchet.test.ts` (run with full history in `free-tests.yml`'s
`free-plan` job, `GSTACK_FREE_SEED_BASE=$(git merge-base HEAD origin/main)`)
fails when a seeded file over 60 seconds is missing from
`scripts/free-test-seed-allowlist.json`, when an allowlist entry is stale (gone,
or no longer over 60 seconds), or when an entry is added relative to the
merge-base copy without a `todo`. An annotated exception is
`{file, reason, todo}` and the test prints it. A merge-base without the
allowlist file counts as the first allowlist. Fix a failure by splitting the
file, then rerun `bun run test:ubicloud --record-durations`.
### Windows free lane
`windows-free-tests.yml` plans on Linux (`--windows-only --ci-plan --shards 6`),
runs one strict serial shard per `windows-latest` job
(`bun run test:windows --ci-run`), and verifies every shard receipt on Linux
(`--windows-only --ci-verify`). Packing uses Windows-measured durations from
`scripts/free-test-durations-windows.json`; refresh them with
`gh workflow run windows-free-tests.yml --ref <branch> -f record_durations=true`,
then download the `free-test-durations-windows` artifact and commit it.
Curation lives in `scripts/lib/windows-curation.ts`. Every Windows plan and
`--list` prints the curated count and the top exclusion reasons (also in the job
summary). Files a probe proved safe despite a `/tmp` literal are force-included
(`WINDOWS_PROBE_SAFE`). `test/windows-native-workflows.test.ts` holds
`WINDOWS_CURATED_FLOOR`; when Windows-safe tests are deleted on purpose, lower it
in the same PR with a reason.
Native qualification campaigns (Windows cookie extraction, Windows launch
diagnostics, Dia on macOS) run only on dispatch of `native-qualification.yml`:
```bash
gh workflow run native-qualification.yml --ref <branch> -f mode=<cookie-native|windows-native-diagnostics|dia-native|dia-launch-comparison|dia-gui-readiness>
```
### Real-setup install fixture
Tests that run the real `setup` use `test/helpers/install-fixture.ts`. Each test
file builds one read-only seed checkout and clones it per test with
copy-on-write file copies (plain copy fallback), never hard links. Writes go
through `put`, `setVersion` and the `fixture*Sync` helpers; files register
`afterEach(cleanupFixtures)` and `afterAll(cleanupSeed)`.
`test/install-fixture.test.ts` is the isolation tripwire: writes in one clone
leave the seed and sibling clones unchanged.
### Test-only timing knobs
Defaults are the product's; tests shrink them instead of waiting real time:
`GSTACK_RENDER_SLACK_MS` (`lib/aside-render.ts`, default 10000),
`GSTACK_MEMORABLE_LOCK_TRIES` (`bin/gstack-memorable`, default 50 tries of 0.1 s),
the `recordGraceMs` option of `test/helpers/office-hours-attempt.ts` (default
5000) and the `timeoutMs` argument of `clearCookieTargetStorage(page, origin,
timeoutMs)` (default 15000).
## Running evals as an agent: detach
Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in
@@ -745,36 +944,51 @@ When **you (an agent/harness)** launch a long eval/benchmark run, run it through
`bin/gstack-detach` — NEVER as a plain backgrounded Bash task. A plain background
task lives in the harness's process group, so a SIGTERM ("polite quit") on a turn
boundary, a stopped Monitor, or an interruption kills the run mid-flight (observed:
`script "test:gate" was terminated by signal SIGTERM` ~40 min into a run). On macOS
`script "test:gate" was terminated by signal SIGTERM` ~40 min into a run of the
since-retired single-process runner). On macOS
the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
(escapes the group SIGTERM) wrapped in `caffeinate -i` (blocks idle-sleep).
- Use the `eval:bg*` scripts (`eval:bg`, `eval:bg:all`, `eval:bg:gate`,
`eval:bg:periodic`) — they wrap the eval command in `gstack-detach` with the
machine-wide `gstack-evals` lock (concurrent worktrees serialize instead of
saturating the shared model API), a per-tier watchdog, and a **run-scoped** log
under `~/.gstack-dev/eval-runs/` (no shared-`/tmp` collision). Each prints its
log path. `eval:bg:gate` / `eval:bg:periodic` run their tier through the
sharded paid runner (`scripts/test-paid-shards.ts`, also exposed as
`test:gate:sharded` / `test:periodic:sharded`): one Bun process per test
file, an external wall-clock timeout that kills the shard's process GROUP
(stray `claude`/`codex` grandchildren included), a per-shard
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
constructor, and an aggregate that separates failed vs timed-out vs
never-started shards — the detach timeouts (the `--timeout` values on
package.json's `eval:bg:gate` / `eval:bg:periodic`;
floor enforced against the live shard census by
test/eval-detach-timeout-floor.test.ts)
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
process count (default 8); `EVALS_CONCURRENCY` is bun's --max-concurrency
WITHIN a shard (default 2) — they are deliberately separate knobs. `eval:list` / `eval:compare` /
`eval:summary` / `eval:flake-rank` read the shard dirs too. Or call
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] --
<cmd>` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never
pass keys in argv).
- Use `bun run eval:bg:<pr|gate|periodic|release>` (`scripts/eval-bg.ts`). It
prints `[eval-bg] backend=<dispatch|local> …` and `gstack-detach LOG <path>` and
returns at once. Both backends run under `gstack-detach` with the machine-wide
`gstack-evals` lock (concurrent worktrees serialize instead of saturating the
shared model API) and a **run-scoped** log under `~/.gstack-dev/eval-runs/`
whose first line names the backend, the run and the tested revision.
- **Dispatch** (default when HEAD is clean and equal to its pushed branch on
`garrytan/gstack`, checked with `gh api`, and `gh` can dispatch): `pr` and
`gate` dispatch `evals.yml` (`evals_all=false` with `base_ref`/`base_sha` for
`pr`, `evals_all=true` for `gate`), `periodic` dispatches `evals-periodic.yml`,
`release` dispatches both. Every dispatch passes `expected_sha` (the workflow
fails fast when the checked-out revision differs) and a `nonce` that appears
in the run name (`E2E Evals [eval-bg <nonce>]`), which is how the log finds
and follows its run. A queued, running or green dispatch with the same
revision and inputs is followed instead of dispatched again. Dispatched runs
are validation runs: fresh and never writing PR receipts, except that a `pr`
dispatch on a branch with an open same-repo PR passes `pr_receipts=<PR>` and
reads that PR's receipts read-only.
- **Local** (dirty tree, unpushed commits, a fork, no `gh`, or `--local` /
`GSTACK_EVAL_BG_MODE=local`): the sharded paid runner
(`scripts/test-paid-shards.ts`) on this machine, capped at
ceil(1.5 × planned serial seconds / `EVALS_JOBS`) + 20 minutes, at most 4 hours
(`--timeout SECS` overrides). One Bun process per test file, an external
wall-clock timeout that kills the shard's process GROUP (stray
`claude`/`codex` grandchildren included), a per-shard
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
constructor, and an aggregate that separates failed vs timed-out vs
never-started shards. `EVALS_JOBS` sets the shard process count (default 8,
2 for `pr`); `EVALS_CONCURRENCY` is bun's --max-concurrency WITHIN a shard
(default 2). Export `ANTHROPIC_API_KEY` first (never pass keys in argv).
- `--dispatch` refuses with the fix ("commit and push, or rerun with --local")
instead of falling back.
`eval:list` / `eval:compare` / `eval:summary` / `eval:pass-rates` read the shard
dirs too. For any other long agent job, call
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] -- <cmd>` directly.
- Then **poll the printed logfile** with a death-aware watcher: break on the
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (success AND failure are
both marked, so silence is never mistaken for success). The detached run survives
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (0 passed, 1 failed,
130 cancelled, 2 other on both backends; silence is never mistaken for success).
`bun run scripts/eval-bg.ts status <log-or-run-id>` reconnects and prints
running, passed, failed, cancelled or incomplete (died without a sentinel). The detached run survives
even if your watcher gets reaped, so re-checking the log always works. Keep
checking until the sentinel appears or the user tells you to stop; a long run is
expected, and a promise to check later is not a result. At each check, report
@@ -782,5 +996,5 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
- Why the lock: a shared dev box with several Conductor worktrees will rate-limit
the model API if two eval suites run at once (15-way concurrency each), which
mass-times-out E2E tests. The lock makes the second run WAIT, not collide.
- Humans running `bun run test:evals` foreground in their own terminal don't need
- Humans running `bun run test:pr` foreground in their own terminal don't need
this — Ctrl-C is intended there. Detachment is for agent-launched runs only.
+12 -4
View File
@@ -26,6 +26,12 @@ The dedicated Windows CI job checks startup, argument forwarding, and runtime
injection separately from static audit and private-state support. POSIX
watchdog execution remains unavailable there.
No qualified runtime or scanner image exists yet, so contained runtime and
scanner execution is unavailable and setup preloads nothing: the scanner
catalog is empty while its build inputs await review, and the runtime catalog's
profiles stay unqualified until a private evaluator, still being built, can
qualify them.
Every successful fresh install or upgrade asks the trusted launcher to preload
the current host platform's qualified runtime and scanner images. This is the
only automatic image acquisition path. It first validates the committed
@@ -48,10 +54,12 @@ egress. `gstack-cso doctor --repo PATH` never downloads; it reports a qualified
profile unavailable until that exact digest and platform are already present
in the local daemon.
The ordinary free suite exercises contracts without requiring Docker. The
`cso-docker-integration` CI job sets `GSTACK_CSO_DOCKER_TESTS=1`; unavailable Docker
or missing containment prerequisites fail that job. Run it locally with a
working local daemon:
The ordinary free suite exercises contracts without requiring Docker. The free
lane's `cso-docker-integration` job sets `GSTACK_CSO_DOCKER_TESTS=1` and runs
`test/cso-docker-integration.test.ts`; unavailable Docker or missing containment
prerequisites fail that job. The node-lifecycle and stack-cold files need a
staged stack image, so they run in the CSO runtime workflows through
`test:cso:docker`. Run all three locally with a working local daemon:
```bash
bun run build:cso
+3 -2
View File
@@ -815,8 +815,9 @@ A mechanical pre-filter shortlists assertion-free probes, source greps, export-l
copies and near-duplicate files before any model reading. Each candidate gets a
retirement card (what it detects, non-test callers with the search command, the
stronger remaining proof, history, what retiring it unlocks, and the validation
command). Contract tests such as SKILL.md goldens and prompt-byte checks are
retained. The report and a JSON sidecar land in `~/.gstack/projects/<slug>/`.
command). Contract tests such as SKILL.md goldens and prompt-byte checks
(machine-read tokens, not English sentences; see [the value bar](test-value-bar.md))
are retained. The report and a JSON sidecar land in `~/.gstack/projects/<slug>/`.
Nothing is edited unless you approve a batch; spawned sessions stay report-only.
Tests marked `gstack:test-value keep reason="..."` are skipped and listed in the
report's appendix.
+18
View File
@@ -476,3 +476,21 @@ release docs sweep); tests 383 (+9,375/−44,379); test helpers 47 (+786/−12,7
- **`test/helpers/claude-pty-runner.unit.test.ts` (3,694 LOC, 223 tests, 0.1 s):** a large self-test of the 5,885-LOC PTY harness. The classifiers it covers (`classifyVisible`, `parseNumberedOptions`, `Step0BoundaryPredicate`…) are live in paid runs, so keep it. The per-capture blocks (`captured F`, `captured G`) belong to the per-incident consolidation lane.
- **Zero-importer helpers** `auq-parallel-worker`, `setup-gbrain-fixture-command`, `emulate-bun-windows-eexist` are loaded by path (preload / generated import). `benchmark-judge` has a production caller (dynamic import in `bin/gstack-model-benchmark`).
- **`browse/src` `__reset*`/`reset*ForTests` exports (12):** standard singleton-reset seams, keep.
## Corrections (2026-10 audit)
Facts above that no longer hold, checked against main at v1.91.19.0:
- `codex-e2e-recommendation-substance` is not census-only: it is no longer in
`PERIODIC_CI_EXCLUDE` and runs in every periodic census.
- The after-metrics are superseded. Five days after release the free suite had
regrown by 132 test files and about 53% of serial time; the
[2026-10 audit](test-audit-2026-10.md) carries current numbers.
- The duration seed again lacked 19 files, including the 115-second
`claude-overlay-setup.test.ts`, which the planner packed as a short file.
- `browse/test/compare-board.test.ts` stayed quarantined through this audit; its
16 tests now pass headless and run in the free suite again.
- `ci-image-tag-binding` now pins `evals.yml`, `evals-periodic.yml` and
`evals-marathon.yml`; `ci-image.yml` is deleted because those workflows push
the same content-hash tag.
- The weak literal pins listed under Lane 5 are still present.
+276
View File
@@ -0,0 +1,276 @@
# Test, eval and CI audit — October 2026
This document records the evidence behind the October 2026 test, eval and CI
wave: the EVAL_POLICY v2 decision memo (D1) with its pre-registered backtest,
and the commands that reproduce every number. Metrics that need post-merge
traffic are measured with `bun run test:health --since-days 7` (merge + 7 days)
and `bun run test:health --since-days 28` (after the fourth scheduled census).
## EVAL_POLICY v2 decision memo (D1)
### Why the census is red
The weekly census is red by arithmetic, not by a handful of broken tests.
Over 22 censuses with trial outcomes the periodic lane recorded 76 failed
trials in 3,550 (about 3.5 failed trials per run) and the gate census 27 in
2,241 (about 1.2 per run). With independent failures, P(green) is e^-3.5 ≈ 3%
for the periodic lane and e^-1.2 ≈ 30% for the gate census; observed green
rates were 0/22 and 8/22. Holding a ~190-case single-trial lane green 80% of
the time needs every case at or below about 0.12% failure per run, which a
live agent judged by one trial does not reach. Fixing the top five chronic
cases leaves the lane red.
Three levers stay inside the no-retry, no-threshold, no-rejudge rule: make
detectors deterministic over structured artifacts (W3), classify INFRA
honestly (W2d, owned by the reliability follow-ups), and make quarantine
reachable for detector, harness and model-latency failures. v2 targets the
third lever.
`test:health` reports the census as three separate numbers, so a policy
change cannot read as a reliability gain: the raw trial failure rate per kind
and failure class, P(lane red) from pooled per-case rates under the recorded
panels, and the same evidence re-aggregated with every case judged on one
trial. Both red probabilities treat every observed failure as
nondeterminism, so they are upper bounds for known-good inputs.
**False-blocking targets.** Periodic lane: at most 1.0 expected failed trial
per census (P(red) ≤ 63%) at the 28-day check; gate census: at most 0.5
(P(red) ≤ 39%). Neither target may be reached by a threshold, retry or
rejudge change. A miss opens a P1 TODOS entry with the per-kind,
per-failure-class breakdown from `test:health`.
### What v2 changes
- **Series identity (W4a, CEO-27, ENG-3).** A case's pass-rate series is keyed
by the bytes it owns plus `HARNESS_VERSION`: its paid test file, its
fixtures, and the prompt files of the skills it names (templates, sections,
manifests and the generated `SKILL.md` the model reads). The series key adds
the model id, the pinned Claude Code CLI version and the policy version.
Shared helpers no longer start new series. Every trial also records the
full consumed-input fingerprint (all touchfiles, globals included) as
`series_fingerprint`, plus `harness_version`, as provenance.
(`caseSeriesIdentitiesV2` in `scripts/eval-trial-series.ts`.)
- **Harness guard.** `scripts/harness-version.json` pins the blob of every
shared execution-harness file: `GLOBAL_TOUCHFILES` plus the session runners,
the PTY harness, the e2e gate and the LLM judge (36 files). A free test
(`test/harness-version.test.ts`) fails when one changes without a decision.
`bun run scripts/bump-harness-version.ts --bump "<reason>"` raises
`HARNESS_VERSION` and starts a new series for every case;
`--non-behavioral "<reason>"` records the new blobs as compatible with no
reset. Entries are one line per file so concurrent PRs that touch different
files merge cleanly; on a conflict, take the higher version, then rerun.
- **Weekly history (W4b).** Weeks for quarantine expiry and drift are
`evals-periodic.yml` runs on `main`: scheduled runs and main dispatches.
Trials from branch census runs in the same window pool into a series `main`
has also run (option (b) below, approved 2026-10-05); a branch identity
`main` has not run is dropped and never becomes the current series.
- **Kinds (W4c).** No kind changes; see the evidence below.
- **Pre-v2 evidence does not qualify.** Readers segment strictly by
`policy_version`: only v2 trials count toward the quarantine entry rule; v1
trials stay visible as display-only. Quarantine is not seeded at merge; a
case enters only after it reaches `entry.minTrials` qualifying v2 trials,
and `test:health` shows qualifying-vs-required counts with "insufficient
evidence" distinct from healthy (DX-12).
### W4c: the named "stochastic" cases are deterministic in the record
Recorded census and PR trials (all 95 recorded revisions, October 2026):
| Case | Kind | Passed / trials | Failures |
|---|---|---|---|
| plan-ceo-finding-floor | rule | 26/27 | 1 timeout (608 s) |
| plan-devex-finding-floor | rule | 27/27 | — |
| plan-eng-finding-floor | rule | 9/9 | — |
| plan-design-finding-floor | rule | 9/9 | — |
| plan-ceo-review-plan-mode | rule | 34/34 | — |
| plan-devex-review-plan-mode | rule | 27/27 | — |
| plan-eng-review-plan-mode | rule | 9/9 | — |
The comments calling `plan-*-finding-floor` "stochastic ask-first" are stale:
the record shows these cases behave like rules. AGENTS.md forbids changing a
kind without pass-rate evidence, and a 3-trial panel would triple their cost
for no measurable gain, so they stay `rule`. `plan-design-review-plan-mode`
(17/22) is handled by its detector repair, not by a kind change.
### Backtest (ENG-3, DX-12)
`bun run scripts/eval-trial-series.ts --backtest --weeks 30 --records <dir>`
replays identities with today's registry over historical trees (cases whose
test file did not exist at a revision are absent there).
**Weekly main revisions** (29 Mondays, 06:00 UTC, 2026-03-16 to 2026-09-28).
A case qualifies at a week when one identity holds long enough to collect 10
trials within the last 10 weekly runs (rule and judge cases record one trial
per run, behavior cases a panel of three):
| Identity | Qualifying at the latest week | Mean qualifying share, last 8 weeks | Median identity changes per case |
|---|---|---|---|
| v1 (own touchfiles minus globals) | 0/195 | 0.3% | 15 |
| v2 (case-owned bytes) | 0/195 | 0.7% | 14 |
| v2, reset on every pinned-harness change | 0/195 | 0.1% | 14 |
| v2 without the generated `SKILL.md` (prototype) | 0/217 | 2.0% | 7 |
**Recorded trials** re-keyed by the v2 identity of the revision that produced
them (blocking cases with at least 10 trials on one series):
| Trial source | Recorded (v1) identity | v2 identity |
|---|---|---|
| 22 census runs (5,665 trials) | 52/210 | 67/210 |
| Census + PR lane, 95 revisions (14,898 trials) | 107/210 | 134/210 |
**Harness churn.** The pinned harness changed in 24 of 28 consecutive-week
transitions, and 33 of the last 60 main commits (2026-08-04 to 2026-10-04)
touched a pinned file. About half of PRs therefore record a
`bump-harness-version` decision, and only `--bump` decisions reset series.
### What the backtest shows
1. v2 makes series honest and longer when trials are dense: on the recorded
census runs, 67 blocking cases reach 10 trials on one series versus 52
under v1, and 134 versus 107 with PR-lane trials included.
2. Under the approved weekly history (main scheduled runs plus main
dispatches, about one run per week), no blocking case reaches 10
qualifying trials at any recent week. A rule case needs ten consecutive
weekly runs on unchanged owned bytes, and a case's own prompt files
change about every two weeks (the generated `SKILL.md` carries the shared
preamble). Excluding the generated `SKILL.md` halves identity churn but
still qualifies at most 2% of cases, and it would let preamble and resolver
prompt changes pool unless every such edit went through the harness guard.
3. So v2 alone does not make quarantine reachable. Reaching it needs one of:
(a) keep the history as approved and accept that quarantine stays rare, so
census redness is fixed at source (W3) and tracked by `test:health`;
(b) pool trials from every v2 census run whose series key matches,
including branch dispatches, while weeks for expiry still count from main
only — the recorded backtest's 67/210, at the cost of pooling branch code
whose unbumped harness edits match; or (c) extra census dispatches on an
unchanged main revision (about $175 recorded each). Garry approved (b) on
2026-10-05, and this PR implements it: pooled branch trials count only
toward a series a `main` run also has.
### Alternatives considered
- **Gating vs trended.** The native CEO voice proposed keeping live-agent
cases trended (reported, never blocking) and gating only on deterministic
detectors. v2 keeps the census blocking because quarantine and the
pass-rate gate already provide a trended path per case; the
false-blocking targets above are the check on that choice. If the 28-day
check misses them, trended reporting for the remaining stochastic cases is
the next proposal.
- **Adaptive sequential panels** (one trial, escalate to three on a failure,
about 1.04× cost across ~190 cases) would be the strongest lever for a
green census. They are not adopted: an escalation triggered by a failure is
a retry, which Garry's rule forbids.
- **Buy vs build.** Hosted eval platforms (Braintrust, LangSmith, Weights &
Biases Weave) store per-case pass-rate history and cost natively, but
gstack's verdicts are produced inside GitHub Actions from Claude Code
sessions with pre-registered panels and receipts, and the history is a few
thousand JSONL lines per week that `gh` already serves. A platform would add
an external dependency and a second source of truth for verdicts without
removing the in-repo policy. Building on the existing artifacts stays the
choice; revisit if history outgrows artifact retention (90 days).
### Rollout and rollback (ENG-14, DX-12)
The reader-compatibility commit lands first: readers score only trials under
their own `EVAL_POLICY.version`, show older-policy trials as display-only,
and ignore newer-policy trials with a printed count. The v2 writer (identity
stamping and `version: 2`) lands last. The supported rollback is reverting
the v2 writer commits; a whole-PR revert is not claimed safe.
On a mixed v1/v2 fixture (10 passing v1 trials, 12 v2 trials with 3
failures), the pre-PR reader at 2db0b3a pools the v2 records as the current
v1 series and fails its gate:
```
pass-rates: policy v1, 22 post-policy trial(s), 0 pre-policy (display only)
FAILING rule gate 9/12 [46.8%–91.1%] ... plan-ceo-review-plan-mode (baseline reset)
ACTION REQUIRED (2): [drift] ... [rule-as-behavior] ...
exit=1
```
The compatibility reader ignores the v2 records and keeps the v1 series:
```
pass-rates: policy v1, 10 post-policy trial(s), 0 pre-policy (display only)
ignored 12 trial(s) recorded under a policy newer than v1: this checkout predates them. Fix: ...
PASSING rule gate 10/10 [72.2%–100%] ... plan-ceo-review-plan-mode
exit=0
```
## Wave evidence
### PR paid lane selection replay (CEO-23, ENG-18)
The replay sends each recorded PR push's changed files through today's
selector and dependency graph:
```bash
gh run list -R garrytan/gstack --workflow evals.yml --json databaseId,headSha,event,createdAt,conclusion -L 200 > runs.json
bun run scripts/replay-pr-selection.ts --runs runs.json
```
Over 168 PR pushes with plan manifests (2026-09-25 to 2026-10-04):
| Measure | Recorded | Replayed |
|---|---:|---:|
| Full-fallback pushes | 165 of 168 (98.2%) | 164 of 168 (97.6%) |
| Misses (a failed case that depends on the diff and is no longer selected) | — | 0 |
| Failed cases that no longer exist (counted apart) | — | 6 |
113 of the 168 diffs exceed the compare API's 300-file cap and are replayed on
their first 300 files. The rate barely moves because the replay uses each push's
cumulative branch diff, and the wave branches of that week touch shared inputs
that correctly select every case: `scripts/` in 156 pushes, `.github/` in 125
(mostly the eval workflows), `bin/` in 120, `make-pdf/` in 90, `test/helpers/` in
86, `design/` in 67, `lib/` in 58, `hosts/` in 56 and `tsconfig*.json` in 37. The
selector removes the single-class fallbacks (duration seeds, free-only workflows,
`scripts/ubicloud/**`, free tests and free fixtures; run 36497566037 now selects
the `pr` profile) and cannot narrow a branch that changes `bin/` or `lib/`;
narrowing those needs touchfile mappings for `bin/`, `lib/`, `hosts/` and
`make-pdf/`. The post-merge `test:health` check measures the fallback rate on
ordinary PRs.
### Paid slices and concurrency (ENG-17)
Live census with `EVALS_ALL=1` (periodic: `--list --slice-budget S --jobs 2`;
gate census: `--emit-plan <file> --slice-budget S --jobs 2 --skip-judges`):
| Lane | Before (540 s) | After (420 s) |
|---|---|---|
| Periodic slices / peak shard processes | 26 / 52 | 33 / 66 (max-parallel 26 → 36) |
| Weekly gate census slices / peak | 12 / 24 | 16 / 32 (max-parallel 16 → 20) |
| CI job ceiling | One value for every slice: 173 min periodic, 254 min gate census | Per slice: ordinary 50 min (30-minute shard wall + 20), periodic outliers 52 and 73 min, the overlay slice 173 min (5 serialized overlay wrappers); gate census 34–82 min |
The PR lane (`evals.yml`) uses the same 420-second budget and per-slice ceilings.
### Free suite
- The Windows lane runs on six `windows-latest` jobs packed by Windows-measured
durations: about 4.6 minutes wall in the free-suite branch's validation runs,
down from 8.1 minutes on one runner.
- Unseeded files pack at the 99th percentile (was the 75th), and the seed growth
ratchet (`test/free-seed-ratchet.test.ts`) keeps files over 60 seconds on a
shrink-only allowlist.
- The bash+zsh portability tests (#2669) run in CI now that the free-suite job
installs zsh.
- Native Windows and Dia qualification campaigns moved out of the PR workflow
into the dispatch-only `native-qualification.yml`.
### Prose pins (D2)
D2 is approved: the prompt-byte contract in
[test-value-bar.md](test-value-bar.md) covers machine-read tokens only, and
sentence pins in 55 free test files became structural checks through
`test/helpers/prompt-structure.ts`. A sentence pin here is a 40-or-more-character
English string literal, or a prose regex of five or more words, asserted against
SKILL.md, template or section text in a free test.
| Measure | main (2db0b3a) | This wave |
|---|---:|---:|
| Sentence pins | 2,332 in 217 files | 1,190 in 209 files |
| Shouting pins (NEVER/MUST/ALWAYS/CRITICAL emphasis) | 8 | 4 |
The 4 remaining shouting pins are enum or marker tokens (`PLAN MODE EXCEPTION —
ALWAYS RUN`, `[CRITICAL]`, `ALWAYS-ON` CLI output), not emphasis. Most remaining
sentence pins sit in QA caller, detector-helper and workflow files that other
work in flight owns.
+21 -5
View File
@@ -20,12 +20,28 @@ OpenClaw's `test-audit` skill (openclaw/openclaw@a214e76,
- **Value bar**: the authoring gate plus the rule that a test which breaks under a
behavior-preserving refactor asserts implementation, unless its exact output is the
declared contract (goldens, prompt bytes, wire formats).
- **Prompt-byte contract**: the machine-read tokens inside a prompt, nothing wider.
A token qualifies when software consumes its exact bytes: a marker or sentinel
line a parser, hook or grader matches; a field name or JSON shape a script reads;
a command line, flag or environment variable the agent runs; an enum value or
status word a caller branches on; a file path or placeholder another step opens;
or an exact string a security boundary emits into model input. An English
sentence is not a prompt-byte contract because the model reads it: rewording it
must not break a test. Pin a sentence only when a recorded eval shows that exact
wording changes behavior, and cite that eval next to the pin.
- **Sentence pin**: a test that asserts an English sentence or clause (40 or more
characters of prose, or a prose regex) against SKILL.md, a `.tmpl` template or a
section file. Convert it to a structural check (the section or heading is
present, steps keep their order, the machine-read token is present in the right
section), or delete it when a behavioral eval already covers the behavior and
name that eval in the commit. Check safety lines on meaning (a case-insensitive
keyword in the right section), never on capitalization or emphasis.
- **Retention bar**: keep a test that independently enforces a public API, protocol,
config, migration, storage, security, platform, default, prompt-byte,
generated-output (SKILL.md golden), package, release or architecture contract; call
order when order is observable; source inspection when it is the cheapest
independent guard. Static or slow is not a reason to delete. Anything reachable
from the package entrypoint is never retired.
config, migration, storage, security, platform, default, prompt-byte (as defined
above), generated-output (SKILL.md golden), package, release or architecture
contract; call order when order is observable; source inspection when it is the
cheapest independent guard. Static or slow is not a reason to delete. Anything
reachable from the package entrypoint is never retired.
- **Value card**: the gate's four answers as one line,
`Value: protects=<...>; fails_when=<...>; why_new=<...>; seam=none`. Each field is
at most 160 UTF-8 bytes in that line (clamped to 157 bytes plus `...`); JSON keeps