Merge remote-tracking branch 'origin/main' into cso-evaluator-support

# Conflicts:
#	CHANGELOG.md
#	VERSION
#	agents-digest/gstack-AGENTS.md
#	package.json
#	test/cso-docker-integration.test.ts
This commit is contained in:
garrytan committed 2026-10-05 01:33:04 +00:00
commit 2d79967403
288 files changed
+13384 -10107

No files matched your search

+1 -1
View File
@@ -93,7 +93,7 @@ gstack/
├── docs/designs/ # Design documents (incl. IMPECCABLE_INTEROP.md = the design detector / catalog / open DESIGN.md record, and fork-port-residual-2026-09/ evaluation evidence)
├── setup-deploy/ # /setup-deploy skill (one-time deploy config)
├── .github/ # CI workflows + shared composite actions (.github/actions/) + Docker image (claude CLI pinned)
│ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, and 8 more (windows, periodic evals, release gates, ci-image)
│ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, plus free-tests, Windows, periodic and marathon evals, CSO release, platform qualification and smaller gates
│ └── docker/ # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium)
├── contrib/ # Contributor-only tools (never installed for users)
│ └── add-host/ # /gstack-contrib-add-host skill
+284 -70
View File
@@ -184,10 +184,9 @@ This does not change the separate CI machine count. Full-suite shards are packed
DURATIONS (LPT, `packShardsByDuration`) when the committed seed
`scripts/free-test-durations.json` exists — refresh it with
`bun run test:ubicloud --record-durations`, which times each file in its own
child on a VM with the CI lane's environment and copies the seed back (CI never
records; a seed recorded where browser or display tests skip underestimates
them). Missing seed → silent hash-shard fallback; corrupt seed → one warning +
fallback; unknown files get 75th-percentile pessimism, and both full-suite and
child on a VM with the CI lane's environment and copies the seed back; see
[Free suite duration seed](#free-suite-duration-seed). Missing seed → silent hash-shard fallback; corrupt seed → one warning +
fallback; unknown files get 99th-percentile pessimism, and both full-suite and
`--ci-plan` runs name them on stderr so a new slow file cannot silently become
the long pole. Packed
shards get duration-aware walls (`max(base, predicted × 3, files × 5s)`). The
@@ -257,15 +256,17 @@ never retry; each case's kind fixes its trials before the run (see "Eval verdict
policy" below). Files in `CASE_SHARDED_FILES` run one registered case per
process (`<file>#<case id>`, an exact `--test-name-pattern`, exactly one executed
case), so a long file of short cases spreads across runners and each case gets
its own SDK semaphore.
its own SDK semaphore. `test/carve-section-loading.test.ts` is one of them: it registers
one periodic case per carved skill (`carve-section-loading-<skill>`, each with
its own touchfiles and history). Run one case with
`bun run scripts/test-paid-shards.ts --tier periodic --case carve-section-loading-<skill>`.
Trial telemetry rides the store: every recorded test carries its 1-based
`attempt` plus, on an isolated trial shard, its `case_id`, `kind`, `trial`,
`panel` and `policy_version`, and each lane's report uploads one
`trial-outcomes` JSONL line per trial. `bun run eval:pass-rates`
(`eval:flake-rank` is an alias) turns that history into per-case pass rates
(see "Pass-rate history" below; the free lane's flake ledger is folded in from
`flakeLedgerPath()` — override with `GSTACK_FLAKE_LEDGER`, the same env var the
CI free lane sets before uploading the ledger as the `flake-ledger` artifact). Census integrity is
turns that history into per-case pass rates
(see "Pass-rate history" below; free-suite flakes are a separate record, see
[Flake ledger](#flake-ledger)). Census integrity is
enforced from the free suite: every `E2E_TOUCHFILES` / `LLM_JUDGE_TOUCHFILES`
key must name a living paid test (`test/touchfiles.test.ts`'s reverse
invariant), and `git show <sha>:path` fixtures are banned — vendor the bytes
@@ -291,7 +292,8 @@ are not collector results and do not establish that an unfinished test passed.
retains the broad census; no case IDs or tier assignments are removed. The plan
records both E2E and judge selections and lists deferred coverage. Executors
receive those selections separately and the report checks actual executed case
counts. Unknown source dependencies restore the broad gate; unmapped prompts
counts. Unknown source dependencies restore the broad gate (see
[PR paid lane fallback](#pr-paid-lane-fallback)); unmapped prompts
without registered coverage require full validation. Known broad-only prompt
changes are explicitly deferred, not counted as PR passes. Weekly/manual runs of
`evals-periodic.yml` execute both complete censuses fresh; manual `evals.yml`
@@ -342,10 +344,22 @@ into the fullest slice whose estimated wall on J FIFO workers stays within S
seconds, else a new slice; a shard with no recorded wall weighs the whole budget
(its own runner), a shard longer than the budget runs alone, and overlays keep
one final one-at-a-time slice. The manifest's `plan` records each slice estimate
and `ciTimeoutMinutes` (every slice's supervised worst case plus 20 minutes
setup); CI derives the matrix (`[range(1; .sliceCount + 1)]`) and job timeout
from it, and executors refuse an `EVALS_JOBS` other than the planned J and run
their shards longest first. `--slices K` keeps the supervised round-robin
and its own CI job ceiling in `sliceCiTimeoutMinutes`: max(2 × slice budget, the
slice's longest shard supervised wall, the serialized overlay envelope) + 20
minutes setup (`CI_SETUP_ALLOWANCE_MINUTES`); `ciTimeoutMinutes` is the largest of
them. CI derives the matrix (`[range(1; .sliceCount + 1)]`) from the manifest and
indexes each slice's ceiling as its job timeout
(`fromJSON(needs.plan-slices.outputs.slice_timeouts)[matrix.slice]`). Executors
refuse an `EVALS_JOBS` other than the planned J and run their shards longest first.
Each executor's first step records its job start (`GSTACK_SLICE_JOB_STARTED_AT`).
The slice deadline is that start + the slice ceiling − a 5-minute upload reserve:
after it the executor starts no more shards. A shard that never started reports
`not_run` (INFRA, `never-started`); a shard still running is killed and reports
`hung` (TIMEOUT, `timed-out`). The executor checkpoints `slice-N.json` after every
shard start and finish, so a job killed before its final write reports from the
checkpoint (`slice N/M ended before its final result …: TIMEOUT`), never as a
missing slice. `--slices K` keeps the supervised round-robin
baseline re-packed by recorded times for local runs. Durations are recorded per
tier (a file's gate and periodic cases differ); refresh one tier from a
downloaded report directory with `--report <dir> --write-durations`. Under `EVALS_ALL` the hollow-shard guard marks exit-0 shards with
@@ -373,8 +387,10 @@ the runner parent and handed to shard children as `GSTACK_CLAUDE_CLI_VERSION`
(never spawned on a test thread), so a TUI-drift flake hunt is a grep, not
archaeology.
**Eval verdict policy** (`EVAL_POLICY` version 1 in
`test/helpers/periodic-exclude-data.ts`, pre-registered 2026-09-29). Paid evals
**Eval verdict policy** (`EVAL_POLICY` version 2 in
`test/helpers/periodic-exclude-data.ts`; version 1 pre-registered 2026-09-29,
version 2 approved 2026-10-04 with the decision memo in
[test-audit-2026-10.md](test-audit-2026-10.md)). Paid evals
never retry. Each live case has exactly one kind in `E2E_KINDS`
(`test/helpers/touchfiles-data.ts`; `test/eval-kinds.test.ts` enforces coverage),
and the kind fixes its trials before the run:
@@ -429,14 +445,19 @@ least 97% over at least 10 trials) without being removed, when an entry is 8
weekly runs old, or when a tier is over its cap.
**Pass-rate history** (`bun run eval:pass-rates`, `scripts/eval-flake-rank.ts`).
It reads the `trial-outcomes` artifact of the last N completed
`evals-periodic.yml` runs on the current branch and `main` (flags: `--case`,
`--runs N`, `--branch`, `--dir`, `--backfill`, `--json`, `--gate`) and prints
It reads the `trial-outcomes` artifact of the last N completed weekly
`evals-periodic.yml` runs, meaning scheduled runs on `main` plus `main`
dispatches, plus completed branch census runs in the same window (flags:
`--case`, `--runs N`, `--branch` to inspect one branch, `--dir`, `--backfill`,
`--json`, `--gate`). A branch trial counts only toward a series `main` has also
run (same case-owned bytes, `HARNESS_VERSION`, model and CLI); a branch never
starts or becomes a case's current series, and weeks for quarantine expiry
count `main` runs only. It prints
per-case per-trial pass rates with 95% Wilson intervals. A series is one case
under one input identity, the hash of its own touchfiles minus
`GLOBAL_TOUCHFILES` (harness edits do not restart it), per model, Claude CLI
version and policy version; a change starts a new series and older ones stay
visible. Labels: INCONCLUSIVE below 10 trials, BROKEN when the latest run is
under one input identity ([Harness version](#harness-version)), per model,
Claude CLI version and policy version
([Pass-rate policy versions](#pass-rate-policy-versions)); a change starts a
new series and older ones stay visible. Labels: INCONCLUSIVE below 10 trials, BROKEN when the latest run is
0/n after a prior interval at or above 95%, FLAKY when failures leave the
interval straddling 95%, FAILING when the whole interval is below it, PASSING
otherwise. `--backfill` imports legacy slice artifacts as pre-policy trials
@@ -530,19 +551,20 @@ its source and policy identifier. Custom drivers must resolve each job instead
of passing their ordinary 1800-second default as an explicit cap;
their outer controller/detach wall must also cover the allocated work and cleanup.
The paid census counts are printed by `--list` for each tier.
`eval:bg:pr` and `eval:bg:periodic` have 92820/67380-second outer caps, above their recomputed floors (PR fallback 78,425 s, periodic 33,821 s including the trial shards); the PR
wrapper covers a full-gate fallback at its default two workers. The broad gate
wrapper reserves 49320 seconds (floor 21,725 s), and release reserves 116700 seconds for both
tiers; free tests recompute each floor from the live shard census, case shards
included. Legacy monolithic
`eval:bg`/`eval:bg:all` retain their shorter 5400/7200-second caps; use the
sharded periodic path for complete coverage.
`eval:bg:<pr|gate|periodic|release>` runs `scripts/eval-bg.ts`. On the local
backend its outer cap is ceil(1.5 × planned serial seconds / `EVALS_JOBS`) + 20
minutes, at most 4 hours, computed from `scripts/paid-test-durations.json` for
the lane's current plan (`--timeout SECS` overrides). On the dispatch backend the
CI job ceilings above bound the run, and the local follower stops after 4 hours.
`test/eval-detach-timeout-floor.test.ts` pins the cap formula, the 4-hour
ceiling, the backend choice, the sentinel on both backends, the conclusion
mapping and dispatch dedupe.
CI plans with `--slice-budget 540 --jobs 2` for the PR gate, the periodic census
CI plans with `--slice-budget 420 --jobs 2` for the PR gate, the periodic census
and the weekly gate census (the gate census also `--skip-judges`), and
`--slice-budget 1 --jobs 1` (one file per runner) for marathon. The live plans
must fit their workflow's `max-parallel` so every slice starts at once, and
`ciTimeoutMinutes` must stay within 360; `test/evals-workflow-wiring.test.ts`
must fit their workflow's `max-parallel` so every slice starts at once, and every
per-slice ceiling must stay within 360 minutes; `test/evals-workflow-wiring.test.ts`
recomputes both from the complete census. Reconciliation rejects missing,
duplicated or misplaced registered work, absent budget records, case shards that
did not execute exactly their case, and reused results outside the PR profile.
@@ -590,6 +612,35 @@ against a temp `GSTACK_INSTALL_DIR` / `GSTACK_SKILLS_DIR`, and
`freeze/bin/check-freeze.sh` with JSON payloads on stdin (including the
`GSTACK_HOME` state-root parity against `bin/gstack-paths`).
### Pass-rate policy versions
Readers score only trials recorded under their own `EVAL_POLICY.version`.
Trials from an older policy are shown display-only. Trials from a newer policy
are ignored with a printed count ("ignored N trial(s) recorded under a policy
newer than vX"). When you see that line, run pass-rates from a checkout at or
after the commit that bumped the version. The supported rollback of a policy
change is reverting its writer commit; the reader-compatibility commit before it
stays.
### Harness version
Under `EVAL_POLICY` v2 a case's series identity is the bytes the case owns (its
paid test file, its fixtures and its skills' prompt files, including the
generated SKILL.md) plus `HARNESS_VERSION`, keyed with model, Claude CLI version
and policy version (`scripts/eval-trial-series.ts`). Each trial also records the
full consumed-input fingerprint as provenance. `scripts/harness-version.json`
pins every shared-harness file (`GLOBAL_TOUCHFILES` plus the session runners, the
PTY harness, the e2e gate and the LLM judge) by blob hash. When
`test/harness-version.test.ts` fails, run one of:
```bash
bun run scripts/bump-harness-version.ts --non-behavioral "<why no verdict can change>" # no series reset
bun run scripts/bump-harness-version.ts --bump "<what behavior changed>" # new series for every case
```
On a merge conflict in the manifest, take the higher version, then rerun the
script.
## Ubicloud VMs (`bun run test:ubicloud`)
`bun run test:ubicloud [test:free args]` runs the free suite on an ephemeral
@@ -605,8 +656,8 @@ checkout to it: tracked files, untracked files that are not ignored, and
`.git`, so uncommitted edits are tested. It then runs
`scripts/ubicloud/setup-free-suite.sh`, which mirrors the `free-suite` CI job
(same Bun pin, Playwright Chromium with its setuid sandbox helper, Xvfb,
poppler, emoji fonts, generated host outputs, gate binaries, and the CSO
helper), and runs `xvfb-run -a bun run test:free` with `GSTACK_EXPECT_BINARIES=1`
poppler, emoji fonts, zsh for the bash+zsh portability arms (#2669), generated
host outputs, gate binaries, and the CSO helper), and runs `xvfb-run -a bun run test:free` with `GSTACK_EXPECT_BINARIES=1`
and `GSTACK_FREE_RETRY_FLAKY=1`. Shard logs are copied to
`.context/ubicloud/<timestamp>/`, and the VM is destroyed on every exit path.
The exit status is the suite's.
@@ -646,17 +697,19 @@ serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so
a broken tree can't masquerade as flaky). The required CI free lane and the
Windows lane set the retry knob too, appending every flaky pass to the JSONL ledger it uploads
(`GSTACK_FLAKE_LEDGER`) — a flaky pass never reds the lane, but it never
disappears either.
disappears either ([Flake ledger](#flake-ledger)).
## Test selection and tiers
Moved verbatim from CLAUDE.md (#2096 size limit).
**Diff-based test selection:** `test:evals` and `test:e2e` auto-select tests based
on `git diff` against the base branch. Each test declares its file dependencies in
`test/helpers/touchfiles.ts`. Changes to global touchfiles (session-runner, eval-store,
touchfiles.ts itself) trigger all tests. Use `EVALS_ALL=1` or the `:all` script
variants to force all tests. Run `eval:select` to preview which tests would run.
**Diff-based test selection:** the sharded paid runner (`test:pr`, `eval:bg:pr`,
`test:gate:sharded`) selects tests based on `git diff` against the base branch.
Each test declares its file dependencies in `test/helpers/touchfiles.ts`. Changes
to global touchfiles (session-runner, eval-store, touchfiles.ts itself) trigger
all tests. Use `EVALS_ALL=1` to force all tests. Run `eval:select` to preview
which tests would run; it applies the PR profile by default, and
`--profile full` shows the plain touchfile selection. Commands by task: [CONTRIBUTING.md](../CONTRIBUTING.md#which-command-do-i-run).
**Two-tier system:** Tests are classified as `gate` or `periodic` in `E2E_TIERS`
(in `test/helpers/touchfiles.ts` — a facade over `touchfiles-data.ts` +
@@ -680,14 +733,61 @@ in `bun test`): a `skill-e2e-*` file named in a touchfiles dep list whose
suite. Files not named in any dep list are reported, not enforced — keep both
in sync.
### PR paid lane fallback
PR selection (`scripts/test-pr-profile.ts`) classifies every changed file. A file
mapped by a paid case's touchfiles selects that case. These files are exempt and
select nothing: docs, any non-paid `*.test.ts(x)`, `FREE_ONLY_PR_FILES` (the
free and paid duration seeds, `scripts/ubicloud/**`, every workflow that runs no
paid evals, and free-only tools), and the fixtures in `FREE_FIXTURES`
(`test/helpers/free-fixtures-data.ts`, each fixture with its free consumers).
`PAID_WORKFLOW_FILES` (`evals.yml`, `evals-periodic.yml`, `evals-marathon.yml`)
and any other unknown file restore the full gate (`prCoverage.mode =
full-fallback`). The job summary and PR comment list each file that caused a
fallback with one of three labels:
- **needs touchfile entry**: a paid test, helper or fixture no case maps; add it
to the owning case in `E2E_TOUCHFILES` (`test/helpers/touchfiles-data.ts`).
- **add to FREE_ONLY_PR_FILES**: a file only free lanes read; add it to the list
in `scripts/test-pr-profile.ts` (or a fixture to `FREE_FIXTURES`).
- **real unknown dependency**: shared code (bin/, lib/, resolvers, hosts, setup)
whose dependents are not mapped; the full gate is the correct result.
`test/free-fixtures.test.ts` holds the inventory: each `test/fixtures/**` path
belongs to a paid touchfile or `FREE_FIXTURES`, and each workflow to the paid or
free-only list. Editing a paid test file runs its own gate cases on the PR when
their Bun test is named by the case id; the file's other cases stay deferred with
a reason that says to name the test by its case id.
### PR receipts, recovery and debounce
Passes and failures of single-trial shards are written as receipts at the
execution boundary, so a cancelled run's finished work survives. The
`recover-receipts` job runs `scripts/recover-receipts.ts` from the base ref with
`actions: read`: it merges the receipts of at most 5 completed, cancelled
`evals.yml` runs of the same PR (oldest first, negatives before passes, so a newer
FAIL blocks an older PASS) and hands the store to `plan-slices`, which keeps
`contents: read` only. Its `recovery.json` checkpoint never advances past an
unresolved run. `plan-slices` turns reuse off when recovery degraded, the
checkpoint is missing or a cancelled run is still unresolved; recovery itself
never fails the run, and the job summary names the reason (rate limit with its
reset time, `gh auth status`, the 60-second budget). The whole store is saved by
`slices-report`.
On a `synchronize` push, when another run for the same PR started in the last 15
minutes, the debounce job waits 90 seconds before planning; a run that outlives
the wait reports "superseded by <sha>". PR labels: `evals-no-debounce` skips the
wait and `evals-fresh` turns receipt reuse off. Labels apply from the next push,
because adding a label does not start `evals.yml`.
## Free suite runner, judge reuse and engine skips
Moved verbatim from CLAUDE.md (#2096 size limit).
`bun run test` routes through `scripts/test-free-shards.ts` (N concurrent
shard processes, serial within each, packed by recorded per-file durations
when `scripts/free-test-durations.json` exists — refresh occasionally with
`bun run test:free --record-durations`; strict-output classification per
when `scripts/free-test-durations.json` exists — refresh it with
`bun run test:ubicloud --record-durations`; strict-output classification per
shard: a shard without bun's terminal summary line FAILS — silent truncation
cannot report green). `TREE_MUTATING` lists the files that still run in their
own trailing serial shard (today only `test/bootstrap-retention.test.ts`, for
@@ -736,6 +836,105 @@ Fix a writer with `usePrivateStateRoot()` (`test/helpers/private-state-root.ts`)
or a child HOME/GSTACK_HOME, and resolve product state paths at write time,
never at import.
### Flake ledger
`GSTACK_FREE_RETRY_FLAKY=1` reruns a failing free test file once, serially. A
clean retry turns the shard green as a flaky pass and appends an entry to that
shard's flake ledger, uploaded as `flake-ledger-<shard>` (90-day retention). The
Monday "Weekly test health" workflow (`bun run test:health --enforce`) reads
main's ledgers through `scripts/lib/ci-history.ts` and fails when a file flakes
in more than 5% of main free-tests runs over a window of at least 20 runs; a
window thinner than 20 runs in the last 7 days widens to the newest 20 runs, and
fewer than 20 runs is report-only. `bun run test:health flake-summary <dir>`
prints a run's ledgers as a table. A flaky file is fixed at source; the retry
never masks it. `bun run eval:pass-rates` lists only the local ledger
(`flakeLedgerPath()`, overridden by `GSTACK_FLAKE_LEDGER`).
The aggregate job of `free-tests.yml` downloads every `flake-ledger-*` artifact
of the run and appends `bun run scripts/test-health-report.ts flake-summary <dir>`
to its job summary. The Windows lane uploads `flake-ledger-windows-<shard>`.
Free-suite flake rules:
- Bun gives a child `stdio: 'pipe'` a unix socketpair. A test that must hold a
write blocked uses a FIFO the test never reads (`test/qa-evidence.test.ts`,
`test/qa-deadline.test.ts`), not a paused child stream: socket capacity follows
`net.core.{w,r}mem_default` and the parent's reader.
- A test that rewrites CI provenance variables (`CI`, `GITHUB_*`, `CI_*`)
restores them. Free shards run many files in one process, and a later browse
daemon without `CI=true` launches Chromium sandboxed.
- Handshakes with a runner wait for the runner's own observable effect (its
progress line, a coalesced read count), not for a fixture side file or a timer.
### Free suite duration seed
`scripts/free-test-durations.json` drives shard packing; files missing from it
pack at the seed's 99th percentile. Acceptable seeds come from
`bun run test:ubicloud --record-durations` (each file timed in its own child on a
VM with the CI lane's environment) or a CI recording. A 4-core laptop or
small-sandbox recording is not acceptable: browser and display tests skip or slow
down there, so the seed misstates them.
- `--ci-plan` warns in the job summary, with the refresh command, when more than 5
files are unseeded; it never fails a PR. `--ci-verify` lists shards that ran
longer than 1.5× their prediction and at least 30 seconds over.
- The weekly test-health run fails its enforced `unseeded-free-files` check
("stale seed: N unseeded file(s) > 5") when more than 5 free test files are
missing from the seed, so contributors without Ubicloud access learn within a
week that it needs a refresh.
- `test/free-seed-ratchet.test.ts` (run with full history in `free-tests.yml`'s
`free-plan` job, `GSTACK_FREE_SEED_BASE=$(git merge-base HEAD origin/main)`)
fails when a seeded file over 60 seconds is missing from
`scripts/free-test-seed-allowlist.json`, when an allowlist entry is stale (gone,
or no longer over 60 seconds), or when an entry is added relative to the
merge-base copy without a `todo`. An annotated exception is
`{file, reason, todo}` and the test prints it. A merge-base without the
allowlist file counts as the first allowlist. Fix a failure by splitting the
file, then rerun `bun run test:ubicloud --record-durations`.
### Windows free lane
`windows-free-tests.yml` plans on Linux (`--windows-only --ci-plan --shards 6`),
runs one strict serial shard per `windows-latest` job
(`bun run test:windows --ci-run`), and verifies every shard receipt on Linux
(`--windows-only --ci-verify`). Packing uses Windows-measured durations from
`scripts/free-test-durations-windows.json`; refresh them with
`gh workflow run windows-free-tests.yml --ref <branch> -f record_durations=true`,
then download the `free-test-durations-windows` artifact and commit it.
Curation lives in `scripts/lib/windows-curation.ts`. Every Windows plan and
`--list` prints the curated count and the top exclusion reasons (also in the job
summary). Files a probe proved safe despite a `/tmp` literal are force-included
(`WINDOWS_PROBE_SAFE`). `test/windows-native-workflows.test.ts` holds
`WINDOWS_CURATED_FLOOR`; when Windows-safe tests are deleted on purpose, lower it
in the same PR with a reason.
Native qualification campaigns (Windows cookie extraction, Windows launch
diagnostics, Dia on macOS) run only on dispatch of `native-qualification.yml`:
```bash
gh workflow run native-qualification.yml --ref <branch> -f mode=<cookie-native|windows-native-diagnostics|dia-native|dia-launch-comparison|dia-gui-readiness>
```
### Real-setup install fixture
Tests that run the real `setup` use `test/helpers/install-fixture.ts`. Each test
file builds one read-only seed checkout and clones it per test with
copy-on-write file copies (plain copy fallback), never hard links. Writes go
through `put`, `setVersion` and the `fixture*Sync` helpers; files register
`afterEach(cleanupFixtures)` and `afterAll(cleanupSeed)`.
`test/install-fixture.test.ts` is the isolation tripwire: writes in one clone
leave the seed and sibling clones unchanged.
### Test-only timing knobs
Defaults are the product's; tests shrink them instead of waiting real time:
`GSTACK_RENDER_SLACK_MS` (`lib/aside-render.ts`, default 10000),
`GSTACK_MEMORABLE_LOCK_TRIES` (`bin/gstack-memorable`, default 50 tries of 0.1 s),
the `recordGraceMs` option of `test/helpers/office-hours-attempt.ts` (default
5000) and the `timeoutMs` argument of `clearCookieTargetStorage(page, origin,
timeoutMs)` (default 15000).
## Running evals as an agent: detach
Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in
@@ -745,36 +944,51 @@ When **you (an agent/harness)** launch a long eval/benchmark run, run it through
`bin/gstack-detach` — NEVER as a plain backgrounded Bash task. A plain background
task lives in the harness's process group, so a SIGTERM ("polite quit") on a turn
boundary, a stopped Monitor, or an interruption kills the run mid-flight (observed:
`script "test:gate" was terminated by signal SIGTERM` ~40 min into a run). On macOS
`script "test:gate" was terminated by signal SIGTERM` ~40 min into a run of the
since-retired single-process runner). On macOS
the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
(escapes the group SIGTERM) wrapped in `caffeinate -i` (blocks idle-sleep).
- Use the `eval:bg*` scripts (`eval:bg`, `eval:bg:all`, `eval:bg:gate`,
`eval:bg:periodic`) — they wrap the eval command in `gstack-detach` with the
machine-wide `gstack-evals` lock (concurrent worktrees serialize instead of
saturating the shared model API), a per-tier watchdog, and a **run-scoped** log
under `~/.gstack-dev/eval-runs/` (no shared-`/tmp` collision). Each prints its
log path. `eval:bg:gate` / `eval:bg:periodic` run their tier through the
sharded paid runner (`scripts/test-paid-shards.ts`, also exposed as
`test:gate:sharded` / `test:periodic:sharded`): one Bun process per test
file, an external wall-clock timeout that kills the shard's process GROUP
(stray `claude`/`codex` grandchildren included), a per-shard
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
constructor, and an aggregate that separates failed vs timed-out vs
never-started shards — the detach timeouts (the `--timeout` values on
package.json's `eval:bg:gate` / `eval:bg:periodic`;
floor enforced against the live shard census by
test/eval-detach-timeout-floor.test.ts)
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
process count (default 8); `EVALS_CONCURRENCY` is bun's --max-concurrency
WITHIN a shard (default 2) — they are deliberately separate knobs. `eval:list` / `eval:compare` /
`eval:summary` / `eval:flake-rank` read the shard dirs too. Or call
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] --
<cmd>` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never
pass keys in argv).
- Use `bun run eval:bg:<pr|gate|periodic|release>` (`scripts/eval-bg.ts`). It
prints `[eval-bg] backend=<dispatch|local> …` and `gstack-detach LOG <path>` and
returns at once. Both backends run under `gstack-detach` with the machine-wide
`gstack-evals` lock (concurrent worktrees serialize instead of saturating the
shared model API) and a **run-scoped** log under `~/.gstack-dev/eval-runs/`
whose first line names the backend, the run and the tested revision.
- **Dispatch** (default when HEAD is clean and equal to its pushed branch on
`garrytan/gstack`, checked with `gh api`, and `gh` can dispatch): `pr` and
`gate` dispatch `evals.yml` (`evals_all=false` with `base_ref`/`base_sha` for
`pr`, `evals_all=true` for `gate`), `periodic` dispatches `evals-periodic.yml`,
`release` dispatches both. Every dispatch passes `expected_sha` (the workflow
fails fast when the checked-out revision differs) and a `nonce` that appears
in the run name (`E2E Evals [eval-bg <nonce>]`), which is how the log finds
and follows its run. A queued, running or green dispatch with the same
revision and inputs is followed instead of dispatched again. Dispatched runs
are validation runs: fresh and never writing PR receipts, except that a `pr`
dispatch on a branch with an open same-repo PR passes `pr_receipts=<PR>` and
reads that PR's receipts read-only.
- **Local** (dirty tree, unpushed commits, a fork, no `gh`, or `--local` /
`GSTACK_EVAL_BG_MODE=local`): the sharded paid runner
(`scripts/test-paid-shards.ts`) on this machine, capped at
ceil(1.5 × planned serial seconds / `EVALS_JOBS`) + 20 minutes, at most 4 hours
(`--timeout SECS` overrides). One Bun process per test file, an external
wall-clock timeout that kills the shard's process GROUP (stray
`claude`/`codex` grandchildren included), a per-shard
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
constructor, and an aggregate that separates failed vs timed-out vs
never-started shards. `EVALS_JOBS` sets the shard process count (default 8,
2 for `pr`); `EVALS_CONCURRENCY` is bun's --max-concurrency WITHIN a shard
(default 2). Export `ANTHROPIC_API_KEY` first (never pass keys in argv).
- `--dispatch` refuses with the fix ("commit and push, or rerun with --local")
instead of falling back.
`eval:list` / `eval:compare` / `eval:summary` / `eval:pass-rates` read the shard
dirs too. For any other long agent job, call
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] -- <cmd>` directly.
- Then **poll the printed logfile** with a death-aware watcher: break on the
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (success AND failure are
both marked, so silence is never mistaken for success). The detached run survives
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (0 passed, 1 failed,
130 cancelled, 2 other on both backends; silence is never mistaken for success).
`bun run scripts/eval-bg.ts status <log-or-run-id>` reconnects and prints
running, passed, failed, cancelled or incomplete (died without a sentinel). The detached run survives
even if your watcher gets reaped, so re-checking the log always works. Keep
checking until the sentinel appears or the user tells you to stop; a long run is
expected, and a promise to check later is not a result. At each check, report
@@ -782,5 +996,5 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
- Why the lock: a shared dev box with several Conductor worktrees will rate-limit
the model API if two eval suites run at once (15-way concurrency each), which
mass-times-out E2E tests. The lock makes the second run WAIT, not collide.
- Humans running `bun run test:evals` foreground in their own terminal don't need
- Humans running `bun run test:pr` foreground in their own terminal don't need
this — Ctrl-C is intended there. Detachment is for agent-launched runs only.
+12 -4
View File
@@ -26,6 +26,12 @@ The dedicated Windows CI job checks startup, argument forwarding, and runtime
injection separately from static audit and private-state support. POSIX
watchdog execution remains unavailable there.
No qualified runtime or scanner image exists yet, so contained runtime and
scanner execution is unavailable and setup preloads nothing: the scanner
catalog is empty while its build inputs await review, and the runtime catalog's
profiles stay unqualified until a private evaluator, still being built, can
qualify them.
Every successful fresh install or upgrade asks the trusted launcher to preload
the current host platform's qualified runtime and scanner images. This is the
only automatic image acquisition path. It first validates the committed
@@ -48,10 +54,12 @@ egress. `gstack-cso doctor --repo PATH` never downloads; it reports a qualified
profile unavailable until that exact digest and platform are already present
in the local daemon.
The ordinary free suite exercises contracts without requiring Docker. The
`cso-docker-integration` CI job sets `GSTACK_CSO_DOCKER_TESTS=1`; unavailable Docker
or missing containment prerequisites fail that job. Run it locally with a
working local daemon:
The ordinary free suite exercises contracts without requiring Docker. The free
lane's `cso-docker-integration` job sets `GSTACK_CSO_DOCKER_TESTS=1` and runs
`test/cso-docker-integration.test.ts`; unavailable Docker or missing containment
prerequisites fail that job. The node-lifecycle and stack-cold files need a
staged stack image, so they run in the CSO runtime workflows through
`test:cso:docker`. Run all three locally with a working local daemon:
```bash
bun run build:cso
+3 -2
View File
@@ -815,8 +815,9 @@ A mechanical pre-filter shortlists assertion-free probes, source greps, export-l
copies and near-duplicate files before any model reading. Each candidate gets a
retirement card (what it detects, non-test callers with the search command, the
stronger remaining proof, history, what retiring it unlocks, and the validation
command). Contract tests such as SKILL.md goldens and prompt-byte checks are
retained. The report and a JSON sidecar land in `~/.gstack/projects/<slug>/`.
command). Contract tests such as SKILL.md goldens and prompt-byte checks
(machine-read tokens, not English sentences; see [the value bar](test-value-bar.md))
are retained. The report and a JSON sidecar land in `~/.gstack/projects/<slug>/`.
Nothing is edited unless you approve a batch; spawned sessions stay report-only.
Tests marked `gstack:test-value keep reason="..."` are skipped and listed in the
report's appendix.
+18
View File
@@ -476,3 +476,21 @@ release docs sweep); tests 383 (+9,375/−44,379); test helpers 47 (+786/−12,7
- **`test/helpers/claude-pty-runner.unit.test.ts` (3,694 LOC, 223 tests, 0.1 s):** a large self-test of the 5,885-LOC PTY harness. The classifiers it covers (`classifyVisible`, `parseNumberedOptions`, `Step0BoundaryPredicate`…) are live in paid runs, so keep it. The per-capture blocks (`captured F`, `captured G`) belong to the per-incident consolidation lane.
- **Zero-importer helpers** `auq-parallel-worker`, `setup-gbrain-fixture-command`, `emulate-bun-windows-eexist` are loaded by path (preload / generated import). `benchmark-judge` has a production caller (dynamic import in `bin/gstack-model-benchmark`).
- **`browse/src` `__reset*`/`reset*ForTests` exports (12):** standard singleton-reset seams, keep.
## Corrections (2026-10 audit)
Facts above that no longer hold, checked against main at v1.91.19.0:
- `codex-e2e-recommendation-substance` is not census-only: it is no longer in
`PERIODIC_CI_EXCLUDE` and runs in every periodic census.
- The after-metrics are superseded. Five days after release the free suite had
regrown by 132 test files and about 53% of serial time; the
[2026-10 audit](test-audit-2026-10.md) carries current numbers.
- The duration seed again lacked 19 files, including the 115-second
`claude-overlay-setup.test.ts`, which the planner packed as a short file.
- `browse/test/compare-board.test.ts` stayed quarantined through this audit; its
16 tests now pass headless and run in the free suite again.
- `ci-image-tag-binding` now pins `evals.yml`, `evals-periodic.yml` and
`evals-marathon.yml`; `ci-image.yml` is deleted because those workflows push
the same content-hash tag.
- The weak literal pins listed under Lane 5 are still present.
+276
View File
@@ -0,0 +1,276 @@
# Test, eval and CI audit — October 2026
This document records the evidence behind the October 2026 test, eval and CI
wave: the EVAL_POLICY v2 decision memo (D1) with its pre-registered backtest,
and the commands that reproduce every number. Metrics that need post-merge
traffic are measured with `bun run test:health --since-days 7` (merge + 7 days)
and `bun run test:health --since-days 28` (after the fourth scheduled census).
## EVAL_POLICY v2 decision memo (D1)
### Why the census is red
The weekly census is red by arithmetic, not by a handful of broken tests.
Over 22 censuses with trial outcomes the periodic lane recorded 76 failed
trials in 3,550 (about 3.5 failed trials per run) and the gate census 27 in
2,241 (about 1.2 per run). With independent failures, P(green) is e^-3.5 ≈ 3%
for the periodic lane and e^-1.2 ≈ 30% for the gate census; observed green
rates were 0/22 and 8/22. Holding a ~190-case single-trial lane green 80% of
the time needs every case at or below about 0.12% failure per run, which a
live agent judged by one trial does not reach. Fixing the top five chronic
cases leaves the lane red.
Three levers stay inside the no-retry, no-threshold, no-rejudge rule: make
detectors deterministic over structured artifacts (W3), classify INFRA
honestly (W2d, owned by the reliability follow-ups), and make quarantine
reachable for detector, harness and model-latency failures. v2 targets the
third lever.
`test:health` reports the census as three separate numbers, so a policy
change cannot read as a reliability gain: the raw trial failure rate per kind
and failure class, P(lane red) from pooled per-case rates under the recorded
panels, and the same evidence re-aggregated with every case judged on one
trial. Both red probabilities treat every observed failure as
nondeterminism, so they are upper bounds for known-good inputs.
**False-blocking targets.** Periodic lane: at most 1.0 expected failed trial
per census (P(red) ≤ 63%) at the 28-day check; gate census: at most 0.5
(P(red) ≤ 39%). Neither target may be reached by a threshold, retry or
rejudge change. A miss opens a P1 TODOS entry with the per-kind,
per-failure-class breakdown from `test:health`.
### What v2 changes
- **Series identity (W4a, CEO-27, ENG-3).** A case's pass-rate series is keyed
by the bytes it owns plus `HARNESS_VERSION`: its paid test file, its
fixtures, and the prompt files of the skills it names (templates, sections,
manifests and the generated `SKILL.md` the model reads). The series key adds
the model id, the pinned Claude Code CLI version and the policy version.
Shared helpers no longer start new series. Every trial also records the
full consumed-input fingerprint (all touchfiles, globals included) as
`series_fingerprint`, plus `harness_version`, as provenance.
(`caseSeriesIdentitiesV2` in `scripts/eval-trial-series.ts`.)
- **Harness guard.** `scripts/harness-version.json` pins the blob of every
shared execution-harness file: `GLOBAL_TOUCHFILES` plus the session runners,
the PTY harness, the e2e gate and the LLM judge (36 files). A free test
(`test/harness-version.test.ts`) fails when one changes without a decision.
`bun run scripts/bump-harness-version.ts --bump "<reason>"` raises
`HARNESS_VERSION` and starts a new series for every case;
`--non-behavioral "<reason>"` records the new blobs as compatible with no
reset. Entries are one line per file so concurrent PRs that touch different
files merge cleanly; on a conflict, take the higher version, then rerun.
- **Weekly history (W4b).** Weeks for quarantine expiry and drift are
`evals-periodic.yml` runs on `main`: scheduled runs and main dispatches.
Trials from branch census runs in the same window pool into a series `main`
has also run (option (b) below, approved 2026-10-05); a branch identity
`main` has not run is dropped and never becomes the current series.
- **Kinds (W4c).** No kind changes; see the evidence below.
- **Pre-v2 evidence does not qualify.** Readers segment strictly by
`policy_version`: only v2 trials count toward the quarantine entry rule; v1
trials stay visible as display-only. Quarantine is not seeded at merge; a
case enters only after it reaches `entry.minTrials` qualifying v2 trials,
and `test:health` shows qualifying-vs-required counts with "insufficient
evidence" distinct from healthy (DX-12).
### W4c: the named "stochastic" cases are deterministic in the record
Recorded census and PR trials (all 95 recorded revisions, October 2026):
| Case | Kind | Passed / trials | Failures |
|---|---|---|---|
| plan-ceo-finding-floor | rule | 26/27 | 1 timeout (608 s) |
| plan-devex-finding-floor | rule | 27/27 | — |
| plan-eng-finding-floor | rule | 9/9 | — |
| plan-design-finding-floor | rule | 9/9 | — |
| plan-ceo-review-plan-mode | rule | 34/34 | — |
| plan-devex-review-plan-mode | rule | 27/27 | — |
| plan-eng-review-plan-mode | rule | 9/9 | — |
The comments calling `plan-*-finding-floor` "stochastic ask-first" are stale:
the record shows these cases behave like rules. AGENTS.md forbids changing a
kind without pass-rate evidence, and a 3-trial panel would triple their cost
for no measurable gain, so they stay `rule`. `plan-design-review-plan-mode`
(17/22) is handled by its detector repair, not by a kind change.
### Backtest (ENG-3, DX-12)
`bun run scripts/eval-trial-series.ts --backtest --weeks 30 --records <dir>`
replays identities with today's registry over historical trees (cases whose
test file did not exist at a revision are absent there).
**Weekly main revisions** (29 Mondays, 06:00 UTC, 2026-03-16 to 2026-09-28).
A case qualifies at a week when one identity holds long enough to collect 10
trials within the last 10 weekly runs (rule and judge cases record one trial
per run, behavior cases a panel of three):
| Identity | Qualifying at the latest week | Mean qualifying share, last 8 weeks | Median identity changes per case |
|---|---|---|---|
| v1 (own touchfiles minus globals) | 0/195 | 0.3% | 15 |
| v2 (case-owned bytes) | 0/195 | 0.7% | 14 |
| v2, reset on every pinned-harness change | 0/195 | 0.1% | 14 |
| v2 without the generated `SKILL.md` (prototype) | 0/217 | 2.0% | 7 |
**Recorded trials** re-keyed by the v2 identity of the revision that produced
them (blocking cases with at least 10 trials on one series):
| Trial source | Recorded (v1) identity | v2 identity |
|---|---|---|
| 22 census runs (5,665 trials) | 52/210 | 67/210 |
| Census + PR lane, 95 revisions (14,898 trials) | 107/210 | 134/210 |
**Harness churn.** The pinned harness changed in 24 of 28 consecutive-week
transitions, and 33 of the last 60 main commits (2026-08-04 to 2026-10-04)
touched a pinned file. About half of PRs therefore record a
`bump-harness-version` decision, and only `--bump` decisions reset series.
### What the backtest shows
1. v2 makes series honest and longer when trials are dense: on the recorded
census runs, 67 blocking cases reach 10 trials on one series versus 52
under v1, and 134 versus 107 with PR-lane trials included.
2. Under the approved weekly history (main scheduled runs plus main
dispatches, about one run per week), no blocking case reaches 10
qualifying trials at any recent week. A rule case needs ten consecutive
weekly runs on unchanged owned bytes, and a case's own prompt files
change about every two weeks (the generated `SKILL.md` carries the shared
preamble). Excluding the generated `SKILL.md` halves identity churn but
still qualifies at most 2% of cases, and it would let preamble and resolver
prompt changes pool unless every such edit went through the harness guard.
3. So v2 alone does not make quarantine reachable. Reaching it needs one of:
(a) keep the history as approved and accept that quarantine stays rare, so
census redness is fixed at source (W3) and tracked by `test:health`;
(b) pool trials from every v2 census run whose series key matches,
including branch dispatches, while weeks for expiry still count from main
only — the recorded backtest's 67/210, at the cost of pooling branch code
whose unbumped harness edits match; or (c) extra census dispatches on an
unchanged main revision (about $175 recorded each). Garry approved (b) on
2026-10-05, and this PR implements it: pooled branch trials count only
toward a series a `main` run also has.
### Alternatives considered
- **Gating vs trended.** The native CEO voice proposed keeping live-agent
cases trended (reported, never blocking) and gating only on deterministic
detectors. v2 keeps the census blocking because quarantine and the
pass-rate gate already provide a trended path per case; the
false-blocking targets above are the check on that choice. If the 28-day
check misses them, trended reporting for the remaining stochastic cases is
the next proposal.
- **Adaptive sequential panels** (one trial, escalate to three on a failure,
about 1.04× cost across ~190 cases) would be the strongest lever for a
green census. They are not adopted: an escalation triggered by a failure is
a retry, which Garry's rule forbids.
- **Buy vs build.** Hosted eval platforms (Braintrust, LangSmith, Weights &
Biases Weave) store per-case pass-rate history and cost natively, but
gstack's verdicts are produced inside GitHub Actions from Claude Code
sessions with pre-registered panels and receipts, and the history is a few
thousand JSONL lines per week that `gh` already serves. A platform would add
an external dependency and a second source of truth for verdicts without
removing the in-repo policy. Building on the existing artifacts stays the
choice; revisit if history outgrows artifact retention (90 days).
### Rollout and rollback (ENG-14, DX-12)
The reader-compatibility commit lands first: readers score only trials under
their own `EVAL_POLICY.version`, show older-policy trials as display-only,
and ignore newer-policy trials with a printed count. The v2 writer (identity
stamping and `version: 2`) lands last. The supported rollback is reverting
the v2 writer commits; a whole-PR revert is not claimed safe.
On a mixed v1/v2 fixture (10 passing v1 trials, 12 v2 trials with 3
failures), the pre-PR reader at 2db0b3a pools the v2 records as the current
v1 series and fails its gate:
```
pass-rates: policy v1, 22 post-policy trial(s), 0 pre-policy (display only)
FAILING rule gate 9/12 [46.8%–91.1%] ... plan-ceo-review-plan-mode (baseline reset)
ACTION REQUIRED (2): [drift] ... [rule-as-behavior] ...
exit=1
```
The compatibility reader ignores the v2 records and keeps the v1 series:
```
pass-rates: policy v1, 10 post-policy trial(s), 0 pre-policy (display only)
ignored 12 trial(s) recorded under a policy newer than v1: this checkout predates them. Fix: ...
PASSING rule gate 10/10 [72.2%–100%] ... plan-ceo-review-plan-mode
exit=0
```
## Wave evidence
### PR paid lane selection replay (CEO-23, ENG-18)
The replay sends each recorded PR push's changed files through today's
selector and dependency graph:
```bash
gh run list -R garrytan/gstack --workflow evals.yml --json databaseId,headSha,event,createdAt,conclusion -L 200 > runs.json
bun run scripts/replay-pr-selection.ts --runs runs.json
```
Over 168 PR pushes with plan manifests (2026-09-25 to 2026-10-04):
| Measure | Recorded | Replayed |
|---|---:|---:|
| Full-fallback pushes | 165 of 168 (98.2%) | 164 of 168 (97.6%) |
| Misses (a failed case that depends on the diff and is no longer selected) | — | 0 |
| Failed cases that no longer exist (counted apart) | — | 6 |
113 of the 168 diffs exceed the compare API's 300-file cap and are replayed on
their first 300 files. The rate barely moves because the replay uses each push's
cumulative branch diff, and the wave branches of that week touch shared inputs
that correctly select every case: `scripts/` in 156 pushes, `.github/` in 125
(mostly the eval workflows), `bin/` in 120, `make-pdf/` in 90, `test/helpers/` in
86, `design/` in 67, `lib/` in 58, `hosts/` in 56 and `tsconfig*.json` in 37. The
selector removes the single-class fallbacks (duration seeds, free-only workflows,
`scripts/ubicloud/**`, free tests and free fixtures; run 36497566037 now selects
the `pr` profile) and cannot narrow a branch that changes `bin/` or `lib/`;
narrowing those needs touchfile mappings for `bin/`, `lib/`, `hosts/` and
`make-pdf/`. The post-merge `test:health` check measures the fallback rate on
ordinary PRs.
### Paid slices and concurrency (ENG-17)
Live census with `EVALS_ALL=1` (periodic: `--list --slice-budget S --jobs 2`;
gate census: `--emit-plan <file> --slice-budget S --jobs 2 --skip-judges`):
| Lane | Before (540 s) | After (420 s) |
|---|---|---|
| Periodic slices / peak shard processes | 26 / 52 | 33 / 66 (max-parallel 26 → 36) |
| Weekly gate census slices / peak | 12 / 24 | 16 / 32 (max-parallel 16 → 20) |
| CI job ceiling | One value for every slice: 173 min periodic, 254 min gate census | Per slice: ordinary 50 min (30-minute shard wall + 20), periodic outliers 52 and 73 min, the overlay slice 173 min (5 serialized overlay wrappers); gate census 34–82 min |
The PR lane (`evals.yml`) uses the same 420-second budget and per-slice ceilings.
### Free suite
- The Windows lane runs on six `windows-latest` jobs packed by Windows-measured
durations: about 4.6 minutes wall in the free-suite branch's validation runs,
down from 8.1 minutes on one runner.
- Unseeded files pack at the 99th percentile (was the 75th), and the seed growth
ratchet (`test/free-seed-ratchet.test.ts`) keeps files over 60 seconds on a
shrink-only allowlist.
- The bash+zsh portability tests (#2669) run in CI now that the free-suite job
installs zsh.
- Native Windows and Dia qualification campaigns moved out of the PR workflow
into the dispatch-only `native-qualification.yml`.
### Prose pins (D2)
D2 is approved: the prompt-byte contract in
[test-value-bar.md](test-value-bar.md) covers machine-read tokens only, and
sentence pins in 55 free test files became structural checks through
`test/helpers/prompt-structure.ts`. A sentence pin here is a 40-or-more-character
English string literal, or a prose regex of five or more words, asserted against
SKILL.md, template or section text in a free test.
| Measure | main (2db0b3a) | This wave |
|---|---:|---:|
| Sentence pins | 2,332 in 217 files | 1,190 in 209 files |
| Shouting pins (NEVER/MUST/ALWAYS/CRITICAL emphasis) | 8 | 4 |
The 4 remaining shouting pins are enum or marker tokens (`PLAN MODE EXCEPTION —
ALWAYS RUN`, `[CRITICAL]`, `ALWAYS-ON` CLI output), not emphasis. Most remaining
sentence pins sit in QA caller, detector-helper and workflow files that other
work in flight owns.
+21 -5
View File
@@ -20,12 +20,28 @@ OpenClaw's `test-audit` skill (openclaw/openclaw@a214e76,
- **Value bar**: the authoring gate plus the rule that a test which breaks under a
behavior-preserving refactor asserts implementation, unless its exact output is the
declared contract (goldens, prompt bytes, wire formats).
- **Prompt-byte contract**: the machine-read tokens inside a prompt, nothing wider.
A token qualifies when software consumes its exact bytes: a marker or sentinel
line a parser, hook or grader matches; a field name or JSON shape a script reads;
a command line, flag or environment variable the agent runs; an enum value or
status word a caller branches on; a file path or placeholder another step opens;
or an exact string a security boundary emits into model input. An English
sentence is not a prompt-byte contract because the model reads it: rewording it
must not break a test. Pin a sentence only when a recorded eval shows that exact
wording changes behavior, and cite that eval next to the pin.
- **Sentence pin**: a test that asserts an English sentence or clause (40 or more
characters of prose, or a prose regex) against SKILL.md, a `.tmpl` template or a
section file. Convert it to a structural check (the section or heading is
present, steps keep their order, the machine-read token is present in the right
section), or delete it when a behavioral eval already covers the behavior and
name that eval in the commit. Check safety lines on meaning (a case-insensitive
keyword in the right section), never on capitalization or emphasis.
- **Retention bar**: keep a test that independently enforces a public API, protocol,
config, migration, storage, security, platform, default, prompt-byte,
generated-output (SKILL.md golden), package, release or architecture contract; call
order when order is observable; source inspection when it is the cheapest
independent guard. Static or slow is not a reason to delete. Anything reachable
from the package entrypoint is never retired.
config, migration, storage, security, platform, default, prompt-byte (as defined
above), generated-output (SKILL.md golden), package, release or architecture
contract; call order when order is observable; source inspection when it is the
cheapest independent guard. Static or slow is not a reason to delete. Anything
reachable from the package entrypoint is never retired.
- **Value card**: the gate's four answers as one line,
`Value: protects=<...>; fails_when=<...>; why_new=<...>; seam=none`. Each field is
at most 160 UTF-8 bytes in that line (clamped to 157 bytes plus `...`); JSON keeps