mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-16 13:21:30 +01:00
Merge remote-tracking branch 'origin/main' into cso-evaluator-support
# Conflicts: # CHANGELOG.md # VERSION # agents-digest/gstack-AGENTS.md # package.json # test/cso-docker-integration.test.ts
This commit is contained in:
commit
2d79967403
288 files changed
+13384
-10107
No files matched your search
@@ -93,7 +93,7 @@ gstack/
|
||||
├── docs/designs/ # Design documents (incl. IMPECCABLE_INTEROP.md = the design detector / catalog / open DESIGN.md record, and fork-port-residual-2026-09/ evaluation evidence)
|
||||
├── setup-deploy/ # /setup-deploy skill (one-time deploy config)
|
||||
├── .github/ # CI workflows + shared composite actions (.github/actions/) + Docker image (claude CLI pinned)
|
||||
│ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, and 8 more (windows, periodic evals, release gates, ci-image)
|
||||
│ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, plus free-tests, Windows, periodic and marathon evals, CSO release, platform qualification and smaller gates
|
||||
│ └── docker/ # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium)
|
||||
├── contrib/ # Contributor-only tools (never installed for users)
|
||||
│ └── add-host/ # /gstack-contrib-add-host skill
|
||||
|
||||
+284
-70
@@ -184,10 +184,9 @@ This does not change the separate CI machine count. Full-suite shards are packed
|
||||
DURATIONS (LPT, `packShardsByDuration`) when the committed seed
|
||||
`scripts/free-test-durations.json` exists — refresh it with
|
||||
`bun run test:ubicloud --record-durations`, which times each file in its own
|
||||
child on a VM with the CI lane's environment and copies the seed back (CI never
|
||||
records; a seed recorded where browser or display tests skip underestimates
|
||||
them). Missing seed → silent hash-shard fallback; corrupt seed → one warning +
|
||||
fallback; unknown files get 75th-percentile pessimism, and both full-suite and
|
||||
child on a VM with the CI lane's environment and copies the seed back; see
|
||||
[Free suite duration seed](#free-suite-duration-seed). Missing seed → silent hash-shard fallback; corrupt seed → one warning +
|
||||
fallback; unknown files get 99th-percentile pessimism, and both full-suite and
|
||||
`--ci-plan` runs name them on stderr so a new slow file cannot silently become
|
||||
the long pole. Packed
|
||||
shards get duration-aware walls (`max(base, predicted × 3, files × 5s)`). The
|
||||
@@ -257,15 +256,17 @@ never retry; each case's kind fixes its trials before the run (see "Eval verdict
|
||||
policy" below). Files in `CASE_SHARDED_FILES` run one registered case per
|
||||
process (`<file>#<case id>`, an exact `--test-name-pattern`, exactly one executed
|
||||
case), so a long file of short cases spreads across runners and each case gets
|
||||
its own SDK semaphore.
|
||||
its own SDK semaphore. `test/carve-section-loading.test.ts` is one of them: it registers
|
||||
one periodic case per carved skill (`carve-section-loading-<skill>`, each with
|
||||
its own touchfiles and history). Run one case with
|
||||
`bun run scripts/test-paid-shards.ts --tier periodic --case carve-section-loading-<skill>`.
|
||||
Trial telemetry rides the store: every recorded test carries its 1-based
|
||||
`attempt` plus, on an isolated trial shard, its `case_id`, `kind`, `trial`,
|
||||
`panel` and `policy_version`, and each lane's report uploads one
|
||||
`trial-outcomes` JSONL line per trial. `bun run eval:pass-rates`
|
||||
(`eval:flake-rank` is an alias) turns that history into per-case pass rates
|
||||
(see "Pass-rate history" below; the free lane's flake ledger is folded in from
|
||||
`flakeLedgerPath()` — override with `GSTACK_FLAKE_LEDGER`, the same env var the
|
||||
CI free lane sets before uploading the ledger as the `flake-ledger` artifact). Census integrity is
|
||||
turns that history into per-case pass rates
|
||||
(see "Pass-rate history" below; free-suite flakes are a separate record, see
|
||||
[Flake ledger](#flake-ledger)). Census integrity is
|
||||
enforced from the free suite: every `E2E_TOUCHFILES` / `LLM_JUDGE_TOUCHFILES`
|
||||
key must name a living paid test (`test/touchfiles.test.ts`'s reverse
|
||||
invariant), and `git show <sha>:path` fixtures are banned — vendor the bytes
|
||||
@@ -291,7 +292,8 @@ are not collector results and do not establish that an unfinished test passed.
|
||||
retains the broad census; no case IDs or tier assignments are removed. The plan
|
||||
records both E2E and judge selections and lists deferred coverage. Executors
|
||||
receive those selections separately and the report checks actual executed case
|
||||
counts. Unknown source dependencies restore the broad gate; unmapped prompts
|
||||
counts. Unknown source dependencies restore the broad gate (see
|
||||
[PR paid lane fallback](#pr-paid-lane-fallback)); unmapped prompts
|
||||
without registered coverage require full validation. Known broad-only prompt
|
||||
changes are explicitly deferred, not counted as PR passes. Weekly/manual runs of
|
||||
`evals-periodic.yml` execute both complete censuses fresh; manual `evals.yml`
|
||||
@@ -342,10 +344,22 @@ into the fullest slice whose estimated wall on J FIFO workers stays within S
|
||||
seconds, else a new slice; a shard with no recorded wall weighs the whole budget
|
||||
(its own runner), a shard longer than the budget runs alone, and overlays keep
|
||||
one final one-at-a-time slice. The manifest's `plan` records each slice estimate
|
||||
and `ciTimeoutMinutes` (every slice's supervised worst case plus 20 minutes
|
||||
setup); CI derives the matrix (`[range(1; .sliceCount + 1)]`) and job timeout
|
||||
from it, and executors refuse an `EVALS_JOBS` other than the planned J and run
|
||||
their shards longest first. `--slices K` keeps the supervised round-robin
|
||||
and its own CI job ceiling in `sliceCiTimeoutMinutes`: max(2 × slice budget, the
|
||||
slice's longest shard supervised wall, the serialized overlay envelope) + 20
|
||||
minutes setup (`CI_SETUP_ALLOWANCE_MINUTES`); `ciTimeoutMinutes` is the largest of
|
||||
them. CI derives the matrix (`[range(1; .sliceCount + 1)]`) from the manifest and
|
||||
indexes each slice's ceiling as its job timeout
|
||||
(`fromJSON(needs.plan-slices.outputs.slice_timeouts)[matrix.slice]`). Executors
|
||||
refuse an `EVALS_JOBS` other than the planned J and run their shards longest first.
|
||||
|
||||
Each executor's first step records its job start (`GSTACK_SLICE_JOB_STARTED_AT`).
|
||||
The slice deadline is that start + the slice ceiling − a 5-minute upload reserve:
|
||||
after it the executor starts no more shards. A shard that never started reports
|
||||
`not_run` (INFRA, `never-started`); a shard still running is killed and reports
|
||||
`hung` (TIMEOUT, `timed-out`). The executor checkpoints `slice-N.json` after every
|
||||
shard start and finish, so a job killed before its final write reports from the
|
||||
checkpoint (`slice N/M ended before its final result …: TIMEOUT`), never as a
|
||||
missing slice. `--slices K` keeps the supervised round-robin
|
||||
baseline re-packed by recorded times for local runs. Durations are recorded per
|
||||
tier (a file's gate and periodic cases differ); refresh one tier from a
|
||||
downloaded report directory with `--report <dir> --write-durations`. Under `EVALS_ALL` the hollow-shard guard marks exit-0 shards with
|
||||
@@ -373,8 +387,10 @@ the runner parent and handed to shard children as `GSTACK_CLAUDE_CLI_VERSION`
|
||||
(never spawned on a test thread), so a TUI-drift flake hunt is a grep, not
|
||||
archaeology.
|
||||
|
||||
**Eval verdict policy** (`EVAL_POLICY` version 1 in
|
||||
`test/helpers/periodic-exclude-data.ts`, pre-registered 2026-09-29). Paid evals
|
||||
**Eval verdict policy** (`EVAL_POLICY` version 2 in
|
||||
`test/helpers/periodic-exclude-data.ts`; version 1 pre-registered 2026-09-29,
|
||||
version 2 approved 2026-10-04 with the decision memo in
|
||||
[test-audit-2026-10.md](test-audit-2026-10.md)). Paid evals
|
||||
never retry. Each live case has exactly one kind in `E2E_KINDS`
|
||||
(`test/helpers/touchfiles-data.ts`; `test/eval-kinds.test.ts` enforces coverage),
|
||||
and the kind fixes its trials before the run:
|
||||
@@ -429,14 +445,19 @@ least 97% over at least 10 trials) without being removed, when an entry is 8
|
||||
weekly runs old, or when a tier is over its cap.
|
||||
|
||||
**Pass-rate history** (`bun run eval:pass-rates`, `scripts/eval-flake-rank.ts`).
|
||||
It reads the `trial-outcomes` artifact of the last N completed
|
||||
`evals-periodic.yml` runs on the current branch and `main` (flags: `--case`,
|
||||
`--runs N`, `--branch`, `--dir`, `--backfill`, `--json`, `--gate`) and prints
|
||||
It reads the `trial-outcomes` artifact of the last N completed weekly
|
||||
`evals-periodic.yml` runs, meaning scheduled runs on `main` plus `main`
|
||||
dispatches, plus completed branch census runs in the same window (flags:
|
||||
`--case`, `--runs N`, `--branch` to inspect one branch, `--dir`, `--backfill`,
|
||||
`--json`, `--gate`). A branch trial counts only toward a series `main` has also
|
||||
run (same case-owned bytes, `HARNESS_VERSION`, model and CLI); a branch never
|
||||
starts or becomes a case's current series, and weeks for quarantine expiry
|
||||
count `main` runs only. It prints
|
||||
per-case per-trial pass rates with 95% Wilson intervals. A series is one case
|
||||
under one input identity, the hash of its own touchfiles minus
|
||||
`GLOBAL_TOUCHFILES` (harness edits do not restart it), per model, Claude CLI
|
||||
version and policy version; a change starts a new series and older ones stay
|
||||
visible. Labels: INCONCLUSIVE below 10 trials, BROKEN when the latest run is
|
||||
under one input identity ([Harness version](#harness-version)), per model,
|
||||
Claude CLI version and policy version
|
||||
([Pass-rate policy versions](#pass-rate-policy-versions)); a change starts a
|
||||
new series and older ones stay visible. Labels: INCONCLUSIVE below 10 trials, BROKEN when the latest run is
|
||||
0/n after a prior interval at or above 95%, FLAKY when failures leave the
|
||||
interval straddling 95%, FAILING when the whole interval is below it, PASSING
|
||||
otherwise. `--backfill` imports legacy slice artifacts as pre-policy trials
|
||||
@@ -530,19 +551,20 @@ its source and policy identifier. Custom drivers must resolve each job instead
|
||||
of passing their ordinary 1800-second default as an explicit cap;
|
||||
their outer controller/detach wall must also cover the allocated work and cleanup.
|
||||
The paid census counts are printed by `--list` for each tier.
|
||||
`eval:bg:pr` and `eval:bg:periodic` have 92820/67380-second outer caps, above their recomputed floors (PR fallback 78,425 s, periodic 33,821 s including the trial shards); the PR
|
||||
wrapper covers a full-gate fallback at its default two workers. The broad gate
|
||||
wrapper reserves 49320 seconds (floor 21,725 s), and release reserves 116700 seconds for both
|
||||
tiers; free tests recompute each floor from the live shard census, case shards
|
||||
included. Legacy monolithic
|
||||
`eval:bg`/`eval:bg:all` retain their shorter 5400/7200-second caps; use the
|
||||
sharded periodic path for complete coverage.
|
||||
`eval:bg:<pr|gate|periodic|release>` runs `scripts/eval-bg.ts`. On the local
|
||||
backend its outer cap is ceil(1.5 × planned serial seconds / `EVALS_JOBS`) + 20
|
||||
minutes, at most 4 hours, computed from `scripts/paid-test-durations.json` for
|
||||
the lane's current plan (`--timeout SECS` overrides). On the dispatch backend the
|
||||
CI job ceilings above bound the run, and the local follower stops after 4 hours.
|
||||
`test/eval-detach-timeout-floor.test.ts` pins the cap formula, the 4-hour
|
||||
ceiling, the backend choice, the sentinel on both backends, the conclusion
|
||||
mapping and dispatch dedupe.
|
||||
|
||||
CI plans with `--slice-budget 540 --jobs 2` for the PR gate, the periodic census
|
||||
CI plans with `--slice-budget 420 --jobs 2` for the PR gate, the periodic census
|
||||
and the weekly gate census (the gate census also `--skip-judges`), and
|
||||
`--slice-budget 1 --jobs 1` (one file per runner) for marathon. The live plans
|
||||
must fit their workflow's `max-parallel` so every slice starts at once, and
|
||||
`ciTimeoutMinutes` must stay within 360; `test/evals-workflow-wiring.test.ts`
|
||||
must fit their workflow's `max-parallel` so every slice starts at once, and every
|
||||
per-slice ceiling must stay within 360 minutes; `test/evals-workflow-wiring.test.ts`
|
||||
recomputes both from the complete census. Reconciliation rejects missing,
|
||||
duplicated or misplaced registered work, absent budget records, case shards that
|
||||
did not execute exactly their case, and reused results outside the PR profile.
|
||||
@@ -590,6 +612,35 @@ against a temp `GSTACK_INSTALL_DIR` / `GSTACK_SKILLS_DIR`, and
|
||||
`freeze/bin/check-freeze.sh` with JSON payloads on stdin (including the
|
||||
`GSTACK_HOME` state-root parity against `bin/gstack-paths`).
|
||||
|
||||
### Pass-rate policy versions
|
||||
|
||||
Readers score only trials recorded under their own `EVAL_POLICY.version`.
|
||||
Trials from an older policy are shown display-only. Trials from a newer policy
|
||||
are ignored with a printed count ("ignored N trial(s) recorded under a policy
|
||||
newer than vX"). When you see that line, run pass-rates from a checkout at or
|
||||
after the commit that bumped the version. The supported rollback of a policy
|
||||
change is reverting its writer commit; the reader-compatibility commit before it
|
||||
stays.
|
||||
|
||||
### Harness version
|
||||
|
||||
Under `EVAL_POLICY` v2 a case's series identity is the bytes the case owns (its
|
||||
paid test file, its fixtures and its skills' prompt files, including the
|
||||
generated SKILL.md) plus `HARNESS_VERSION`, keyed with model, Claude CLI version
|
||||
and policy version (`scripts/eval-trial-series.ts`). Each trial also records the
|
||||
full consumed-input fingerprint as provenance. `scripts/harness-version.json`
|
||||
pins every shared-harness file (`GLOBAL_TOUCHFILES` plus the session runners, the
|
||||
PTY harness, the e2e gate and the LLM judge) by blob hash. When
|
||||
`test/harness-version.test.ts` fails, run one of:
|
||||
|
||||
```bash
|
||||
bun run scripts/bump-harness-version.ts --non-behavioral "<why no verdict can change>" # no series reset
|
||||
bun run scripts/bump-harness-version.ts --bump "<what behavior changed>" # new series for every case
|
||||
```
|
||||
|
||||
On a merge conflict in the manifest, take the higher version, then rerun the
|
||||
script.
|
||||
|
||||
## Ubicloud VMs (`bun run test:ubicloud`)
|
||||
|
||||
`bun run test:ubicloud [test:free args]` runs the free suite on an ephemeral
|
||||
@@ -605,8 +656,8 @@ checkout to it: tracked files, untracked files that are not ignored, and
|
||||
`.git`, so uncommitted edits are tested. It then runs
|
||||
`scripts/ubicloud/setup-free-suite.sh`, which mirrors the `free-suite` CI job
|
||||
(same Bun pin, Playwright Chromium with its setuid sandbox helper, Xvfb,
|
||||
poppler, emoji fonts, generated host outputs, gate binaries, and the CSO
|
||||
helper), and runs `xvfb-run -a bun run test:free` with `GSTACK_EXPECT_BINARIES=1`
|
||||
poppler, emoji fonts, zsh for the bash+zsh portability arms (#2669), generated
|
||||
host outputs, gate binaries, and the CSO helper), and runs `xvfb-run -a bun run test:free` with `GSTACK_EXPECT_BINARIES=1`
|
||||
and `GSTACK_FREE_RETRY_FLAKY=1`. Shard logs are copied to
|
||||
`.context/ubicloud/<timestamp>/`, and the VM is destroyed on every exit path.
|
||||
The exit status is the suite's.
|
||||
@@ -646,17 +697,19 @@ serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so
|
||||
a broken tree can't masquerade as flaky). The required CI free lane and the
|
||||
Windows lane set the retry knob too, appending every flaky pass to the JSONL ledger it uploads
|
||||
(`GSTACK_FLAKE_LEDGER`) — a flaky pass never reds the lane, but it never
|
||||
disappears either.
|
||||
disappears either ([Flake ledger](#flake-ledger)).
|
||||
|
||||
## Test selection and tiers
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit).
|
||||
|
||||
**Diff-based test selection:** `test:evals` and `test:e2e` auto-select tests based
|
||||
on `git diff` against the base branch. Each test declares its file dependencies in
|
||||
`test/helpers/touchfiles.ts`. Changes to global touchfiles (session-runner, eval-store,
|
||||
touchfiles.ts itself) trigger all tests. Use `EVALS_ALL=1` or the `:all` script
|
||||
variants to force all tests. Run `eval:select` to preview which tests would run.
|
||||
**Diff-based test selection:** the sharded paid runner (`test:pr`, `eval:bg:pr`,
|
||||
`test:gate:sharded`) selects tests based on `git diff` against the base branch.
|
||||
Each test declares its file dependencies in `test/helpers/touchfiles.ts`. Changes
|
||||
to global touchfiles (session-runner, eval-store, touchfiles.ts itself) trigger
|
||||
all tests. Use `EVALS_ALL=1` to force all tests. Run `eval:select` to preview
|
||||
which tests would run; it applies the PR profile by default, and
|
||||
`--profile full` shows the plain touchfile selection. Commands by task: [CONTRIBUTING.md](../CONTRIBUTING.md#which-command-do-i-run).
|
||||
|
||||
**Two-tier system:** Tests are classified as `gate` or `periodic` in `E2E_TIERS`
|
||||
(in `test/helpers/touchfiles.ts` — a facade over `touchfiles-data.ts` +
|
||||
@@ -680,14 +733,61 @@ in `bun test`): a `skill-e2e-*` file named in a touchfiles dep list whose
|
||||
suite. Files not named in any dep list are reported, not enforced — keep both
|
||||
in sync.
|
||||
|
||||
### PR paid lane fallback
|
||||
|
||||
PR selection (`scripts/test-pr-profile.ts`) classifies every changed file. A file
|
||||
mapped by a paid case's touchfiles selects that case. These files are exempt and
|
||||
select nothing: docs, any non-paid `*.test.ts(x)`, `FREE_ONLY_PR_FILES` (the
|
||||
free and paid duration seeds, `scripts/ubicloud/**`, every workflow that runs no
|
||||
paid evals, and free-only tools), and the fixtures in `FREE_FIXTURES`
|
||||
(`test/helpers/free-fixtures-data.ts`, each fixture with its free consumers).
|
||||
`PAID_WORKFLOW_FILES` (`evals.yml`, `evals-periodic.yml`, `evals-marathon.yml`)
|
||||
and any other unknown file restore the full gate (`prCoverage.mode =
|
||||
full-fallback`). The job summary and PR comment list each file that caused a
|
||||
fallback with one of three labels:
|
||||
|
||||
- **needs touchfile entry**: a paid test, helper or fixture no case maps; add it
|
||||
to the owning case in `E2E_TOUCHFILES` (`test/helpers/touchfiles-data.ts`).
|
||||
- **add to FREE_ONLY_PR_FILES**: a file only free lanes read; add it to the list
|
||||
in `scripts/test-pr-profile.ts` (or a fixture to `FREE_FIXTURES`).
|
||||
- **real unknown dependency**: shared code (bin/, lib/, resolvers, hosts, setup)
|
||||
whose dependents are not mapped; the full gate is the correct result.
|
||||
|
||||
`test/free-fixtures.test.ts` holds the inventory: each `test/fixtures/**` path
|
||||
belongs to a paid touchfile or `FREE_FIXTURES`, and each workflow to the paid or
|
||||
free-only list. Editing a paid test file runs its own gate cases on the PR when
|
||||
their Bun test is named by the case id; the file's other cases stay deferred with
|
||||
a reason that says to name the test by its case id.
|
||||
|
||||
### PR receipts, recovery and debounce
|
||||
|
||||
Passes and failures of single-trial shards are written as receipts at the
|
||||
execution boundary, so a cancelled run's finished work survives. The
|
||||
`recover-receipts` job runs `scripts/recover-receipts.ts` from the base ref with
|
||||
`actions: read`: it merges the receipts of at most 5 completed, cancelled
|
||||
`evals.yml` runs of the same PR (oldest first, negatives before passes, so a newer
|
||||
FAIL blocks an older PASS) and hands the store to `plan-slices`, which keeps
|
||||
`contents: read` only. Its `recovery.json` checkpoint never advances past an
|
||||
unresolved run. `plan-slices` turns reuse off when recovery degraded, the
|
||||
checkpoint is missing or a cancelled run is still unresolved; recovery itself
|
||||
never fails the run, and the job summary names the reason (rate limit with its
|
||||
reset time, `gh auth status`, the 60-second budget). The whole store is saved by
|
||||
`slices-report`.
|
||||
|
||||
On a `synchronize` push, when another run for the same PR started in the last 15
|
||||
minutes, the debounce job waits 90 seconds before planning; a run that outlives
|
||||
the wait reports "superseded by <sha>". PR labels: `evals-no-debounce` skips the
|
||||
wait and `evals-fresh` turns receipt reuse off. Labels apply from the next push,
|
||||
because adding a label does not start `evals.yml`.
|
||||
|
||||
## Free suite runner, judge reuse and engine skips
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit).
|
||||
|
||||
`bun run test` routes through `scripts/test-free-shards.ts` (N concurrent
|
||||
shard processes, serial within each, packed by recorded per-file durations
|
||||
when `scripts/free-test-durations.json` exists — refresh occasionally with
|
||||
`bun run test:free --record-durations`; strict-output classification per
|
||||
when `scripts/free-test-durations.json` exists — refresh it with
|
||||
`bun run test:ubicloud --record-durations`; strict-output classification per
|
||||
shard: a shard without bun's terminal summary line FAILS — silent truncation
|
||||
cannot report green). `TREE_MUTATING` lists the files that still run in their
|
||||
own trailing serial shard (today only `test/bootstrap-retention.test.ts`, for
|
||||
@@ -736,6 +836,105 @@ Fix a writer with `usePrivateStateRoot()` (`test/helpers/private-state-root.ts`)
|
||||
or a child HOME/GSTACK_HOME, and resolve product state paths at write time,
|
||||
never at import.
|
||||
|
||||
### Flake ledger
|
||||
|
||||
`GSTACK_FREE_RETRY_FLAKY=1` reruns a failing free test file once, serially. A
|
||||
clean retry turns the shard green as a flaky pass and appends an entry to that
|
||||
shard's flake ledger, uploaded as `flake-ledger-<shard>` (90-day retention). The
|
||||
Monday "Weekly test health" workflow (`bun run test:health --enforce`) reads
|
||||
main's ledgers through `scripts/lib/ci-history.ts` and fails when a file flakes
|
||||
in more than 5% of main free-tests runs over a window of at least 20 runs; a
|
||||
window thinner than 20 runs in the last 7 days widens to the newest 20 runs, and
|
||||
fewer than 20 runs is report-only. `bun run test:health flake-summary <dir>`
|
||||
prints a run's ledgers as a table. A flaky file is fixed at source; the retry
|
||||
never masks it. `bun run eval:pass-rates` lists only the local ledger
|
||||
(`flakeLedgerPath()`, overridden by `GSTACK_FLAKE_LEDGER`).
|
||||
|
||||
The aggregate job of `free-tests.yml` downloads every `flake-ledger-*` artifact
|
||||
of the run and appends `bun run scripts/test-health-report.ts flake-summary <dir>`
|
||||
to its job summary. The Windows lane uploads `flake-ledger-windows-<shard>`.
|
||||
|
||||
Free-suite flake rules:
|
||||
|
||||
- Bun gives a child `stdio: 'pipe'` a unix socketpair. A test that must hold a
|
||||
write blocked uses a FIFO the test never reads (`test/qa-evidence.test.ts`,
|
||||
`test/qa-deadline.test.ts`), not a paused child stream: socket capacity follows
|
||||
`net.core.{w,r}mem_default` and the parent's reader.
|
||||
- A test that rewrites CI provenance variables (`CI`, `GITHUB_*`, `CI_*`)
|
||||
restores them. Free shards run many files in one process, and a later browse
|
||||
daemon without `CI=true` launches Chromium sandboxed.
|
||||
- Handshakes with a runner wait for the runner's own observable effect (its
|
||||
progress line, a coalesced read count), not for a fixture side file or a timer.
|
||||
|
||||
### Free suite duration seed
|
||||
|
||||
`scripts/free-test-durations.json` drives shard packing; files missing from it
|
||||
pack at the seed's 99th percentile. Acceptable seeds come from
|
||||
`bun run test:ubicloud --record-durations` (each file timed in its own child on a
|
||||
VM with the CI lane's environment) or a CI recording. A 4-core laptop or
|
||||
small-sandbox recording is not acceptable: browser and display tests skip or slow
|
||||
down there, so the seed misstates them.
|
||||
|
||||
- `--ci-plan` warns in the job summary, with the refresh command, when more than 5
|
||||
files are unseeded; it never fails a PR. `--ci-verify` lists shards that ran
|
||||
longer than 1.5× their prediction and at least 30 seconds over.
|
||||
- The weekly test-health run fails its enforced `unseeded-free-files` check
|
||||
("stale seed: N unseeded file(s) > 5") when more than 5 free test files are
|
||||
missing from the seed, so contributors without Ubicloud access learn within a
|
||||
week that it needs a refresh.
|
||||
- `test/free-seed-ratchet.test.ts` (run with full history in `free-tests.yml`'s
|
||||
`free-plan` job, `GSTACK_FREE_SEED_BASE=$(git merge-base HEAD origin/main)`)
|
||||
fails when a seeded file over 60 seconds is missing from
|
||||
`scripts/free-test-seed-allowlist.json`, when an allowlist entry is stale (gone,
|
||||
or no longer over 60 seconds), or when an entry is added relative to the
|
||||
merge-base copy without a `todo`. An annotated exception is
|
||||
`{file, reason, todo}` and the test prints it. A merge-base without the
|
||||
allowlist file counts as the first allowlist. Fix a failure by splitting the
|
||||
file, then rerun `bun run test:ubicloud --record-durations`.
|
||||
|
||||
### Windows free lane
|
||||
|
||||
`windows-free-tests.yml` plans on Linux (`--windows-only --ci-plan --shards 6`),
|
||||
runs one strict serial shard per `windows-latest` job
|
||||
(`bun run test:windows --ci-run`), and verifies every shard receipt on Linux
|
||||
(`--windows-only --ci-verify`). Packing uses Windows-measured durations from
|
||||
`scripts/free-test-durations-windows.json`; refresh them with
|
||||
`gh workflow run windows-free-tests.yml --ref <branch> -f record_durations=true`,
|
||||
then download the `free-test-durations-windows` artifact and commit it.
|
||||
|
||||
Curation lives in `scripts/lib/windows-curation.ts`. Every Windows plan and
|
||||
`--list` prints the curated count and the top exclusion reasons (also in the job
|
||||
summary). Files a probe proved safe despite a `/tmp` literal are force-included
|
||||
(`WINDOWS_PROBE_SAFE`). `test/windows-native-workflows.test.ts` holds
|
||||
`WINDOWS_CURATED_FLOOR`; when Windows-safe tests are deleted on purpose, lower it
|
||||
in the same PR with a reason.
|
||||
|
||||
Native qualification campaigns (Windows cookie extraction, Windows launch
|
||||
diagnostics, Dia on macOS) run only on dispatch of `native-qualification.yml`:
|
||||
|
||||
```bash
|
||||
gh workflow run native-qualification.yml --ref <branch> -f mode=<cookie-native|windows-native-diagnostics|dia-native|dia-launch-comparison|dia-gui-readiness>
|
||||
```
|
||||
|
||||
### Real-setup install fixture
|
||||
|
||||
Tests that run the real `setup` use `test/helpers/install-fixture.ts`. Each test
|
||||
file builds one read-only seed checkout and clones it per test with
|
||||
copy-on-write file copies (plain copy fallback), never hard links. Writes go
|
||||
through `put`, `setVersion` and the `fixture*Sync` helpers; files register
|
||||
`afterEach(cleanupFixtures)` and `afterAll(cleanupSeed)`.
|
||||
`test/install-fixture.test.ts` is the isolation tripwire: writes in one clone
|
||||
leave the seed and sibling clones unchanged.
|
||||
|
||||
### Test-only timing knobs
|
||||
|
||||
Defaults are the product's; tests shrink them instead of waiting real time:
|
||||
`GSTACK_RENDER_SLACK_MS` (`lib/aside-render.ts`, default 10000),
|
||||
`GSTACK_MEMORABLE_LOCK_TRIES` (`bin/gstack-memorable`, default 50 tries of 0.1 s),
|
||||
the `recordGraceMs` option of `test/helpers/office-hours-attempt.ts` (default
|
||||
5000) and the `timeoutMs` argument of `clearCookieTargetStorage(page, origin,
|
||||
timeoutMs)` (default 15000).
|
||||
|
||||
## Running evals as an agent: detach
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in
|
||||
@@ -745,36 +944,51 @@ When **you (an agent/harness)** launch a long eval/benchmark run, run it through
|
||||
`bin/gstack-detach` — NEVER as a plain backgrounded Bash task. A plain background
|
||||
task lives in the harness's process group, so a SIGTERM ("polite quit") on a turn
|
||||
boundary, a stopped Monitor, or an interruption kills the run mid-flight (observed:
|
||||
`script "test:gate" was terminated by signal SIGTERM` ~40 min into a run). On macOS
|
||||
`script "test:gate" was terminated by signal SIGTERM` ~40 min into a run of the
|
||||
since-retired single-process runner). On macOS
|
||||
the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
|
||||
(escapes the group SIGTERM) wrapped in `caffeinate -i` (blocks idle-sleep).
|
||||
|
||||
- Use the `eval:bg*` scripts (`eval:bg`, `eval:bg:all`, `eval:bg:gate`,
|
||||
`eval:bg:periodic`) — they wrap the eval command in `gstack-detach` with the
|
||||
machine-wide `gstack-evals` lock (concurrent worktrees serialize instead of
|
||||
saturating the shared model API), a per-tier watchdog, and a **run-scoped** log
|
||||
under `~/.gstack-dev/eval-runs/` (no shared-`/tmp` collision). Each prints its
|
||||
log path. `eval:bg:gate` / `eval:bg:periodic` run their tier through the
|
||||
sharded paid runner (`scripts/test-paid-shards.ts`, also exposed as
|
||||
`test:gate:sharded` / `test:periodic:sharded`): one Bun process per test
|
||||
file, an external wall-clock timeout that kills the shard's process GROUP
|
||||
(stray `claude`/`codex` grandchildren included), a per-shard
|
||||
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
|
||||
constructor, and an aggregate that separates failed vs timed-out vs
|
||||
never-started shards — the detach timeouts (the `--timeout` values on
|
||||
package.json's `eval:bg:gate` / `eval:bg:periodic`;
|
||||
floor enforced against the live shard census by
|
||||
test/eval-detach-timeout-floor.test.ts)
|
||||
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
|
||||
process count (default 8); `EVALS_CONCURRENCY` is bun's --max-concurrency
|
||||
WITHIN a shard (default 2) — they are deliberately separate knobs. `eval:list` / `eval:compare` /
|
||||
`eval:summary` / `eval:flake-rank` read the shard dirs too. Or call
|
||||
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] --
|
||||
<cmd>` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never
|
||||
pass keys in argv).
|
||||
- Use `bun run eval:bg:<pr|gate|periodic|release>` (`scripts/eval-bg.ts`). It
|
||||
prints `[eval-bg] backend=<dispatch|local> …` and `gstack-detach LOG <path>` and
|
||||
returns at once. Both backends run under `gstack-detach` with the machine-wide
|
||||
`gstack-evals` lock (concurrent worktrees serialize instead of saturating the
|
||||
shared model API) and a **run-scoped** log under `~/.gstack-dev/eval-runs/`
|
||||
whose first line names the backend, the run and the tested revision.
|
||||
- **Dispatch** (default when HEAD is clean and equal to its pushed branch on
|
||||
`garrytan/gstack`, checked with `gh api`, and `gh` can dispatch): `pr` and
|
||||
`gate` dispatch `evals.yml` (`evals_all=false` with `base_ref`/`base_sha` for
|
||||
`pr`, `evals_all=true` for `gate`), `periodic` dispatches `evals-periodic.yml`,
|
||||
`release` dispatches both. Every dispatch passes `expected_sha` (the workflow
|
||||
fails fast when the checked-out revision differs) and a `nonce` that appears
|
||||
in the run name (`E2E Evals [eval-bg <nonce>]`), which is how the log finds
|
||||
and follows its run. A queued, running or green dispatch with the same
|
||||
revision and inputs is followed instead of dispatched again. Dispatched runs
|
||||
are validation runs: fresh and never writing PR receipts, except that a `pr`
|
||||
dispatch on a branch with an open same-repo PR passes `pr_receipts=<PR>` and
|
||||
reads that PR's receipts read-only.
|
||||
- **Local** (dirty tree, unpushed commits, a fork, no `gh`, or `--local` /
|
||||
`GSTACK_EVAL_BG_MODE=local`): the sharded paid runner
|
||||
(`scripts/test-paid-shards.ts`) on this machine, capped at
|
||||
ceil(1.5 × planned serial seconds / `EVALS_JOBS`) + 20 minutes, at most 4 hours
|
||||
(`--timeout SECS` overrides). One Bun process per test file, an external
|
||||
wall-clock timeout that kills the shard's process GROUP (stray
|
||||
`claude`/`codex` grandchildren included), a per-shard
|
||||
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
|
||||
constructor, and an aggregate that separates failed vs timed-out vs
|
||||
never-started shards. `EVALS_JOBS` sets the shard process count (default 8,
|
||||
2 for `pr`); `EVALS_CONCURRENCY` is bun's --max-concurrency WITHIN a shard
|
||||
(default 2). Export `ANTHROPIC_API_KEY` first (never pass keys in argv).
|
||||
- `--dispatch` refuses with the fix ("commit and push, or rerun with --local")
|
||||
instead of falling back.
|
||||
`eval:list` / `eval:compare` / `eval:summary` / `eval:pass-rates` read the shard
|
||||
dirs too. For any other long agent job, call
|
||||
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] -- <cmd>` directly.
|
||||
- Then **poll the printed logfile** with a death-aware watcher: break on the
|
||||
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (success AND failure are
|
||||
both marked, so silence is never mistaken for success). The detached run survives
|
||||
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (0 passed, 1 failed,
|
||||
130 cancelled, 2 other on both backends; silence is never mistaken for success).
|
||||
`bun run scripts/eval-bg.ts status <log-or-run-id>` reconnects and prints
|
||||
running, passed, failed, cancelled or incomplete (died without a sentinel). The detached run survives
|
||||
even if your watcher gets reaped, so re-checking the log always works. Keep
|
||||
checking until the sentinel appears or the user tells you to stop; a long run is
|
||||
expected, and a promise to check later is not a result. At each check, report
|
||||
@@ -782,5 +996,5 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
|
||||
- Why the lock: a shared dev box with several Conductor worktrees will rate-limit
|
||||
the model API if two eval suites run at once (15-way concurrency each), which
|
||||
mass-times-out E2E tests. The lock makes the second run WAIT, not collide.
|
||||
- Humans running `bun run test:evals` foreground in their own terminal don't need
|
||||
- Humans running `bun run test:pr` foreground in their own terminal don't need
|
||||
this — Ctrl-C is intended there. Detachment is for agent-launched runs only.
|
||||
@@ -26,6 +26,12 @@ The dedicated Windows CI job checks startup, argument forwarding, and runtime
|
||||
injection separately from static audit and private-state support. POSIX
|
||||
watchdog execution remains unavailable there.
|
||||
|
||||
No qualified runtime or scanner image exists yet, so contained runtime and
|
||||
scanner execution is unavailable and setup preloads nothing: the scanner
|
||||
catalog is empty while its build inputs await review, and the runtime catalog's
|
||||
profiles stay unqualified until a private evaluator, still being built, can
|
||||
qualify them.
|
||||
|
||||
Every successful fresh install or upgrade asks the trusted launcher to preload
|
||||
the current host platform's qualified runtime and scanner images. This is the
|
||||
only automatic image acquisition path. It first validates the committed
|
||||
@@ -48,10 +54,12 @@ egress. `gstack-cso doctor --repo PATH` never downloads; it reports a qualified
|
||||
profile unavailable until that exact digest and platform are already present
|
||||
in the local daemon.
|
||||
|
||||
The ordinary free suite exercises contracts without requiring Docker. The
|
||||
`cso-docker-integration` CI job sets `GSTACK_CSO_DOCKER_TESTS=1`; unavailable Docker
|
||||
or missing containment prerequisites fail that job. Run it locally with a
|
||||
working local daemon:
|
||||
The ordinary free suite exercises contracts without requiring Docker. The free
|
||||
lane's `cso-docker-integration` job sets `GSTACK_CSO_DOCKER_TESTS=1` and runs
|
||||
`test/cso-docker-integration.test.ts`; unavailable Docker or missing containment
|
||||
prerequisites fail that job. The node-lifecycle and stack-cold files need a
|
||||
staged stack image, so they run in the CSO runtime workflows through
|
||||
`test:cso:docker`. Run all three locally with a working local daemon:
|
||||
|
||||
```bash
|
||||
bun run build:cso
|
||||
|
||||
+3
-2
@@ -815,8 +815,9 @@ A mechanical pre-filter shortlists assertion-free probes, source greps, export-l
|
||||
copies and near-duplicate files before any model reading. Each candidate gets a
|
||||
retirement card (what it detects, non-test callers with the search command, the
|
||||
stronger remaining proof, history, what retiring it unlocks, and the validation
|
||||
command). Contract tests such as SKILL.md goldens and prompt-byte checks are
|
||||
retained. The report and a JSON sidecar land in `~/.gstack/projects/<slug>/`.
|
||||
command). Contract tests such as SKILL.md goldens and prompt-byte checks
|
||||
(machine-read tokens, not English sentences; see [the value bar](test-value-bar.md))
|
||||
are retained. The report and a JSON sidecar land in `~/.gstack/projects/<slug>/`.
|
||||
Nothing is edited unless you approve a batch; spawned sessions stay report-only.
|
||||
Tests marked `gstack:test-value keep reason="..."` are skipped and listed in the
|
||||
report's appendix.
|
||||
|
||||
@@ -476,3 +476,21 @@ release docs sweep); tests 383 (+9,375/−44,379); test helpers 47 (+786/−12,7
|
||||
- **`test/helpers/claude-pty-runner.unit.test.ts` (3,694 LOC, 223 tests, 0.1 s):** a large self-test of the 5,885-LOC PTY harness. The classifiers it covers (`classifyVisible`, `parseNumberedOptions`, `Step0BoundaryPredicate`…) are live in paid runs, so keep it. The per-capture blocks (`captured F`, `captured G`) belong to the per-incident consolidation lane.
|
||||
- **Zero-importer helpers** `auq-parallel-worker`, `setup-gbrain-fixture-command`, `emulate-bun-windows-eexist` are loaded by path (preload / generated import). `benchmark-judge` has a production caller (dynamic import in `bin/gstack-model-benchmark`).
|
||||
- **`browse/src` `__reset*`/`reset*ForTests` exports (12):** standard singleton-reset seams, keep.
|
||||
|
||||
## Corrections (2026-10 audit)
|
||||
|
||||
Facts above that no longer hold, checked against main at v1.91.19.0:
|
||||
|
||||
- `codex-e2e-recommendation-substance` is not census-only: it is no longer in
|
||||
`PERIODIC_CI_EXCLUDE` and runs in every periodic census.
|
||||
- The after-metrics are superseded. Five days after release the free suite had
|
||||
regrown by 132 test files and about 53% of serial time; the
|
||||
[2026-10 audit](test-audit-2026-10.md) carries current numbers.
|
||||
- The duration seed again lacked 19 files, including the 115-second
|
||||
`claude-overlay-setup.test.ts`, which the planner packed as a short file.
|
||||
- `browse/test/compare-board.test.ts` stayed quarantined through this audit; its
|
||||
16 tests now pass headless and run in the free suite again.
|
||||
- `ci-image-tag-binding` now pins `evals.yml`, `evals-periodic.yml` and
|
||||
`evals-marathon.yml`; `ci-image.yml` is deleted because those workflows push
|
||||
the same content-hash tag.
|
||||
- The weak literal pins listed under Lane 5 are still present.
|
||||
@@ -0,0 +1,276 @@
|
||||
# Test, eval and CI audit — October 2026
|
||||
|
||||
This document records the evidence behind the October 2026 test, eval and CI
|
||||
wave: the EVAL_POLICY v2 decision memo (D1) with its pre-registered backtest,
|
||||
and the commands that reproduce every number. Metrics that need post-merge
|
||||
traffic are measured with `bun run test:health --since-days 7` (merge + 7 days)
|
||||
and `bun run test:health --since-days 28` (after the fourth scheduled census).
|
||||
|
||||
## EVAL_POLICY v2 decision memo (D1)
|
||||
|
||||
### Why the census is red
|
||||
|
||||
The weekly census is red by arithmetic, not by a handful of broken tests.
|
||||
Over 22 censuses with trial outcomes the periodic lane recorded 76 failed
|
||||
trials in 3,550 (about 3.5 failed trials per run) and the gate census 27 in
|
||||
2,241 (about 1.2 per run). With independent failures, P(green) is e^-3.5 ≈ 3%
|
||||
for the periodic lane and e^-1.2 ≈ 30% for the gate census; observed green
|
||||
rates were 0/22 and 8/22. Holding a ~190-case single-trial lane green 80% of
|
||||
the time needs every case at or below about 0.12% failure per run, which a
|
||||
live agent judged by one trial does not reach. Fixing the top five chronic
|
||||
cases leaves the lane red.
|
||||
|
||||
Three levers stay inside the no-retry, no-threshold, no-rejudge rule: make
|
||||
detectors deterministic over structured artifacts (W3), classify INFRA
|
||||
honestly (W2d, owned by the reliability follow-ups), and make quarantine
|
||||
reachable for detector, harness and model-latency failures. v2 targets the
|
||||
third lever.
|
||||
|
||||
`test:health` reports the census as three separate numbers, so a policy
|
||||
change cannot read as a reliability gain: the raw trial failure rate per kind
|
||||
and failure class, P(lane red) from pooled per-case rates under the recorded
|
||||
panels, and the same evidence re-aggregated with every case judged on one
|
||||
trial. Both red probabilities treat every observed failure as
|
||||
nondeterminism, so they are upper bounds for known-good inputs.
|
||||
|
||||
**False-blocking targets.** Periodic lane: at most 1.0 expected failed trial
|
||||
per census (P(red) ≤ 63%) at the 28-day check; gate census: at most 0.5
|
||||
(P(red) ≤ 39%). Neither target may be reached by a threshold, retry or
|
||||
rejudge change. A miss opens a P1 TODOS entry with the per-kind,
|
||||
per-failure-class breakdown from `test:health`.
|
||||
|
||||
### What v2 changes
|
||||
|
||||
- **Series identity (W4a, CEO-27, ENG-3).** A case's pass-rate series is keyed
|
||||
by the bytes it owns plus `HARNESS_VERSION`: its paid test file, its
|
||||
fixtures, and the prompt files of the skills it names (templates, sections,
|
||||
manifests and the generated `SKILL.md` the model reads). The series key adds
|
||||
the model id, the pinned Claude Code CLI version and the policy version.
|
||||
Shared helpers no longer start new series. Every trial also records the
|
||||
full consumed-input fingerprint (all touchfiles, globals included) as
|
||||
`series_fingerprint`, plus `harness_version`, as provenance.
|
||||
(`caseSeriesIdentitiesV2` in `scripts/eval-trial-series.ts`.)
|
||||
- **Harness guard.** `scripts/harness-version.json` pins the blob of every
|
||||
shared execution-harness file: `GLOBAL_TOUCHFILES` plus the session runners,
|
||||
the PTY harness, the e2e gate and the LLM judge (36 files). A free test
|
||||
(`test/harness-version.test.ts`) fails when one changes without a decision.
|
||||
`bun run scripts/bump-harness-version.ts --bump "<reason>"` raises
|
||||
`HARNESS_VERSION` and starts a new series for every case;
|
||||
`--non-behavioral "<reason>"` records the new blobs as compatible with no
|
||||
reset. Entries are one line per file so concurrent PRs that touch different
|
||||
files merge cleanly; on a conflict, take the higher version, then rerun.
|
||||
- **Weekly history (W4b).** Weeks for quarantine expiry and drift are
|
||||
`evals-periodic.yml` runs on `main`: scheduled runs and main dispatches.
|
||||
Trials from branch census runs in the same window pool into a series `main`
|
||||
has also run (option (b) below, approved 2026-10-05); a branch identity
|
||||
`main` has not run is dropped and never becomes the current series.
|
||||
- **Kinds (W4c).** No kind changes; see the evidence below.
|
||||
- **Pre-v2 evidence does not qualify.** Readers segment strictly by
|
||||
`policy_version`: only v2 trials count toward the quarantine entry rule; v1
|
||||
trials stay visible as display-only. Quarantine is not seeded at merge; a
|
||||
case enters only after it reaches `entry.minTrials` qualifying v2 trials,
|
||||
and `test:health` shows qualifying-vs-required counts with "insufficient
|
||||
evidence" distinct from healthy (DX-12).
|
||||
|
||||
### W4c: the named "stochastic" cases are deterministic in the record
|
||||
|
||||
Recorded census and PR trials (all 95 recorded revisions, October 2026):
|
||||
|
||||
| Case | Kind | Passed / trials | Failures |
|
||||
|---|---|---|---|
|
||||
| plan-ceo-finding-floor | rule | 26/27 | 1 timeout (608 s) |
|
||||
| plan-devex-finding-floor | rule | 27/27 | — |
|
||||
| plan-eng-finding-floor | rule | 9/9 | — |
|
||||
| plan-design-finding-floor | rule | 9/9 | — |
|
||||
| plan-ceo-review-plan-mode | rule | 34/34 | — |
|
||||
| plan-devex-review-plan-mode | rule | 27/27 | — |
|
||||
| plan-eng-review-plan-mode | rule | 9/9 | — |
|
||||
|
||||
The comments calling `plan-*-finding-floor` "stochastic ask-first" are stale:
|
||||
the record shows these cases behave like rules. AGENTS.md forbids changing a
|
||||
kind without pass-rate evidence, and a 3-trial panel would triple their cost
|
||||
for no measurable gain, so they stay `rule`. `plan-design-review-plan-mode`
|
||||
(17/22) is handled by its detector repair, not by a kind change.
|
||||
|
||||
### Backtest (ENG-3, DX-12)
|
||||
|
||||
`bun run scripts/eval-trial-series.ts --backtest --weeks 30 --records <dir>`
|
||||
replays identities with today's registry over historical trees (cases whose
|
||||
test file did not exist at a revision are absent there).
|
||||
|
||||
**Weekly main revisions** (29 Mondays, 06:00 UTC, 2026-03-16 to 2026-09-28).
|
||||
A case qualifies at a week when one identity holds long enough to collect 10
|
||||
trials within the last 10 weekly runs (rule and judge cases record one trial
|
||||
per run, behavior cases a panel of three):
|
||||
|
||||
| Identity | Qualifying at the latest week | Mean qualifying share, last 8 weeks | Median identity changes per case |
|
||||
|---|---|---|---|
|
||||
| v1 (own touchfiles minus globals) | 0/195 | 0.3% | 15 |
|
||||
| v2 (case-owned bytes) | 0/195 | 0.7% | 14 |
|
||||
| v2, reset on every pinned-harness change | 0/195 | 0.1% | 14 |
|
||||
| v2 without the generated `SKILL.md` (prototype) | 0/217 | 2.0% | 7 |
|
||||
|
||||
**Recorded trials** re-keyed by the v2 identity of the revision that produced
|
||||
them (blocking cases with at least 10 trials on one series):
|
||||
|
||||
| Trial source | Recorded (v1) identity | v2 identity |
|
||||
|---|---|---|
|
||||
| 22 census runs (5,665 trials) | 52/210 | 67/210 |
|
||||
| Census + PR lane, 95 revisions (14,898 trials) | 107/210 | 134/210 |
|
||||
|
||||
**Harness churn.** The pinned harness changed in 24 of 28 consecutive-week
|
||||
transitions, and 33 of the last 60 main commits (2026-08-04 to 2026-10-04)
|
||||
touched a pinned file. About half of PRs therefore record a
|
||||
`bump-harness-version` decision, and only `--bump` decisions reset series.
|
||||
|
||||
### What the backtest shows
|
||||
|
||||
1. v2 makes series honest and longer when trials are dense: on the recorded
|
||||
census runs, 67 blocking cases reach 10 trials on one series versus 52
|
||||
under v1, and 134 versus 107 with PR-lane trials included.
|
||||
2. Under the approved weekly history (main scheduled runs plus main
|
||||
dispatches, about one run per week), no blocking case reaches 10
|
||||
qualifying trials at any recent week. A rule case needs ten consecutive
|
||||
weekly runs on unchanged owned bytes, and a case's own prompt files
|
||||
change about every two weeks (the generated `SKILL.md` carries the shared
|
||||
preamble). Excluding the generated `SKILL.md` halves identity churn but
|
||||
still qualifies at most 2% of cases, and it would let preamble and resolver
|
||||
prompt changes pool unless every such edit went through the harness guard.
|
||||
3. So v2 alone does not make quarantine reachable. Reaching it needs one of:
|
||||
(a) keep the history as approved and accept that quarantine stays rare, so
|
||||
census redness is fixed at source (W3) and tracked by `test:health`;
|
||||
(b) pool trials from every v2 census run whose series key matches,
|
||||
including branch dispatches, while weeks for expiry still count from main
|
||||
only — the recorded backtest's 67/210, at the cost of pooling branch code
|
||||
whose unbumped harness edits match; or (c) extra census dispatches on an
|
||||
unchanged main revision (about $175 recorded each). Garry approved (b) on
|
||||
2026-10-05, and this PR implements it: pooled branch trials count only
|
||||
toward a series a `main` run also has.
|
||||
|
||||
### Alternatives considered
|
||||
|
||||
- **Gating vs trended.** The native CEO voice proposed keeping live-agent
|
||||
cases trended (reported, never blocking) and gating only on deterministic
|
||||
detectors. v2 keeps the census blocking because quarantine and the
|
||||
pass-rate gate already provide a trended path per case; the
|
||||
false-blocking targets above are the check on that choice. If the 28-day
|
||||
check misses them, trended reporting for the remaining stochastic cases is
|
||||
the next proposal.
|
||||
- **Adaptive sequential panels** (one trial, escalate to three on a failure,
|
||||
about 1.04× cost across ~190 cases) would be the strongest lever for a
|
||||
green census. They are not adopted: an escalation triggered by a failure is
|
||||
a retry, which Garry's rule forbids.
|
||||
- **Buy vs build.** Hosted eval platforms (Braintrust, LangSmith, Weights &
|
||||
Biases Weave) store per-case pass-rate history and cost natively, but
|
||||
gstack's verdicts are produced inside GitHub Actions from Claude Code
|
||||
sessions with pre-registered panels and receipts, and the history is a few
|
||||
thousand JSONL lines per week that `gh` already serves. A platform would add
|
||||
an external dependency and a second source of truth for verdicts without
|
||||
removing the in-repo policy. Building on the existing artifacts stays the
|
||||
choice; revisit if history outgrows artifact retention (90 days).
|
||||
|
||||
### Rollout and rollback (ENG-14, DX-12)
|
||||
|
||||
The reader-compatibility commit lands first: readers score only trials under
|
||||
their own `EVAL_POLICY.version`, show older-policy trials as display-only,
|
||||
and ignore newer-policy trials with a printed count. The v2 writer (identity
|
||||
stamping and `version: 2`) lands last. The supported rollback is reverting
|
||||
the v2 writer commits; a whole-PR revert is not claimed safe.
|
||||
|
||||
On a mixed v1/v2 fixture (10 passing v1 trials, 12 v2 trials with 3
|
||||
failures), the pre-PR reader at 2db0b3a pools the v2 records as the current
|
||||
v1 series and fails its gate:
|
||||
|
||||
```
|
||||
pass-rates: policy v1, 22 post-policy trial(s), 0 pre-policy (display only)
|
||||
FAILING rule gate 9/12 [46.8%–91.1%] ... plan-ceo-review-plan-mode (baseline reset)
|
||||
ACTION REQUIRED (2): [drift] ... [rule-as-behavior] ...
|
||||
exit=1
|
||||
```
|
||||
|
||||
The compatibility reader ignores the v2 records and keeps the v1 series:
|
||||
|
||||
```
|
||||
pass-rates: policy v1, 10 post-policy trial(s), 0 pre-policy (display only)
|
||||
ignored 12 trial(s) recorded under a policy newer than v1: this checkout predates them. Fix: ...
|
||||
PASSING rule gate 10/10 [72.2%–100%] ... plan-ceo-review-plan-mode
|
||||
exit=0
|
||||
```
|
||||
|
||||
## Wave evidence
|
||||
|
||||
### PR paid lane selection replay (CEO-23, ENG-18)
|
||||
|
||||
The replay sends each recorded PR push's changed files through today's
|
||||
selector and dependency graph:
|
||||
|
||||
```bash
|
||||
gh run list -R garrytan/gstack --workflow evals.yml --json databaseId,headSha,event,createdAt,conclusion -L 200 > runs.json
|
||||
bun run scripts/replay-pr-selection.ts --runs runs.json
|
||||
```
|
||||
|
||||
Over 168 PR pushes with plan manifests (2026-09-25 to 2026-10-04):
|
||||
|
||||
| Measure | Recorded | Replayed |
|
||||
|---|---:|---:|
|
||||
| Full-fallback pushes | 165 of 168 (98.2%) | 164 of 168 (97.6%) |
|
||||
| Misses (a failed case that depends on the diff and is no longer selected) | — | 0 |
|
||||
| Failed cases that no longer exist (counted apart) | — | 6 |
|
||||
|
||||
113 of the 168 diffs exceed the compare API's 300-file cap and are replayed on
|
||||
their first 300 files. The rate barely moves because the replay uses each push's
|
||||
cumulative branch diff, and the wave branches of that week touch shared inputs
|
||||
that correctly select every case: `scripts/` in 156 pushes, `.github/` in 125
|
||||
(mostly the eval workflows), `bin/` in 120, `make-pdf/` in 90, `test/helpers/` in
|
||||
86, `design/` in 67, `lib/` in 58, `hosts/` in 56 and `tsconfig*.json` in 37. The
|
||||
selector removes the single-class fallbacks (duration seeds, free-only workflows,
|
||||
`scripts/ubicloud/**`, free tests and free fixtures; run 36497566037 now selects
|
||||
the `pr` profile) and cannot narrow a branch that changes `bin/` or `lib/`;
|
||||
narrowing those needs touchfile mappings for `bin/`, `lib/`, `hosts/` and
|
||||
`make-pdf/`. The post-merge `test:health` check measures the fallback rate on
|
||||
ordinary PRs.
|
||||
|
||||
### Paid slices and concurrency (ENG-17)
|
||||
|
||||
Live census with `EVALS_ALL=1` (periodic: `--list --slice-budget S --jobs 2`;
|
||||
gate census: `--emit-plan <file> --slice-budget S --jobs 2 --skip-judges`):
|
||||
|
||||
| Lane | Before (540 s) | After (420 s) |
|
||||
|---|---|---|
|
||||
| Periodic slices / peak shard processes | 26 / 52 | 33 / 66 (max-parallel 26 → 36) |
|
||||
| Weekly gate census slices / peak | 12 / 24 | 16 / 32 (max-parallel 16 → 20) |
|
||||
| CI job ceiling | One value for every slice: 173 min periodic, 254 min gate census | Per slice: ordinary 50 min (30-minute shard wall + 20), periodic outliers 52 and 73 min, the overlay slice 173 min (5 serialized overlay wrappers); gate census 34–82 min |
|
||||
|
||||
The PR lane (`evals.yml`) uses the same 420-second budget and per-slice ceilings.
|
||||
|
||||
### Free suite
|
||||
|
||||
- The Windows lane runs on six `windows-latest` jobs packed by Windows-measured
|
||||
durations: about 4.6 minutes wall in the free-suite branch's validation runs,
|
||||
down from 8.1 minutes on one runner.
|
||||
- Unseeded files pack at the 99th percentile (was the 75th), and the seed growth
|
||||
ratchet (`test/free-seed-ratchet.test.ts`) keeps files over 60 seconds on a
|
||||
shrink-only allowlist.
|
||||
- The bash+zsh portability tests (#2669) run in CI now that the free-suite job
|
||||
installs zsh.
|
||||
- Native Windows and Dia qualification campaigns moved out of the PR workflow
|
||||
into the dispatch-only `native-qualification.yml`.
|
||||
|
||||
### Prose pins (D2)
|
||||
|
||||
D2 is approved: the prompt-byte contract in
|
||||
[test-value-bar.md](test-value-bar.md) covers machine-read tokens only, and
|
||||
sentence pins in 55 free test files became structural checks through
|
||||
`test/helpers/prompt-structure.ts`. A sentence pin here is a 40-or-more-character
|
||||
English string literal, or a prose regex of five or more words, asserted against
|
||||
SKILL.md, template or section text in a free test.
|
||||
|
||||
| Measure | main (2db0b3a) | This wave |
|
||||
|---|---:|---:|
|
||||
| Sentence pins | 2,332 in 217 files | 1,190 in 209 files |
|
||||
| Shouting pins (NEVER/MUST/ALWAYS/CRITICAL emphasis) | 8 | 4 |
|
||||
|
||||
The 4 remaining shouting pins are enum or marker tokens (`PLAN MODE EXCEPTION —
|
||||
ALWAYS RUN`, `[CRITICAL]`, `ALWAYS-ON` CLI output), not emphasis. Most remaining
|
||||
sentence pins sit in QA caller, detector-helper and workflow files that other
|
||||
work in flight owns.
|
||||
+21
-5
@@ -20,12 +20,28 @@ OpenClaw's `test-audit` skill (openclaw/openclaw@a214e76,
|
||||
- **Value bar**: the authoring gate plus the rule that a test which breaks under a
|
||||
behavior-preserving refactor asserts implementation, unless its exact output is the
|
||||
declared contract (goldens, prompt bytes, wire formats).
|
||||
- **Prompt-byte contract**: the machine-read tokens inside a prompt, nothing wider.
|
||||
A token qualifies when software consumes its exact bytes: a marker or sentinel
|
||||
line a parser, hook or grader matches; a field name or JSON shape a script reads;
|
||||
a command line, flag or environment variable the agent runs; an enum value or
|
||||
status word a caller branches on; a file path or placeholder another step opens;
|
||||
or an exact string a security boundary emits into model input. An English
|
||||
sentence is not a prompt-byte contract because the model reads it: rewording it
|
||||
must not break a test. Pin a sentence only when a recorded eval shows that exact
|
||||
wording changes behavior, and cite that eval next to the pin.
|
||||
- **Sentence pin**: a test that asserts an English sentence or clause (40 or more
|
||||
characters of prose, or a prose regex) against SKILL.md, a `.tmpl` template or a
|
||||
section file. Convert it to a structural check (the section or heading is
|
||||
present, steps keep their order, the machine-read token is present in the right
|
||||
section), or delete it when a behavioral eval already covers the behavior and
|
||||
name that eval in the commit. Check safety lines on meaning (a case-insensitive
|
||||
keyword in the right section), never on capitalization or emphasis.
|
||||
- **Retention bar**: keep a test that independently enforces a public API, protocol,
|
||||
config, migration, storage, security, platform, default, prompt-byte,
|
||||
generated-output (SKILL.md golden), package, release or architecture contract; call
|
||||
order when order is observable; source inspection when it is the cheapest
|
||||
independent guard. Static or slow is not a reason to delete. Anything reachable
|
||||
from the package entrypoint is never retired.
|
||||
config, migration, storage, security, platform, default, prompt-byte (as defined
|
||||
above), generated-output (SKILL.md golden), package, release or architecture
|
||||
contract; call order when order is observable; source inspection when it is the
|
||||
cheapest independent guard. Static or slow is not a reason to delete. Anything
|
||||
reachable from the package entrypoint is never retired.
|
||||
- **Value card**: the gate's four answers as one line,
|
||||
`Value: protects=<...>; fails_when=<...>; why_new=<...>; seam=none`. Each field is
|
||||
at most 160 UTF-8 bytes in that line (clamped to 157 bytes plus `...`); JSON keeps
|
||||
|
||||
Reference in new issue
Block a user