CalvinBackup/gstack
1
0 Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-10-31 22:14:25 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge wave/w-followups

Browse Source
This commit is contained in:
garrytan committed 2026-10-04 19:05:29 +00:00
2 parents ad22250b4d a1162e3d94
commit 4ea2c6eed7
16 files changed
+1056 -177
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/TESTING_INTERNALS.md
+107 -8
View File
@@ -96,9 +96,13 @@ against the previous finalized run (in-flight `_partial` files are never used as
a baseline, so a run can't compare against itself).
The periodic overlay fixtures use a versioned behavior gate with efficacy
reported separately. See [Overlay benchmark contract v2](OVERLAY_BENCHMARK_CONTRACT.md)
reported separately. See [Overlay benchmark contract v4](OVERLAY_BENCHMARK_CONTRACT.md)
for exact correctness requirements, retired fanout cases, immutable evidence,
and the limits of a passing result.
and the limits of a passing result. Under v4 a completed wrong answer is a
measured correctness result in either arm (`taskCorrect: false` with
`answerError`); only an overlay-ON wrong answer fails the case, so an
overlay-OFF wrong answer stays in the comparison. v3 verdicts keep their
original meaning.
## Coverage ownership
@@ -365,7 +369,22 @@ report and tracking issue, and nothing requires it. evals-periodic.yml runs ALL
periodic-tier files weekly (the coverage contract) minus the reasoned
exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking
required per entry; removal re-activates the file), plus a weekly
`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. The CI
`EVALS_ALL` gate census. A red census writes one report on every ref, to the
step summary and the `census-report-a<attempt>` artifact: the run's branch, sha
and event, both lanes' summaries, every pass-rate alarm line, the red ledger
and session headroom (`eval:pass-rates --reds` / `--headroom --runs 10 --branch
<ref>`), the all-green estimate and a link to the red-census guide, with
report-derived text passed through `scripts/lib/published-text.ts`. Only a run
on `refs/heads/main` comments on (or opens) the one tracking issue and closes it
on the next green run; the marathon lane does the same with its own issue and
`marathon-report-a<attempt>` artifact. `test/evals-tracking-issue.test.ts`
pins the main-only guard and runs both report scripts. The host-run Codex job
also carries the live Codex multi-block case (`codex-multiblock-live`: `./setup
--host codex` from the job's checkout into a fresh `CODEX_HOME`, the router in
`codex debug prompt-input`, and a later /learn block rooted by its own
prelude), and the periodic lane runs the design default-model smoke
(`design-model-smoke`), a visible SKIPPED with no coverage credit when
`OPENAI_API_KEY` is absent. The CI
image pins the claude CLI to an exact version (`.github/docker/Dockerfile.ci`,
enforced by `test/ci-image-cli-pin.test.ts` — bumps ride PRs that run the PTY
gate), and every eval-store run records `claude --version`, resolved once in
@@ -373,6 +392,7 @@ the runner parent and handed to shard children as `GSTACK_CLAUDE_CLI_VERSION`
(never spawned on a test thread), so a TUI-drift flake hunt is a grep, not
archaeology.
<a id="eval-verdict-policy"></a>
**Eval verdict policy** (`EVAL_POLICY` version 1 in
`test/helpers/periodic-exclude-data.ts`, pre-registered 2026-09-29). Paid evals
never retry. Each live case has exactly one kind in `E2E_KINDS`
@@ -397,6 +417,11 @@ and the kind fixes its trials before the run:
refusal counts as an unscored panel only when every sample refused.
`callJudge`'s 429 backoff happens before any model output and is transport,
not a verdict retry. The workflow-judge cache stores whole panels only.
The `rule` case auq-matrix uses the same panel inside its one execution: each
skill's single native capture has its recommendation substance scored by 3
judge samples, gated on their mean against the unchanged minimum of 4, and a
failed sample fails the panel without resampling (approved 2026-10-04 for
censuses after the change).
`panelVerdict()` (`test/helpers/eval-store.ts`) is the single verdict
function the report, `collector-outcomes.json`, the PR comment and pass-rates
@@ -414,6 +439,58 @@ reported. Changing any `EVAL_POLICY` constant after seeing census results needs
Garry's re-approval, a `version` bump and a fresh census;
`test/periodic-exclude-policy.test.ts` pins the approved values.
<a id="failure-causes"></a>
**Failure causes and details** (diagnostic only; verdicts unchanged under v1).
Every failed trial record keeps `failure_class` (`assertion`, `contract`,
`timeout`, `infra`: the field verdicts, INFRA re-dispatch and pass rates read)
and its 300-character `error` exactly as before. `failureCauseOf()`
(`test/helpers/eval-store.ts`, called by both record builders, including the
JUnit census for PTY and JUnit-only cases) adds an optional `failure_cause`
with a one-line `failure_cause_evidence`. It never changes `failure_class`.
The highest-precedence cause with evidence wins:
| `failure_cause` | Recorded when | First step |
|---|---|---|
| `contract` | `expectContract()` stamped a contract violation | fix the product; a violation fails a behavior panel outright |
| `pre_turn_infra` | the trial failed before the first model turn (`failure_class: 'infra'`) | check the runner and API status; an all-INFRA census is re-dispatched once |
| `api_error` | a session ended on an API error (for PTY, the `API Error:` panel on an idle turn) or the runner's exit reason was `error_api` | read the evidence line; a provider outage is a named red |
| `refusal` | the structured stop reason was a refusal (phrases only as a last resort; quoted refusal text in a tool result does not count) | check whether the fixture prompt invites the refusal |
| `provider_stall` | a session that streamed partial messages saw no event of any kind, thinking deltas included, for `STALL_WINDOW_MS` (120 s) while a model request was in flight and no tool, permission prompt, hook or subagent was outstanding; sessions without partial messages are never stalls | read the session ledger row; still a failed trial under v1 |
| `session_timeout` | a runner's armed session timeout fired | `eval:pass-rates --headroom`; cut work, never raise the budget |
| `observer_timeout` | the test's observer gave up waiting for an outcome (`outcome=timeout`) | read the transcript tail: a missed detector or a product that never answered |
| `assertion` | a check failed after the session ran (`session completed; check failed` when it ended normally) | the Expected/Received values or judge dimensions |
| `unknown` | the trial failed with no evidence for any cause above | inspect the shard directory |
A detector, harness or product fault is a human diagnosis, never a cause.
Readers before this field accept the new records (a frozen v1 reader runs over
them in `test/eval-store.test.ts`); an unknown cause from a newer gstack is
rejected with `unknown failure_cause "<x>" (written by a newer gstack; update
this checkout to read it)`.
`failure_detail` carries the values `error` cannot hold: a failed Bun matcher's
`{expected, received}`, or for a judge one `{dimension, mean, threshold,
samples, rationale}` entry per failing dimension, each part capped at 200
characters. `sessions[]` (at most 32) summarizes the trial's session-ledger
rows. `cost_known: false` marks a record or trial whose harness captured no
billing: PTY and Codex sessions never bill, and an SDK session bills only with a
terminal `total_cost_usd`. An absent field means known. Census and marathon
cost lines print the known sum and `N of M trial(s) cost unknown`, never a bare
total over unknowns.
<a id="session-ledger"></a>
**Session ledger.** Every runner (session-runner, the Agent SDK runner
including `runSharedInteractive`, the PTY session and `runRecordedCodexEval`)
appends one best-effort line per armed session to
`$GSTACK_EVAL_DIR/session-ledger.jsonl` (`test/helpers/session-ledger.ts`):
a stable key `<runner>:<label>#n`, the exact `budget_ms` it armed, monotonic
`elapsed_ms`, the `end` reason, sanitized evidence, `billed`, and a compact
liveness summary (longest in-request silence, the last event kind, any tool open
at the end). `claude -p` sessions stream partial messages for this summary;
transcripts keep their shape and size (full stream diagnostics only with
`publicStreamDiagnostics`). A failed ledger write never fails a trial; a trial
without ledger rows has no `sessions` and its headroom is `unknown`, never
scored. The ledger rides in each slice artifact beside the shard results.
**Quarantine** (`CASE_QUARANTINE`, same file). An entry needs: a per-trial rate
below 95% over at least 10 post-policy trials of the case's current input
identity (pre-policy backfill may justify only an initial entry, labeled as
@@ -430,8 +507,11 @@ weekly runs old, or when a tier is over its cap.
**Pass-rate history** (`bun run eval:pass-rates`, `scripts/eval-flake-rank.ts`).
It reads the `trial-outcomes` artifact of the last N completed
`evals-periodic.yml` runs on the current branch and `main` (flags: `--case`,
`--runs N`, `--branch`, `--dir`, `--backfill`, `--json`, `--gate`) and prints
`evals-periodic.yml` runs on the given `--branch` (default: the current branch)
and `main` (flags: `--case`, `--runs N`, `--branch`, `--repo`, `--workflow`,
`--dir`, `--backfill`, `--json`, `--gate`, and the views `--headroom`, `--reds`,
`--run <id>`; `--help` works offline, and an unknown flag or case id exits 2
before anything is fetched; every view prints its `scope:` line) and prints
per-case per-trial pass rates with 95% Wilson intervals. A series is one case
under one input identity, the hash of its own touchfiles minus
`GLOBAL_TOUCHFILES` (harness edits do not restart it), per model, Claude CLI
@@ -447,8 +527,26 @@ only, when a non-quarantined blocking case meets the entry rule (proposing an
entry), when a `rule` case does (rule case behaving like behavior: fix or
reclassify), when a blocking case's current identity is significantly below its
previous one (one-sided Fisher exact, α = 0.05, at least 6 trials each side,
Holm-controlled across the cases tested), and on the quarantine rules above.
History that cannot be fetched fails the gate closed.
Holm-controlled across the cases tested), on the quarantine rules above, and
with a `[headroom]` alarm when a case's slowest session ran above
`HEADROOM_FAIL` (85%) of the budget it armed. History that cannot be fetched
fails the gate closed.
The views (`scripts/lib/eval-history.ts`): `--headroom` lists each case's
slowest armed session against its budget (warn above `HEADROOM_WARN`, 75%;
`insufficient` below 3 samples; timed-out samples marked censored; `unknown`
without a session ledger, with the case wall beside it as a labelled upper
bound) and the census critical path from the slice job timings. `--reds` lists
verdict reds per census lane by failure class and cause, pooled and shrunk
per-case red rates, and the all-green probability Π(1 − p_i) over the latest
census's cases with an interval, labelled an approximation because verdicts are
not proven independent. `--run <id>` is one census's triage: each red with its
values and history, and the transcript evidence it downloads from only the
slices its reds name (a slice artifact over 64 MB is reported, not fetched).
The three thresholds `HEADROOM_WARN`, `HEADROOM_FAIL` and `STALL_WINDOW_MS`
live in `test/helpers/eval-budgets.ts`, outside `EVAL_POLICY`; a slow case gets
less work, never a larger budget. The red-census walkthrough is
[docs/evals/census-red.md](evals/census-red.md).
**The arithmetic.** With per-trial pass rate p, the chance a single case goes
red (a false red while the product works, the catch rate once it has
@@ -483,7 +581,8 @@ and judge verdicts at p_rule:
The rule term dominates: a green lane on a working product needs rule cases to
be near-deterministic (0.999), which is why failing detectors are converted to
outcome checks and product defects are fixed or named, and why each census
reports its expected lane false-red from the measured rates.
reports its all-green probability from the measured rates (`eval:pass-rates
--reds`, carried into the census report).
**Timeout policy.** Paid tests use the tiers in
`test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG);
docs/evals/census-red.md
+134
View File
@@ -0,0 +1,134 @@
# A census went red
A periodic census (`evals-periodic.yml`: the weekly periodic lane plus the
gate census) or a marathon run reports each red verdict as one line. This page
takes one red line to a diagnosis, using free commands first. A paid run is
allowed only after a repair.
Where the report is:
- **Main** (the scheduled weekly run, or a dispatch on `main`): a comment on the
open "Weekly periodic evals: red lane needs triage" issue (marathon: "Weekly
marathon evals: red lane needs triage").
- **Any branch**: the run's step summary and its `census-report-a<attempt>`
artifact (marathon: `marathon-report-a<attempt>`). A branch census never
comments on or closes main's issue.
## Prerequisites
- A checkout of the census branch with `bun install` done. Bun 1.4.0 is the
tested version.
- `gh` authenticated with read access to the repository's Actions (or
`GH_TOKEN` set). `eval:pass-rates` downloads artifacts through the REST zip
route, which also works where `gh run download` is blocked.
- The run id: the number at the end of the run URL in the report's first line.
## 1. Read the red line
A red line in the census report looks like this (one line; a behavior panel
with one failed trial shown):
```
✗ plan-review-prosons-neutral-neg behavior FAIL 1/3 (✗✗✓) t1: assertion (session completed; check failed) — expect(received).not.toMatch(expected) · cause assertion: session completed; check failed Expected: not /taste call/i · Received: "a taste call"; t2: assertion [t1@slice 7, t2@slice 9, t3@slice 9, attempt 1] evidence: paid-slice-7-a1/shards/skill-e2e-plan-prosons--plan-review-prosons-neutral-neg.t1 (fetch: bun run eval:pass-rates --run 37193478719 --case plan-review-prosons-neutral-neg) after a repair: bun run scripts/test-paid-shards.ts --tier periodic --case plan-review-prosons-neutral-neg --trials 3
```
| Field | What it says |
|---|---|
| case, kind, verdict | the registry id, its kind (`rule`, `behavior`, `judge`) and the verdict with each trial's outcome; the brackets name the slice and attempt of each trial |
| `failure_class` | the class verdicts read: `assertion`, `contract`, `timeout` or `infra` |
| `cause` | what the machine observed (`failure_cause`, glossary in [TESTING_INTERNALS](../TESTING_INTERNALS.md#failure-causes)), with its evidence line |
| Expected / Received | the failed matcher's values; for a judge, each failing dimension's mean, threshold, sample count and lowest rationale |
| `evidence:` | the slice artifact and shard directory holding the transcript |
| `after a repair:` | the command that runs only this case, once a repair exists |
The cause is a label. Under EVAL_POLICY v1 it never changes a verdict: a
`provider_stall` or `refusal` is still a failed trial. Whether an assertion
failure was a detector, harness or product fault is your diagnosis, not a
machine cause.
## 2. Inspect the evidence (free)
```bash
bun run eval:pass-rates --run <run-id> # every red of that census
bun run eval:pass-rates --run <run-id> --case <id> # one case
```
Expected output: one block per red, with its history and a local evidence path.
```
run 37198445662 (main @ 1a2b3c4d5, 2026-10-04T11:29:00Z): 2 verdict red(s)
✗ qa-only/SKILL.md workflow periodic/full FAIL (history: 7/8 verdicts green)
t1: assertion / cause assertion: session completed; check failed clarity 3.67 < 4 (3 samples) "…"
evidence: ~/.gstack/eval-pass-rates-cache/garrytan-gstack/37198445662/paid-slice-5-a1/shards/skill-llm-eval
```
Only the slices the reds name are downloaded, into
`<state root>/eval-pass-rates-cache/<owner>-<repo>/<run-id>/`. The shard
directory holds the eval record JSON (transcript, judge scores), `junit.xml`
and, for runs after the session ledger landed, `session-ledger.jsonl` (each
session's armed budget, elapsed time, end reason and liveness summary). A slice
artifact over 64 MB is reported as `too large to fetch`; download it from the
run page or with `gh api repos/<owner>/<repo>/actions/artifacts/<id>/zip > slice.zip`.
## 3. Put it in context (free)
```bash
bun run eval:pass-rates --reds --runs 10 --branch <branch> # reds per census by class and cause, all-green probability
bun run eval:pass-rates --headroom --runs 10 --branch <branch> # slowest session per case vs its armed budget
bun run eval:pass-rates --case <id> --runs 10 # the case's pass rate per input series
```
`--reds` ends with the all-green probability: Π(1 − p_i) over the latest
census's cases, with per-case red rates shrunk toward the pooled rate. It is an
approximation (verdicts are not proven independent). A case that is red in
several censuses is a repeat offender; its fix is structural (a replay corpus,
a structured signal or a speedup), not a one-off spelling patch.
What incomplete data means:
| Shown | Meaning |
|---|---|
| headroom `unknown (no session ledger)` | the trials predate the session ledger, or the runner wrote none; the case wall beside it is an upper bound, not a session clock |
| headroom `insufficient` | fewer than 3 session samples; never read as a pass |
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
| cause `unrecorded` | the trial was written before `failure_cause` existed |
| cause `unknown` | the trial failed with no evidence for any other cause |
| `cost unknown` | the harness (PTY, Codex) records no billing; only the known sum is shown |
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
## 4. Replay and repair (free)
Diagnose before changing code: product defect, test or detector defect, harness
defect, or infrastructure. Then reproduce with the smallest free check:
```bash
bun test test/detector-corpus-<case>.test.ts # cases with a replay corpus
bun run eval:pass-rates --reds --dir <evidence dir> # re-read downloaded artifacts offline
```
The repair lands with a free regression test that fails before it and passes
after it (`AGENTS.md`, "Validation discipline").
## 5. After a repair: one paid run
Run the line's `after a repair:` command, which selects only that case:
```bash
bun run scripts/test-paid-shards.ts --tier periodic --case <id> # a rule case: one diagnostic trial
bun run scripts/test-paid-shards.ts --tier periodic --case <id> --trials 3 # a behavior case: its 3-trial panel
```
A judge prints an `EVALS_JUDGE_SELECTION_JSON=... bun test <file> -t '<pattern>'`
command instead; a changed judge gets one 3-sample panel.
Rules (see [TESTING_INTERNALS](../TESTING_INTERNALS.md#eval-verdict-policy)):
- A paid run follows a concrete repair. A red census is never rerun on
unchanged inputs.
- A diagnostic run never forms a verdict. The next census on the repaired head
decides.
- A census whose every red is machine-classified INFRA or INCOMPLETE is
re-dispatched once automatically; both runs stay reported.
- An unresolved red is listed as a named red with its diagnosis. A case enters
`CASE_QUARANTINE` only under its entry rule, and a product defect is never
quarantined.
docs/troubleshooting.md
+168 -132
View File
@@ -242,6 +242,35 @@ contains no parsable version. /ship stops instead of inventing `0.0.0.0`.
**Expected result.** `gstack-version-bump classify` prints the current version.
<a id="review-base-stale"></a>
### `BASE_REFRESH: stale <rev>` / `Base coverage: stale at <rev>`
**Meaning.** /review could not fetch the base branch (offline, no credentials,
or a read-only `.git` such as Codex's sandbox), so it reviewed against your
local `origin/<base>` at `<rev>`.
**Fix.** Run `git fetch origin <base>` outside the sandbox and rerun /review if
the base has moved.
<a id="review-fingerprint-tmpdir"></a>
### `gstack-wtree: cannot create a temp index under <dir>; set TMPDIR to a writable directory` / `gstack-review-log: cannot create a private object directory under <dir>; no working-tree fingerprint. Set TMPDIR to a writable directory.`
**Meaning.** /review fingerprints the working tree in a private temporary
object directory (Codex's sandbox keeps `.git` read-only). The temp directory
could not be created, so this review has no fingerprint and later steps cannot
reuse its evidence.
**Fix.** `export TMPDIR=<a writable directory>`, then run /review again.
<a id="review-fingerprint-stage"></a>
### `gstack-wtree: cannot stage the working tree: <git error>`
**Meaning.** The fingerprint's objects could not be written. Standalone
`gstack-wtree` writes into the repository's object store.
**Fix.** Run it through /review (`gstack-review-log` supplies a private object
directory), or make `.git/objects` writable.
<a id="plan-audit-not-run"></a>
### `Plan completion audit: not run (no plan is bound to this branch and no docs/designs/ file matches). Fix: ...`
@@ -307,30 +336,6 @@ Or stop the reminder: `gstack-slug --adopt-legacy --dismiss <slug>`.
---
<a id="review-base-stale"></a>
### `BASE_REFRESH: stale <rev>` / `Base coverage: stale at <rev>`
**Meaning.** /review could not fetch the base branch (offline, no credentials,
or a read-only `.git` such as Codex's sandbox), so it reviewed against your
local `origin/<base>` at `<rev>`.
**Fix.** Run `git fetch origin <base>` outside the sandbox and rerun /review if
the base has moved.
<a id="review-fingerprint-objects"></a>
### `gstack-review-log: cannot create a private object directory under <dir>; no working-tree fingerprint.` / `gstack-wtree: cannot create a temp index under <dir>`
**Meaning.** The working-tree fingerprint needs a writable temp directory.
**Fix.** `export TMPDIR=<writable dir>` and rerun.
### `gstack-wtree: cannot stage the working tree: ...` / `gstack-wtree: cannot write the tree: ...`
**Meaning.** The object directory could not be written. `gstack-review-log`
supplies a private one; standalone `gstack-wtree` writes to `.git/objects`.
**Fix.** Run it through `gstack-review-log`, or make `.git/objects` writable.
## Design skills
<a id="design-not-available"></a>
@@ -353,6 +358,15 @@ If the message names a missing timeout tool, install coreutils (`gtimeout`) or p
**Expected result.** The next design skill prints `DESIGN_READY`.
<a id="design-image-model-invalid"></a>
### `GSTACK_DESIGN_IMAGE_MODEL="<value>" is not a gpt-image model name (expected something like gpt-image-2); fix it or unset it to use gpt-image-2`
**Meaning.** The design binary refuses an image tool model override that is not
a `gpt-image-*` model name, before sending any request.
**Fix.** `export GSTACK_DESIGN_IMAGE_MODEL=gpt-image-2` (or another gpt-image
model your key can use), or `unset GSTACK_DESIGN_IMAGE_MODEL`.
<a id="design-taste-profile-unavailable"></a>
### `TASTE_PROFILE_UNAVAILABLE: could not resolve the project slug (gstack-slug failed). Fix: run ./setup.`
@@ -375,15 +389,6 @@ pick a new `--output`), then copy the recovery file to where you want it.
---
<a id="design-image-model-invalid"></a>
### `GSTACK_DESIGN_IMAGE_MODEL="..." is not a gpt-image model name`
**Meaning.** The image model override must look like `gpt-image-2`; anything
else is refused before a request is sent. The live model check always tests
the defaults, whatever this is set to.
**Fix.** Set a gpt-image model name, or `unset GSTACK_DESIGN_IMAGE_MODEL`.
## Codex and other env-var hosts
<a id="gstack-no-install-found"></a>
@@ -446,30 +451,35 @@ stopped instead of touching your project.
## Setup and auto-update
<a id="bun-too-old"></a>
### `gstack needs Bun 1.3.3 or newer (1.4.0 recommended); found X at <path>. Nothing was installed or changed.`
### `gstack needs Bun 1.3.3 or newer (1.4.0 recommended); found <version> at <path>. Nothing was installed or changed.`
Also: `warning: gstack is tested on Bun 1.4.0 (CI pin); found X` and
`warning: could not read the Bun version (...); continuing.`
**Meaning.** Bun older than 1.3.3 accepts but ignores the flags that stop
**Meaning.** Bun older than 1.3.3 silently ignores the build flags that stop
gstack's compiled tools from reading a project's `.env`, so setup refuses it
before writing anything. Bun 1.3.3 to 1.3.x works but is untested; an
unreadable version only warns.
before writing anything. The same link appears on two warnings that do not stop
setup: `warning: gstack is tested on Bun 1.4.0 (CI pin); found <version>` (1.3.3
up to 1.4.0 works but is untested) and `warning: could not read the Bun version
(...)` (setup continued; check `bun --version`).
**Fix.** `bun upgrade`, then `./setup`.
**Fix.**
```bash
bun upgrade
./setup
```
**Expected result.** `bun --version` prints 1.4.0 or newer and setup finishes
with no Bun warning.
<a id="auto-update-bun-too-old"></a>
### `gstack auto-update: update held (bun-too-old: found Bun X at <path>; gstack V needs F or newer); nothing was installed or changed.`
### `gstack auto-update: update held (bun-too-old: found Bun <version> at <path>; gstack <version> needs <floor> or newer); nothing was installed or changed. Fix now: ...`
Also `/gstack-upgrade`'s `BUN_TOO_OLD: bun-too-old: ...; nothing was changed`.
**Meaning.** Team-mode auto-update fetched a release whose setup needs a newer
Bun than the one setup would run. It left your checkout and installed skills at
the current revision and prints this line once per session start. (An older
auto-updater running its first update cannot make this check.)
**Meaning.** The incoming release needs a newer Bun than the one setup would
run, so the update stopped before moving your checkout. Your installed version
keeps working. The first update run by an auto-updater older than this check
cannot do it.
**Fix.** `bun upgrade`. The next session start resumes the update, or run
`cd <gstack checkout> && git pull --ff-only && ./setup` (or `/gstack-upgrade`).
**Fix.** `bun upgrade`; the next session start resumes the update. To update
now: `bun upgrade && cd <gstack checkout> && git pull --ff-only && ./setup`.
<a id="auto-update-incomplete"></a>
### `gstack auto-update: setup did not finish (<reason>); installed skills may be out of date. gstack retries automatically. Fix now: cd <dir> && ./setup`
@@ -581,16 +591,19 @@ after `--`).
**Fix.** `cd <gstack checkout> && ./setup` (or `/unfreeze`).
---
<a id="browse-chromium-pid-unrecorded"></a>
### `[browse] Could not record the Chromium PID (CDP SystemInfo.getProcessInfo: ...); browse stop cannot reap a surviving Chromium.`
### `[browse] Could not record the Chromium PID (CDP SystemInfo.getProcessInfo: <error>); browse stop cannot reap a surviving Chromium.`
**Meaning.** gstack reads the PID of a browser it launched over CDP so `browse
stop` can clean up a Chromium that outlives the server. That read failed, so
this session's browser is not reaped automatically.
**Meaning.** For a headless Chromium it launched, browse reads the browser's
process id over CDP so `browse stop` can kill a browser that outlives the
server. That read failed or took longer than 2 seconds, so this session's
browser is not recorded. Browsers gstack did not launch are never recorded or
killed.
**Fix.** After `browse stop`, check `ps` for a leftover Chromium and kill it by hand.
**Fix.** After `browse stop`, check for a leftover browser with
`ps aux | grep -i chrom` and end it with `kill <pid>`.
---
## Memory and gbrain
@@ -665,6 +678,104 @@ gstack-gbrain-sync --prune-gone-worktrees
---
## Eval reports and pass-rates
These come from the paid eval census reports and `bun run eval:pass-rates`.
The walkthrough for a red census is [docs/evals/census-red.md](evals/census-red.md).
<a id="pass-rates-unknown-flag"></a>
### `eval:pass-rates: unknown flag <flag>` / `eval:pass-rates: unknown case <id> (not an E2E, judge or paid test file id)`
**Meaning.** The flag or case id is not recognized. The command exits 2 before
fetching anything.
**Fix.** `bun run eval:pass-rates --help` lists the flags. `--case` takes a
registry id from `test/helpers/touchfiles-data.ts` or a paid test file path.
<a id="failure-cause-newer"></a>
### `unknown failure_cause "<x>" (written by a newer gstack; update this checkout to read it)`
**Meaning.** A trial record names a failure cause this checkout does not know.
A newer gstack wrote it.
**Fix.** `git pull` (or rebase your branch onto `main`), then rerun the command.
<a id="paid-case-several-owners"></a>
### `--case <id>: registered by <file>, <file>; it needs exactly one`
**Meaning.** More than one paid test file registers the case, so
`test-paid-shards.ts --case` cannot pick one file.
**Fix.** Run the red shard's file directly; the report's `after a repair:` line
prints it: `EVALS=1 EVALS_TIER=<tier> bun test <file>`.
<a id="evidence-too-large"></a>
### `evidence: <artifact> too large to fetch (<N> MB)`
**Meaning.** The slice artifact holding this red's transcript is over 64 MB, so
`eval:pass-rates --run` does not download it.
**Fix.** Download it from the run page, or
`gh api repos/<owner>/<repo>/actions/artifacts/<artifact id>/zip > slice.zip`.
<a id="headroom-alarm"></a>
### `[headroom] <case> session <key>: max <s> of <s> (<pct>) over <n> sample(s), above the 85% cap. Cut work in the skill or fixture; budgets are never raised ...`
**Meaning.** `eval:pass-rates --gate` (the weekly report) found a case whose
slowest session used more than 85% of the timeout it armed. It is one timeout
away from a red.
**Fix.** Cut work in the skill or fixture. Budgets are never raised. Check the
case with `bun run eval:pass-rates --headroom --case <id>`.
<a id="cost-unknown"></a>
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
**Meaning.** PTY and Codex sessions record no billing, so their cost is
unknown. Only the known sum is shown.
**Fix.** None needed; this is informational.
<a id="cause-provider-stall"></a>
### `· cause provider_stall: no stream event for <N>s ...`
**Meaning.** The session streamed partial messages, then received no event of
any kind for at least 120 seconds while a model request was in flight and no
tool, permission prompt, hook or subagent was outstanding. Under EVAL_POLICY v1
it is still a failed trial.
**Fix.** Inspect it with `bun run eval:pass-rates --run <run id> --case <id>`.
A paid rerun happens only after a repair.
<a id="overlay-wrong-answer"></a>
### Overlay record with `taskCorrect: false` and `answerError` (contract v4)
**Meaning.** The trial completed with a wrong answer. Under overlay contract v4
that is a valid measurement in the comparison; only an overlay-ON wrong answer
fails the case.
**Fix.** Read `answerError` in the trial JSON. No rerun is needed.
<a id="detector-corpora-stale"></a>
### `[detector-corpora] <N> stale entries (refresh from a current census when one fails)`
**Meaning.** Some replay-corpus entries came from an older input series of
their case. This is informational; the corpus still replays them.
**Fix.** When that case goes red in a census, add the new capture to its corpus
and check it with `bun test test/detector-corpus-<case>.test.ts`.
<a id="auq-substance-panel"></a>
### `recommendation substance mean <x> < 4 over samples [<a>,<b>,<c>] (boilerplate/weak)`
**Meaning.** The auq-matrix case scores each captured question's recommendation
with a 3-sample judge panel; their mean was below 4.
**Fix.** Read the logged samples and the captured question. A fix goes in the
skill text, followed by one diagnostic run of auq-matrix.
---
## Push guard (redaction)
<a id="redact-postgres-default-pair"></a>
@@ -685,78 +796,3 @@ Bypass once: `GSTACK_REDACT_PREPUSH=skip git push`.
**Fix.** Write it as `version: 1.2.3.4`, `"version": "..."` or `v1.2.3.4`.
MEDIUM findings do not block pushes.
---
## Evals and census reports (contributors)
<a id="pass-rates-unknown-flag"></a>
### `eval:pass-rates: unknown flag --x` / `unknown case <id>`
**Meaning.** The flag or case id is not recognized; it exits 2 before fetching anything.
**Fix.** `bun run eval:pass-rates --help`; use a registry case id or a paid test file path.
<a id="unknown-failure-cause"></a>
### `unknown failure_cause "<x>" (written by a newer gstack; update this checkout to read it)`
**Fix.** `git pull` (or rebase your branch) and rerun.
<a id="case-registered-by-several"></a>
### `--case <id>: registered by a, b; it needs exactly one`
**Fix.** Run the red shard's file: `EVALS=1 EVALS_TIER=<tier> bun test <file>`
(the report's "after a repair:" line prints it).
<a id="evidence-too-large"></a>
### `evidence: <artifact> too large to fetch (N MB)`
**Fix.** Download it from the run page, or
`gh api repos/<owner>/<repo>/actions/artifacts/<id>/zip > slice.zip`.
<a id="headroom-alarm"></a>
### `[headroom] <case> session <key>: max Xs of Ys (Z%) ... above the 85% cap`
**Meaning.** The case's slowest session used more than 85% of the timeout it armed.
**Fix.** Cut work in the skill or fixture. Budgets are never raised.
<a id="cost-unknown"></a>
### `cost unknown (no billing captured for any of N trial(s))` / `... N of M trial(s) cost unknown`
**Meaning.** PTY and Codex harnesses record no billing, so only the known sum is
shown. Informational.
<a id="provider-stall"></a>
### Red line `· cause provider_stall: no stream event for Ns ...`
**Meaning.** The model request went silent for at least 120 s with no tool,
permission prompt, hook or subagent outstanding. Under EVAL_POLICY v1 this is
still a failure.
**Fix.** Inspect it with `bun run eval:pass-rates --run <id> --case <id>`. A
paid rerun only after a repair.
<a id="overlay-contract-v4"></a>
### `overlay ... contract-v4 ... taskCorrect: false` with `answerError` set
**Meaning.** The trial completed with a wrong answer; that is a valid
measurement in the comparison. Only an overlay-on wrong answer fails the case.
**Fix.** Read `answerError` in the trial JSON. No rerun needed.
<a id="detector-corpora-stale"></a>
### `[detector-corpora] N stale entries (refresh from a current census when one fails)`
**Meaning.** Corpus entries came from an older input series of their case. Informational.
**Fix.** When that case goes red in a census, add the new capture and check it
with `bun test test/detector-corpus-<case>.test.ts`.
<a id="auq-substance-mean"></a>
### `recommendation substance mean X < 4 over samples [a,b,c]` (auq-matrix)
**Meaning.** The 3-sample judge panel's mean was below the threshold.
**Fix.** Read the logged samples and the captured question; fix the skill
text, then run one diagnostic.