mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 14:06:50 +01:00
Garry Tanandcapy-ai[bot]
b004c06dbc
* fix(ship-queue): fill trains at 3 READY PRs or one hour, and price every case shard's trial
The 5-PR / 3-hour fill left READY PRs waiting most of a day; .gstack/ship-policy.json
now fills at max_size 3 or fill_hours 1, whichever comes first (fillDue already
dispatches at either bound). train verdict printed cost=$24.41 for a train whose slice
report said cost $32.92 known + 54 of 136 trial(s) cost unknown: E2E records named by a
display label ("/benchmark workflow") were never attributed to their case id, so the
per-case sum was short. A case shard's records now belong to its one case; a multi-case
shard with display-labelled records keeps those trials unknown. The ATTEMPT: line names
how many cases have no cost instead of printing a partial sum as the total.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(ci): record the paid-report.ts harness decision and keep ship-queue-train-cost.test.ts off the Windows subset
scripts/lib/paid-report.ts is a pinned shared-harness file, so test/harness-version.test.ts wants a HARNESS_VERSION decision for its edit: non-behavioral, because a case shard's records are attributed to its one case for the trial's cost, class and error fields while no case's outcome, panel verdict or selection changes.
test/ship-queue-train-cost.test.ts drives bin/gstack-ship-queue against the #!/bin/sh gh shim, which windows-latest cannot run (windows-free-shard 2 on #3478 at d5e6a7d), so it joins test/ship-queue-train.test.ts in the Windows exclusion list.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: pin a failed display-labelled trial's fields and the $0.00 cost line; say so in the CHANGELOG
The widened record filter also feeds failedRecord, so a failed display-labelled trial in a case shard now takes failure_class, exit_reason and error from its record (train-gate classes a red infra record INFRA instead of VERDICT). The CHANGELOG entry said only the cost fields changed; it now says both. A train priced at zero prints cost=$0.00.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore(harness): say which panel class the paid-report decision moves
The non-behavioral note now names the one verdict-side effect of attributing a case shard's records to its case: a red whose own record says infra or timeout is classed INFRA instead of VERDICT, which changes rerun and bisect eligibility and not a pass rate.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: document one-case-shard cost attribution and the train verdict cost line; put the undercounted train costs on record
/document-release (ship-owned audit) added the attribution rule and the ATTEMPT: cost-line form to docs/TESTING_INTERNALS.md, docs/troubleshooting.md#cost-unknown and docs/evals/census-red.md. The CHANGELOG headline now says every train cost the coordinator reported until this fix was the undercounted verdict figure (about $24 per train against about $33 real).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: the train verdict prints the known+unknown cost form only when some but not every case is priced
The second /document-release audit tightened docs/TESTING_INTERNALS.md and docs/troubleshooting.md#cost-unknown: a train record with no priced case prints no cost at all (classifyGate's cost is null). The CHANGELOG bullet says the same.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* v1.91.255.0 stamp
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
135 lines
7.1 KiB
Markdown
135 lines
7.1 KiB
Markdown
# A census went red
|
||
|
||
A periodic census (`evals-periodic.yml`: the weekly periodic lane plus the
|
||
gate census) or a marathon run reports each red verdict as one line. This page
|
||
takes one red line to a diagnosis, using free commands first. A paid run is
|
||
allowed only after a repair.
|
||
|
||
Where the report is:
|
||
|
||
- **Main** (the scheduled weekly run, or a dispatch on `main`): a comment on the
|
||
open "Weekly periodic evals: red lane needs triage" issue (marathon: "Weekly
|
||
marathon evals: red lane needs triage").
|
||
- **Any branch**: the run's step summary and its `census-report-a<attempt>`
|
||
artifact (marathon: `marathon-report-a<attempt>`). A branch census never
|
||
comments on or closes main's issue.
|
||
|
||
## Prerequisites
|
||
|
||
- A checkout of the census branch with `bun install` done. Bun 1.4.2 is the
|
||
tested version.
|
||
- `gh` authenticated with read access to the repository's Actions (or
|
||
`GH_TOKEN` set). `eval:pass-rates` downloads artifacts through the REST zip
|
||
route, which also works where `gh run download` is blocked.
|
||
- The run id: the number at the end of the run URL in the report's first line.
|
||
|
||
## 1. Read the red line
|
||
|
||
A red line in the census report looks like this (one line; a behavior panel
|
||
with one failed trial shown):
|
||
|
||
```
|
||
✗ plan-review-prosons-neutral-neg behavior FAIL 1/3 (✗✗✓) t1: assertion (session completed; check failed) — expect(received).not.toMatch(expected) · cause assertion: session completed; check failed Expected: not /taste call/i · Received: "a taste call"; t2: assertion [t1@slice 7, t2@slice 9, t3@slice 9, attempt 1] evidence: paid-slice-7-a1/shards/skill-e2e-plan-prosons--plan-review-prosons-neutral-neg.t1 (fetch: bun run eval:pass-rates --run 37193478719 --case plan-review-prosons-neutral-neg) after a repair: bun run scripts/test-paid-shards.ts --tier periodic --case plan-review-prosons-neutral-neg --trials 3
|
||
```
|
||
|
||
| Field | What it says |
|
||
|---|---|
|
||
| case, kind, verdict | the registry id, its kind (`rule`, `behavior`, `judge`) and the verdict with each trial's outcome; the brackets name the slice and attempt of each trial |
|
||
| `failure_class` | the class verdicts read: `assertion`, `contract`, `timeout` or `infra` |
|
||
| `cause` | what the machine observed (`failure_cause`, glossary in [TESTING_INTERNALS](../TESTING_INTERNALS.md#failure-causes)), with its evidence line |
|
||
| Expected / Received | the failed matcher's values; for a judge, each failing dimension's gating median (the mean before EVAL_POLICY v3), threshold, sample count and lowest rationale |
|
||
| `evidence:` | the slice artifact and shard directory holding the transcript |
|
||
| `after a repair:` | the command that runs only this case, once a repair exists |
|
||
|
||
The cause is a label. Under EVAL_POLICY v1 it never changes a verdict: a
|
||
`provider_stall` or `refusal` is still a failed trial. Whether an assertion
|
||
failure was a detector, harness or product fault is your diagnosis, not a
|
||
machine cause.
|
||
|
||
## 2. Inspect the evidence (free)
|
||
|
||
```bash
|
||
bun run eval:pass-rates --run <run-id> # every red of that census
|
||
bun run eval:pass-rates --run <run-id> --case <id> # one case
|
||
```
|
||
|
||
Expected output: one block per red, with its history and a local evidence path.
|
||
|
||
```
|
||
run 37198445662 (main @ 1a2b3c4d5, 2026-10-04T11:29:00Z): 2 verdict red(s)
|
||
✗ qa-only/SKILL.md workflow periodic/full FAIL (history: 7/8 verdicts green)
|
||
t1: assertion / cause assertion: session completed; check failed clarity 3.67 < 4 (3 samples) "…"
|
||
evidence: ~/.gstack/eval-pass-rates-cache/garrytan-gstack/37198445662/paid-slice-5-a1/shards/skill-llm-eval
|
||
```
|
||
|
||
Only the slices the reds name are downloaded, into
|
||
`<state root>/eval-pass-rates-cache/<owner>-<repo>/<run-id>/`. The shard
|
||
directory holds the eval record JSON (transcript, judge scores), `junit.xml`
|
||
and, for runs after the session ledger landed, `session-ledger.jsonl` (each
|
||
session's armed budget, elapsed time, end reason and liveness summary). A slice
|
||
artifact over 64 MB is reported as `too large to fetch`; download it from the
|
||
run page or with `gh api repos/<owner>/<repo>/actions/artifacts/<id>/zip > slice.zip`.
|
||
|
||
## 3. Put it in context (free)
|
||
|
||
```bash
|
||
bun run eval:pass-rates --reds --runs 10 --branch <branch> # reds per census by class and cause, all-green probability
|
||
bun run eval:pass-rates --headroom --runs 10 --branch <branch> # slowest session per case vs its armed budget
|
||
bun run eval:pass-rates --case <id> --runs 10 # the case's pass rate per input series
|
||
```
|
||
|
||
`--reds` ends with the all-green probability: Π(1 − p_i) over the latest
|
||
census's cases, with per-case red rates shrunk toward the pooled rate. It is an
|
||
approximation (verdicts are not proven independent). A case that is red in
|
||
several censuses is a repeat offender; its fix is structural (a replay corpus,
|
||
a structured signal or a speedup), not a one-off spelling patch.
|
||
|
||
What incomplete data means:
|
||
|
||
| Shown | Meaning |
|
||
|---|---|
|
||
| headroom `unknown (no session ledger)` | the trials predate the session ledger, or the runner wrote none; the case wall beside it is an upper bound, not a session clock |
|
||
| headroom `insufficient` | fewer than 3 session samples; never read as a pass |
|
||
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
|
||
| cause `unrecorded` | the trial was written before `failure_cause` existed |
|
||
| cause `unknown` | the trial failed with no evidence for any other cause |
|
||
| `cost unknown` | the harness (PTY, Codex) records no billing, or the trial sits in a multi-case shard whose records are named by display label and cannot be attributed to it; only the known sum is shown |
|
||
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
|
||
|
||
## 4. Replay and repair (free)
|
||
|
||
Diagnose before changing code: product defect, test or detector defect, harness
|
||
defect, or infrastructure. Then reproduce with the smallest free check:
|
||
|
||
```bash
|
||
bun test test/detector-corpus-<case>.test.ts # cases with a replay corpus
|
||
bun run eval:pass-rates --reds --dir <evidence dir> # re-read downloaded artifacts offline
|
||
```
|
||
|
||
The repair lands with a free regression test that fails before it and passes
|
||
after it (`AGENTS.md`, "Validation discipline").
|
||
|
||
## 5. After a repair: one paid run
|
||
|
||
Run the line's `after a repair:` command, which selects only that case:
|
||
|
||
```bash
|
||
bun run scripts/test-paid-shards.ts --tier periodic --case <id> # a rule case: one diagnostic trial
|
||
bun run scripts/test-paid-shards.ts --tier periodic --case <id> --trials 3 # a behavior case: its 3-trial panel
|
||
```
|
||
|
||
A judge prints an `EVALS_JUDGE_SELECTION_JSON=... bun test <file> -t '<pattern>'`
|
||
command instead; a changed judge gets one 3-sample panel.
|
||
|
||
Rules (see [TESTING_INTERNALS](../TESTING_INTERNALS.md#eval-verdict-policy)):
|
||
|
||
- A paid run follows a concrete repair. A red census is never rerun on
|
||
unchanged inputs.
|
||
- A diagnostic run never forms a verdict. The next census on the repaired head
|
||
decides.
|
||
- A census whose every red is machine-classified INFRA or INCOMPLETE is
|
||
re-dispatched once automatically; both runs stay reported.
|
||
- An unresolved red is listed as a named red with its diagnosis. A case enters
|
||
`CASE_QUARANTINE` only under its entry rule, and a product defect is never
|
||
quarantined.
|