v1.93.11.0 stamp: merge main daf661374f into #3872 (land)

This commit is contained in:
garrytan committed 2026-11-02 17:44:22 +00:00
commit 964f0cc2e9
184 files changed
+3881 -516

No files matched your search

+114 -14
View File
@@ -651,15 +651,37 @@ this checkout to read it)`.
samples, rationale}` entry per failing dimension (since EVAL_POLICY v3 `mean`
holds the panel's gating median), each part capped at 200
characters. `sessions[]` (at most 32) summarizes the trial's session-ledger
rows. `cost_known: false` marks a record or trial whose harness captured no
billing: PTY and Codex sessions never bill, and an SDK session bills only with a
terminal `total_cost_usd`. It also marks a trial no result record could be
attributed to: a record named by the case id or carrying `case_id` is that
case's, and a record named by a display label (`/qa fix loop`) is the shard's
only case's when the shard ran one case, so a multi-case shard whose records
carry display labels leaves those trials cost-unknown rather than guessing
which case paid (`junitCensus` in `scripts/lib/paid-report.ts`). An absent
field means known. Census and marathon cost lines print the known sum and
rows. `cost_known: false` marks a record or trial whose cost is not fully
known; an absent field means known. A record carries it when its harness
captured no billing (a Codex session that completed no turn or whose model has
no price row). A rule case's trial in the JUnit census (`caseCost` in
`scripts/lib/paid-report.ts`) costs its records' sum plus what no record
carries: each ledger session without a terminal result (`billed: false`) adds
its `streamed_cost_usd`, and a trial with no record adds each billed session's
own `cost_usd` (Codex). The trial is cost-unknown when any record says
`cost_known: false`, when any of those sessions has no priced cost (a PTY
session whose transcript does not price, a `claude -p` session that ended
without a result, an aborted Agent SDK session with no usage or an unpriced
model), or when it has neither records
nor sessions. A case in `MODEL_FREE_CASES` (`timeline-event-flow`, which runs
no model) with neither is a known $0. A panel trial (`behavior`, `judge`) from
an isolated trial shard keeps the older rule (`trialCostKnown` in
`test/helpers/eval-store.ts`): known only with at least one record and every
session billed, and streamed cost is not added.
Records are attributed to a case this way: a record named by the case id or
carrying `case_id` is that case's, and a record named by a display label
(`/qa fix loop`) is the shard's only case's when the shard ran one case. In a
multi-case shard, a display-label record whose `runSkillTest` test name is a
registered case id carries `cost_case_id` (`recordE2E`), and the census adds
its cost to that case only: the failed record, failure class and model still
come from the records named by case id, so no verdict changes. A display-label
record without `cost_case_id` in a multi-case shard is not attributed rather
than guessed, and its case is cost-unknown when nothing else prices it
(`junitCensus`). The report's headline `cost $<x>` adds the streamed session
cost no collector record carries, so a newly priced trial counts once.
Census and marathon cost lines print the known sum and
`N of M trial(s) cost unknown`, and a train verdict's `ATTEMPT:` line prints
`cost=$<known> known + <n> of <m> case(s) cost unknown` when some but not
every case in the train record has a cost (a record with no priced case prints
@@ -671,13 +693,33 @@ including `runSharedInteractive`, the PTY session and `runRecordedCodexEval`)
appends one best-effort line per armed session to
`$GSTACK_EVAL_DIR/session-ledger.jsonl` (`test/helpers/session-ledger.ts`):
a stable key `<runner>:<label>#n`, the exact `budget_ms` it armed, monotonic
`elapsed_ms`, the `end` reason, sanitized evidence, `billed`, and a compact
`elapsed_ms`, the `end` reason, sanitized evidence, `billed`, the billed
`model`, `cost_usd` and token classes of a Codex session, and a compact
liveness summary (longest in-request silence, the last event kind, any tool open
at the end). `claude -p` sessions stream partial messages for this summary;
transcripts keep their shape and size (full stream diagnostics only with
`publicStreamDiagnostics`). A failed ledger write never fails a trial; a trial
without ledger rows has no `sessions` and its headroom is `unknown`, never
scored. The ledger rides in each slice artifact beside the shard results.
scored. An Agent SDK session that ends without a terminal result (aborted,
thrown) also records `streamed_cost_usd`: its streamed assistant usage priced
by `streamedUsageCostUsd` (`lib/pricing.ts`), each message id once at its
largest output, subagent turns included, `<synthetic>` messages skipped,
5-minute cache writes at 1.25x and 1-hour writes at 2x the input rate. The
field is absent when no message carries usage or any message's model has no
price row, so an unpriced session stays cost-unknown, never $0. A PTY session
records `streamed_cost_usd` from its transcript the same way. Both PTY ledger
writers price `<CLAUDE_CONFIG_DIR>/projects/*/<id>.jsonl` plus
`<id>/subagents/*.jsonl` (`test/helpers/pty/transcript-cost.ts`), where the
id is the caller's `--session-id`, else the one transcript that appeared
during this launch for its cwd. The launcher adds no argument. Pricing never
takes the whole directory, because concurrent PTY tests in one process share
one hermetic config dir; two new transcripts for one cwd are ambiguous and stay
unknown. A non-hermetic or resumed launch, a caller that opts out
(`priceTranscript: false`, the native AskUserQuestion capture, which never
reads model transcripts), a missing transcript, no usage, an unpriced model or
a read error leaves the field absent (cost-unknown), and a pricing failure
never fails the test. The row keeps `billed: false`. The ledger rides in each slice
artifact beside the shard results.
**Quarantine** (`CASE_QUARANTINE`, same file). An entry needs: a per-trial rate
below 95% over at least 10 post-policy trials of the case's current input
@@ -1232,9 +1274,11 @@ observation never removes a command and a missing or empty artifact never erases
one; removing a command is a reviewed edit. While the map is in shadow mode it
changes no selection, so a PR that changes only the map selects no paid case.
Dollar figures are lower bounds. Known cost excludes PTY trials, whose cost is
unknown: about 39% of observed case runs in the committed map. The shadow block
and `test:health` both say so.
Dollar figures are lower bounds. Known cost excludes trials whose cost is
unknown: about 39% of observed case runs in the committed map, mostly PTY
trials recorded before PTY sessions were priced from their transcripts
([Failure causes and details](#failure-causes)). The shadow block and
`test:health` both say so.
- **Shadow block:** `prCoverage.shadow` in the paid-plan manifest lists the gate
cases the map would select, would drop and would add, a provenance line per
@@ -1750,6 +1794,32 @@ lower-case words in path components under `test/fixtures/`; a substring match
separate change with its own touchfile updates; the list is here so that change
can be scoped.
## Agent SDK system prompt
An Agent SDK caller that leaves out `systemPrompt` doesn't get Claude Code's
prompt. It gets the SDK's bare stub, and the stub drops the line that tells the
model to run `/skill` through the Skill tool. A skill arm then never loads its
skill, so a "no difference" result says nothing about the skill (FOUN-32 found
this in another crew's null result).
**Every caller says which prompt it uses.** `runAgentSdkTest` and every direct
SDK `query()` call pass the Claude Code preset,
`{ type: 'preset', preset: 'claude_code' }` (also exported as
`CLAUDE_CODE_PRESET` from `test/helpers/sdk-system-prompt.ts`; an overlay arm
adds `append`), or name why not: `{ type: 'sdk-default', reason }` for a stub
provider or a connectivity probe, `{ type: 'custom', text, reason }` for an arm
that replaces the prompt. The runner refuses a missing, empty, bare-string or
unnamed prompt before it starts, and
`test/sdk-system-prompt-tripwire.test.ts` scans test/ and scripts/ for the same
mistake in free CI. Arms that compare loadouts, the no-gstack arm included,
keep the same preset and differ only in what the arm loads.
**Record it, and treat a null on the stub as a method bug.** Every eval report
and results table names the prompt each arm ran under (`claude_code`,
`claude_code + append`, or the named reason). A null or negative result from an
arm that ran on the SDK default is FIX-METHOD under R48 ("Why null?"): fix the
prompt and rerun before reading it as KEEP or CUT-CANDIDATE.
## No-gstack arm
Garry, 2026-10-31: "all of our head to head tests have to test no gstack from
@@ -1843,6 +1913,36 @@ it as the diagnostic. Candidate folders and git identity name a project, never
the arm. The model panel (`test/skill-e2e-model-panel.test.ts`) runs a
`<model>#no-gstack` arm beside each model (GST-468, GST-469).
## Frontier models under test
Garry, 2026-11-01 (R54): "I think you are spending too much time with Haiku and
Sonnet, whereas GBrain and GStack are really designed to be used with frontier
models like Opus and 6.1 Sol... just transfer from Haiku to Sonnet, who gives a
shit about that, man?"
**Models under test are frontier models.** Today that's `claude-opus-5-5` and
`gpt-6.1-sol`, plus at most one low-cost primary the preregistration names. Haiku
and Sonnet are not models under test. They aren't stand-ins for a frontier model,
and an eval has no tier-transfer arms (Haiku against Sonnet, say) unless its
preregistration says it is asking about that tier.
**Verdicts need a frontier result.** A result measured only on Haiku or Sonnet
can't carry a KEEP, FIX-METHOD or CUT-CANDIDATE verdict under R48. Rerun it on a
frontier model first.
**What the rule leaves alone.** Judges, cross-family review voices and a
harness's shipped housekeeping model aren't models under test, so they keep the
model they already use.
**The check.** `gstack-eval-plan dry-run` prints `CHECK: frontier` whenever an arm
is Haiku or Sonnet. When every arm is one, the dry run fails unless
`--tier-question "<what the preregistration asks about that tier>"` is passed.
When only some arms are, it warns. Either way it prints the whole plan's price
on `claude-opus-5-5` and `gpt-6.1-sol`. The dry run doesn't enforce that number:
compare it with the brief cap before any arm is swapped. `pilot start` refuses a
pilot whose every arm is Haiku or Sonnet unless its dry run recorded a tier
question.
## Running evals as an agent: detach
Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in
+3 -1
View File
@@ -434,7 +434,9 @@ A scheduled automation starts one lane a day with the
runs' flake-records artifacts and the free suites' retried cases, writes one ledger line
per event, groups by case, and fixes the worst repeat first with a PR that shows the
failure red before green. It never reruns a gate and never touches a threshold, trial
count or case kind. On Capy, create it with the automation tool, scheduled in the owner's
count or case kind. Where a repository runs `gstack-flaky`, the lane also reads
`gstack-flaky quarantine report --ledger <downloaded ledgers>` and fixes expiring quarantine
entries before anything else ([flaky.md](flaky.md#the-daily-flaky-pass)). On Capy, create it with the automation tool, scheduled in the owner's
timezone, running as a thread in the crew so its report lands in `reports/`.
The lane's side of the same rule: in CI the gate reruns a case that failed on its own
+2 -1
View File
@@ -89,6 +89,7 @@ After weighing Codex's argument, we chose to roll back CEO EXPANSION and ship an
"sample_size": 47,
"signal_events": 19,
"contributing_events": {"scope_appetite": 9, "risk_tolerance": 6, "...": "..."},
"hinted_events": 3,
"diversity": {
"skills_covered": 5,
"question_ids_covered": 14,
@@ -105,7 +106,7 @@ After weighing Codex's argument, we chose to roll back CEO EXPANSION and ship an
}
```
`signal_events` counts the logged answers that moved any dimension, and `contributing_events` counts, per dimension, the logged answers that moved it (an answer that writes two deltas to one dimension counts once), so a dimension with no evidence (0) is visible beside one whose evidence balances out; both print 0.5 (#2623). Each value is the sigmoid of its accumulated deltas, and each UTC day's sum per dimension (by the row's `ts`) is clamped to the largest single delta in the signal map (currently ±0.06), so one burst of answers can't pin a dimension; `--trace <dim>` lists every contributing answer, each clamped day and the total. The bound keys on the day, not on `session_id`: the PostToolUse hook and the agent log different ids for the same review (the host's conversation id and the preamble's run id), and an agent can leave the `SESSION_ID` placeholder unsubstituted, so no logged id reliably marks one review. The cost is that several reviews on one day count as one. `contributing_events` still counts an answer logged by both writers twice. Only an ISO 8601 date or date-time `ts` counts as dated (a time without a zone is read as UTC), every other row shares one undated bucket, and `diversity.days_span` counts the same UTC days. `--check-mismatch`, `--gap`, the stored `gap` and /plan-tune's profile view compare a dimension only when its `contributing_events` is at least 3; `--gap` lists the others under `low_evidence` and the stored `gap` omits them (#3436).
`signal_events` counts the logged answers that moved any dimension, and `contributing_events` counts, per dimension, the logged answers that moved it (an answer that writes two deltas to one dimension counts once), so a dimension with no evidence (0) is visible beside one whose evidence balances out; both print 0.5 (#2623). Each value is the sigmoid of its accumulated deltas, and each UTC day's sum per dimension (by the row's `ts`) is clamped to the largest single delta in the signal map (currently ±0.06), so one burst of answers can't pin a dimension; `--trace <dim>` lists every contributing answer, each clamped day and the total. The bound keys on the day, not on `session_id`: the PostToolUse hook and the agent log different ids for the same review (the host's conversation id and the preamble's run id), and an agent can leave the `SESSION_ID` placeholder unsubstituted, so no logged id reliably marks one review. The cost is that several reviews on one day count as one. `contributing_events` still counts an answer logged by both writers twice. Only an ISO 8601 date or date-time `ts` counts as dated (a time without a zone is read as UTC), every other row shares one undated bucket, and `diversity.days_span` counts the same UTC days. `--check-mismatch`, `--gap`, the stored `gap` and /plan-tune's profile view compare a dimension only when its `contributing_events` is at least 3; `--gap` lists the others under `low_evidence` and the stored `gap` omits them (#3436). An unregistered `{skill}-{slug}` id moves a dimension only through a signal hint (#3442): the asked question carries `<gstack-qid:{id};signal={signal_key}>` and the log row a matching `signal_key`, chosen from `gstack-question-preference --signal-keys`. Derive uses the hint only when the id has no registry entry, the key is in the signal map and the answer matches one of that signal's option keys; `hinted_events` counts those signal events, and `--trace` marks their rows `(hinted)`.
**Diversity check** (Codex #13): `inferred` is considered "enough data" only when `sample_size >= 20 AND skills_covered >= 3 AND question_ids_covered >= 8 AND days_span >= 7`. Below this, `/plan-tune profile` shows "not enough observed data yet" instead of a potentially-misleading inferred value.
+1 -1
View File
@@ -93,7 +93,7 @@ What incomplete data means:
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
| cause `unrecorded` | the trial was written before `failure_cause` existed |
| cause `unknown` | the trial failed with no evidence for any other cause |
| `cost unknown` | the harness (PTY, Codex) records no billing, or the trial sits in a multi-case shard whose records are named by display label and cannot be attributed to it; only the known sum is shown |
| `cost unknown` | a session's cost was not captured (a PTY session without a priced transcript, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown |
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
## 4. Replay and repair (free)
+117 -8
View File
@@ -6,7 +6,9 @@ rerun is a **flaky pass**: the job stays green and the flake is recorded on a le
the daily flaky pass can fix it at the source. A test that fails twice is red.
This is CREW rule 16 ("run the failed test again and note that it is flaky")
for any runner that writes JUnit: Playwright, `bun test`, jest, vitest and
pytest. gstack's own free suite keeps its file-level retry
pytest. A test that keeps flaking while it is fixed can be
[quarantined](#quarantine): it still runs, with up to 3 attempts, and never counts
as a pass. gstack's own free suite keeps its file-level retry
(`GSTACK_FREE_RETRY_FLAKY`, [Flake ledger](TESTING_INTERNALS.md#flake-ledger)).
## Get started in GitHub Actions
@@ -74,6 +76,104 @@ It cannot see a coverage threshold configured without console output (for
example `coverageReporter = ["lcov"]` in `bunfig.toml`), which is why this
scope is opt-in.
## Quarantine
A test that keeps flaking while someone fixes it can be **quarantined** in
`.gstack/quarantine.json` at the repository root. Quarantine is a larger
attempt budget, never a skip: the test still runs on every run, is listed as
quarantined in the output, and never counts as a pass.
**How a quarantined test is judged.** The verdict rule from
[Rerun scope](#rerun-scope) does not change: an attempt counts only when the
runner itself reports a full green run. When every test an attempt failed is a
live quarantined test, and nothing else is wrong with that attempt (no kill, no
failure outside the report, no first-run test missing), the whole command may
run again, up to 3 attempts in total. The run is green only when one of those
attempts is fully green. A quarantined test that fails all 3 is a **hard
break**, and the run is red. A test that is not quarantined keeps its one
rerun. A red attempt never turns green because its only reported failure is a
quarantined test, because the runner's exit status can come from things the
report does not show (a coverage threshold, a fail-on-flaky policy, a global
setup). With `--rerun-scope test` the quarantined test gets the same 3
attempts on the targeted path, with the same fallback to the whole command.
When the runner retries on its own (Playwright `retries`), its attempts are
the budget and nothing is rerun again.
**Missing results.** A quarantined id missing from a run that included its
file (and, for Playwright, its project) is red. A run that did not include the
file, such as a shard or a diff selection, is not.
**Read from the base branch.** `run --base <ref>` reads the registry from
that commit instead of the working tree. Pull-request CI passes its target
branch, so a change cannot quarantine the test it breaks. `check --base`
compares against the merge base, so check out full history for it:
```yaml
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- name: Quarantine registry
run: '"$RUNNER_TEMP/gstack/bin/gstack-flaky" quarantine check --base "origin/${{ github.base_ref }}"'
- name: E2E
run: |
"$RUNNER_TEMP/gstack/bin/gstack-flaky" run --base "origin/${{ github.base_ref }}" --junit results/junit.xml -- npx playwright test --reporter=line,junit
```
### Commands
| Command | What it does |
|---|---|
| `gstack-flaky quarantine add <id> --owner <who> --ledger <url> --class <class> --reason <text> --evidence <url>… --junit <file> [--days <n>]` | Adds an entry that expires in `--days` (default 7, at most 14). `--junit` counts the suite for the cap and confirms the id exists; `gstack-flaky ids` prints the ids. |
| `gstack-flaky quarantine renew <id> --owner <who> --evidence <url> [--days <n>]` | One renewal of up to 7 days from the later of now and the current expiry, with evidence not already on the entry and the owner named again. A second renewal is refused. |
| `gstack-flaky quarantine release <id> --reason <text> --pr <url>` | Moves the entry to `released` with the date, the reason and the PR. |
| `gstack-flaky quarantine check [--base <ref>] [--junit <file>]` | Validates every entry and the cap; with `--base`, also what this change did since the merge base. Exits 1 with one `QUARANTINE_REFUSED:` line per problem. |
| `gstack-flaky quarantine report [--ledger <file\|dir>…] [--base <ref>]` | Lists expired entries, then live entries soonest-expiring first with days left. With downloaded ledgers it adds recent outcomes and marks entries with 10 consecutive clean passes `READY_TO_RELEASE`; without `--ledger` it says the exit rule was not checked. |
### Rules
- **Every entry is owned and expires.** An entry holds `owner`, `ledger`
(the ledger line or issue tracking the fix), `failureClass`, `reason`,
`evidence`, `enteredAt`, `expiresAt`, `renewals` and `suiteTests` (the
suite size `add` counted). It lasts 7 days by default and at most 14 from
entry, plus one renewal of at most 7 days.
- **Expired means not quarantined.** An expired entry stops applying without
an edit: `run` treats the test as an ordinary test and prints
`FLAKY_QUARANTINE_EXPIRED` with the renew and release commands. An entry
that breaks a rule (for example one that lasts 30 days) is ignored the same
way, with `FLAKY_QUARANTINE_IGNORED`.
- **Cap.** Live entries are capped at 10% of the suite's tests, at least 1,
and 5 for a suite under 50 tests. `add` refuses past the cap; `check`
enforces it against `--junit` when given, else against the smallest
`suiteTests` on a live entry.
- **Classes.** `detector`, `harness`, `timing` and `model-latency`.
`product` is refused (a product defect is fixed, not quarantined), and so is
`infra` (an infrastructure red is rerun or fixed as infrastructure).
- **`check --base` refuses** an entry added in the same change that edits its
test's file, an entry that disappears without a release record, a release
record that was removed, and a changed `enteredAt` or rewritten renewal
history.
- **Exit rule.** 10 consecutive clean passes (`quarantined-pass` with no
failed attempt) on the ledgers since the entry was added. Release the entry
with the fix PR.
- **Separate from `CASE_QUARANTINE`.** gstack's own paid-eval quarantine
(`CASE_QUARANTINE` and `EVAL_POLICY`,
[TESTING_INTERNALS.md](TESTING_INTERNALS.md)) is a different registry with
its own rules for eval cases. The two share no schema, and `gstack-flaky`
neither reads nor writes it.
### The daily flaky pass
Download the ledger artifacts of the last week's runs and read them with
`report`; fix expired and `EXPIRING` entries first (fix the test, or renew
once with new evidence), release `READY_TO_RELEASE` entries, then fix the
worst repeat on the ledger:
```bash
for id in $(gh run list --workflow e2e.yml --limit 50 --json databaseId -q '.[].databaseId'); do
gh run download "$id" --pattern 'flaky-junit-ledger-*' --dir "ledgers/$id" || true
done
gstack-flaky quarantine report --ledger ledgers
```
## Rules
- **One rerun, small failures only.** A failed test gets one rerun when the
@@ -101,8 +201,9 @@ scope is opt-in.
- **Ids.** A test is `<file>::<project>::<describe > title>`, with the project
empty outside Playwright, so a test in two browser projects is two tests.
`gstack-flaky ids --junit <file>` prints them.
- **Ledger.** One JSON line per flaky pass (`kind: flaky-pass`) and per test
that failed its rerun (`rerun-fail`), with the tree, commit, branch, run URL,
- **Ledger.** One JSON line per flaky pass (`kind: flaky-pass`), per test
that failed its rerun (`rerun-fail`) and per quarantined test on every run
(`quarantined-pass`, `quarantined-fail`, `hard-break`), with the tree, commit, branch, run URL,
attempts, the rerun's scope (`test` or `command`) and the first lines of
the error. Ids, paths and failure text go through gstack's redaction engine
first; when the scan cannot run, they are withheld. The path is `--ledger`,
@@ -113,15 +214,17 @@ scope is opt-in.
## Output and exit codes
One `FLAKY: <flaky-pass|fail|rerun-fail> <id> attempts=<…> (<reason>)` line per
test that failed at least once, `FLAKY_RERUN: scope=command (<why>)` when the
rerun repeated the whole command, then
`FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>`.
One `FLAKY: <flaky-pass|fail|rerun-fail|quarantined-pass|quarantined-fail|hard-break> <id> attempts=<…> (<reason>)`
line per test that failed at least once and per quarantined test,
`FLAKY_RERUN: scope=command (<why>)` when the rerun repeated the whole command, then
`FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> quarantined=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>`.
`failed` counts every outcome but `flaky-pass` and `quarantined-pass`; `quarantined`
counts the quarantined tests' outcomes, which are never counted as passes.
| Exit | Meaning |
|---|---|
| 0 | Every test passed, flaky passes included |
| 1 | A test failed (after its rerun, or with no rerun), including `FLAKY_UNREPORTED_FAILURES` |
| 1 | A test failed (after its rerun, or with no rerun), including `FLAKY_UNREPORTED_FAILURES`, a quarantined test's hard break or missing result, and a refused `quarantine` command |
| 2 | Usage error, including a `--junit` path that is not a `.xml` file, or an existing file there that is not a readable JUnit report |
| 3 | No usable report, a crash, or a killed runner (a code below on stderr) |
@@ -135,3 +238,9 @@ rerun repeated the whole command, then
| `FLAKY_COMMAND_NOT_STARTED` | 3 | The command could not be started | Check the argv after `--` |
| `FLAKY_UNREPORTED_FAILURES` | 1 | The runner counted failures or errors its JUnit omits | Fix the load error or the error outside any test: a real red |
| `FLAKY_LEDGER_WRITE_FAILED` | — | The ledger could not be appended (warning) | Check the path; the verdict stands |
| `FLAKY_QUARANTINE_INVALID` | 3 | `.gstack/quarantine.json` is not valid JSON or not a schema 1 registry | Fix the file; `quarantine check` names the problem |
| `FLAKY_QUARANTINE_BASE` | 3 | `--base` does not resolve to a commit (exit 2 for a value starting with `-`), or `check` finds no merge base | Fetch the base branch; `check --base` needs `fetch-depth: 0` |
| `FLAKY_QUARANTINE_REFUSED` | 1 | `add`, `renew` or `release` broke a rule | Read the reason; [Quarantine](#quarantine) lists the rules |
| `FLAKY_QUARANTINE_USAGE` | 2 | An unknown `quarantine` command or option, or a missing value | `gstack-flaky quarantine --help` |
| `FLAKY_QUARANTINE_EXPIRED` | — | Notice: an expired entry no longer applies | Fix the test, renew once, or release; the line prints both commands |
| `FLAKY_QUARANTINE_IGNORED` | — | Notice: an entry breaks a rule and does not apply | Fix the entry; `quarantine check` names the rule |
+12 -3
View File
@@ -74,7 +74,7 @@ the byte-pinned `/ship` renders in `test/fixtures/golden/`).
| Key | Meaning |
|---|---|
| `runner` | the local test command with `{files}` (gstack: `bun test {files}`); unset, it is detected (bun, vitest, jest, pytest, `node --test`, and `flutter test`/`dart test` per Dart package root, through fvm when that package or an ancestor pins it with `.fvmrc` or `.fvm/fvm_config.json` and fvm is on PATH; `cargo test -p <name>` per Rust crate, run from the root of the workspace the crate is a member of (Cargo's `members` globs, `exclude` and `package.workspace`), else the crate directory; a JS/TS test whose nearest package.json is a nested package declaring vitest or jest (a dependency or its `test` script) runs from that package, with package-relative paths, through the package manager its `packageManager` field or the nearest lockfile up to the repo root names: `pnpm exec`, `yarn`, `bun x`, else `npx`; every other JS/TS test keeps the root detection); nothing detected, a Dart SDK, cargo or package manager missing from PATH, a `.dart` file with no `pubspec.yaml` above it, a `.rs` file outside any crate, or one in a crate whose name Cargo would reject (anything but letters, digits, `_` and `-`), is `incomplete` (the runnable groups still run, so a real failure still fails) |
| `runner` | the local test command with `{files}` (gstack: `bun test --timeout=30000 {files}`, the 30 s per-test timeout CI uses); unset, it is detected (bun, vitest, jest, pytest, `node --test`, and `flutter test`/`dart test` per Dart package root, through fvm when that package or an ancestor pins it with `.fvmrc` or `.fvm/fvm_config.json` and fvm is on PATH; `cargo test -p <name>` per Rust crate, run from the root of the workspace the crate is a member of (Cargo's `members` globs, `exclude` and `package.workspace`), else the crate directory; a JS/TS test whose nearest package.json is a nested package declaring vitest or jest (a dependency or its `test` script) runs from that package, with package-relative paths, through the package manager its `packageManager` field or the nearest lockfile up to the repo root names: `pnpm exec`, `yarn`, `bun x`, else `npx`; every other JS/TS test keeps the root detection); nothing detected, a Dart SDK, cargo or package manager missing from PATH, a `.dart` file with no `pubspec.yaml` above it, a `.rs` file outside any crate, or one in a crate whose name Cargo would reject (anything but letters, digits, `_` and `-`), is `incomplete` (the runnable groups still run, so a real failure still fails) |
| `dependencies` | glob → test files for subprocess targets, runtime reads and generated inputs the import graph cannot see; `[]` declares "no lane needed" (docs, licences). A `[]` that matches a deleted test does not skip it: the deleted test still runs its owning suite, and a file naming it that `[]` covers (a durations list) counts as covered. With this key present, even empty, the policy [declares lanes](#lane-requirements) |
| `empty_lanes` | extra globs for the docs/no-tests empty lane. Any Markdown file already qualifies with no pin file when no test depends on it (the same import graph as other files: a quoted path, a `join()` of path segments, an import, or a `dependencies` entry). reStructuredText and AsciiDoc count as Markdown here. README, CHANGELOG, CONTRIBUTING, LICENSE, AGENTS and CLAUDE qualify even without `.md`, and so does licence text under any vendored name (`roboto-LICENSE.txt`, `LICENSE-MIT`, `licenses/*.txt`). Source code and files under a test directory never qualify. A test that reads the file still selects that test |
| `selection` | `workflow.yml/job` → a command printing the lane's file list, for matrices a planner computes (gstack: `scripts/test-free-shards.ts --list`) |
@@ -762,7 +762,15 @@ workflow file is the lane's own `<workflow.yml>`. A run of another workflow
refuses with `REUSE: refused reason=workflow_mismatch — run <id> attempt <n>
ran .github/workflows/<other>.yml, not .github/workflows/<lane's>.yml`; a
run gh cannot read refuses with `api_error`, and a URL that is not an Actions
run URL or id on this repository with `run_ref_invalid`. The run must also
run URL or id on this repository with `run_ref_invalid`. A reusable-workflow
lane (one whose file another workflow calls through `workflow_call`) also clears from a run
of a caller: a workflow in the branch's own `.github/workflows/` with a job
that says `uses: ./.github/workflows/<lane's>.yml` (GST-971), read at
pregate.json's recorded tree, as long as the lane's own job succeeded in the
run under that calling job (`<calling job> / <lane job>`); else it refuses
with `caller_job_skipped`. Fix lines only suggest `gh workflow run` for
a file with a `workflow_dispatch` trigger.
The run must also
have finished green (GSTA-203 PR 8): a run that is still queued or running
refuses with `run_in_progress`, and one that concluded anything but `success`
(`failure`, `cancelled`, `timed_out`, …) with `run_not_success`; each fix line
@@ -770,7 +778,8 @@ says to rerun the workflow and clear with the green run. The check reads the
run's newest attempt, so a `/job/<id>` or `/attempts/<n>` link is judged by
the whole run, not that job or attempt, and the lane records
`https://github.com/<repo>/actions/runs/<id>/attempts/<n>` for the attempt it
judged rather than the link you passed. The job, tree and age are not
judged rather than the link you passed. The job (apart from a caller run's
`<calling job> / <lane job>` above), tree and age are not
checked there; that is what a run profile adds. `test_backend/<backend>` lanes and `--bundle-lane` are
unchanged. When the base config or the GitHub origin cannot be read, `--run`
fails, and so do `--run-url` and `--bundle-lane` unless the base was read and
+26 -7
View File
@@ -98,9 +98,19 @@ incomplete coverage, which blocks until the reviewer is rerun.
finishes with its open notes listed in the PR body and receipt, or stops for
the owner when a blocking finding is open, or was fixed in round 2 and is
unverified, or a round-2 fix edited a gate-class path, naming it (path,
severity, problem). A third round is the owner's
severity, problem). The cap limits fix rounds, not Step 11's voices: on gate
class a stop waits for Step 11's adversarial, outside and targeted voices
(`NEXT: voices`), so gate code never reaches the owner's "ship with open
notes" without them (GST-967). A third round is the owner's
grant of one scoped fix-check (recorded in `review_loop_exception`), never the
loop's. The cap approves nothing.
loop's: once `gstack-approvals add` records the grant (its ask or scope
naming `review_loop_grant` and the stop head's sha),
`gstack-review-plan --grant <approval id> --range-base <round 2's PASS_HEAD>`
plans that fix-check in place of the stop after a round-2 full pass with a
fixed blocking finding or a gate-class edit, and refuses any other row,
including a grant for a stop head before `--range-base` (an earlier /ship).
A fix-check never earns another, so the grant cannot repeat, and a stop after
a round-2 fix-check still goes to the owner. The cap approves nothing.
<a id="transitions"></a>
### Transitions
@@ -122,12 +132,19 @@ incomplete coverage, which blocks until the reviewer is rerun.
| a full pass that leaves a blocking finding open (fixed or not) | stop: name it for the owner |
| a fix-check that found nothing blocking | done |
| a fix-check that found a blocking finding | stop: name it for the owner |
| a stop (any row above) on gate class before Step 11's voices ran | Step 11's voices (adversarial, outside, targeted), then ask again with `--adversarial-ran` |
| a round-2 full pass that stops for a fixed blocking finding or a gate-class edit, with the owner's recorded grant (`--grant <approval id>`) | one scoped fix-check (round 3), after Step 11 on gate class |
A **fix-check** is one reviewer reading the touched hunks, the fix list, the
test results and the callers of every changed symbol, at full effort. It is
logged as a `review-fix-check` row, so it never counts as a pass. No army, no
Red Team and no outside voice run on it.
`NEXT: voices` runs Step 11 as `done` does, from the last full pass's
manifest; /ship then asks the plan again with `--adversarial-ran` (a blocking
finding the voices queued counts as `--blocking-open`, never a new Step 9), and
the stop or the granted fix-check it names stands.
<a id="scoped-pass"></a>
A **scoped pass** is pass 2 after a blocking fix when pass 1 and the diff both
ran below full depth (no gate class, no depth error, no `--depth-floor full`).
@@ -384,10 +401,11 @@ block without new floor words. The reader is read-only, and `NO FINDINGS` is a
result. `gstack-review-dispatch render --role targeted --classes <classes>`
renders its brief; `verify --targeted-out <file>` writes the record that
`gstack-ship-receipt write --review-targeted @<file>` puts in the receipt.
`gstack-ship-queue ready` and `train build` print
`gstack-ship-queue ready`, `train build` and `land` print
`REVIEW_TARGETED: required=<classes|none> receipt=<classes|none|legacy>
verdict=ok|missing` and refuse nothing yet; enforcement comes in a later
release.
verdict=ok|missing|failed` and refuse
[`QUEUE_RECEIPT_TARGETED_MISSING`](troubleshooting.md#queue-receipt-targeted-missing)
when a required class has no done pass covering it.
Over the last 100 non-release commits on main (8663c3d2..8be7f213), money hit
21, gate 18, auth 3 and secret 8. The gate class first used the gate paths,
@@ -539,10 +557,11 @@ ones, and the arm lost blockers. `bin/gstack-review-dispatch` closes both ends:
3. `collect` hashes each reader's raw output and appends its findings to
`merged.jsonl` with the label as written (a JSON line's `severity`, or the most
severe of a prose line's Critical, High, Medium, Low, P0-P3, FIXABLE or INVESTIGATE
wherever they stand, headings and table rows included; a top-level list item under a
wherever they stand, headings and table rows included, except that a line with its own `Severity` field takes that
field's label (any letter case; a field stating more than one label, "High/Critical", counts as none), so a severity word in the finding's text does not raise it, and a label leading the line stays its floor (GST-976); a top-level list item under a
label-only heading takes the stronger of the two). A `Severity:` or `Blocks:` line
continues the finding above it until a list item, heading, bold paragraph opener or a
second line for the same field starts another: `Severity` raises its label,
second line for the same field starts another: `Severity` sets its label (a label leading the finding stays its floor),
`Blocks: yes` makes it CRITICAL and `Blocks: no` never lowers it; an inline
`Blocks: yes` is CRITICAL too, and with no finding above it is one. The native
adversarial voice's FIXABLE collects as CRITICAL (normalizer v2, GSTA-167). An unnumbered, unlabeled bullet with no file:line
+19 -9
View File
@@ -246,8 +246,10 @@ result is not a current pass, and (`QUEUE_RECEIPT_MISSING`,
`QUEUE_RECEIPT_STALE`) when the PR body carries no /ship receipt for that
head: /ship runs before every READY ([details](#ship-before-ready)). It also
refuses `QUEUE_RECEIPT_DEPTH_RAISED` when the receipt's review depth is below
what the head needs and `QUEUE_RECEIPT_EXCEPTION_INVALID` when an owner
exception the receipt records does not stand ([the ship receipt](#ship-receipt)).
what the head needs, `QUEUE_RECEIPT_TARGETED_MISSING` when a targeted class
the head needs has no done targeted pass in the receipt, and
`QUEUE_RECEIPT_EXCEPTION_INVALID` when an owner exception the receipt records
does not stand ([the ship receipt](#ship-receipt)).
Send the block as printed.
**2. Lane behind another PR: gate ahead.** Gate the tree the queue will have
@@ -683,8 +685,10 @@ slot. The queue enforces this at every step from the PR body alone:
head being READY'd (or its `pr` is another PR); the informational
`SHIP_RECEIPT:` line still prints. It refuses `QUEUE_RECEIPT_DEPTH_RAISED`
(with a `REVIEW_DEPTH:` line) when the receipt records a review depth below
what the head needs, and `QUEUE_RECEIPT_EXCEPTION_INVALID` when an owner
exception it records does not stand ([the ship receipt](#ship-receipt)).
what the head needs, `QUEUE_RECEIPT_TARGETED_MISSING` (with a
`REVIEW_TARGETED:` line) when a targeted class the head needs has no done
targeted pass in the receipt, and `QUEUE_RECEIPT_EXCEPTION_INVALID` when an
owner exception it records does not stand ([the ship receipt](#ship-receipt)).
There is no flag around it.
- `train build` refuses a pin whose PR has no current receipt with the same
codes, naming the PR, before anything is written, pushed or dispatched,
@@ -803,9 +807,10 @@ its record. Refusals, all before any spend: `QUEUE_TRAIN_UNTRUSTED` (a fork
head), `QUEUE_TRAIN_MOVED` (a head moved from its pin),
`QUEUE_RECEIPT_MISSING` and `QUEUE_RECEIPT_STALE` (the PR body has no /ship
receipt for the pin; see [/ship before READY](#ship-before-ready)),
`QUEUE_RECEIPT_DEPTH_RAISED` and `QUEUE_RECEIPT_EXCEPTION_INVALID` (the
receipt's review depth is below what the pin needs, or an owner exception it
records does not stand; see [the ship receipt](#ship-receipt)),
`QUEUE_RECEIPT_DEPTH_RAISED`, `QUEUE_RECEIPT_TARGETED_MISSING` and
`QUEUE_RECEIPT_EXCEPTION_INVALID` (the receipt's review depth is below what
the pin needs, a targeted class the pin needs has no done targeted pass, or an
owner exception it records does not stand; see [the ship receipt](#ship-receipt)),
`QUEUE_NOT_READY` (no READY receipt at that head), `QUEUE_TRAIN_CONFLICT` (the printed
`QUEUE_TRAIN_CONFLICT: train <id> stopped at #<pr> (conflicts: <files>) after
merging #<a>,#<b>; nothing dispatched, no branch pushed; fix: …` line names
@@ -1222,7 +1227,12 @@ block `gstack-approvals` renders (its heading, the Date | Asked | Exact words
table and the `Source: gstack-approvals record` line) quotes the owner's
words, not this PR's numbers, so it is skipped: "one Step 11 pass" in that table
is not a `pass` claim. Text you write under the heading after that block, and
a `## Approvals` example inside a code fence, are still checked. Data PRs (evals,
a `## Approvals` example inside a code fence, are still checked. The
`Review-loop exception: …` line the receipt's own exception requires is
skipped the same way (GST-977): it quotes the owner's reason, so "outside 1 and
2 skip detector" is not a `2 skip` claim. Only a line that matches the rendered
line exactly is skipped, inside a code fence too, as the receipt check accepts
a fenced copy; the same words inside a longer line are still checked. Data PRs (evals,
measurements, sweeps) run it before READY so the body's numbers are the run's.
<a id="review-scope"></a>
@@ -1524,7 +1534,7 @@ Stages, each journaled in `.gstack/tmp/restamp-journal.json`:
1. **version**: `gstack-version-bump write --version <X>` in the staging worktree (VERSION, the manifest, npm lockfiles); skipped when `restamp_command` owns the version. The manifest gets the npm-valid three-part form, except when the policy lists it in `stamp_paths` or `stamp_files` and its copy at the base holds four parts (gbrain's convention): then it keeps the full version (`--manifest-form full`) and the stage detail says so.
2. **release_tool**: the policy command, under the contract below.
3. **changelog**: the PR's `[Unreleased]` or old-version section moves to the top under `<X>`, above its predecessor's entry. A duplicate heading the PR introduces (`CHANGELOG_DUPLICATE_HEADING`: the headings are counted against the base's CHANGELOG at `origin/<base>` and at the fork point, so a version the base already heads twice passes as long as the PR adds no further copy) and a top entry that is not `<X>` (`CHANGELOG_TOP_MISMATCH`) are errors; the section being re-headed itself may exist only once.
4. **stamp_paths**: exact `(was X)`, `since: X` and `vX` rewrites, plus every `stamp_patterns` template, within `stamp_paths` and `stamp_files`. A quote around X in the first two forms is part of the form and is kept as written: `since: 'X'`, `since: "X"`, `(was 'X')` and `(was "X")` move like the bare ones (the shape of a quoted source row such as gbrain's `{ since: '0.60.468.0', … }`), an opened quote must close right after X (`since: '1.2.3-beta'` and `since: '1.2.3"` are not the form, in the writer and in the tree receipt alike), and `vX` already accepts a quote before the `v`. Which X moves is decided by provenance. The branch's own tentative stamp (its VERSION when it moved past base) and `--was <X>`, when given, are each rewritten on every line of every stamp file. Beyond that, only lines the branch itself added are read, by their position in the file (`git diff --unified=0` from the fork point), so a base line with the same text, or a CRLF checkout whose diff is LF-normalized, is told apart: a stamp-form mention there of a version newer than the version at the branch's fork point is a tentative stamp and moves (this is what finds a quoted row after a lost slot and a merge of the base, since the base's release is newer than the fork point), and when the branch has no stamp of its own because its VERSION equals the base's and its fork point is the base itself (a branch rebased onto the base after it reached X, or one never stamped) a mention of the base version on a line it added is its stale stamp and moves too, while the same version on a line the base already carries is the base's release and stays. The stage detail names both counts per file (`VERSION equals base X, so X moved only on lines this branch added and stayed on the base's own lines (<file>: N moved, M kept)`), so the stamp commit shows which lines moved and why. A line the base added since the fork point is the base's even when that diff shows it (a merge of the base on the branch, committed or still staged, brings every line the base added since the fork into the diff from the fork point, main's own release keys included): it is left out, the detail says `N line(s) the base added since the fork point are the base's, not this branch's, and stay (<file>: N)`, and a base diff git cannot run refuses the stage (`POLICY_SOURCE_UNAVAILABLE`, `cannot diff <since>..<base> for the stamp_paths stage`) rather than reading as an empty set that would move the base's lines again. The rule reads line ownership, not meaning: a branch-added line inside `stamp_paths` that means the base's own release in a stamp form (`Requires vX or later` written on a branch forked at X) moves with it, which the `--dry-run` diff and the per-file counts show before anything is written; write such a reference outside a stamp form, or keep that file out of `stamp_paths`. `--was X` is the opposite tool: it rewrites X everywhere, the base's own lines included, and re-heads the CHANGELOG section `## [X]`. Only the repo's own version width counts, so a 3-digit dependency pin in a 4-digit repo is never a candidate.
4. **stamp_paths**: exact `(was X)`, `since: X` and `vX` rewrites, plus every `stamp_patterns` template, within `stamp_paths` and `stamp_files`. A quote around X in the first two forms is part of the form and is kept as written: `since: 'X'`, `since: "X"`, `(was 'X')` and `(was "X")` move like the bare ones (the shape of a quoted source row such as gbrain's `{ since: '0.60.468.0', … }`), an opened quote must close right after X (`since: '1.2.3-beta'` and `since: '1.2.3"` are not the form, in the writer and in the tree receipt alike), and `vX` already accepts a quote before the `v`. Which X moves is decided by provenance. The branch's own tentative stamp (its VERSION when it moved past base) and `--was <X>`, when given, are each rewritten on every line of every stamp file. Beyond that, only lines the branch itself added are read, by their position in the file (`git diff --unified=0` from the fork point), so a base line with the same text, or a CRLF checkout whose diff is LF-normalized, is told apart: a stamp-form mention there of a version newer than the version at the branch's fork point is a tentative stamp and moves (this is what finds a quoted row after a lost slot and a merge of the base, since the base's release is newer than the fork point), and so does one of the fork-point version itself (a row the branch wrote at the version it forked from instead of a next version is its own stale stamp; a branch-added mention older than the fork point stays), and when the branch has no stamp of its own because its VERSION equals the base's and its fork point is the base itself (a branch rebased onto the base after it reached X, or one never stamped) a mention of the base version on a line it added is its stale stamp and moves too, while the same version on a line the base already carries is the base's release and stays. The stage detail names both counts per file (`VERSION equals base X, so X moved only on lines this branch added and stayed on the base's own lines (<file>: N moved, M kept)`), so the stamp commit shows which lines moved and why. A line the base added since the fork point is the base's even when that diff shows it (a merge of the base on the branch, committed or still staged, brings every line the base added since the fork into the diff from the fork point, main's own release keys included): it is left out, the detail says `N line(s) the base added since the fork point are the base's, not this branch's, and stay (<file>: N)`, and a base diff git cannot run refuses the stage (`POLICY_SOURCE_UNAVAILABLE`, `cannot diff <since>..<base> for the stamp_paths stage`) rather than reading as an empty set that would move the base's lines again. The rule reads line ownership, not meaning: a branch-added line inside `stamp_paths` that means the base's own release in a stamp form (`Requires vX or later` written on a branch forked at X) moves with it, which the `--dry-run` diff and the per-file counts show before anything is written; write such a reference outside a stamp form, or keep that file out of `stamp_paths`. `--was X` is the opposite tool: it rewrites X everywhere, the base's own lines included, and re-heads the CHANGELOG section `## [X]`. Only the repo's own version width counts, so a 3-digit dependency pin in a 4-digit repo is never a candidate. The checks that accept a stamp commit read the same bound as the writer (`stampOnlyLine` with the branch's side in `lib/stamp-only.ts`): the tree receipt behind `stamp`, `verify` and the gate-ahead replay, the merge train's stamp and verify checks (each reading the slot's predecessor, the gated commit's first parent, as the base), and finish's route each take a moved version on a line the branch added at or above the fork-point version, and on any other line at or above the gated head's VERSION. A base line whose text duplicates a line the branch added gets the branch's bound too, because the checks match branch lines by text, not position (GST-979). The /ship receipt carry can still refuse the move (`RECEIPT_CARRY_PRODUCT_CHANGED`), and when the branch keeps its VERSION claim, carry and finish still route the move as behavior (GST-980).
5. **mirror**: tier 2; declared and validated, reported as not run.
6. **apply**: every changed file is written to the branch only after all stages validated, each against the fingerprinted preimage; a file edited meanwhile is `RESTAMP_CONFLICT` and the applied writes roll back (rollback refuses to overwrite a file whose bytes differ from the recorded postimage).
+4 -1
View File
@@ -961,7 +961,10 @@ arms, decoys, stop rules, cap) and `--check` refuses to continue while a field i
blank (`PREREG_INCOMPLETE`). `arms` resolves the `eval-arms` model role against
`lib/pricing.ts` and stops on an unpriced model (`EVAL_PRICE_MISSING`); `dry-run`
runs every row through the stub model for $0 and flags malformed rows
(`EVAL_DRY_RUN_FAILED`); `pilot start|settle|release|report` runs three priced
(`EVAL_DRY_RUN_FAILED`), and refuses a plan whose every arm is Haiku or Sonnet
unless `--tier-question` names what it asks about that tier, re-costing it on
frontier models ([frontier models under test](TESTING_INTERNALS.md#frontier-models-under-test));
`pilot start|settle|release|report` runs three priced
items, records actual spend, extrapolates each priced arm to the full set against
the cap (`EVAL_BUDGET_EXHAUSTED`) and prints the spend table the approval question
shows. An arm on a subscription with no extra charge is declared with
+1 -1
View File
@@ -19,7 +19,7 @@ Beyond the slash-command skills, gstack ships standalone CLIs for workflows that
| `gstack-review-log` | **Review-pass receipts** — `--start <skill>` captures the working-tree fingerprint before a diff review; `'<JSON>' --finish <token>` consumes that single-use, repository/branch/skill-scoped receipt. Binding requires matching start/end content and reviewer-reported `completed:true` and `converged:true`; it is not independent proof that a model read the source. `--check-shared-libs <token>` reads a current finding as JSON on stdin and checks prior Skip decisions against the actual branch, capture and eligible raw source blobs. It returns `reusable:false` when proof is missing or unsafe, including older records without logger-versioned coverage. Final review logging computes shared-code fingerprints and coverage rather than trusting supplied proof. |
| `gstack-review-read` | **Review freshness** — emits review records with computed `review_freshness.status` and `reason`: CURRENT, STALE, or UNVERIFIED for diff reviews. `/ship` and `/land-and-deploy` use the same grade; a matching commit alone never certifies a diff review. [Dashboard rules](skills.md#review-readiness-dashboard). |
| `gstack-evidence` | **Verification-evidence ledger** — `run --label <lane> -- <cmd>` transparently wraps any test command (the child's exit code always passes through) and records what ran against which working-tree fingerprint (tracked and untracked files that are not gitignored, minus an untracked `.bun/` folder at the repo root, which Bun can create during a run); a run whose files changed while it ran is recorded without a fingerprint and with up to five of the changed paths, which `check` names in its STALE reason. `check` grades each label FRESH/STALE/MISSING with `--expect-cmd`, `--max-age`, and `--allow-paths` binding. /ship and /land-and-deploy cite fresh evidence instead of re-running suites. Per-run logs are 0600, capped at 2MB, pruned after 30 days; the ledger and logs stay machine-local by design. |
| `gstack-flaky` | **Flaky-test rerun for any JUnit runner** — `run --junit <file.xml> -- <command>` runs your test command (Playwright, `bun test`, jest, vitest, pytest) and, when at most `--max-reruns` (3) tests failed and the runner made no retries of its own, reruns the whole command once (`--rerun-scope test` opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (`GSTACK_FLAKE_LEDGER_JUNIT`, default `<state root>/flake-ledger-junit.jsonl`). A missing or malformed report, a crash or a killed runner never passes. `ids --junit <file>` prints a report's test ids. Exit 0 pass, 1 a test failed, 2 usage, 3 no usable report. [Guide](flaky.md). |
| `gstack-flaky` | **Flaky-test rerun for any JUnit runner** — `run --junit <file.xml> -- <command>` runs your test command (Playwright, `bun test`, jest, vitest, pytest) and, when at most `--max-reruns` (3) tests failed and the runner made no retries of its own, reruns the whole command once (`--rerun-scope test` opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (`GSTACK_FLAKE_LEDGER_JUNIT`, default `<state root>/flake-ledger-junit.jsonl`). A missing or malformed report, a crash or a killed runner never passes. `quarantine add|renew|release|check|report` keeps `.gstack/quarantine.json`: a quarantined test still runs, gets up to 3 attempts of the whole command, passes only on a full green attempt and is never counted as a pass (`run --base <ref>` reads the registry from the base branch). `ids --junit <file>` prints a report's test ids. Exit 0 pass, 1 a test failed, 2 usage, 3 no usable report. [Guide](flaky.md). |
| `gstack-pr-wait` | **PR event watcher** — `gstack-pr-wait <pr> [--repo owner/name] [--until green\|ready\|any-change] [--interval 20] [--timeout 60] [--exclude CHECK,...]` polls one pull request through `gh` and prints one stdout line per real event (a check failed with its link, all checks finished, a comment or review from someone other than the author, a review bot finished on the head, the base moved and its new commits touch this PR's files, BEHIND or DIRTY, merged or closed), with progress on stderr, so a monitor wakes the agent once per event instead of on a timer. `green` gates on every check on the head (an advisory bot's review is never waited for, its checks gate like any other), as `gstack-ci-gate` does; a check the base branch requires (rulesets and classic branch protection) is pending until it appears, never green by omission; two identical polls confirm a check set is complete; `--exclude` leaves a named non-required check out of the verdict as `gstack-ci-gate --exclude` does (a required name is refused and gates). Exit 0 when the condition is met, 1 when it needs action or `--timeout` passes (`PR_WAIT_BLOCKED`), 2 on bad input, a second watcher for the same PR or a local error (`PR_WAIT_REFUSED`). State under `<state root>/pr-wait/`, baseline discarded after 6 hours. Review bots are configuration: `gstack-config set pr_wait_gating_bots greptile,macroscope`, `pr_wait_advisory_bots coderabbit` (advisory bots are reported, never waited on). `/land-and-deploy` Step 3 is one call of it. Design ported from tstack's pr-wait (MIT, Theo Browne; NOTICE.md). [Exit codes](troubleshooting.md#gstack-pr-wait). |
| `gstack-issue-guard` | **Tracker-text trust envelope** — fetches GitHub issue/PR text (`issue <n>`, `pr-body`, `pr-comments`, or `--stdin`) and wraps it in a labeled envelope so agents treat it as data: injection-shaped lines get labeled even through fullwidth and invisible-character evasion, and forged envelope banners are defused. Every tracker-text ingress in gstack routes through it, enforced by a CI scanner. |
| `gstack-ios-qa-daemon` | **iOS QA daemon** — Mac-side broker between an agent and a connected iPhone or iPad over USB CoreDevice. Loopback by default; `--tailnet` opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on `~/.gstack/ios-qa-daemon.pid`. See [docs/howto-ios-testing-with-gstack.md](howto-ios-testing-with-gstack.md). |
+4 -3
View File
@@ -10,7 +10,7 @@ gstack has two support tiers. Full support gets the whole test suite, paid evals
## Minimal support
- **Windows** (Git Bash or MSYS): with a default Git for Windows install, `./setup` installs, and the core skills start without errors. The core skills are `/ship`, `/review`, `/qa-only`, `/investigate` and `/browse` on gstack's own browser. Once the `windows-smoke` CI job lands, it checks this on every pull request (see Windows CI below).
- **Windows** (Git Bash or MSYS): with a default Git for Windows install, `./setup` installs, and the core skills start without errors. The core skills are `/ship`, `/review`, `/qa-only`, `/investigate` and `/browse` on gstack's own browser. One required CI job, `windows-smoke`, checks this on every pull request (see Windows CI below).
- **Gemini models:** the core workflows run to the end without errors when Gemini is the host agent or a reviewer. We check this by hand or with one cheap smoke run, not with an eval cell.
That is the whole promise. Other skills, optional parts such as `/cso`'s native helper, non-default shell settings and speed are not covered.
@@ -23,8 +23,9 @@ That is the whole promise. Other skills, optional parts such as `/cso`'s native
## Windows CI
- **Today:** the six-shard Windows free suite (the `Windows Free Tests` workflow, `windows-free-tests`) runs on every pull request and every push to main, and it is a required check.
- **Once `windows-smoke` lands:** `windows-smoke` becomes the required Windows check on pull requests. The six-shard suite then runs nightly on main, or when someone starts it by hand, and never blocks a pull request. A red nightly run gets a fix only when it shows an install, startup or core-workflow break.
- `windows-smoke` is the required Windows check on pull requests.
- The six-shard Windows free suite (the `Windows Free Tests` workflow, `windows-free-tests`) runs nightly on main, or when someone starts it by hand, and never blocks a pull request.
- A red nightly run gets a fix only when it shows an install, startup or core-workflow break.
## What doesn't change
+41 -8
View File
@@ -2592,12 +2592,24 @@ case with `bun run eval:pass-rates --headroom --case <id>`.
<a id="cost-unknown"></a>
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
**Meaning.** PTY and Codex sessions record no billing, so their cost is
unknown. So is a trial in a multi-case shard whose result records are named by
a display label rather than the case id, because no record can be attributed
to it (a one-case shard's records are its case's). Only the known sum is
shown. A train verdict's `ATTEMPT:` line reports the same as `cost=$<x> known
+ <n> of <m> case(s) cost unknown`, a bare `cost=$<x>` only when every case
**Meaning.** Some trials' cost was not captured, so only the known sum is
shown. A trial is cost-unknown when it ran a PTY session whose transcript
could not be priced (a non-hermetic or resumed launch, the native
AskUserQuestion capture, two new transcripts for one cwd, no transcript or
usage, or a model with no price row), a
Codex session that completed no turn or whose model has no price row, a
`claude -p` session that ended without a result, or an aborted Agent SDK
session whose streamed usage could not be priced (no usage, or a model with no
price row). A PTY or aborted Agent SDK session that does price counts its
`streamed_cost_usd` instead. A trial is also unknown when no record and no
session can be attributed to it, for example a multi-case shard's
display-label record without `cost_case_id` (a one-case shard's records are
its case's); a model-free case (`timeline-event-flow`) is a known $0. A panel
trial (`behavior`, `judge`) is unknown whenever any of its sessions billed
nothing, even one with a priced `streamed_cost_usd`. The rules are in
[Failure causes and details](TESTING_INTERNALS.md#failure-causes). A train
verdict's `ATTEMPT:` line reports the same as `cost=$<x> known + <n> of <m>
case(s) cost unknown`, a bare `cost=$<x>` only when every case
in the train record is priced, and no cost at all when none is.
**Fix.** None needed; this is informational.
@@ -3855,7 +3867,10 @@ repository), `api_error` (gh could not read the run), `workflow_mismatch` (a
run of another workflow), `run_in_progress` (still queued or running) or
`run_not_success` (it failed, was cancelled or timed out). Each prints a fix
line; after the last three, clear again with a finished green run of the
lane's own workflow ([run evidence](pregate.md#run-evidence)).
lane's own workflow ([run evidence](pregate.md#run-evidence)). For a lane whose
workflow runs only through `workflow_call`, a run of a workflow whose job says
`uses: ./.github/workflows/<lane's>.yml` counts as a run of the lane (GST-971);
it refuses `caller_job_skipped` when its `<calling job> / <lane job>` job did not succeed.
Another tree is accepted in exactly one case: it differs from `pregate.json`'s
tree only in the repo's stamp rule (`.gstack/ship-policy.json` on the base
@@ -5486,6 +5501,23 @@ body no longer carries the exception's line word for word
/ship again and record the owner's acceptance with `--docs` or
`--review-loop-exception` ([ship-policy.md](ship-policy.md#ship-receipt)).
<a id="queue-receipt-targeted-missing"></a>
### `QUEUE_RECEIPT_TARGETED_MISSING`
**Meaning.** `ready`, `train build` or `land` recomputed the targeted
classes the head needs (money, gate, auth, secret) from git, and the receipt's
`review_targeted` does not cover every one of them with a `done` pass. The
`REVIEW_TARGETED:` line prints the required classes, the classes a done pass
covered (`legacy` when the receipt records no targeted pass) and
`verdict=missing`, or `verdict=failed` when the recorded pass failed. A
head that needs no class passes whatever the receipt records. The receipt is
unchanged.
**Fix.** Run /ship on the head the READY names so the targeted reader runs on
every required class and `gstack-ship-receipt write --review-targeted` records
a done pass ([review-policy.md](review-policy.md#targeted-pass-on-money-gate-auth-and-secret-code)).
Never edit `review_targeted` by hand.
<a id="queue-screen-stale-ref"></a>
### `QUEUE_SCREEN_STALE_REF`
@@ -5883,7 +5915,8 @@ body that names a counter the receipt records (`passes=3` against
against `gate.passed: 41`) with a different value. The receipt is written from
the run; the prose was typed, so a reader of the body would learn a number the
run did not produce. The `PROSE:` line names each field with both values. The
`## Approvals` table `gstack-approvals` renders is never read as a claim.
`## Approvals` table `gstack-approvals` renders, and the `Review-loop exception: …`
line the receipt's exception requires, are never read as claims.
**Fix.** Correct the body to the receipt's number. When the receipt is the
stale one (the run was repeated), rerun `/ship` so the receipt is rewritten