mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-15 12:57:22 +01:00
v1.93.17.0 stamp: merge main 9cc40e0d03 into #3811 (land)
This commit is contained in:
commit
182314528e
96 files changed
+3349
-267
No files matched your search
+124
-15
@@ -206,7 +206,16 @@ GSTA-203 chose them by a 3-run A/B against 20 × `ubicloud-standard-4` at k = 1
|
||||
(the PR that set them has the table). A `workflow_dispatch` run may
|
||||
override them with `free_runner` (`ubicloud-standard-2`, `-4` or `-8`),
|
||||
`free_shards` (1–40) and `free_jobs` (1–8) for runner-sizing experiments only;
|
||||
its job summary says so, and it is never cited as a PR or train Free Tests result. Each shard prints
|
||||
its job summary says so, and it is never cited as a PR or train Free Tests result.
|
||||
Each `free-suite` job's GitHub timeout comes from the plan, not a fixed number
|
||||
(#3850): `--ci-plan --job-setup-seconds $FREE_SUITE_SETUP_SECONDS` emits every
|
||||
shard's `timeout_minutes` as the shard's runner wall (the packed wall above),
|
||||
plus the flaky-retry pass's wall (at most 5 files, so the 6-minute base), plus
|
||||
the job's setup and teardown (`ciShardJobTimeoutMinutes`). So GitHub never kills
|
||||
a hung file before the runner names it and retries it. A shard that would need
|
||||
more than 90 minutes fails the plan with `FREE_JOB_TIMEOUT_OVER_CAP`; plan more
|
||||
shards or a higher k instead. The plan step prints the emitted timeouts and adds
|
||||
them to its job summary. Each shard prints
|
||||
`RESOURCES: peak_used_mib=… min_free_disk_mib=… samples=…` from a memory and
|
||||
disk sample every 2 seconds, even when the suite fails. The public
|
||||
`TREE_MUTATING` map now classifies exclusive host-state fixtures; its sole
|
||||
@@ -651,15 +660,37 @@ this checkout to read it)`.
|
||||
samples, rationale}` entry per failing dimension (since EVAL_POLICY v3 `mean`
|
||||
holds the panel's gating median), each part capped at 200
|
||||
characters. `sessions[]` (at most 32) summarizes the trial's session-ledger
|
||||
rows. `cost_known: false` marks a record or trial whose harness captured no
|
||||
billing: PTY and Codex sessions never bill, and an SDK session bills only with a
|
||||
terminal `total_cost_usd`. It also marks a trial no result record could be
|
||||
attributed to: a record named by the case id or carrying `case_id` is that
|
||||
case's, and a record named by a display label (`/qa fix loop`) is the shard's
|
||||
only case's when the shard ran one case, so a multi-case shard whose records
|
||||
carry display labels leaves those trials cost-unknown rather than guessing
|
||||
which case paid (`junitCensus` in `scripts/lib/paid-report.ts`). An absent
|
||||
field means known. Census and marathon cost lines print the known sum and
|
||||
rows. `cost_known: false` marks a record or trial whose cost is not fully
|
||||
known; an absent field means known. A record carries it when its harness
|
||||
captured no billing (a Codex session that completed no turn or whose model has
|
||||
no price row). A rule case's trial in the JUnit census (`caseCost` in
|
||||
`scripts/lib/paid-report.ts`) costs its records' sum plus what no record
|
||||
carries: each ledger session without a terminal result (`billed: false`) adds
|
||||
its `streamed_cost_usd`, and a trial with no record adds each billed session's
|
||||
own `cost_usd` (Codex). The trial is cost-unknown when any record says
|
||||
`cost_known: false`, when any of those sessions has no priced cost (a PTY
|
||||
session whose transcript does not price, a `claude -p` session that ended
|
||||
without a result, an aborted Agent SDK session with no usage or an unpriced
|
||||
model), or when it has neither records
|
||||
nor sessions. A case in `MODEL_FREE_CASES` (`timeline-event-flow`, which runs
|
||||
no model) with neither is a known $0. A panel trial (`behavior`, `judge`) from
|
||||
an isolated trial shard keeps the older rule (`trialCostKnown` in
|
||||
`test/helpers/eval-store.ts`): known only with at least one record and every
|
||||
session billed, and streamed cost is not added.
|
||||
|
||||
Records are attributed to a case this way: a record named by the case id or
|
||||
carrying `case_id` is that case's, and a record named by a display label
|
||||
(`/qa fix loop`) is the shard's only case's when the shard ran one case. In a
|
||||
multi-case shard, a display-label record whose `runSkillTest` test name is a
|
||||
registered case id carries `cost_case_id` (`recordE2E`), and the census adds
|
||||
its cost to that case only: the failed record, failure class and model still
|
||||
come from the records named by case id, so no verdict changes. A display-label
|
||||
record without `cost_case_id` in a multi-case shard is not attributed rather
|
||||
than guessed, and its case is cost-unknown when nothing else prices it
|
||||
(`junitCensus`). The report's headline `cost $<x>` adds the streamed session
|
||||
cost no collector record carries, so a newly priced trial counts once.
|
||||
|
||||
Census and marathon cost lines print the known sum and
|
||||
`N of M trial(s) cost unknown`, and a train verdict's `ATTEMPT:` line prints
|
||||
`cost=$<known> known + <n> of <m> case(s) cost unknown` when some but not
|
||||
every case in the train record has a cost (a record with no priced case prints
|
||||
@@ -671,13 +702,33 @@ including `runSharedInteractive`, the PTY session and `runRecordedCodexEval`)
|
||||
appends one best-effort line per armed session to
|
||||
`$GSTACK_EVAL_DIR/session-ledger.jsonl` (`test/helpers/session-ledger.ts`):
|
||||
a stable key `<runner>:<label>#n`, the exact `budget_ms` it armed, monotonic
|
||||
`elapsed_ms`, the `end` reason, sanitized evidence, `billed`, and a compact
|
||||
`elapsed_ms`, the `end` reason, sanitized evidence, `billed`, the billed
|
||||
`model`, `cost_usd` and token classes of a Codex session, and a compact
|
||||
liveness summary (longest in-request silence, the last event kind, any tool open
|
||||
at the end). `claude -p` sessions stream partial messages for this summary;
|
||||
transcripts keep their shape and size (full stream diagnostics only with
|
||||
`publicStreamDiagnostics`). A failed ledger write never fails a trial; a trial
|
||||
without ledger rows has no `sessions` and its headroom is `unknown`, never
|
||||
scored. The ledger rides in each slice artifact beside the shard results.
|
||||
scored. An Agent SDK session that ends without a terminal result (aborted,
|
||||
thrown) also records `streamed_cost_usd`: its streamed assistant usage priced
|
||||
by `streamedUsageCostUsd` (`lib/pricing.ts`), each message id once at its
|
||||
largest output, subagent turns included, `<synthetic>` messages skipped,
|
||||
5-minute cache writes at 1.25x and 1-hour writes at 2x the input rate. The
|
||||
field is absent when no message carries usage or any message's model has no
|
||||
price row, so an unpriced session stays cost-unknown, never $0. A PTY session
|
||||
records `streamed_cost_usd` from its transcript the same way. Both PTY ledger
|
||||
writers price `<CLAUDE_CONFIG_DIR>/projects/*/<id>.jsonl` plus
|
||||
`<id>/subagents/*.jsonl` (`test/helpers/pty/transcript-cost.ts`), where the
|
||||
id is the caller's `--session-id`, else the one transcript that appeared
|
||||
during this launch for its cwd. The launcher adds no argument. Pricing never
|
||||
takes the whole directory, because concurrent PTY tests in one process share
|
||||
one hermetic config dir; two new transcripts for one cwd are ambiguous and stay
|
||||
unknown. A non-hermetic or resumed launch, a caller that opts out
|
||||
(`priceTranscript: false`, the native AskUserQuestion capture, which never
|
||||
reads model transcripts), a missing transcript, no usage, an unpriced model or
|
||||
a read error leaves the field absent (cost-unknown), and a pricing failure
|
||||
never fails the test. The row keeps `billed: false`. The ledger rides in each slice
|
||||
artifact beside the shard results.
|
||||
|
||||
**Quarantine** (`CASE_QUARANTINE`, same file). An entry needs: a per-trial rate
|
||||
below 95% over at least 10 post-policy trials of the case's current input
|
||||
@@ -1232,9 +1283,11 @@ observation never removes a command and a missing or empty artifact never erases
|
||||
one; removing a command is a reviewed edit. While the map is in shadow mode it
|
||||
changes no selection, so a PR that changes only the map selects no paid case.
|
||||
|
||||
Dollar figures are lower bounds. Known cost excludes PTY trials, whose cost is
|
||||
unknown: about 39% of observed case runs in the committed map. The shadow block
|
||||
and `test:health` both say so.
|
||||
Dollar figures are lower bounds. Known cost excludes trials whose cost is
|
||||
unknown: about 39% of observed case runs in the committed map, mostly PTY
|
||||
trials recorded before PTY sessions were priced from their transcripts
|
||||
([Failure causes and details](#failure-causes)). The shadow block and
|
||||
`test:health` both say so.
|
||||
|
||||
- **Shadow block:** `prCoverage.shadow` in the paid-plan manifest lists the gate
|
||||
cases the map would select, would drop and would add, a provenance line per
|
||||
@@ -1750,6 +1803,32 @@ lower-case words in path components under `test/fixtures/`; a substring match
|
||||
separate change with its own touchfile updates; the list is here so that change
|
||||
can be scoped.
|
||||
|
||||
## Agent SDK system prompt
|
||||
|
||||
An Agent SDK caller that leaves out `systemPrompt` doesn't get Claude Code's
|
||||
prompt. It gets the SDK's bare stub, and the stub drops the line that tells the
|
||||
model to run `/skill` through the Skill tool. A skill arm then never loads its
|
||||
skill, so a "no difference" result says nothing about the skill (FOUN-32 found
|
||||
this in another crew's null result).
|
||||
|
||||
**Every caller says which prompt it uses.** `runAgentSdkTest` and every direct
|
||||
SDK `query()` call pass the Claude Code preset,
|
||||
`{ type: 'preset', preset: 'claude_code' }` (also exported as
|
||||
`CLAUDE_CODE_PRESET` from `test/helpers/sdk-system-prompt.ts`; an overlay arm
|
||||
adds `append`), or name why not: `{ type: 'sdk-default', reason }` for a stub
|
||||
provider or a connectivity probe, `{ type: 'custom', text, reason }` for an arm
|
||||
that replaces the prompt. The runner refuses a missing, empty, bare-string or
|
||||
unnamed prompt before it starts, and
|
||||
`test/sdk-system-prompt-tripwire.test.ts` scans test/ and scripts/ for the same
|
||||
mistake in free CI. Arms that compare loadouts, the no-gstack arm included,
|
||||
keep the same preset and differ only in what the arm loads.
|
||||
|
||||
**Record it, and treat a null on the stub as a method bug.** Every eval report
|
||||
and results table names the prompt each arm ran under (`claude_code`,
|
||||
`claude_code + append`, or the named reason). A null or negative result from an
|
||||
arm that ran on the SDK default is FIX-METHOD under R48 ("Why null?"): fix the
|
||||
prompt and rerun before reading it as KEEP or CUT-CANDIDATE.
|
||||
|
||||
## No-gstack arm
|
||||
|
||||
Garry, 2026-10-31: "all of our head to head tests have to test no gstack from
|
||||
@@ -1843,6 +1922,36 @@ it as the diagnostic. Candidate folders and git identity name a project, never
|
||||
the arm. The model panel (`test/skill-e2e-model-panel.test.ts`) runs a
|
||||
`<model>#no-gstack` arm beside each model (GST-468, GST-469).
|
||||
|
||||
## Frontier models under test
|
||||
|
||||
Garry, 2026-11-01 (R54): "I think you are spending too much time with Haiku and
|
||||
Sonnet, whereas GBrain and GStack are really designed to be used with frontier
|
||||
models like Opus and 6.1 Sol... just transfer from Haiku to Sonnet, who gives a
|
||||
shit about that, man?"
|
||||
|
||||
**Models under test are frontier models.** Today that's `claude-opus-5-5` and
|
||||
`gpt-6.1-sol`, plus at most one low-cost primary the preregistration names. Haiku
|
||||
and Sonnet are not models under test. They aren't stand-ins for a frontier model,
|
||||
and an eval has no tier-transfer arms (Haiku against Sonnet, say) unless its
|
||||
preregistration says it is asking about that tier.
|
||||
|
||||
**Verdicts need a frontier result.** A result measured only on Haiku or Sonnet
|
||||
can't carry a KEEP, FIX-METHOD or CUT-CANDIDATE verdict under R48. Rerun it on a
|
||||
frontier model first.
|
||||
|
||||
**What the rule leaves alone.** Judges, cross-family review voices and a
|
||||
harness's shipped housekeeping model aren't models under test, so they keep the
|
||||
model they already use.
|
||||
|
||||
**The check.** `gstack-eval-plan dry-run` prints `CHECK: frontier` whenever an arm
|
||||
is Haiku or Sonnet. When every arm is one, the dry run fails unless
|
||||
`--tier-question "<what the preregistration asks about that tier>"` is passed.
|
||||
When only some arms are, it warns. Either way it prints the whole plan's price
|
||||
on `claude-opus-5-5` and `gpt-6.1-sol`. The dry run doesn't enforce that number:
|
||||
compare it with the brief cap before any arm is swapped. `pilot start` refuses a
|
||||
pilot whose every arm is Haiku or Sonnet unless its dry run recorded a tier
|
||||
question.
|
||||
|
||||
## Running evals as an agent: detach
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in
|
||||
|
||||
+3
-1
@@ -434,7 +434,9 @@ A scheduled automation starts one lane a day with the
|
||||
runs' flake-records artifacts and the free suites' retried cases, writes one ledger line
|
||||
per event, groups by case, and fixes the worst repeat first with a PR that shows the
|
||||
failure red before green. It never reruns a gate and never touches a threshold, trial
|
||||
count or case kind. On Capy, create it with the automation tool, scheduled in the owner's
|
||||
count or case kind. Where a repository runs `gstack-flaky`, the lane also reads
|
||||
`gstack-flaky quarantine report --ledger <downloaded ledgers>` and fixes expiring quarantine
|
||||
entries before anything else ([flaky.md](flaky.md#the-daily-flaky-pass)). On Capy, create it with the automation tool, scheduled in the owner's
|
||||
timezone, running as a thread in the crew so its report lands in `reports/`.
|
||||
|
||||
The lane's side of the same rule: in CI the gate reruns a case that failed on its own
|
||||
|
||||
@@ -472,8 +472,9 @@ make <dir> writable, or set GSTACK_STATE_ROOT to a writable directory
|
||||
(LAUNCH_FILE_UNWRITABLE)`, and a failing run prints no `GSTACK_LAUNCH_FILE` line
|
||||
and removes the file an earlier passing run left beside the verdict it rewrites
|
||||
(a profile check that keeps a fuller ok verdict leaves that verdict's file).
|
||||
With `--model <id>`, or `GSTACK_MODEL=<id>` in its environment (what the Capy
|
||||
skill passes), an ok run other than `--for review-voice` also prints `LOADOUT: …`
|
||||
With `--model <id>`, `GSTACK_MODEL=<id>` in its environment, or else an
|
||||
`agent_model` in the session's `GSTACK_PREFERENCES_FILE` outside an eval run (GST-982, with a
|
||||
`LOADOUT_ASSUMED:` note), an ok run other than `--for review-voice` also prints `LOADOUT: …`
|
||||
and `SKILL_ROOT: <dir>` just before `GSTACK_LAUNCH`. For a loadout other than
|
||||
`full`, `GSTACK_LAUNCH_FILE` then names `<verdict file without .json>.loadout-<name>-<digest>.env`,
|
||||
a copy of the verdict's launch file that adds `GSTACK_LOADOUT_SKILL_ROOT`; the
|
||||
|
||||
@@ -93,7 +93,7 @@ What incomplete data means:
|
||||
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
|
||||
| cause `unrecorded` | the trial was written before `failure_cause` existed |
|
||||
| cause `unknown` | the trial failed with no evidence for any other cause |
|
||||
| `cost unknown` | the harness (PTY, Codex) records no billing, or the trial sits in a multi-case shard whose records are named by display label and cannot be attributed to it; only the known sum is shown |
|
||||
| `cost unknown` | a session's cost was not captured (a PTY session without a priced transcript, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown |
|
||||
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
|
||||
|
||||
## 4. Replay and repair (free)
|
||||
|
||||
+117
-8
@@ -6,7 +6,9 @@ rerun is a **flaky pass**: the job stays green and the flake is recorded on a le
|
||||
the daily flaky pass can fix it at the source. A test that fails twice is red.
|
||||
This is CREW rule 16 ("run the failed test again and note that it is flaky")
|
||||
for any runner that writes JUnit: Playwright, `bun test`, jest, vitest and
|
||||
pytest. gstack's own free suite keeps its file-level retry
|
||||
pytest. A test that keeps flaking while it is fixed can be
|
||||
[quarantined](#quarantine): it still runs, with up to 3 attempts, and never counts
|
||||
as a pass. gstack's own free suite keeps its file-level retry
|
||||
(`GSTACK_FREE_RETRY_FLAKY`, [Flake ledger](TESTING_INTERNALS.md#flake-ledger)).
|
||||
|
||||
## Get started in GitHub Actions
|
||||
@@ -74,6 +76,104 @@ It cannot see a coverage threshold configured without console output (for
|
||||
example `coverageReporter = ["lcov"]` in `bunfig.toml`), which is why this
|
||||
scope is opt-in.
|
||||
|
||||
## Quarantine
|
||||
|
||||
A test that keeps flaking while someone fixes it can be **quarantined** in
|
||||
`.gstack/quarantine.json` at the repository root. Quarantine is a larger
|
||||
attempt budget, never a skip: the test still runs on every run, is listed as
|
||||
quarantined in the output, and never counts as a pass.
|
||||
|
||||
**How a quarantined test is judged.** The verdict rule from
|
||||
[Rerun scope](#rerun-scope) does not change: an attempt counts only when the
|
||||
runner itself reports a full green run. When every test an attempt failed is a
|
||||
live quarantined test, and nothing else is wrong with that attempt (no kill, no
|
||||
failure outside the report, no first-run test missing), the whole command may
|
||||
run again, up to 3 attempts in total. The run is green only when one of those
|
||||
attempts is fully green. A quarantined test that fails all 3 is a **hard
|
||||
break**, and the run is red. A test that is not quarantined keeps its one
|
||||
rerun. A red attempt never turns green because its only reported failure is a
|
||||
quarantined test, because the runner's exit status can come from things the
|
||||
report does not show (a coverage threshold, a fail-on-flaky policy, a global
|
||||
setup). With `--rerun-scope test` the quarantined test gets the same 3
|
||||
attempts on the targeted path, with the same fallback to the whole command.
|
||||
When the runner retries on its own (Playwright `retries`), its attempts are
|
||||
the budget and nothing is rerun again.
|
||||
|
||||
**Missing results.** A quarantined id missing from a run that included its
|
||||
file (and, for Playwright, its project) is red. A run that did not include the
|
||||
file, such as a shard or a diff selection, is not.
|
||||
|
||||
**Read from the base branch.** `run --base <ref>` reads the registry from
|
||||
that commit instead of the working tree. Pull-request CI passes its target
|
||||
branch, so a change cannot quarantine the test it breaks. `check --base`
|
||||
compares against the merge base, so check out full history for it:
|
||||
|
||||
```yaml
|
||||
- uses: actions/checkout@v4
|
||||
with: { fetch-depth: 0 }
|
||||
- name: Quarantine registry
|
||||
run: '"$RUNNER_TEMP/gstack/bin/gstack-flaky" quarantine check --base "origin/${{ github.base_ref }}"'
|
||||
- name: E2E
|
||||
run: |
|
||||
"$RUNNER_TEMP/gstack/bin/gstack-flaky" run --base "origin/${{ github.base_ref }}" --junit results/junit.xml -- npx playwright test --reporter=line,junit
|
||||
```
|
||||
|
||||
### Commands
|
||||
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
| `gstack-flaky quarantine add <id> --owner <who> --ledger <url> --class <class> --reason <text> --evidence <url>… --junit <file> [--days <n>]` | Adds an entry that expires in `--days` (default 7, at most 14). `--junit` counts the suite for the cap and confirms the id exists; `gstack-flaky ids` prints the ids. |
|
||||
| `gstack-flaky quarantine renew <id> --owner <who> --evidence <url> [--days <n>]` | One renewal of up to 7 days from the later of now and the current expiry, with evidence not already on the entry and the owner named again. A second renewal is refused. |
|
||||
| `gstack-flaky quarantine release <id> --reason <text> --pr <url>` | Moves the entry to `released` with the date, the reason and the PR. |
|
||||
| `gstack-flaky quarantine check [--base <ref>] [--junit <file>]` | Validates every entry and the cap; with `--base`, also what this change did since the merge base. Exits 1 with one `QUARANTINE_REFUSED:` line per problem. |
|
||||
| `gstack-flaky quarantine report [--ledger <file\|dir>…] [--base <ref>]` | Lists expired entries, then live entries soonest-expiring first with days left. With downloaded ledgers it adds recent outcomes and marks entries with 10 consecutive clean passes `READY_TO_RELEASE`; without `--ledger` it says the exit rule was not checked. |
|
||||
|
||||
### Rules
|
||||
|
||||
- **Every entry is owned and expires.** An entry holds `owner`, `ledger`
|
||||
(the ledger line or issue tracking the fix), `failureClass`, `reason`,
|
||||
`evidence`, `enteredAt`, `expiresAt`, `renewals` and `suiteTests` (the
|
||||
suite size `add` counted). It lasts 7 days by default and at most 14 from
|
||||
entry, plus one renewal of at most 7 days.
|
||||
- **Expired means not quarantined.** An expired entry stops applying without
|
||||
an edit: `run` treats the test as an ordinary test and prints
|
||||
`FLAKY_QUARANTINE_EXPIRED` with the renew and release commands. An entry
|
||||
that breaks a rule (for example one that lasts 30 days) is ignored the same
|
||||
way, with `FLAKY_QUARANTINE_IGNORED`.
|
||||
- **Cap.** Live entries are capped at 10% of the suite's tests, at least 1,
|
||||
and 5 for a suite under 50 tests. `add` refuses past the cap; `check`
|
||||
enforces it against `--junit` when given, else against the smallest
|
||||
`suiteTests` on a live entry.
|
||||
- **Classes.** `detector`, `harness`, `timing` and `model-latency`.
|
||||
`product` is refused (a product defect is fixed, not quarantined), and so is
|
||||
`infra` (an infrastructure red is rerun or fixed as infrastructure).
|
||||
- **`check --base` refuses** an entry added in the same change that edits its
|
||||
test's file, an entry that disappears without a release record, a release
|
||||
record that was removed, and a changed `enteredAt` or rewritten renewal
|
||||
history.
|
||||
- **Exit rule.** 10 consecutive clean passes (`quarantined-pass` with no
|
||||
failed attempt) on the ledgers since the entry was added. Release the entry
|
||||
with the fix PR.
|
||||
- **Separate from `CASE_QUARANTINE`.** gstack's own paid-eval quarantine
|
||||
(`CASE_QUARANTINE` and `EVAL_POLICY`,
|
||||
[TESTING_INTERNALS.md](TESTING_INTERNALS.md)) is a different registry with
|
||||
its own rules for eval cases. The two share no schema, and `gstack-flaky`
|
||||
neither reads nor writes it.
|
||||
|
||||
### The daily flaky pass
|
||||
|
||||
Download the ledger artifacts of the last week's runs and read them with
|
||||
`report`; fix expired and `EXPIRING` entries first (fix the test, or renew
|
||||
once with new evidence), release `READY_TO_RELEASE` entries, then fix the
|
||||
worst repeat on the ledger:
|
||||
|
||||
```bash
|
||||
for id in $(gh run list --workflow e2e.yml --limit 50 --json databaseId -q '.[].databaseId'); do
|
||||
gh run download "$id" --pattern 'flaky-junit-ledger-*' --dir "ledgers/$id" || true
|
||||
done
|
||||
gstack-flaky quarantine report --ledger ledgers
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **One rerun, small failures only.** A failed test gets one rerun when the
|
||||
@@ -101,8 +201,9 @@ scope is opt-in.
|
||||
- **Ids.** A test is `<file>::<project>::<describe > title>`, with the project
|
||||
empty outside Playwright, so a test in two browser projects is two tests.
|
||||
`gstack-flaky ids --junit <file>` prints them.
|
||||
- **Ledger.** One JSON line per flaky pass (`kind: flaky-pass`) and per test
|
||||
that failed its rerun (`rerun-fail`), with the tree, commit, branch, run URL,
|
||||
- **Ledger.** One JSON line per flaky pass (`kind: flaky-pass`), per test
|
||||
that failed its rerun (`rerun-fail`) and per quarantined test on every run
|
||||
(`quarantined-pass`, `quarantined-fail`, `hard-break`), with the tree, commit, branch, run URL,
|
||||
attempts, the rerun's scope (`test` or `command`) and the first lines of
|
||||
the error. Ids, paths and failure text go through gstack's redaction engine
|
||||
first; when the scan cannot run, they are withheld. The path is `--ledger`,
|
||||
@@ -113,15 +214,17 @@ scope is opt-in.
|
||||
|
||||
## Output and exit codes
|
||||
|
||||
One `FLAKY: <flaky-pass|fail|rerun-fail> <id> attempts=<…> (<reason>)` line per
|
||||
test that failed at least once, `FLAKY_RERUN: scope=command (<why>)` when the
|
||||
rerun repeated the whole command, then
|
||||
`FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>`.
|
||||
One `FLAKY: <flaky-pass|fail|rerun-fail|quarantined-pass|quarantined-fail|hard-break> <id> attempts=<…> (<reason>)`
|
||||
line per test that failed at least once and per quarantined test,
|
||||
`FLAKY_RERUN: scope=command (<why>)` when the rerun repeated the whole command, then
|
||||
`FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> quarantined=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>`.
|
||||
`failed` counts every outcome but `flaky-pass` and `quarantined-pass`; `quarantined`
|
||||
counts the quarantined tests' outcomes, which are never counted as passes.
|
||||
|
||||
| Exit | Meaning |
|
||||
|---|---|
|
||||
| 0 | Every test passed, flaky passes included |
|
||||
| 1 | A test failed (after its rerun, or with no rerun), including `FLAKY_UNREPORTED_FAILURES` |
|
||||
| 1 | A test failed (after its rerun, or with no rerun), including `FLAKY_UNREPORTED_FAILURES`, a quarantined test's hard break or missing result, and a refused `quarantine` command |
|
||||
| 2 | Usage error, including a `--junit` path that is not a `.xml` file, or an existing file there that is not a readable JUnit report |
|
||||
| 3 | No usable report, a crash, or a killed runner (a code below on stderr) |
|
||||
|
||||
@@ -135,3 +238,9 @@ rerun repeated the whole command, then
|
||||
| `FLAKY_COMMAND_NOT_STARTED` | 3 | The command could not be started | Check the argv after `--` |
|
||||
| `FLAKY_UNREPORTED_FAILURES` | 1 | The runner counted failures or errors its JUnit omits | Fix the load error or the error outside any test: a real red |
|
||||
| `FLAKY_LEDGER_WRITE_FAILED` | — | The ledger could not be appended (warning) | Check the path; the verdict stands |
|
||||
| `FLAKY_QUARANTINE_INVALID` | 3 | `.gstack/quarantine.json` is not valid JSON or not a schema 1 registry | Fix the file; `quarantine check` names the problem |
|
||||
| `FLAKY_QUARANTINE_BASE` | 3 | `--base` does not resolve to a commit (exit 2 for a value starting with `-`), or `check` finds no merge base | Fetch the base branch; `check --base` needs `fetch-depth: 0` |
|
||||
| `FLAKY_QUARANTINE_REFUSED` | 1 | `add`, `renew` or `release` broke a rule | Read the reason; [Quarantine](#quarantine) lists the rules |
|
||||
| `FLAKY_QUARANTINE_USAGE` | 2 | An unknown `quarantine` command or option, or a missing value | `gstack-flaky quarantine --help` |
|
||||
| `FLAKY_QUARANTINE_EXPIRED` | — | Notice: an expired entry no longer applies | Fix the test, renew once, or release; the line prints both commands |
|
||||
| `FLAKY_QUARANTINE_IGNORED` | — | Notice: an entry breaks a rule and does not apply | Fix the entry; `quarantine check` names the rule |
|
||||
+8
-7
@@ -9,7 +9,7 @@ One model has its own loadout: Claude Opus 5.5 gets `lite` ([below](#lite-on-cla
|
||||
| Term | Meaning |
|
||||
|---|---|
|
||||
| Module | One of four kinds. `skill:<dir>` is a whole skill. `section:<skill>/<id>` is one entry of `<skill>/sections/manifest.json`. `overlay:<name>` is one `model-overlays/<name>.md`; excluding a base overlay also removes the base's text from every overlay that inherits it; the inheriting overlay's own text stays. `preamble:<id>` is one piece of the shared preamble (`PREAMBLE_PIECES` in `scripts/resolvers/preamble.ts`). |
|
||||
| Pinned | A module no loadout can remove where `full` renders it: the start command, plan-mode rules, the question format, completion status, the unattended profile, /ship's review army and adversarial pass, the router and `/gstack-upgrade`. A skill's `preamble-tier` still applies, so a tier-1 skill has no question-format section in any loadout. |
|
||||
| Pinned | A module no loadout can remove where `full` renders it: the start command, plan-mode rules, the question format, completion status, the Claims Need Evidence rule, the unattended profile, /ship's review army and adversarial pass, the router and `/gstack-upgrade`. A skill's `preamble-tier` still applies, so a tier-1 skill has no question-format section in any loadout. |
|
||||
| Rung | How much scaffolding a model gets: `frontier`, `strong`, `mid`, `light`, or `unknown`. Model ids are matched after lowercasing and stripping a `provider/` prefix (`anthropic/claude-haiku-5-5` is `mid`). Rungs are separate from the frontier/smart tiers in `lib/model-catalog.ts`, which choose which model runs a role. |
|
||||
| Loadout | `{ base, exclude, include, preamble_tier_max, note }`. `full` (include `*`) is always present. An unknown model gets `default.unknown`, which is `full`. |
|
||||
| Model map | `models` in `modules/loadouts.json`: an exact model id to a loadout, checked before rungs. The id also matches its dated snapshot (`-YYYYMMDD`) and `-latest`, never a point release or any other suffix (`claude-opus-5-5` matches `claude-opus-5-5-20261101`, not `claude-opus-5-50` or `claude-opus-5-5-1`). |
|
||||
@@ -40,21 +40,22 @@ bin/gstack-modules install --full-root "$HOME/.claude/skills" --model claude-opu
|
||||
|
||||
## Lite on Claude Opus 5.5
|
||||
|
||||
`lite` is `full` minus eleven coaching pieces of the shared preamble: voice, writing style, completeness, confusion, evidence, context health, question tuning, search before building, repo mode, brain health and context recovery. It keeps every pinned piece, the upgrade check (the PROACTIVE and skill-prefix rules) and brain sync (the privacy stop-gate). Only `claude-opus-5-5` maps to it, because only Opus 5.5 has been measured: office-hours showed no loss on Opus 5.5 (GSTA-161 preamble run), and the land-and-deploy confirmation (GSTA-161, PR 1b) compared full, lite and no gstack on the same tasks. Haiku 5.5 leaned slightly against a larger cut on the same office-hours run, so small models stay on `full`. Two skills point at a preamble section lite drops: office-hours and plan-ceo-review name ETHOS.md through the "Search Before Building" section, so under lite they find ETHOS.md in the gstack checkout instead.
|
||||
`lite` is `full` minus ten coaching pieces of the shared preamble: voice, writing style, completeness, confusion, context health, question tuning, search before building, repo mode, brain health and context recovery. It keeps every pinned piece, including the Claims Need Evidence rule (GST-944: the honest-work design never retires it while gstack reports evidence, and it adds 804 bytes back to lite's 16,175-byte shared preamble), the upgrade check (the PROACTIVE and skill-prefix rules) and brain sync (the privacy stop-gate). Only `claude-opus-5-5` maps to it, because only Opus 5.5 has been measured: office-hours showed no loss on Opus 5.5 (GSTA-161 preamble run), and the land-and-deploy confirmation (GSTA-161, PR 1b) compared full, lite and no gstack on the same tasks. Haiku 5.5 leaned slightly against a larger cut on the same office-hours run, so small models stay on `full`. Two skills point at a preamble section lite drops: office-hours and plan-ceo-review name ETHOS.md through the "Search Before Building" section, so under lite they find ETHOS.md in the gstack checkout instead.
|
||||
|
||||
### How gstack picks a loadout
|
||||
|
||||
gstack never guesses the model. It uses, in order:
|
||||
gstack does not detect the running model. It uses, in order:
|
||||
|
||||
1. An explicit loadout: `./setup --loadout <name>` on Claude Code, or `bin/gstack-modules install --loadout <name>`. It is served as asked and bound to no model.
|
||||
2. An explicit model id: `--model <id>` on the Capy installer (`scripts/install.sh --model <id>`, which passes it to `bin/gstack-capy-install`), else `GSTACK_MODEL`, which the installer also reads.
|
||||
3. Nothing: `full`.
|
||||
3. On Capy only, with neither and outside an eval run: the owner's `agent_model` (this machine's `config.yaml`, else the Drive skill's `preferences.yaml`), as `gstack-models resolve --role agent` resolves it (`opus` is the newest Opus). This is an assumption, not a detection, and the installer prints `LOADOUT_ASSUMED:` beside it. `host` or no preference means no model.
|
||||
4. Nothing: `full`.
|
||||
|
||||
A model id is lowercased, its `provider/` prefix and a trailing `[…]` are dropped, and it must then be a plain id (`^[a-z0-9][a-z0-9._-]{0,79}$`). An unmapped id serves `full`; an id that is not plain also serves `full` and prints `LOADOUT_INVALID_MODEL`. Nothing reads Claude Code settings or an inherited loadout variable, so a Sonnet subagent of an Opus session gets `full` unless its own install names Opus.
|
||||
A model id is lowercased, its `provider/` prefix and a trailing `[…]` are dropped, and it must then be a plain id (`^[a-z0-9][a-z0-9._-]{0,79}$`). An unmapped id serves `full`; an id that is not plain also serves `full` and prints `LOADOUT_INVALID_MODEL`. Nothing reads Claude Code settings or an inherited loadout variable, so on Claude Code a Sonnet subagent of an Opus session gets `full` unless its own install names Opus. On Capy every session shares the owner's preference, so a session on another model sets `GSTACK_MODEL=<its id>` to get its own loadout.
|
||||
|
||||
### Capy
|
||||
|
||||
The Capy skill passes the thread's model id in the environment: `GSTACK_MODEL=<id> bash scripts/install.sh --check --for <skill>`. The variable, unlike `--model`, is safe on a checkout whose installer predates loadouts: that installer ignores it instead of refusing an unknown argument. The installer prints two lines before the launch lines:
|
||||
A plain `bash scripts/install.sh --check --for <skill>` picks the loadout from the owner's `agent_model` preference, so the skill needs no model sentence (`source=preferences` on the line below, or `source=config` when this machine's `config.yaml` sets `agent_model`). The preference names the model Garry starts sessions on, not the model reading the skill, so the installer says so in a `LOADOUT_ASSUMED:` line, and it skips the preference inside an eval run (`GSTACK_EVAL_DIR` or `EVALS_RUN_ID` set), where the model under test may differ. A session on another model names its own with `GSTACK_MODEL=<id>`, which wins over the preference and, unlike `--model`, is safe on a checkout whose installer predates loadouts: that installer ignores it instead of refusing an unknown argument. The installer prints two lines before the launch lines:
|
||||
|
||||
```
|
||||
LOADOUT: lite model=claude-opus-5-5 source=GSTACK_MODEL digest=3f2a9c01d4e7 reused=yes
|
||||
@@ -63,7 +64,7 @@ SKILL_ROOT: /home/user/.gstack/loadouts/lite-3f2a9c01d4e7
|
||||
|
||||
Read skills from `SKILL_ROOT` (`<root>/gstack/SKILL.md` routes, `<root>/<skill>/SKILL.md` runs) and source the printed `GSTACK_LAUNCH_FILE`, which adds `GSTACK_LOADOUT_SKILL_ROOT`. For `full`, `SKILL_ROOT` is `~/.claude/skills` and the launch file is the install verdict's own.
|
||||
|
||||
The lite tree is published by `bin/gstack-modules install` under `<state root>/loadouts/<loadout>-<digest12>/`, beside the full install; the checkout and its committed SKILL.md files stay the full render. The digest covers the selection, the bound model, the full root, host, overlay, plan mode and the source tree (the git HEAD tree plus any uncommitted change), so an upgrade or a different model gets a new folder and an unchanged one is reused after its files are re-hashed against `loadout.lock.json`. Each folder is a complete skill tree: the rendered SKILL.md and section files plus every tracked runtime asset of each skill (templates, references, scripts). A published folder is never replaced. Pruning removes only folders it published (a `<loadout>-<digest12>` name with an install lock) beyond the three most recently used per loadout, and never one used in the last day. When `--full-root` is not `~/.claude/skills`, the tree's command paths point at that install's `gstack` folder. Two Capy threads with different models on one machine share one install verdict and get one launch file each (`<verdict>.loadout-<name>-<digest>.env` for lite). Any failure serves `full` and prints `LOADOUT_FALLBACK`.
|
||||
The lite tree is published by `bin/gstack-modules install` under `<state root>/loadouts/<loadout>-<digest12>/`, beside the full install; the checkout and its committed SKILL.md files stay the full render. It carries over two of the full render's install settings: gbrain detection (`gbrain-detection.json` in the state root, so the brain context-load and save-results blocks appear when gbrain is detected) and `gstack-config` `disabled_skills` (the router omits those skills). The digest covers the selection, the bound model, the full root, host, overlay, plan mode, gbrain detection, disabled skills and the source tree (the git HEAD tree plus any uncommitted change), so an upgrade or a different model gets a new folder and an unchanged one is reused after its files are re-hashed against `loadout.lock.json`. Each folder is a complete skill tree: the rendered SKILL.md and section files plus every tracked runtime asset of each skill (templates, references, scripts). A published folder is never replaced. Pruning removes only folders it published (a `<loadout>-<digest12>` name with an install lock) beyond the three most recently used per loadout, and never one used in the last day. When `--full-root` is not `~/.claude/skills`, the tree's command paths point at that install's `gstack` folder. Two Capy threads with different models on one machine share one install verdict and get one launch file each (`<verdict>.loadout-<name>-<digest>.env` for lite). Any failure serves `full` and prints `LOADOUT_FALLBACK`.
|
||||
|
||||
### The start line carries the model
|
||||
|
||||
|
||||
+16
-4
@@ -74,7 +74,7 @@ the byte-pinned `/ship` renders in `test/fixtures/golden/`).
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `runner` | the local test command with `{files}` (gstack: `bun test --timeout=30000 {files}`, the 30 s per-test timeout CI uses); unset, it is detected (bun, vitest, jest, pytest, `node --test`, and `flutter test`/`dart test` per Dart package root, through fvm when that package or an ancestor pins it with `.fvmrc` or `.fvm/fvm_config.json` and fvm is on PATH; `cargo test -p <name>` per Rust crate, run from the root of the workspace the crate is a member of (Cargo's `members` globs, `exclude` and `package.workspace`), else the crate directory; a JS/TS test whose nearest package.json is a nested package declaring vitest or jest (a dependency or its `test` script) runs from that package, with package-relative paths, through the package manager its `packageManager` field or the nearest lockfile up to the repo root names: `pnpm exec`, `yarn`, `bun x`, else `npx`; every other JS/TS test keeps the root detection); nothing detected, a Dart SDK, cargo or package manager missing from PATH, a `.dart` file with no `pubspec.yaml` above it, a `.rs` file outside any crate, or one in a crate whose name Cargo would reject (anything but letters, digits, `_` and `-`), is `incomplete` (the runnable groups still run, so a real failure still fails) |
|
||||
| `runner` | the local test command with `{files}` (gstack: `bun test --timeout=30000 {files}`, the 30 s per-test timeout CI uses); unset, it is detected (bun, vitest or jest for JS/TS files, pytest when python3 can import it (else `pytest` or `python3` is named missing), `node --test`, and `flutter test`/`dart test` per Dart package root, through fvm when that package or an ancestor pins it with `.fvmrc` or `.fvm/fvm_config.json` and fvm is on PATH; `cargo test -p <name>` per Rust crate, run from the root of the workspace the crate is a member of (Cargo's `members` globs, `exclude` and `package.workspace`), else the crate directory; a JS/TS test whose nearest package.json is a nested package declaring vitest or jest (a dependency or its `test` script) runs from that package, with package-relative paths, through the package manager its `packageManager` field or the nearest lockfile up to the repo root names: `pnpm exec`, `yarn`, `bun x`, else `npx`; a JS/TS test file that imports `@playwright/test` at the start of a line runs as `playwright test --reporter=list` from its nearest package through that same package manager, whatever the unit runner is, a type-only import (`import type …`, or one whose every specifier is `type X`) does not count, file arguments are passed as escaped regexes anchored to the path (Playwright reads them as regexes), and a run where every test fails and Playwright says its browser is not installed is `PREGATE_ENVIRONMENT` (a partial install, where some tests pass, stays the diff's failure); every other test keeps the root detection, split by language when no one runner fits them all, so `.mjs` and `.py` root tests each get their own runner); nothing detected, a Dart SDK, cargo or package manager missing from PATH, a `.dart` file with no `pubspec.yaml` above it, a `.rs` file outside any crate, or one in a crate whose name Cargo would reject (anything but letters, digits, `_` and `-`), is `incomplete` (the runnable groups still run, so a real failure still fails) |
|
||||
| `dependencies` | glob → test files for subprocess targets, runtime reads and generated inputs the import graph cannot see; `[]` declares "no lane needed" (docs, licences). A `[]` that matches a deleted test does not skip it: the deleted test still runs its owning suite, and a file naming it that `[]` covers (a durations list) counts as covered. With this key present, even empty, the policy [declares lanes](#lane-requirements) |
|
||||
| `empty_lanes` | extra globs for the docs/no-tests empty lane. Any Markdown file already qualifies with no pin file when no test depends on it (the same import graph as other files: a quoted path, a `join()` of path segments, an import, or a `dependencies` entry). reStructuredText and AsciiDoc count as Markdown here. README, CHANGELOG, CONTRIBUTING, LICENSE, AGENTS and CLAUDE qualify even without `.md`, and so does licence text under any vendored name (`roboto-LICENSE.txt`, `LICENSE-MIT`, `licenses/*.txt`). Source code and files under a test directory never qualify. A test that reads the file still selects that test |
|
||||
| `selection` | `workflow.yml/job` → a command printing the lane's file list, for matrices a planner computes (gstack: `scripts/test-free-shards.ts --list`) |
|
||||
@@ -220,6 +220,14 @@ Two stages with separate clocks, both printed:
|
||||
other line holding the old value still fail. Only the row itself is exempt:
|
||||
the old value after its closing bracket on the same line still fails. Rows in object form (`{ v: 45 }`) or spread over several
|
||||
lines are not recognized as history rows.
|
||||
A one-character old value (`2`) counts only in a statement that names the
|
||||
constant: on the name's line, or on up to three lines that continue it (the
|
||||
line before ends with `(`, `[`, `{`, `,`, `.`, `=` or another operator, or the
|
||||
line starts with `.`, `)`, `]`, `}` or an operator), never on a comment line, plus the history rows
|
||||
above. A `$2` in a shell stub, a comment, or `export default 2` in a fixture
|
||||
elsewhere in the file is not a consumer. A call-site name such as `attempts`
|
||||
or `limit` is an ordinary word, so for it the file must also name the changed
|
||||
module.
|
||||
|
||||
- `timing`: added or changed test lines that compare an elapsed wall-clock or
|
||||
CPU time against a bound, or sleep and then assert, print
|
||||
@@ -437,7 +445,7 @@ test-tier command:
|
||||
Per-file lines come from a progress adapter where the runner's reporter
|
||||
provides them: `bun test` (its `<file>:` headers) is the only adapter today;
|
||||
a one-file run (each serial file) reports its file from the launch for any
|
||||
runner. Other runners (node --test, vitest, jest, pytest, flutter test, dart test, cargo test) get heartbeats
|
||||
runner. Other runners (node --test, vitest, jest, pytest, playwright test, flutter test, dart test, cargo test) get heartbeats
|
||||
only. A heartbeat follows each 60 s with no progress line;
|
||||
- is killed as **stalled** when its log does not grow for the stall timeout
|
||||
(`--stall-timeout <s>`, else `stall_timeout_s`, else 300; `0` disables): the
|
||||
@@ -994,7 +1002,7 @@ failure the branch caused carries `cause: "diff"` (`PREGATE_LANE_FAILED`,
|
||||
|
||||
| Kind | Evidence |
|
||||
|---|---|
|
||||
| `missing-binary` | exit 127, `command not found`, a spawn `ENOENT`, a runner tool `not on PATH` (dart, flutter, cargo, a package manager) |
|
||||
| `missing-binary` | exit 127, `command not found`, a spawn `ENOENT`, a runner tool `not on PATH` (dart, flutter, cargo, a package manager, `python3`, or `pytest` when python3 cannot import it), Playwright's browsers not installed |
|
||||
| `network` | `ENOTFOUND`, `ECONNREFUSED`, `ECONNRESET`, `ETIMEDOUT`, `EAI_AGAIN`, `Could not resolve host`, `fetch failed`, a TLS handshake or certificate failure |
|
||||
| `quota` | `quota`, `429 Too Many Requests`, `rate limit`, `insufficient_quota` (a vCPU quota on the remote backend) |
|
||||
| `disk` | `ENOSPC`, `No space left on device`, `EDQUOT` |
|
||||
@@ -1005,7 +1013,11 @@ The classifier is conservative. A local lane that exited non-zero is read from
|
||||
its output **only when its runner reported no test summary** (nothing ran) and
|
||||
the runner was not seen starting a test file; a run whose summary counts `N
|
||||
ran, M failed` is the diff's failure whatever its log says, so a test that
|
||||
failed on `ECONNREFUSED` is still a red test. A remote gate that exited
|
||||
failed on `ECONNREFUSED` is still a red test. One exception: when every
|
||||
failed run is a Playwright run in which every test failed and Playwright says
|
||||
its browsers are not installed, the lane is `missing-binary` with `proof:
|
||||
"output"` (a partial install, where some tests pass, stays the diff's
|
||||
failure). A remote gate that exited
|
||||
without a `PREGATE_REMOTE:` line is read from its **preamble only** (its first
|
||||
40 lines, before any test output could stream): a VM refused for quota or an
|
||||
unreachable API there never ran the tests, so it is environment and the
|
||||
|
||||
+19
-2
@@ -98,9 +98,19 @@ incomplete coverage, which blocks until the reviewer is rerun.
|
||||
finishes with its open notes listed in the PR body and receipt, or stops for
|
||||
the owner when a blocking finding is open, or was fixed in round 2 and is
|
||||
unverified, or a round-2 fix edited a gate-class path, naming it (path,
|
||||
severity, problem). A third round is the owner's
|
||||
severity, problem). The cap limits fix rounds, not Step 11's voices: on gate
|
||||
class a stop waits for Step 11's adversarial, outside and targeted voices
|
||||
(`NEXT: voices`), so gate code never reaches the owner's "ship with open
|
||||
notes" without them (GST-967). A third round is the owner's
|
||||
grant of one scoped fix-check (recorded in `review_loop_exception`), never the
|
||||
loop's. The cap approves nothing.
|
||||
loop's: once `gstack-approvals add` records the grant (its ask or scope
|
||||
naming `review_loop_grant` and the stop head's sha),
|
||||
`gstack-review-plan --grant <approval id> --range-base <round 2's PASS_HEAD>`
|
||||
plans that fix-check in place of the stop after a round-2 full pass with a
|
||||
fixed blocking finding or a gate-class edit, and refuses any other row,
|
||||
including a grant for a stop head before `--range-base` (an earlier /ship).
|
||||
A fix-check never earns another, so the grant cannot repeat, and a stop after
|
||||
a round-2 fix-check still goes to the owner. The cap approves nothing.
|
||||
|
||||
<a id="transitions"></a>
|
||||
### Transitions
|
||||
@@ -122,12 +132,19 @@ incomplete coverage, which blocks until the reviewer is rerun.
|
||||
| a full pass that leaves a blocking finding open (fixed or not) | stop: name it for the owner |
|
||||
| a fix-check that found nothing blocking | done |
|
||||
| a fix-check that found a blocking finding | stop: name it for the owner |
|
||||
| a stop (any row above) on gate class before Step 11's voices ran | Step 11's voices (adversarial, outside, targeted), then ask again with `--adversarial-ran` |
|
||||
| a round-2 full pass that stops for a fixed blocking finding or a gate-class edit, with the owner's recorded grant (`--grant <approval id>`) | one scoped fix-check (round 3), after Step 11 on gate class |
|
||||
|
||||
A **fix-check** is one reviewer reading the touched hunks, the fix list, the
|
||||
test results and the callers of every changed symbol, at full effort. It is
|
||||
logged as a `review-fix-check` row, so it never counts as a pass. No army, no
|
||||
Red Team and no outside voice run on it.
|
||||
|
||||
`NEXT: voices` runs Step 11 as `done` does, from the last full pass's
|
||||
manifest; /ship then asks the plan again with `--adversarial-ran` (a blocking
|
||||
finding the voices queued counts as `--blocking-open`, never a new Step 9), and
|
||||
the stop or the granted fix-check it names stands.
|
||||
|
||||
<a id="scoped-pass"></a>
|
||||
A **scoped pass** is pass 2 after a blocking fix when pass 1 and the diff both
|
||||
ran below full depth (no gate class, no depth error, no `--depth-floor full`).
|
||||
|
||||
+6
-1
@@ -1234,7 +1234,12 @@ block `gstack-approvals` renders (its heading, the Date | Asked | Exact words
|
||||
table and the `Source: gstack-approvals record` line) quotes the owner's
|
||||
words, not this PR's numbers, so it is skipped: "one Step 11 pass" in that table
|
||||
is not a `pass` claim. Text you write under the heading after that block, and
|
||||
a `## Approvals` example inside a code fence, are still checked. Data PRs (evals,
|
||||
a `## Approvals` example inside a code fence, are still checked. The
|
||||
`Review-loop exception: …` line the receipt's own exception requires is
|
||||
skipped the same way (GST-977): it quotes the owner's reason, so "outside 1 and
|
||||
2 skip detector" is not a `2 skip` claim. Only a line that matches the rendered
|
||||
line exactly is skipped, inside a code fence too, as the receipt check accepts
|
||||
a fenced copy; the same words inside a longer line are still checked. Data PRs (evals,
|
||||
measurements, sweeps) run it before READY so the body's numbers are the run's.
|
||||
|
||||
<a id="review-scope"></a>
|
||||
|
||||
+4
-1
@@ -961,7 +961,10 @@ arms, decoys, stop rules, cap) and `--check` refuses to continue while a field i
|
||||
blank (`PREREG_INCOMPLETE`). `arms` resolves the `eval-arms` model role against
|
||||
`lib/pricing.ts` and stops on an unpriced model (`EVAL_PRICE_MISSING`); `dry-run`
|
||||
runs every row through the stub model for $0 and flags malformed rows
|
||||
(`EVAL_DRY_RUN_FAILED`); `pilot start|settle|release|report` runs three priced
|
||||
(`EVAL_DRY_RUN_FAILED`), and refuses a plan whose every arm is Haiku or Sonnet
|
||||
unless `--tier-question` names what it asks about that tier, re-costing it on
|
||||
frontier models ([frontier models under test](TESTING_INTERNALS.md#frontier-models-under-test));
|
||||
`pilot start|settle|release|report` runs three priced
|
||||
items, records actual spend, extrapolates each priced arm to the full set against
|
||||
the cap (`EVAL_BUDGET_EXHAUSTED`) and prints the spend table the approval question
|
||||
shows. An arm on a subscription with no extra charge is declared with
|
||||
|
||||
@@ -19,7 +19,7 @@ Beyond the slash-command skills, gstack ships standalone CLIs for workflows that
|
||||
| `gstack-review-log` | **Review-pass receipts** — `--start <skill>` captures the working-tree fingerprint before a diff review; `'<JSON>' --finish <token>` consumes that single-use, repository/branch/skill-scoped receipt. Binding requires matching start/end content and reviewer-reported `completed:true` and `converged:true`; it is not independent proof that a model read the source. `--check-shared-libs <token>` reads a current finding as JSON on stdin and checks prior Skip decisions against the actual branch, capture and eligible raw source blobs. It returns `reusable:false` when proof is missing or unsafe, including older records without logger-versioned coverage. Final review logging computes shared-code fingerprints and coverage rather than trusting supplied proof. |
|
||||
| `gstack-review-read` | **Review freshness** — emits review records with computed `review_freshness.status` and `reason`: CURRENT, STALE, or UNVERIFIED for diff reviews. `/ship` and `/land-and-deploy` use the same grade; a matching commit alone never certifies a diff review. [Dashboard rules](skills.md#review-readiness-dashboard). |
|
||||
| `gstack-evidence` | **Verification-evidence ledger** — `run --label <lane> -- <cmd>` transparently wraps any test command (the child's exit code always passes through) and records what ran against which working-tree fingerprint (tracked and untracked files that are not gitignored, minus an untracked `.bun/` folder at the repo root, which Bun can create during a run); a run whose files changed while it ran is recorded without a fingerprint and with up to five of the changed paths, which `check` names in its STALE reason. `check` grades each label FRESH/STALE/MISSING with `--expect-cmd`, `--max-age`, and `--allow-paths` binding. /ship and /land-and-deploy cite fresh evidence instead of re-running suites. Per-run logs are 0600, capped at 2MB, pruned after 30 days; the ledger and logs stay machine-local by design. |
|
||||
| `gstack-flaky` | **Flaky-test rerun for any JUnit runner** — `run --junit <file.xml> -- <command>` runs your test command (Playwright, `bun test`, jest, vitest, pytest) and, when at most `--max-reruns` (3) tests failed and the runner made no retries of its own, reruns the whole command once (`--rerun-scope test` opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (`GSTACK_FLAKE_LEDGER_JUNIT`, default `<state root>/flake-ledger-junit.jsonl`). A missing or malformed report, a crash or a killed runner never passes. `ids --junit <file>` prints a report's test ids. Exit 0 pass, 1 a test failed, 2 usage, 3 no usable report. [Guide](flaky.md). |
|
||||
| `gstack-flaky` | **Flaky-test rerun for any JUnit runner** — `run --junit <file.xml> -- <command>` runs your test command (Playwright, `bun test`, jest, vitest, pytest) and, when at most `--max-reruns` (3) tests failed and the runner made no retries of its own, reruns the whole command once (`--rerun-scope test` opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (`GSTACK_FLAKE_LEDGER_JUNIT`, default `<state root>/flake-ledger-junit.jsonl`). A missing or malformed report, a crash or a killed runner never passes. `quarantine add|renew|release|check|report` keeps `.gstack/quarantine.json`: a quarantined test still runs, gets up to 3 attempts of the whole command, passes only on a full green attempt and is never counted as a pass (`run --base <ref>` reads the registry from the base branch). `ids --junit <file>` prints a report's test ids. Exit 0 pass, 1 a test failed, 2 usage, 3 no usable report. [Guide](flaky.md). |
|
||||
| `gstack-pr-wait` | **PR event watcher** — `gstack-pr-wait <pr> [--repo owner/name] [--until green\|ready\|any-change] [--interval 20] [--timeout 60] [--exclude CHECK,...]` polls one pull request through `gh` and prints one stdout line per real event (a check failed with its link, all checks finished, a comment or review from someone other than the author, a review bot finished on the head, the base moved and its new commits touch this PR's files, BEHIND or DIRTY, merged or closed), with progress on stderr, so a monitor wakes the agent once per event instead of on a timer. `green` gates on every check on the head (an advisory bot's review is never waited for, its checks gate like any other), as `gstack-ci-gate` does; a check the base branch requires (rulesets and classic branch protection) is pending until it appears, never green by omission; two identical polls confirm a check set is complete; `--exclude` leaves a named non-required check out of the verdict as `gstack-ci-gate --exclude` does (a required name is refused and gates). Exit 0 when the condition is met, 1 when it needs action or `--timeout` passes (`PR_WAIT_BLOCKED`), 2 on bad input, a second watcher for the same PR or a local error (`PR_WAIT_REFUSED`). State under `<state root>/pr-wait/`, baseline discarded after 6 hours. Review bots are configuration: `gstack-config set pr_wait_gating_bots greptile,macroscope`, `pr_wait_advisory_bots coderabbit` (advisory bots are reported, never waited on). `/land-and-deploy` Step 3 is one call of it. Design ported from tstack's pr-wait (MIT, Theo Browne; NOTICE.md). [Exit codes](troubleshooting.md#gstack-pr-wait). |
|
||||
| `gstack-issue-guard` | **Tracker-text trust envelope** — fetches GitHub issue/PR text (`issue <n>`, `pr-body`, `pr-comments`, or `--stdin`) and wraps it in a labeled envelope so agents treat it as data: injection-shaped lines get labeled even through fullwidth and invisible-character evasion, and forged envelope banners are defused. Every tracker-text ingress in gstack routes through it, enforced by a CI scanner. |
|
||||
| `gstack-ios-qa-daemon` | **iOS QA daemon** — Mac-side broker between an agent and a connected iPhone or iPad over USB CoreDevice. Loopback by default; `--tailnet` opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on `~/.gstack/ios-qa-daemon.pid`. See [docs/howto-ios-testing-with-gstack.md](howto-ios-testing-with-gstack.md). |
|
||||
|
||||
+3
-1
@@ -105,7 +105,9 @@ lives in `projects/<slug>/`. `bin/gstack-slug` picks the slug from the git
|
||||
- Without a remote, the project root's directory name is used.
|
||||
|
||||
A bucket can also hold a user-owned `pregate.json` policy, for a repo that keeps
|
||||
`.gstack/` out of git (see [pregate v1](pregate.md#pregate-v1)).
|
||||
`.gstack/` out of git (see [pregate v1](pregate.md#pregate-v1)), and
|
||||
`pregate/runs/<run id>/`, the scratch each pregate run writes (lane and guard
|
||||
logs, the remote gate's file list); old runs are pruned when a new one starts.
|
||||
|
||||
The first slug gstack resolves in a directory sticks (it is cached under
|
||||
`slug-cache/`), so a project keeps its bucket when it adds a remote later.
|
||||
|
||||
@@ -10,7 +10,7 @@ gstack has two support tiers. Full support gets the whole test suite, paid evals
|
||||
|
||||
## Minimal support
|
||||
|
||||
- **Windows** (Git Bash or MSYS): with a default Git for Windows install, `./setup` installs, and the core skills start without errors. The core skills are `/ship`, `/review`, `/qa-only`, `/investigate` and `/browse` on gstack's own browser. Once the `windows-smoke` CI job lands, it checks this on every pull request (see Windows CI below).
|
||||
- **Windows** (Git Bash or MSYS): with a default Git for Windows install, `./setup` installs, and the core skills start without errors. The core skills are `/ship`, `/review`, `/qa-only`, `/investigate` and `/browse` on gstack's own browser. One required CI job, `windows-smoke`, checks this on every pull request (see Windows CI below).
|
||||
- **Gemini models:** the core workflows run to the end without errors when Gemini is the host agent or a reviewer. We check this by hand or with one cheap smoke run, not with an eval cell.
|
||||
|
||||
That is the whole promise. Other skills, optional parts such as `/cso`'s native helper, non-default shell settings and speed are not covered.
|
||||
@@ -23,8 +23,9 @@ That is the whole promise. Other skills, optional parts such as `/cso`'s native
|
||||
|
||||
## Windows CI
|
||||
|
||||
- **Today:** the six-shard Windows free suite (the `Windows Free Tests` workflow, `windows-free-tests`) runs on every pull request and every push to main, and it is a required check.
|
||||
- **Once `windows-smoke` lands:** `windows-smoke` becomes the required Windows check on pull requests. The six-shard suite then runs nightly on main, or when someone starts it by hand, and never blocks a pull request. A red nightly run gets a fix only when it shows an install, startup or core-workflow break.
|
||||
- `windows-smoke` is the required Windows check on pull requests.
|
||||
- The six-shard Windows free suite (the `Windows Free Tests` workflow, `windows-free-tests`) runs nightly on main, or when someone starts it by hand, and never blocks a pull request.
|
||||
- A red nightly run gets a fix only when it shows an install, startup or core-workflow break.
|
||||
|
||||
## What doesn't change
|
||||
|
||||
|
||||
+27
-8
@@ -570,6 +570,9 @@ the base has moved.
|
||||
**Meaning.** The working tree changed between the pass's `gstack-review-log --start` and
|
||||
its logged row: an edit, commit or merge landed while reviewers were still reading. The
|
||||
reviewers may not have seen that change, so the row's `review_binding.state` is `changed`.
|
||||
Review records fingerprint with `gstack-wtree --review`, which skips untracked root dot-folders
|
||||
that hold no tracked file (never `.github/` or `.gstack/`), so a test lane's scratch folder alone
|
||||
does not move the tree.
|
||||
Step 11.5 cannot bind it and inserts `9 → 10 → 11 → 11.5`. Logging the same results again
|
||||
cannot repair it.
|
||||
|
||||
@@ -2592,12 +2595,24 @@ case with `bun run eval:pass-rates --headroom --case <id>`.
|
||||
<a id="cost-unknown"></a>
|
||||
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
|
||||
|
||||
**Meaning.** PTY and Codex sessions record no billing, so their cost is
|
||||
unknown. So is a trial in a multi-case shard whose result records are named by
|
||||
a display label rather than the case id, because no record can be attributed
|
||||
to it (a one-case shard's records are its case's). Only the known sum is
|
||||
shown. A train verdict's `ATTEMPT:` line reports the same as `cost=$<x> known
|
||||
+ <n> of <m> case(s) cost unknown`, a bare `cost=$<x>` only when every case
|
||||
**Meaning.** Some trials' cost was not captured, so only the known sum is
|
||||
shown. A trial is cost-unknown when it ran a PTY session whose transcript
|
||||
could not be priced (a non-hermetic or resumed launch, the native
|
||||
AskUserQuestion capture, two new transcripts for one cwd, no transcript or
|
||||
usage, or a model with no price row), a
|
||||
Codex session that completed no turn or whose model has no price row, a
|
||||
`claude -p` session that ended without a result, or an aborted Agent SDK
|
||||
session whose streamed usage could not be priced (no usage, or a model with no
|
||||
price row). A PTY or aborted Agent SDK session that does price counts its
|
||||
`streamed_cost_usd` instead. A trial is also unknown when no record and no
|
||||
session can be attributed to it, for example a multi-case shard's
|
||||
display-label record without `cost_case_id` (a one-case shard's records are
|
||||
its case's); a model-free case (`timeline-event-flow`) is a known $0. A panel
|
||||
trial (`behavior`, `judge`) is unknown whenever any of its sessions billed
|
||||
nothing, even one with a priced `streamed_cost_usd`. The rules are in
|
||||
[Failure causes and details](TESTING_INTERNALS.md#failure-causes). A train
|
||||
verdict's `ATTEMPT:` line reports the same as `cost=$<x> known + <n> of <m>
|
||||
case(s) cost unknown`, a bare `cost=$<x>` only when every case
|
||||
in the train record is priced, and no cost at all when none is.
|
||||
|
||||
**Fix.** None needed; this is informational.
|
||||
@@ -3920,7 +3935,10 @@ and the table prints `environment: <kind> — <evidence>`. A test that failed an
|
||||
whose log also names a connection error is **not** this code: the runner's own
|
||||
summary counted a failure, so it is `PREGATE_LANE_FAILED` with `cause:
|
||||
"diff"`; nor is a runner, guard or gate script whose own binary is present but
|
||||
exits 127. `proof: "launch"` is certain (the command could not start, its
|
||||
exits 127. One exception: a Playwright lane where every test failed and
|
||||
Playwright says its browsers are not installed is `missing-binary` with
|
||||
`proof: "output"` (`npx playwright install`); a partial install stays the
|
||||
diff's failure. `proof: "launch"` is certain (the command could not start, its
|
||||
binary is off PATH, the runtime or a gstack tool refused); `proof: "output"`
|
||||
means a rule matched the command's text before any test was counted, which the
|
||||
diff's own output could have written, so the line says `(inferred from output)`
|
||||
@@ -5903,7 +5921,8 @@ body that names a counter the receipt records (`passes=3` against
|
||||
against `gate.passed: 41`) with a different value. The receipt is written from
|
||||
the run; the prose was typed, so a reader of the body would learn a number the
|
||||
run did not produce. The `PROSE:` line names each field with both values. The
|
||||
`## Approvals` table `gstack-approvals` renders is never read as a claim.
|
||||
`## Approvals` table `gstack-approvals` renders, and the `Review-loop exception: …`
|
||||
line the receipt's exception requires, are never read as claims.
|
||||
|
||||
**Fix.** Correct the body to the receipt's number. When the receipt is the
|
||||
stale one (the run was repeated), rerun `/ship` so the receipt is rewritten
|
||||
|
||||
Reference in new issue
Block a user