mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-13 20:07:08 +01:00
v1.93.11.0 stamp: merge main daf661374f into #3872 (land)
This commit is contained in:
commit
964f0cc2e9
184 files changed
+3881
-516
No files matched your search
+114
-14
@@ -651,15 +651,37 @@ this checkout to read it)`.
|
||||
samples, rationale}` entry per failing dimension (since EVAL_POLICY v3 `mean`
|
||||
holds the panel's gating median), each part capped at 200
|
||||
characters. `sessions[]` (at most 32) summarizes the trial's session-ledger
|
||||
rows. `cost_known: false` marks a record or trial whose harness captured no
|
||||
billing: PTY and Codex sessions never bill, and an SDK session bills only with a
|
||||
terminal `total_cost_usd`. It also marks a trial no result record could be
|
||||
attributed to: a record named by the case id or carrying `case_id` is that
|
||||
case's, and a record named by a display label (`/qa fix loop`) is the shard's
|
||||
only case's when the shard ran one case, so a multi-case shard whose records
|
||||
carry display labels leaves those trials cost-unknown rather than guessing
|
||||
which case paid (`junitCensus` in `scripts/lib/paid-report.ts`). An absent
|
||||
field means known. Census and marathon cost lines print the known sum and
|
||||
rows. `cost_known: false` marks a record or trial whose cost is not fully
|
||||
known; an absent field means known. A record carries it when its harness
|
||||
captured no billing (a Codex session that completed no turn or whose model has
|
||||
no price row). A rule case's trial in the JUnit census (`caseCost` in
|
||||
`scripts/lib/paid-report.ts`) costs its records' sum plus what no record
|
||||
carries: each ledger session without a terminal result (`billed: false`) adds
|
||||
its `streamed_cost_usd`, and a trial with no record adds each billed session's
|
||||
own `cost_usd` (Codex). The trial is cost-unknown when any record says
|
||||
`cost_known: false`, when any of those sessions has no priced cost (a PTY
|
||||
session whose transcript does not price, a `claude -p` session that ended
|
||||
without a result, an aborted Agent SDK session with no usage or an unpriced
|
||||
model), or when it has neither records
|
||||
nor sessions. A case in `MODEL_FREE_CASES` (`timeline-event-flow`, which runs
|
||||
no model) with neither is a known $0. A panel trial (`behavior`, `judge`) from
|
||||
an isolated trial shard keeps the older rule (`trialCostKnown` in
|
||||
`test/helpers/eval-store.ts`): known only with at least one record and every
|
||||
session billed, and streamed cost is not added.
|
||||
|
||||
Records are attributed to a case this way: a record named by the case id or
|
||||
carrying `case_id` is that case's, and a record named by a display label
|
||||
(`/qa fix loop`) is the shard's only case's when the shard ran one case. In a
|
||||
multi-case shard, a display-label record whose `runSkillTest` test name is a
|
||||
registered case id carries `cost_case_id` (`recordE2E`), and the census adds
|
||||
its cost to that case only: the failed record, failure class and model still
|
||||
come from the records named by case id, so no verdict changes. A display-label
|
||||
record without `cost_case_id` in a multi-case shard is not attributed rather
|
||||
than guessed, and its case is cost-unknown when nothing else prices it
|
||||
(`junitCensus`). The report's headline `cost $<x>` adds the streamed session
|
||||
cost no collector record carries, so a newly priced trial counts once.
|
||||
|
||||
Census and marathon cost lines print the known sum and
|
||||
`N of M trial(s) cost unknown`, and a train verdict's `ATTEMPT:` line prints
|
||||
`cost=$<known> known + <n> of <m> case(s) cost unknown` when some but not
|
||||
every case in the train record has a cost (a record with no priced case prints
|
||||
@@ -671,13 +693,33 @@ including `runSharedInteractive`, the PTY session and `runRecordedCodexEval`)
|
||||
appends one best-effort line per armed session to
|
||||
`$GSTACK_EVAL_DIR/session-ledger.jsonl` (`test/helpers/session-ledger.ts`):
|
||||
a stable key `<runner>:<label>#n`, the exact `budget_ms` it armed, monotonic
|
||||
`elapsed_ms`, the `end` reason, sanitized evidence, `billed`, and a compact
|
||||
`elapsed_ms`, the `end` reason, sanitized evidence, `billed`, the billed
|
||||
`model`, `cost_usd` and token classes of a Codex session, and a compact
|
||||
liveness summary (longest in-request silence, the last event kind, any tool open
|
||||
at the end). `claude -p` sessions stream partial messages for this summary;
|
||||
transcripts keep their shape and size (full stream diagnostics only with
|
||||
`publicStreamDiagnostics`). A failed ledger write never fails a trial; a trial
|
||||
without ledger rows has no `sessions` and its headroom is `unknown`, never
|
||||
scored. The ledger rides in each slice artifact beside the shard results.
|
||||
scored. An Agent SDK session that ends without a terminal result (aborted,
|
||||
thrown) also records `streamed_cost_usd`: its streamed assistant usage priced
|
||||
by `streamedUsageCostUsd` (`lib/pricing.ts`), each message id once at its
|
||||
largest output, subagent turns included, `<synthetic>` messages skipped,
|
||||
5-minute cache writes at 1.25x and 1-hour writes at 2x the input rate. The
|
||||
field is absent when no message carries usage or any message's model has no
|
||||
price row, so an unpriced session stays cost-unknown, never $0. A PTY session
|
||||
records `streamed_cost_usd` from its transcript the same way. Both PTY ledger
|
||||
writers price `<CLAUDE_CONFIG_DIR>/projects/*/<id>.jsonl` plus
|
||||
`<id>/subagents/*.jsonl` (`test/helpers/pty/transcript-cost.ts`), where the
|
||||
id is the caller's `--session-id`, else the one transcript that appeared
|
||||
during this launch for its cwd. The launcher adds no argument. Pricing never
|
||||
takes the whole directory, because concurrent PTY tests in one process share
|
||||
one hermetic config dir; two new transcripts for one cwd are ambiguous and stay
|
||||
unknown. A non-hermetic or resumed launch, a caller that opts out
|
||||
(`priceTranscript: false`, the native AskUserQuestion capture, which never
|
||||
reads model transcripts), a missing transcript, no usage, an unpriced model or
|
||||
a read error leaves the field absent (cost-unknown), and a pricing failure
|
||||
never fails the test. The row keeps `billed: false`. The ledger rides in each slice
|
||||
artifact beside the shard results.
|
||||
|
||||
**Quarantine** (`CASE_QUARANTINE`, same file). An entry needs: a per-trial rate
|
||||
below 95% over at least 10 post-policy trials of the case's current input
|
||||
@@ -1232,9 +1274,11 @@ observation never removes a command and a missing or empty artifact never erases
|
||||
one; removing a command is a reviewed edit. While the map is in shadow mode it
|
||||
changes no selection, so a PR that changes only the map selects no paid case.
|
||||
|
||||
Dollar figures are lower bounds. Known cost excludes PTY trials, whose cost is
|
||||
unknown: about 39% of observed case runs in the committed map. The shadow block
|
||||
and `test:health` both say so.
|
||||
Dollar figures are lower bounds. Known cost excludes trials whose cost is
|
||||
unknown: about 39% of observed case runs in the committed map, mostly PTY
|
||||
trials recorded before PTY sessions were priced from their transcripts
|
||||
([Failure causes and details](#failure-causes)). The shadow block and
|
||||
`test:health` both say so.
|
||||
|
||||
- **Shadow block:** `prCoverage.shadow` in the paid-plan manifest lists the gate
|
||||
cases the map would select, would drop and would add, a provenance line per
|
||||
@@ -1750,6 +1794,32 @@ lower-case words in path components under `test/fixtures/`; a substring match
|
||||
separate change with its own touchfile updates; the list is here so that change
|
||||
can be scoped.
|
||||
|
||||
## Agent SDK system prompt
|
||||
|
||||
An Agent SDK caller that leaves out `systemPrompt` doesn't get Claude Code's
|
||||
prompt. It gets the SDK's bare stub, and the stub drops the line that tells the
|
||||
model to run `/skill` through the Skill tool. A skill arm then never loads its
|
||||
skill, so a "no difference" result says nothing about the skill (FOUN-32 found
|
||||
this in another crew's null result).
|
||||
|
||||
**Every caller says which prompt it uses.** `runAgentSdkTest` and every direct
|
||||
SDK `query()` call pass the Claude Code preset,
|
||||
`{ type: 'preset', preset: 'claude_code' }` (also exported as
|
||||
`CLAUDE_CODE_PRESET` from `test/helpers/sdk-system-prompt.ts`; an overlay arm
|
||||
adds `append`), or name why not: `{ type: 'sdk-default', reason }` for a stub
|
||||
provider or a connectivity probe, `{ type: 'custom', text, reason }` for an arm
|
||||
that replaces the prompt. The runner refuses a missing, empty, bare-string or
|
||||
unnamed prompt before it starts, and
|
||||
`test/sdk-system-prompt-tripwire.test.ts` scans test/ and scripts/ for the same
|
||||
mistake in free CI. Arms that compare loadouts, the no-gstack arm included,
|
||||
keep the same preset and differ only in what the arm loads.
|
||||
|
||||
**Record it, and treat a null on the stub as a method bug.** Every eval report
|
||||
and results table names the prompt each arm ran under (`claude_code`,
|
||||
`claude_code + append`, or the named reason). A null or negative result from an
|
||||
arm that ran on the SDK default is FIX-METHOD under R48 ("Why null?"): fix the
|
||||
prompt and rerun before reading it as KEEP or CUT-CANDIDATE.
|
||||
|
||||
## No-gstack arm
|
||||
|
||||
Garry, 2026-10-31: "all of our head to head tests have to test no gstack from
|
||||
@@ -1843,6 +1913,36 @@ it as the diagnostic. Candidate folders and git identity name a project, never
|
||||
the arm. The model panel (`test/skill-e2e-model-panel.test.ts`) runs a
|
||||
`<model>#no-gstack` arm beside each model (GST-468, GST-469).
|
||||
|
||||
## Frontier models under test
|
||||
|
||||
Garry, 2026-11-01 (R54): "I think you are spending too much time with Haiku and
|
||||
Sonnet, whereas GBrain and GStack are really designed to be used with frontier
|
||||
models like Opus and 6.1 Sol... just transfer from Haiku to Sonnet, who gives a
|
||||
shit about that, man?"
|
||||
|
||||
**Models under test are frontier models.** Today that's `claude-opus-5-5` and
|
||||
`gpt-6.1-sol`, plus at most one low-cost primary the preregistration names. Haiku
|
||||
and Sonnet are not models under test. They aren't stand-ins for a frontier model,
|
||||
and an eval has no tier-transfer arms (Haiku against Sonnet, say) unless its
|
||||
preregistration says it is asking about that tier.
|
||||
|
||||
**Verdicts need a frontier result.** A result measured only on Haiku or Sonnet
|
||||
can't carry a KEEP, FIX-METHOD or CUT-CANDIDATE verdict under R48. Rerun it on a
|
||||
frontier model first.
|
||||
|
||||
**What the rule leaves alone.** Judges, cross-family review voices and a
|
||||
harness's shipped housekeeping model aren't models under test, so they keep the
|
||||
model they already use.
|
||||
|
||||
**The check.** `gstack-eval-plan dry-run` prints `CHECK: frontier` whenever an arm
|
||||
is Haiku or Sonnet. When every arm is one, the dry run fails unless
|
||||
`--tier-question "<what the preregistration asks about that tier>"` is passed.
|
||||
When only some arms are, it warns. Either way it prints the whole plan's price
|
||||
on `claude-opus-5-5` and `gpt-6.1-sol`. The dry run doesn't enforce that number:
|
||||
compare it with the brief cap before any arm is swapped. `pilot start` refuses a
|
||||
pilot whose every arm is Haiku or Sonnet unless its dry run recorded a tier
|
||||
question.
|
||||
|
||||
## Running evals as an agent: detach
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in
|
||||
|
||||
+3
-1
@@ -434,7 +434,9 @@ A scheduled automation starts one lane a day with the
|
||||
runs' flake-records artifacts and the free suites' retried cases, writes one ledger line
|
||||
per event, groups by case, and fixes the worst repeat first with a PR that shows the
|
||||
failure red before green. It never reruns a gate and never touches a threshold, trial
|
||||
count or case kind. On Capy, create it with the automation tool, scheduled in the owner's
|
||||
count or case kind. Where a repository runs `gstack-flaky`, the lane also reads
|
||||
`gstack-flaky quarantine report --ledger <downloaded ledgers>` and fixes expiring quarantine
|
||||
entries before anything else ([flaky.md](flaky.md#the-daily-flaky-pass)). On Capy, create it with the automation tool, scheduled in the owner's
|
||||
timezone, running as a thread in the crew so its report lands in `reports/`.
|
||||
|
||||
The lane's side of the same rule: in CI the gate reruns a case that failed on its own
|
||||
|
||||
@@ -89,6 +89,7 @@ After weighing Codex's argument, we chose to roll back CEO EXPANSION and ship an
|
||||
"sample_size": 47,
|
||||
"signal_events": 19,
|
||||
"contributing_events": {"scope_appetite": 9, "risk_tolerance": 6, "...": "..."},
|
||||
"hinted_events": 3,
|
||||
"diversity": {
|
||||
"skills_covered": 5,
|
||||
"question_ids_covered": 14,
|
||||
@@ -105,7 +106,7 @@ After weighing Codex's argument, we chose to roll back CEO EXPANSION and ship an
|
||||
}
|
||||
```
|
||||
|
||||
`signal_events` counts the logged answers that moved any dimension, and `contributing_events` counts, per dimension, the logged answers that moved it (an answer that writes two deltas to one dimension counts once), so a dimension with no evidence (0) is visible beside one whose evidence balances out; both print 0.5 (#2623). Each value is the sigmoid of its accumulated deltas, and each UTC day's sum per dimension (by the row's `ts`) is clamped to the largest single delta in the signal map (currently ±0.06), so one burst of answers can't pin a dimension; `--trace <dim>` lists every contributing answer, each clamped day and the total. The bound keys on the day, not on `session_id`: the PostToolUse hook and the agent log different ids for the same review (the host's conversation id and the preamble's run id), and an agent can leave the `SESSION_ID` placeholder unsubstituted, so no logged id reliably marks one review. The cost is that several reviews on one day count as one. `contributing_events` still counts an answer logged by both writers twice. Only an ISO 8601 date or date-time `ts` counts as dated (a time without a zone is read as UTC), every other row shares one undated bucket, and `diversity.days_span` counts the same UTC days. `--check-mismatch`, `--gap`, the stored `gap` and /plan-tune's profile view compare a dimension only when its `contributing_events` is at least 3; `--gap` lists the others under `low_evidence` and the stored `gap` omits them (#3436).
|
||||
`signal_events` counts the logged answers that moved any dimension, and `contributing_events` counts, per dimension, the logged answers that moved it (an answer that writes two deltas to one dimension counts once), so a dimension with no evidence (0) is visible beside one whose evidence balances out; both print 0.5 (#2623). Each value is the sigmoid of its accumulated deltas, and each UTC day's sum per dimension (by the row's `ts`) is clamped to the largest single delta in the signal map (currently ±0.06), so one burst of answers can't pin a dimension; `--trace <dim>` lists every contributing answer, each clamped day and the total. The bound keys on the day, not on `session_id`: the PostToolUse hook and the agent log different ids for the same review (the host's conversation id and the preamble's run id), and an agent can leave the `SESSION_ID` placeholder unsubstituted, so no logged id reliably marks one review. The cost is that several reviews on one day count as one. `contributing_events` still counts an answer logged by both writers twice. Only an ISO 8601 date or date-time `ts` counts as dated (a time without a zone is read as UTC), every other row shares one undated bucket, and `diversity.days_span` counts the same UTC days. `--check-mismatch`, `--gap`, the stored `gap` and /plan-tune's profile view compare a dimension only when its `contributing_events` is at least 3; `--gap` lists the others under `low_evidence` and the stored `gap` omits them (#3436). An unregistered `{skill}-{slug}` id moves a dimension only through a signal hint (#3442): the asked question carries `<gstack-qid:{id};signal={signal_key}>` and the log row a matching `signal_key`, chosen from `gstack-question-preference --signal-keys`. Derive uses the hint only when the id has no registry entry, the key is in the signal map and the answer matches one of that signal's option keys; `hinted_events` counts those signal events, and `--trace` marks their rows `(hinted)`.
|
||||
|
||||
**Diversity check** (Codex #13): `inferred` is considered "enough data" only when `sample_size >= 20 AND skills_covered >= 3 AND question_ids_covered >= 8 AND days_span >= 7`. Below this, `/plan-tune profile` shows "not enough observed data yet" instead of a potentially-misleading inferred value.
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@ What incomplete data means:
|
||||
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
|
||||
| cause `unrecorded` | the trial was written before `failure_cause` existed |
|
||||
| cause `unknown` | the trial failed with no evidence for any other cause |
|
||||
| `cost unknown` | the harness (PTY, Codex) records no billing, or the trial sits in a multi-case shard whose records are named by display label and cannot be attributed to it; only the known sum is shown |
|
||||
| `cost unknown` | a session's cost was not captured (a PTY session without a priced transcript, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown |
|
||||
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
|
||||
|
||||
## 4. Replay and repair (free)
|
||||
|
||||
+117
-8
@@ -6,7 +6,9 @@ rerun is a **flaky pass**: the job stays green and the flake is recorded on a le
|
||||
the daily flaky pass can fix it at the source. A test that fails twice is red.
|
||||
This is CREW rule 16 ("run the failed test again and note that it is flaky")
|
||||
for any runner that writes JUnit: Playwright, `bun test`, jest, vitest and
|
||||
pytest. gstack's own free suite keeps its file-level retry
|
||||
pytest. A test that keeps flaking while it is fixed can be
|
||||
[quarantined](#quarantine): it still runs, with up to 3 attempts, and never counts
|
||||
as a pass. gstack's own free suite keeps its file-level retry
|
||||
(`GSTACK_FREE_RETRY_FLAKY`, [Flake ledger](TESTING_INTERNALS.md#flake-ledger)).
|
||||
|
||||
## Get started in GitHub Actions
|
||||
@@ -74,6 +76,104 @@ It cannot see a coverage threshold configured without console output (for
|
||||
example `coverageReporter = ["lcov"]` in `bunfig.toml`), which is why this
|
||||
scope is opt-in.
|
||||
|
||||
## Quarantine
|
||||
|
||||
A test that keeps flaking while someone fixes it can be **quarantined** in
|
||||
`.gstack/quarantine.json` at the repository root. Quarantine is a larger
|
||||
attempt budget, never a skip: the test still runs on every run, is listed as
|
||||
quarantined in the output, and never counts as a pass.
|
||||
|
||||
**How a quarantined test is judged.** The verdict rule from
|
||||
[Rerun scope](#rerun-scope) does not change: an attempt counts only when the
|
||||
runner itself reports a full green run. When every test an attempt failed is a
|
||||
live quarantined test, and nothing else is wrong with that attempt (no kill, no
|
||||
failure outside the report, no first-run test missing), the whole command may
|
||||
run again, up to 3 attempts in total. The run is green only when one of those
|
||||
attempts is fully green. A quarantined test that fails all 3 is a **hard
|
||||
break**, and the run is red. A test that is not quarantined keeps its one
|
||||
rerun. A red attempt never turns green because its only reported failure is a
|
||||
quarantined test, because the runner's exit status can come from things the
|
||||
report does not show (a coverage threshold, a fail-on-flaky policy, a global
|
||||
setup). With `--rerun-scope test` the quarantined test gets the same 3
|
||||
attempts on the targeted path, with the same fallback to the whole command.
|
||||
When the runner retries on its own (Playwright `retries`), its attempts are
|
||||
the budget and nothing is rerun again.
|
||||
|
||||
**Missing results.** A quarantined id missing from a run that included its
|
||||
file (and, for Playwright, its project) is red. A run that did not include the
|
||||
file, such as a shard or a diff selection, is not.
|
||||
|
||||
**Read from the base branch.** `run --base <ref>` reads the registry from
|
||||
that commit instead of the working tree. Pull-request CI passes its target
|
||||
branch, so a change cannot quarantine the test it breaks. `check --base`
|
||||
compares against the merge base, so check out full history for it:
|
||||
|
||||
```yaml
|
||||
- uses: actions/checkout@v4
|
||||
with: { fetch-depth: 0 }
|
||||
- name: Quarantine registry
|
||||
run: '"$RUNNER_TEMP/gstack/bin/gstack-flaky" quarantine check --base "origin/${{ github.base_ref }}"'
|
||||
- name: E2E
|
||||
run: |
|
||||
"$RUNNER_TEMP/gstack/bin/gstack-flaky" run --base "origin/${{ github.base_ref }}" --junit results/junit.xml -- npx playwright test --reporter=line,junit
|
||||
```
|
||||
|
||||
### Commands
|
||||
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
| `gstack-flaky quarantine add <id> --owner <who> --ledger <url> --class <class> --reason <text> --evidence <url>… --junit <file> [--days <n>]` | Adds an entry that expires in `--days` (default 7, at most 14). `--junit` counts the suite for the cap and confirms the id exists; `gstack-flaky ids` prints the ids. |
|
||||
| `gstack-flaky quarantine renew <id> --owner <who> --evidence <url> [--days <n>]` | One renewal of up to 7 days from the later of now and the current expiry, with evidence not already on the entry and the owner named again. A second renewal is refused. |
|
||||
| `gstack-flaky quarantine release <id> --reason <text> --pr <url>` | Moves the entry to `released` with the date, the reason and the PR. |
|
||||
| `gstack-flaky quarantine check [--base <ref>] [--junit <file>]` | Validates every entry and the cap; with `--base`, also what this change did since the merge base. Exits 1 with one `QUARANTINE_REFUSED:` line per problem. |
|
||||
| `gstack-flaky quarantine report [--ledger <file\|dir>…] [--base <ref>]` | Lists expired entries, then live entries soonest-expiring first with days left. With downloaded ledgers it adds recent outcomes and marks entries with 10 consecutive clean passes `READY_TO_RELEASE`; without `--ledger` it says the exit rule was not checked. |
|
||||
|
||||
### Rules
|
||||
|
||||
- **Every entry is owned and expires.** An entry holds `owner`, `ledger`
|
||||
(the ledger line or issue tracking the fix), `failureClass`, `reason`,
|
||||
`evidence`, `enteredAt`, `expiresAt`, `renewals` and `suiteTests` (the
|
||||
suite size `add` counted). It lasts 7 days by default and at most 14 from
|
||||
entry, plus one renewal of at most 7 days.
|
||||
- **Expired means not quarantined.** An expired entry stops applying without
|
||||
an edit: `run` treats the test as an ordinary test and prints
|
||||
`FLAKY_QUARANTINE_EXPIRED` with the renew and release commands. An entry
|
||||
that breaks a rule (for example one that lasts 30 days) is ignored the same
|
||||
way, with `FLAKY_QUARANTINE_IGNORED`.
|
||||
- **Cap.** Live entries are capped at 10% of the suite's tests, at least 1,
|
||||
and 5 for a suite under 50 tests. `add` refuses past the cap; `check`
|
||||
enforces it against `--junit` when given, else against the smallest
|
||||
`suiteTests` on a live entry.
|
||||
- **Classes.** `detector`, `harness`, `timing` and `model-latency`.
|
||||
`product` is refused (a product defect is fixed, not quarantined), and so is
|
||||
`infra` (an infrastructure red is rerun or fixed as infrastructure).
|
||||
- **`check --base` refuses** an entry added in the same change that edits its
|
||||
test's file, an entry that disappears without a release record, a release
|
||||
record that was removed, and a changed `enteredAt` or rewritten renewal
|
||||
history.
|
||||
- **Exit rule.** 10 consecutive clean passes (`quarantined-pass` with no
|
||||
failed attempt) on the ledgers since the entry was added. Release the entry
|
||||
with the fix PR.
|
||||
- **Separate from `CASE_QUARANTINE`.** gstack's own paid-eval quarantine
|
||||
(`CASE_QUARANTINE` and `EVAL_POLICY`,
|
||||
[TESTING_INTERNALS.md](TESTING_INTERNALS.md)) is a different registry with
|
||||
its own rules for eval cases. The two share no schema, and `gstack-flaky`
|
||||
neither reads nor writes it.
|
||||
|
||||
### The daily flaky pass
|
||||
|
||||
Download the ledger artifacts of the last week's runs and read them with
|
||||
`report`; fix expired and `EXPIRING` entries first (fix the test, or renew
|
||||
once with new evidence), release `READY_TO_RELEASE` entries, then fix the
|
||||
worst repeat on the ledger:
|
||||
|
||||
```bash
|
||||
for id in $(gh run list --workflow e2e.yml --limit 50 --json databaseId -q '.[].databaseId'); do
|
||||
gh run download "$id" --pattern 'flaky-junit-ledger-*' --dir "ledgers/$id" || true
|
||||
done
|
||||
gstack-flaky quarantine report --ledger ledgers
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **One rerun, small failures only.** A failed test gets one rerun when the
|
||||
@@ -101,8 +201,9 @@ scope is opt-in.
|
||||
- **Ids.** A test is `<file>::<project>::<describe > title>`, with the project
|
||||
empty outside Playwright, so a test in two browser projects is two tests.
|
||||
`gstack-flaky ids --junit <file>` prints them.
|
||||
- **Ledger.** One JSON line per flaky pass (`kind: flaky-pass`) and per test
|
||||
that failed its rerun (`rerun-fail`), with the tree, commit, branch, run URL,
|
||||
- **Ledger.** One JSON line per flaky pass (`kind: flaky-pass`), per test
|
||||
that failed its rerun (`rerun-fail`) and per quarantined test on every run
|
||||
(`quarantined-pass`, `quarantined-fail`, `hard-break`), with the tree, commit, branch, run URL,
|
||||
attempts, the rerun's scope (`test` or `command`) and the first lines of
|
||||
the error. Ids, paths and failure text go through gstack's redaction engine
|
||||
first; when the scan cannot run, they are withheld. The path is `--ledger`,
|
||||
@@ -113,15 +214,17 @@ scope is opt-in.
|
||||
|
||||
## Output and exit codes
|
||||
|
||||
One `FLAKY: <flaky-pass|fail|rerun-fail> <id> attempts=<…> (<reason>)` line per
|
||||
test that failed at least once, `FLAKY_RERUN: scope=command (<why>)` when the
|
||||
rerun repeated the whole command, then
|
||||
`FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>`.
|
||||
One `FLAKY: <flaky-pass|fail|rerun-fail|quarantined-pass|quarantined-fail|hard-break> <id> attempts=<…> (<reason>)`
|
||||
line per test that failed at least once and per quarantined test,
|
||||
`FLAKY_RERUN: scope=command (<why>)` when the rerun repeated the whole command, then
|
||||
`FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> quarantined=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>`.
|
||||
`failed` counts every outcome but `flaky-pass` and `quarantined-pass`; `quarantined`
|
||||
counts the quarantined tests' outcomes, which are never counted as passes.
|
||||
|
||||
| Exit | Meaning |
|
||||
|---|---|
|
||||
| 0 | Every test passed, flaky passes included |
|
||||
| 1 | A test failed (after its rerun, or with no rerun), including `FLAKY_UNREPORTED_FAILURES` |
|
||||
| 1 | A test failed (after its rerun, or with no rerun), including `FLAKY_UNREPORTED_FAILURES`, a quarantined test's hard break or missing result, and a refused `quarantine` command |
|
||||
| 2 | Usage error, including a `--junit` path that is not a `.xml` file, or an existing file there that is not a readable JUnit report |
|
||||
| 3 | No usable report, a crash, or a killed runner (a code below on stderr) |
|
||||
|
||||
@@ -135,3 +238,9 @@ rerun repeated the whole command, then
|
||||
| `FLAKY_COMMAND_NOT_STARTED` | 3 | The command could not be started | Check the argv after `--` |
|
||||
| `FLAKY_UNREPORTED_FAILURES` | 1 | The runner counted failures or errors its JUnit omits | Fix the load error or the error outside any test: a real red |
|
||||
| `FLAKY_LEDGER_WRITE_FAILED` | — | The ledger could not be appended (warning) | Check the path; the verdict stands |
|
||||
| `FLAKY_QUARANTINE_INVALID` | 3 | `.gstack/quarantine.json` is not valid JSON or not a schema 1 registry | Fix the file; `quarantine check` names the problem |
|
||||
| `FLAKY_QUARANTINE_BASE` | 3 | `--base` does not resolve to a commit (exit 2 for a value starting with `-`), or `check` finds no merge base | Fetch the base branch; `check --base` needs `fetch-depth: 0` |
|
||||
| `FLAKY_QUARANTINE_REFUSED` | 1 | `add`, `renew` or `release` broke a rule | Read the reason; [Quarantine](#quarantine) lists the rules |
|
||||
| `FLAKY_QUARANTINE_USAGE` | 2 | An unknown `quarantine` command or option, or a missing value | `gstack-flaky quarantine --help` |
|
||||
| `FLAKY_QUARANTINE_EXPIRED` | — | Notice: an expired entry no longer applies | Fix the test, renew once, or release; the line prints both commands |
|
||||
| `FLAKY_QUARANTINE_IGNORED` | — | Notice: an entry breaks a rule and does not apply | Fix the entry; `quarantine check` names the rule |
|
||||
+12
-3
@@ -74,7 +74,7 @@ the byte-pinned `/ship` renders in `test/fixtures/golden/`).
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `runner` | the local test command with `{files}` (gstack: `bun test {files}`); unset, it is detected (bun, vitest, jest, pytest, `node --test`, and `flutter test`/`dart test` per Dart package root, through fvm when that package or an ancestor pins it with `.fvmrc` or `.fvm/fvm_config.json` and fvm is on PATH; `cargo test -p <name>` per Rust crate, run from the root of the workspace the crate is a member of (Cargo's `members` globs, `exclude` and `package.workspace`), else the crate directory; a JS/TS test whose nearest package.json is a nested package declaring vitest or jest (a dependency or its `test` script) runs from that package, with package-relative paths, through the package manager its `packageManager` field or the nearest lockfile up to the repo root names: `pnpm exec`, `yarn`, `bun x`, else `npx`; every other JS/TS test keeps the root detection); nothing detected, a Dart SDK, cargo or package manager missing from PATH, a `.dart` file with no `pubspec.yaml` above it, a `.rs` file outside any crate, or one in a crate whose name Cargo would reject (anything but letters, digits, `_` and `-`), is `incomplete` (the runnable groups still run, so a real failure still fails) |
|
||||
| `runner` | the local test command with `{files}` (gstack: `bun test --timeout=30000 {files}`, the 30 s per-test timeout CI uses); unset, it is detected (bun, vitest, jest, pytest, `node --test`, and `flutter test`/`dart test` per Dart package root, through fvm when that package or an ancestor pins it with `.fvmrc` or `.fvm/fvm_config.json` and fvm is on PATH; `cargo test -p <name>` per Rust crate, run from the root of the workspace the crate is a member of (Cargo's `members` globs, `exclude` and `package.workspace`), else the crate directory; a JS/TS test whose nearest package.json is a nested package declaring vitest or jest (a dependency or its `test` script) runs from that package, with package-relative paths, through the package manager its `packageManager` field or the nearest lockfile up to the repo root names: `pnpm exec`, `yarn`, `bun x`, else `npx`; every other JS/TS test keeps the root detection); nothing detected, a Dart SDK, cargo or package manager missing from PATH, a `.dart` file with no `pubspec.yaml` above it, a `.rs` file outside any crate, or one in a crate whose name Cargo would reject (anything but letters, digits, `_` and `-`), is `incomplete` (the runnable groups still run, so a real failure still fails) |
|
||||
| `dependencies` | glob → test files for subprocess targets, runtime reads and generated inputs the import graph cannot see; `[]` declares "no lane needed" (docs, licences). A `[]` that matches a deleted test does not skip it: the deleted test still runs its owning suite, and a file naming it that `[]` covers (a durations list) counts as covered. With this key present, even empty, the policy [declares lanes](#lane-requirements) |
|
||||
| `empty_lanes` | extra globs for the docs/no-tests empty lane. Any Markdown file already qualifies with no pin file when no test depends on it (the same import graph as other files: a quoted path, a `join()` of path segments, an import, or a `dependencies` entry). reStructuredText and AsciiDoc count as Markdown here. README, CHANGELOG, CONTRIBUTING, LICENSE, AGENTS and CLAUDE qualify even without `.md`, and so does licence text under any vendored name (`roboto-LICENSE.txt`, `LICENSE-MIT`, `licenses/*.txt`). Source code and files under a test directory never qualify. A test that reads the file still selects that test |
|
||||
| `selection` | `workflow.yml/job` → a command printing the lane's file list, for matrices a planner computes (gstack: `scripts/test-free-shards.ts --list`) |
|
||||
@@ -762,7 +762,15 @@ workflow file is the lane's own `<workflow.yml>`. A run of another workflow
|
||||
refuses with `REUSE: refused reason=workflow_mismatch — run <id> attempt <n>
|
||||
ran .github/workflows/<other>.yml, not .github/workflows/<lane's>.yml`; a
|
||||
run gh cannot read refuses with `api_error`, and a URL that is not an Actions
|
||||
run URL or id on this repository with `run_ref_invalid`. The run must also
|
||||
run URL or id on this repository with `run_ref_invalid`. A reusable-workflow
|
||||
lane (one whose file another workflow calls through `workflow_call`) also clears from a run
|
||||
of a caller: a workflow in the branch's own `.github/workflows/` with a job
|
||||
that says `uses: ./.github/workflows/<lane's>.yml` (GST-971), read at
|
||||
pregate.json's recorded tree, as long as the lane's own job succeeded in the
|
||||
run under that calling job (`<calling job> / <lane job>`); else it refuses
|
||||
with `caller_job_skipped`. Fix lines only suggest `gh workflow run` for
|
||||
a file with a `workflow_dispatch` trigger.
|
||||
The run must also
|
||||
have finished green (GSTA-203 PR 8): a run that is still queued or running
|
||||
refuses with `run_in_progress`, and one that concluded anything but `success`
|
||||
(`failure`, `cancelled`, `timed_out`, …) with `run_not_success`; each fix line
|
||||
@@ -770,7 +778,8 @@ says to rerun the workflow and clear with the green run. The check reads the
|
||||
run's newest attempt, so a `/job/<id>` or `/attempts/<n>` link is judged by
|
||||
the whole run, not that job or attempt, and the lane records
|
||||
`https://github.com/<repo>/actions/runs/<id>/attempts/<n>` for the attempt it
|
||||
judged rather than the link you passed. The job, tree and age are not
|
||||
judged rather than the link you passed. The job (apart from a caller run's
|
||||
`<calling job> / <lane job>` above), tree and age are not
|
||||
checked there; that is what a run profile adds. `test_backend/<backend>` lanes and `--bundle-lane` are
|
||||
unchanged. When the base config or the GitHub origin cannot be read, `--run`
|
||||
fails, and so do `--run-url` and `--bundle-lane` unless the base was read and
|
||||
|
||||
+26
-7
@@ -98,9 +98,19 @@ incomplete coverage, which blocks until the reviewer is rerun.
|
||||
finishes with its open notes listed in the PR body and receipt, or stops for
|
||||
the owner when a blocking finding is open, or was fixed in round 2 and is
|
||||
unverified, or a round-2 fix edited a gate-class path, naming it (path,
|
||||
severity, problem). A third round is the owner's
|
||||
severity, problem). The cap limits fix rounds, not Step 11's voices: on gate
|
||||
class a stop waits for Step 11's adversarial, outside and targeted voices
|
||||
(`NEXT: voices`), so gate code never reaches the owner's "ship with open
|
||||
notes" without them (GST-967). A third round is the owner's
|
||||
grant of one scoped fix-check (recorded in `review_loop_exception`), never the
|
||||
loop's. The cap approves nothing.
|
||||
loop's: once `gstack-approvals add` records the grant (its ask or scope
|
||||
naming `review_loop_grant` and the stop head's sha),
|
||||
`gstack-review-plan --grant <approval id> --range-base <round 2's PASS_HEAD>`
|
||||
plans that fix-check in place of the stop after a round-2 full pass with a
|
||||
fixed blocking finding or a gate-class edit, and refuses any other row,
|
||||
including a grant for a stop head before `--range-base` (an earlier /ship).
|
||||
A fix-check never earns another, so the grant cannot repeat, and a stop after
|
||||
a round-2 fix-check still goes to the owner. The cap approves nothing.
|
||||
|
||||
<a id="transitions"></a>
|
||||
### Transitions
|
||||
@@ -122,12 +132,19 @@ incomplete coverage, which blocks until the reviewer is rerun.
|
||||
| a full pass that leaves a blocking finding open (fixed or not) | stop: name it for the owner |
|
||||
| a fix-check that found nothing blocking | done |
|
||||
| a fix-check that found a blocking finding | stop: name it for the owner |
|
||||
| a stop (any row above) on gate class before Step 11's voices ran | Step 11's voices (adversarial, outside, targeted), then ask again with `--adversarial-ran` |
|
||||
| a round-2 full pass that stops for a fixed blocking finding or a gate-class edit, with the owner's recorded grant (`--grant <approval id>`) | one scoped fix-check (round 3), after Step 11 on gate class |
|
||||
|
||||
A **fix-check** is one reviewer reading the touched hunks, the fix list, the
|
||||
test results and the callers of every changed symbol, at full effort. It is
|
||||
logged as a `review-fix-check` row, so it never counts as a pass. No army, no
|
||||
Red Team and no outside voice run on it.
|
||||
|
||||
`NEXT: voices` runs Step 11 as `done` does, from the last full pass's
|
||||
manifest; /ship then asks the plan again with `--adversarial-ran` (a blocking
|
||||
finding the voices queued counts as `--blocking-open`, never a new Step 9), and
|
||||
the stop or the granted fix-check it names stands.
|
||||
|
||||
<a id="scoped-pass"></a>
|
||||
A **scoped pass** is pass 2 after a blocking fix when pass 1 and the diff both
|
||||
ran below full depth (no gate class, no depth error, no `--depth-floor full`).
|
||||
@@ -384,10 +401,11 @@ block without new floor words. The reader is read-only, and `NO FINDINGS` is a
|
||||
result. `gstack-review-dispatch render --role targeted --classes <classes>`
|
||||
renders its brief; `verify --targeted-out <file>` writes the record that
|
||||
`gstack-ship-receipt write --review-targeted @<file>` puts in the receipt.
|
||||
`gstack-ship-queue ready` and `train build` print
|
||||
`gstack-ship-queue ready`, `train build` and `land` print
|
||||
`REVIEW_TARGETED: required=<classes|none> receipt=<classes|none|legacy>
|
||||
verdict=ok|missing` and refuse nothing yet; enforcement comes in a later
|
||||
release.
|
||||
verdict=ok|missing|failed` and refuse
|
||||
[`QUEUE_RECEIPT_TARGETED_MISSING`](troubleshooting.md#queue-receipt-targeted-missing)
|
||||
when a required class has no done pass covering it.
|
||||
|
||||
Over the last 100 non-release commits on main (8663c3d2..8be7f213), money hit
|
||||
21, gate 18, auth 3 and secret 8. The gate class first used the gate paths,
|
||||
@@ -539,10 +557,11 @@ ones, and the arm lost blockers. `bin/gstack-review-dispatch` closes both ends:
|
||||
3. `collect` hashes each reader's raw output and appends its findings to
|
||||
`merged.jsonl` with the label as written (a JSON line's `severity`, or the most
|
||||
severe of a prose line's Critical, High, Medium, Low, P0-P3, FIXABLE or INVESTIGATE
|
||||
wherever they stand, headings and table rows included; a top-level list item under a
|
||||
wherever they stand, headings and table rows included, except that a line with its own `Severity` field takes that
|
||||
field's label (any letter case; a field stating more than one label, "High/Critical", counts as none), so a severity word in the finding's text does not raise it, and a label leading the line stays its floor (GST-976); a top-level list item under a
|
||||
label-only heading takes the stronger of the two). A `Severity:` or `Blocks:` line
|
||||
continues the finding above it until a list item, heading, bold paragraph opener or a
|
||||
second line for the same field starts another: `Severity` raises its label,
|
||||
second line for the same field starts another: `Severity` sets its label (a label leading the finding stays its floor),
|
||||
`Blocks: yes` makes it CRITICAL and `Blocks: no` never lowers it; an inline
|
||||
`Blocks: yes` is CRITICAL too, and with no finding above it is one. The native
|
||||
adversarial voice's FIXABLE collects as CRITICAL (normalizer v2, GSTA-167). An unnumbered, unlabeled bullet with no file:line
|
||||
|
||||
+19
-9
@@ -246,8 +246,10 @@ result is not a current pass, and (`QUEUE_RECEIPT_MISSING`,
|
||||
`QUEUE_RECEIPT_STALE`) when the PR body carries no /ship receipt for that
|
||||
head: /ship runs before every READY ([details](#ship-before-ready)). It also
|
||||
refuses `QUEUE_RECEIPT_DEPTH_RAISED` when the receipt's review depth is below
|
||||
what the head needs and `QUEUE_RECEIPT_EXCEPTION_INVALID` when an owner
|
||||
exception the receipt records does not stand ([the ship receipt](#ship-receipt)).
|
||||
what the head needs, `QUEUE_RECEIPT_TARGETED_MISSING` when a targeted class
|
||||
the head needs has no done targeted pass in the receipt, and
|
||||
`QUEUE_RECEIPT_EXCEPTION_INVALID` when an owner exception the receipt records
|
||||
does not stand ([the ship receipt](#ship-receipt)).
|
||||
Send the block as printed.
|
||||
|
||||
**2. Lane behind another PR: gate ahead.** Gate the tree the queue will have
|
||||
@@ -683,8 +685,10 @@ slot. The queue enforces this at every step from the PR body alone:
|
||||
head being READY'd (or its `pr` is another PR); the informational
|
||||
`SHIP_RECEIPT:` line still prints. It refuses `QUEUE_RECEIPT_DEPTH_RAISED`
|
||||
(with a `REVIEW_DEPTH:` line) when the receipt records a review depth below
|
||||
what the head needs, and `QUEUE_RECEIPT_EXCEPTION_INVALID` when an owner
|
||||
exception it records does not stand ([the ship receipt](#ship-receipt)).
|
||||
what the head needs, `QUEUE_RECEIPT_TARGETED_MISSING` (with a
|
||||
`REVIEW_TARGETED:` line) when a targeted class the head needs has no done
|
||||
targeted pass in the receipt, and `QUEUE_RECEIPT_EXCEPTION_INVALID` when an
|
||||
owner exception it records does not stand ([the ship receipt](#ship-receipt)).
|
||||
There is no flag around it.
|
||||
- `train build` refuses a pin whose PR has no current receipt with the same
|
||||
codes, naming the PR, before anything is written, pushed or dispatched,
|
||||
@@ -803,9 +807,10 @@ its record. Refusals, all before any spend: `QUEUE_TRAIN_UNTRUSTED` (a fork
|
||||
head), `QUEUE_TRAIN_MOVED` (a head moved from its pin),
|
||||
`QUEUE_RECEIPT_MISSING` and `QUEUE_RECEIPT_STALE` (the PR body has no /ship
|
||||
receipt for the pin; see [/ship before READY](#ship-before-ready)),
|
||||
`QUEUE_RECEIPT_DEPTH_RAISED` and `QUEUE_RECEIPT_EXCEPTION_INVALID` (the
|
||||
receipt's review depth is below what the pin needs, or an owner exception it
|
||||
records does not stand; see [the ship receipt](#ship-receipt)),
|
||||
`QUEUE_RECEIPT_DEPTH_RAISED`, `QUEUE_RECEIPT_TARGETED_MISSING` and
|
||||
`QUEUE_RECEIPT_EXCEPTION_INVALID` (the receipt's review depth is below what
|
||||
the pin needs, a targeted class the pin needs has no done targeted pass, or an
|
||||
owner exception it records does not stand; see [the ship receipt](#ship-receipt)),
|
||||
`QUEUE_NOT_READY` (no READY receipt at that head), `QUEUE_TRAIN_CONFLICT` (the printed
|
||||
`QUEUE_TRAIN_CONFLICT: train <id> stopped at #<pr> (conflicts: <files>) after
|
||||
merging #<a>,#<b>; nothing dispatched, no branch pushed; fix: …` line names
|
||||
@@ -1222,7 +1227,12 @@ block `gstack-approvals` renders (its heading, the Date | Asked | Exact words
|
||||
table and the `Source: gstack-approvals record` line) quotes the owner's
|
||||
words, not this PR's numbers, so it is skipped: "one Step 11 pass" in that table
|
||||
is not a `pass` claim. Text you write under the heading after that block, and
|
||||
a `## Approvals` example inside a code fence, are still checked. Data PRs (evals,
|
||||
a `## Approvals` example inside a code fence, are still checked. The
|
||||
`Review-loop exception: …` line the receipt's own exception requires is
|
||||
skipped the same way (GST-977): it quotes the owner's reason, so "outside 1 and
|
||||
2 skip detector" is not a `2 skip` claim. Only a line that matches the rendered
|
||||
line exactly is skipped, inside a code fence too, as the receipt check accepts
|
||||
a fenced copy; the same words inside a longer line are still checked. Data PRs (evals,
|
||||
measurements, sweeps) run it before READY so the body's numbers are the run's.
|
||||
|
||||
<a id="review-scope"></a>
|
||||
@@ -1524,7 +1534,7 @@ Stages, each journaled in `.gstack/tmp/restamp-journal.json`:
|
||||
1. **version**: `gstack-version-bump write --version <X>` in the staging worktree (VERSION, the manifest, npm lockfiles); skipped when `restamp_command` owns the version. The manifest gets the npm-valid three-part form, except when the policy lists it in `stamp_paths` or `stamp_files` and its copy at the base holds four parts (gbrain's convention): then it keeps the full version (`--manifest-form full`) and the stage detail says so.
|
||||
2. **release_tool**: the policy command, under the contract below.
|
||||
3. **changelog**: the PR's `[Unreleased]` or old-version section moves to the top under `<X>`, above its predecessor's entry. A duplicate heading the PR introduces (`CHANGELOG_DUPLICATE_HEADING`: the headings are counted against the base's CHANGELOG at `origin/<base>` and at the fork point, so a version the base already heads twice passes as long as the PR adds no further copy) and a top entry that is not `<X>` (`CHANGELOG_TOP_MISMATCH`) are errors; the section being re-headed itself may exist only once.
|
||||
4. **stamp_paths**: exact `(was X)`, `since: X` and `vX` rewrites, plus every `stamp_patterns` template, within `stamp_paths` and `stamp_files`. A quote around X in the first two forms is part of the form and is kept as written: `since: 'X'`, `since: "X"`, `(was 'X')` and `(was "X")` move like the bare ones (the shape of a quoted source row such as gbrain's `{ since: '0.60.468.0', … }`), an opened quote must close right after X (`since: '1.2.3-beta'` and `since: '1.2.3"` are not the form, in the writer and in the tree receipt alike), and `vX` already accepts a quote before the `v`. Which X moves is decided by provenance. The branch's own tentative stamp (its VERSION when it moved past base) and `--was <X>`, when given, are each rewritten on every line of every stamp file. Beyond that, only lines the branch itself added are read, by their position in the file (`git diff --unified=0` from the fork point), so a base line with the same text, or a CRLF checkout whose diff is LF-normalized, is told apart: a stamp-form mention there of a version newer than the version at the branch's fork point is a tentative stamp and moves (this is what finds a quoted row after a lost slot and a merge of the base, since the base's release is newer than the fork point), and when the branch has no stamp of its own because its VERSION equals the base's and its fork point is the base itself (a branch rebased onto the base after it reached X, or one never stamped) a mention of the base version on a line it added is its stale stamp and moves too, while the same version on a line the base already carries is the base's release and stays. The stage detail names both counts per file (`VERSION equals base X, so X moved only on lines this branch added and stayed on the base's own lines (<file>: N moved, M kept)`), so the stamp commit shows which lines moved and why. A line the base added since the fork point is the base's even when that diff shows it (a merge of the base on the branch, committed or still staged, brings every line the base added since the fork into the diff from the fork point, main's own release keys included): it is left out, the detail says `N line(s) the base added since the fork point are the base's, not this branch's, and stay (<file>: N)`, and a base diff git cannot run refuses the stage (`POLICY_SOURCE_UNAVAILABLE`, `cannot diff <since>..<base> for the stamp_paths stage`) rather than reading as an empty set that would move the base's lines again. The rule reads line ownership, not meaning: a branch-added line inside `stamp_paths` that means the base's own release in a stamp form (`Requires vX or later` written on a branch forked at X) moves with it, which the `--dry-run` diff and the per-file counts show before anything is written; write such a reference outside a stamp form, or keep that file out of `stamp_paths`. `--was X` is the opposite tool: it rewrites X everywhere, the base's own lines included, and re-heads the CHANGELOG section `## [X]`. Only the repo's own version width counts, so a 3-digit dependency pin in a 4-digit repo is never a candidate.
|
||||
4. **stamp_paths**: exact `(was X)`, `since: X` and `vX` rewrites, plus every `stamp_patterns` template, within `stamp_paths` and `stamp_files`. A quote around X in the first two forms is part of the form and is kept as written: `since: 'X'`, `since: "X"`, `(was 'X')` and `(was "X")` move like the bare ones (the shape of a quoted source row such as gbrain's `{ since: '0.60.468.0', … }`), an opened quote must close right after X (`since: '1.2.3-beta'` and `since: '1.2.3"` are not the form, in the writer and in the tree receipt alike), and `vX` already accepts a quote before the `v`. Which X moves is decided by provenance. The branch's own tentative stamp (its VERSION when it moved past base) and `--was <X>`, when given, are each rewritten on every line of every stamp file. Beyond that, only lines the branch itself added are read, by their position in the file (`git diff --unified=0` from the fork point), so a base line with the same text, or a CRLF checkout whose diff is LF-normalized, is told apart: a stamp-form mention there of a version newer than the version at the branch's fork point is a tentative stamp and moves (this is what finds a quoted row after a lost slot and a merge of the base, since the base's release is newer than the fork point), and so does one of the fork-point version itself (a row the branch wrote at the version it forked from instead of a next version is its own stale stamp; a branch-added mention older than the fork point stays), and when the branch has no stamp of its own because its VERSION equals the base's and its fork point is the base itself (a branch rebased onto the base after it reached X, or one never stamped) a mention of the base version on a line it added is its stale stamp and moves too, while the same version on a line the base already carries is the base's release and stays. The stage detail names both counts per file (`VERSION equals base X, so X moved only on lines this branch added and stayed on the base's own lines (<file>: N moved, M kept)`), so the stamp commit shows which lines moved and why. A line the base added since the fork point is the base's even when that diff shows it (a merge of the base on the branch, committed or still staged, brings every line the base added since the fork into the diff from the fork point, main's own release keys included): it is left out, the detail says `N line(s) the base added since the fork point are the base's, not this branch's, and stay (<file>: N)`, and a base diff git cannot run refuses the stage (`POLICY_SOURCE_UNAVAILABLE`, `cannot diff <since>..<base> for the stamp_paths stage`) rather than reading as an empty set that would move the base's lines again. The rule reads line ownership, not meaning: a branch-added line inside `stamp_paths` that means the base's own release in a stamp form (`Requires vX or later` written on a branch forked at X) moves with it, which the `--dry-run` diff and the per-file counts show before anything is written; write such a reference outside a stamp form, or keep that file out of `stamp_paths`. `--was X` is the opposite tool: it rewrites X everywhere, the base's own lines included, and re-heads the CHANGELOG section `## [X]`. Only the repo's own version width counts, so a 3-digit dependency pin in a 4-digit repo is never a candidate. The checks that accept a stamp commit read the same bound as the writer (`stampOnlyLine` with the branch's side in `lib/stamp-only.ts`): the tree receipt behind `stamp`, `verify` and the gate-ahead replay, the merge train's stamp and verify checks (each reading the slot's predecessor, the gated commit's first parent, as the base), and finish's route each take a moved version on a line the branch added at or above the fork-point version, and on any other line at or above the gated head's VERSION. A base line whose text duplicates a line the branch added gets the branch's bound too, because the checks match branch lines by text, not position (GST-979). The /ship receipt carry can still refuse the move (`RECEIPT_CARRY_PRODUCT_CHANGED`), and when the branch keeps its VERSION claim, carry and finish still route the move as behavior (GST-980).
|
||||
5. **mirror**: tier 2; declared and validated, reported as not run.
|
||||
6. **apply**: every changed file is written to the branch only after all stages validated, each against the fingerprinted preimage; a file edited meanwhile is `RESTAMP_CONFLICT` and the applied writes roll back (rollback refuses to overwrite a file whose bytes differ from the recorded postimage).
|
||||
|
||||
|
||||
+4
-1
@@ -961,7 +961,10 @@ arms, decoys, stop rules, cap) and `--check` refuses to continue while a field i
|
||||
blank (`PREREG_INCOMPLETE`). `arms` resolves the `eval-arms` model role against
|
||||
`lib/pricing.ts` and stops on an unpriced model (`EVAL_PRICE_MISSING`); `dry-run`
|
||||
runs every row through the stub model for $0 and flags malformed rows
|
||||
(`EVAL_DRY_RUN_FAILED`); `pilot start|settle|release|report` runs three priced
|
||||
(`EVAL_DRY_RUN_FAILED`), and refuses a plan whose every arm is Haiku or Sonnet
|
||||
unless `--tier-question` names what it asks about that tier, re-costing it on
|
||||
frontier models ([frontier models under test](TESTING_INTERNALS.md#frontier-models-under-test));
|
||||
`pilot start|settle|release|report` runs three priced
|
||||
items, records actual spend, extrapolates each priced arm to the full set against
|
||||
the cap (`EVAL_BUDGET_EXHAUSTED`) and prints the spend table the approval question
|
||||
shows. An arm on a subscription with no extra charge is declared with
|
||||
|
||||
@@ -19,7 +19,7 @@ Beyond the slash-command skills, gstack ships standalone CLIs for workflows that
|
||||
| `gstack-review-log` | **Review-pass receipts** — `--start <skill>` captures the working-tree fingerprint before a diff review; `'<JSON>' --finish <token>` consumes that single-use, repository/branch/skill-scoped receipt. Binding requires matching start/end content and reviewer-reported `completed:true` and `converged:true`; it is not independent proof that a model read the source. `--check-shared-libs <token>` reads a current finding as JSON on stdin and checks prior Skip decisions against the actual branch, capture and eligible raw source blobs. It returns `reusable:false` when proof is missing or unsafe, including older records without logger-versioned coverage. Final review logging computes shared-code fingerprints and coverage rather than trusting supplied proof. |
|
||||
| `gstack-review-read` | **Review freshness** — emits review records with computed `review_freshness.status` and `reason`: CURRENT, STALE, or UNVERIFIED for diff reviews. `/ship` and `/land-and-deploy` use the same grade; a matching commit alone never certifies a diff review. [Dashboard rules](skills.md#review-readiness-dashboard). |
|
||||
| `gstack-evidence` | **Verification-evidence ledger** — `run --label <lane> -- <cmd>` transparently wraps any test command (the child's exit code always passes through) and records what ran against which working-tree fingerprint (tracked and untracked files that are not gitignored, minus an untracked `.bun/` folder at the repo root, which Bun can create during a run); a run whose files changed while it ran is recorded without a fingerprint and with up to five of the changed paths, which `check` names in its STALE reason. `check` grades each label FRESH/STALE/MISSING with `--expect-cmd`, `--max-age`, and `--allow-paths` binding. /ship and /land-and-deploy cite fresh evidence instead of re-running suites. Per-run logs are 0600, capped at 2MB, pruned after 30 days; the ledger and logs stay machine-local by design. |
|
||||
| `gstack-flaky` | **Flaky-test rerun for any JUnit runner** — `run --junit <file.xml> -- <command>` runs your test command (Playwright, `bun test`, jest, vitest, pytest) and, when at most `--max-reruns` (3) tests failed and the runner made no retries of its own, reruns the whole command once (`--rerun-scope test` opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (`GSTACK_FLAKE_LEDGER_JUNIT`, default `<state root>/flake-ledger-junit.jsonl`). A missing or malformed report, a crash or a killed runner never passes. `ids --junit <file>` prints a report's test ids. Exit 0 pass, 1 a test failed, 2 usage, 3 no usable report. [Guide](flaky.md). |
|
||||
| `gstack-flaky` | **Flaky-test rerun for any JUnit runner** — `run --junit <file.xml> -- <command>` runs your test command (Playwright, `bun test`, jest, vitest, pytest) and, when at most `--max-reruns` (3) tests failed and the runner made no retries of its own, reruns the whole command once (`--rerun-scope test` opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (`GSTACK_FLAKE_LEDGER_JUNIT`, default `<state root>/flake-ledger-junit.jsonl`). A missing or malformed report, a crash or a killed runner never passes. `quarantine add|renew|release|check|report` keeps `.gstack/quarantine.json`: a quarantined test still runs, gets up to 3 attempts of the whole command, passes only on a full green attempt and is never counted as a pass (`run --base <ref>` reads the registry from the base branch). `ids --junit <file>` prints a report's test ids. Exit 0 pass, 1 a test failed, 2 usage, 3 no usable report. [Guide](flaky.md). |
|
||||
| `gstack-pr-wait` | **PR event watcher** — `gstack-pr-wait <pr> [--repo owner/name] [--until green\|ready\|any-change] [--interval 20] [--timeout 60] [--exclude CHECK,...]` polls one pull request through `gh` and prints one stdout line per real event (a check failed with its link, all checks finished, a comment or review from someone other than the author, a review bot finished on the head, the base moved and its new commits touch this PR's files, BEHIND or DIRTY, merged or closed), with progress on stderr, so a monitor wakes the agent once per event instead of on a timer. `green` gates on every check on the head (an advisory bot's review is never waited for, its checks gate like any other), as `gstack-ci-gate` does; a check the base branch requires (rulesets and classic branch protection) is pending until it appears, never green by omission; two identical polls confirm a check set is complete; `--exclude` leaves a named non-required check out of the verdict as `gstack-ci-gate --exclude` does (a required name is refused and gates). Exit 0 when the condition is met, 1 when it needs action or `--timeout` passes (`PR_WAIT_BLOCKED`), 2 on bad input, a second watcher for the same PR or a local error (`PR_WAIT_REFUSED`). State under `<state root>/pr-wait/`, baseline discarded after 6 hours. Review bots are configuration: `gstack-config set pr_wait_gating_bots greptile,macroscope`, `pr_wait_advisory_bots coderabbit` (advisory bots are reported, never waited on). `/land-and-deploy` Step 3 is one call of it. Design ported from tstack's pr-wait (MIT, Theo Browne; NOTICE.md). [Exit codes](troubleshooting.md#gstack-pr-wait). |
|
||||
| `gstack-issue-guard` | **Tracker-text trust envelope** — fetches GitHub issue/PR text (`issue <n>`, `pr-body`, `pr-comments`, or `--stdin`) and wraps it in a labeled envelope so agents treat it as data: injection-shaped lines get labeled even through fullwidth and invisible-character evasion, and forged envelope banners are defused. Every tracker-text ingress in gstack routes through it, enforced by a CI scanner. |
|
||||
| `gstack-ios-qa-daemon` | **iOS QA daemon** — Mac-side broker between an agent and a connected iPhone or iPad over USB CoreDevice. Loopback by default; `--tailnet` opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on `~/.gstack/ios-qa-daemon.pid`. See [docs/howto-ios-testing-with-gstack.md](howto-ios-testing-with-gstack.md). |
|
||||
|
||||
@@ -10,7 +10,7 @@ gstack has two support tiers. Full support gets the whole test suite, paid evals
|
||||
|
||||
## Minimal support
|
||||
|
||||
- **Windows** (Git Bash or MSYS): with a default Git for Windows install, `./setup` installs, and the core skills start without errors. The core skills are `/ship`, `/review`, `/qa-only`, `/investigate` and `/browse` on gstack's own browser. Once the `windows-smoke` CI job lands, it checks this on every pull request (see Windows CI below).
|
||||
- **Windows** (Git Bash or MSYS): with a default Git for Windows install, `./setup` installs, and the core skills start without errors. The core skills are `/ship`, `/review`, `/qa-only`, `/investigate` and `/browse` on gstack's own browser. One required CI job, `windows-smoke`, checks this on every pull request (see Windows CI below).
|
||||
- **Gemini models:** the core workflows run to the end without errors when Gemini is the host agent or a reviewer. We check this by hand or with one cheap smoke run, not with an eval cell.
|
||||
|
||||
That is the whole promise. Other skills, optional parts such as `/cso`'s native helper, non-default shell settings and speed are not covered.
|
||||
@@ -23,8 +23,9 @@ That is the whole promise. Other skills, optional parts such as `/cso`'s native
|
||||
|
||||
## Windows CI
|
||||
|
||||
- **Today:** the six-shard Windows free suite (the `Windows Free Tests` workflow, `windows-free-tests`) runs on every pull request and every push to main, and it is a required check.
|
||||
- **Once `windows-smoke` lands:** `windows-smoke` becomes the required Windows check on pull requests. The six-shard suite then runs nightly on main, or when someone starts it by hand, and never blocks a pull request. A red nightly run gets a fix only when it shows an install, startup or core-workflow break.
|
||||
- `windows-smoke` is the required Windows check on pull requests.
|
||||
- The six-shard Windows free suite (the `Windows Free Tests` workflow, `windows-free-tests`) runs nightly on main, or when someone starts it by hand, and never blocks a pull request.
|
||||
- A red nightly run gets a fix only when it shows an install, startup or core-workflow break.
|
||||
|
||||
## What doesn't change
|
||||
|
||||
|
||||
+41
-8
@@ -2592,12 +2592,24 @@ case with `bun run eval:pass-rates --headroom --case <id>`.
|
||||
<a id="cost-unknown"></a>
|
||||
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
|
||||
|
||||
**Meaning.** PTY and Codex sessions record no billing, so their cost is
|
||||
unknown. So is a trial in a multi-case shard whose result records are named by
|
||||
a display label rather than the case id, because no record can be attributed
|
||||
to it (a one-case shard's records are its case's). Only the known sum is
|
||||
shown. A train verdict's `ATTEMPT:` line reports the same as `cost=$<x> known
|
||||
+ <n> of <m> case(s) cost unknown`, a bare `cost=$<x>` only when every case
|
||||
**Meaning.** Some trials' cost was not captured, so only the known sum is
|
||||
shown. A trial is cost-unknown when it ran a PTY session whose transcript
|
||||
could not be priced (a non-hermetic or resumed launch, the native
|
||||
AskUserQuestion capture, two new transcripts for one cwd, no transcript or
|
||||
usage, or a model with no price row), a
|
||||
Codex session that completed no turn or whose model has no price row, a
|
||||
`claude -p` session that ended without a result, or an aborted Agent SDK
|
||||
session whose streamed usage could not be priced (no usage, or a model with no
|
||||
price row). A PTY or aborted Agent SDK session that does price counts its
|
||||
`streamed_cost_usd` instead. A trial is also unknown when no record and no
|
||||
session can be attributed to it, for example a multi-case shard's
|
||||
display-label record without `cost_case_id` (a one-case shard's records are
|
||||
its case's); a model-free case (`timeline-event-flow`) is a known $0. A panel
|
||||
trial (`behavior`, `judge`) is unknown whenever any of its sessions billed
|
||||
nothing, even one with a priced `streamed_cost_usd`. The rules are in
|
||||
[Failure causes and details](TESTING_INTERNALS.md#failure-causes). A train
|
||||
verdict's `ATTEMPT:` line reports the same as `cost=$<x> known + <n> of <m>
|
||||
case(s) cost unknown`, a bare `cost=$<x>` only when every case
|
||||
in the train record is priced, and no cost at all when none is.
|
||||
|
||||
**Fix.** None needed; this is informational.
|
||||
@@ -3855,7 +3867,10 @@ repository), `api_error` (gh could not read the run), `workflow_mismatch` (a
|
||||
run of another workflow), `run_in_progress` (still queued or running) or
|
||||
`run_not_success` (it failed, was cancelled or timed out). Each prints a fix
|
||||
line; after the last three, clear again with a finished green run of the
|
||||
lane's own workflow ([run evidence](pregate.md#run-evidence)).
|
||||
lane's own workflow ([run evidence](pregate.md#run-evidence)). For a lane whose
|
||||
workflow runs only through `workflow_call`, a run of a workflow whose job says
|
||||
`uses: ./.github/workflows/<lane's>.yml` counts as a run of the lane (GST-971);
|
||||
it refuses `caller_job_skipped` when its `<calling job> / <lane job>` job did not succeed.
|
||||
|
||||
Another tree is accepted in exactly one case: it differs from `pregate.json`'s
|
||||
tree only in the repo's stamp rule (`.gstack/ship-policy.json` on the base
|
||||
@@ -5486,6 +5501,23 @@ body no longer carries the exception's line word for word
|
||||
/ship again and record the owner's acceptance with `--docs` or
|
||||
`--review-loop-exception` ([ship-policy.md](ship-policy.md#ship-receipt)).
|
||||
|
||||
<a id="queue-receipt-targeted-missing"></a>
|
||||
### `QUEUE_RECEIPT_TARGETED_MISSING`
|
||||
|
||||
**Meaning.** `ready`, `train build` or `land` recomputed the targeted
|
||||
classes the head needs (money, gate, auth, secret) from git, and the receipt's
|
||||
`review_targeted` does not cover every one of them with a `done` pass. The
|
||||
`REVIEW_TARGETED:` line prints the required classes, the classes a done pass
|
||||
covered (`legacy` when the receipt records no targeted pass) and
|
||||
`verdict=missing`, or `verdict=failed` when the recorded pass failed. A
|
||||
head that needs no class passes whatever the receipt records. The receipt is
|
||||
unchanged.
|
||||
|
||||
**Fix.** Run /ship on the head the READY names so the targeted reader runs on
|
||||
every required class and `gstack-ship-receipt write --review-targeted` records
|
||||
a done pass ([review-policy.md](review-policy.md#targeted-pass-on-money-gate-auth-and-secret-code)).
|
||||
Never edit `review_targeted` by hand.
|
||||
|
||||
<a id="queue-screen-stale-ref"></a>
|
||||
### `QUEUE_SCREEN_STALE_REF`
|
||||
|
||||
@@ -5883,7 +5915,8 @@ body that names a counter the receipt records (`passes=3` against
|
||||
against `gate.passed: 41`) with a different value. The receipt is written from
|
||||
the run; the prose was typed, so a reader of the body would learn a number the
|
||||
run did not produce. The `PROSE:` line names each field with both values. The
|
||||
`## Approvals` table `gstack-approvals` renders is never read as a claim.
|
||||
`## Approvals` table `gstack-approvals` renders, and the `Review-loop exception: …`
|
||||
line the receipt's exception requires, are never read as claims.
|
||||
|
||||
**Fix.** Correct the body to the receipt's number. When the receipt is the
|
||||
stale one (the run was repeated), rerun `/ship` so the receipt is rewritten
|
||||
|
||||
Reference in new issue
Block a user