docs(designs): gate-cost-4x plan and measurement (GSTA-49)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-18 04:16:18 +00:00
1 parent fd13cf25e8
commit 1a6dd2f7ed
2 files changed
+212

No files matched your search

@@ -0,0 +1,111 @@
# Paid PR gate cost: where one full gate's money goes (2026-10-17)
Thread GSTA-49. Free, read-only measurement over recorded artifacts. Base: main fd13cf2 (v1.91.133.0).
## Data
- Nine complete full gates (108 rule cases + 24 judges each) from 2026-10-17/18: runs 39055436252, 39069323673, 39079802260, 39082422480, 39083793205, 39085410841, 39086339074, 39087248425, 39088415319. Every paid-slice artifact (eval JSON with the full stream-json transcript and the CLI's `result` event), session ledgers, trial outcomes and report verdicts. Recorded cost per gate: $90.93 to $94.70 (mean $93.14).
- Selection replay: every one of the 92 squash-merged PR commits on main since 2026-10-03, run through today's selector (`computePaidCaseSelection`, profile pr) on fd13cf2, plus GSTA-47's per-run CSV (505 runs) for spend weights.
- Prices: lib/pricing.ts. A least-squares fit over 93 Fable sessions' `modelUsage.costUSD` reproduces $10 input, $50 output, $0.25 cache read and $12.50 cache write per million tokens exactly.
## 1. Who the money goes to
| spender | $ per gate | share |
|---|---|---|
| Model under test, claude-fable-5-1 (claude -p and Agent SDK sessions) | 90.46 | 97.1% |
| plan-review-report's pinned claude-opus-4-7 session (also model under test) | 1.13 | 1.2% |
| 24 LLM judges (skill-llm-eval, $0.06 each, 3 samples) | 1.44 | 1.5% |
| Claude Code's own Haiku calls, opus-4-8 side calls | 0.09 | 0.1% |
| **recorded total** | **93.14** | |
| Not recorded: 8 PTY cases, review-gate-integrity, diagram-triplet, auq-format-gate (1,371 s of sessions) | ~12.9 (estimated at the measured $0.0094 per session-second) | |
There are no LLM fixture actors in the gate: every fixture actor is scripted. Judges and side models together are 2.8% of a gate. **Cheaper judges or actors can save at most about $1.15 per gate (Sonnet 5.5 judges), about $35 a week after the rest of this plan.**
## 2. Token classes (Fable, per gate)
| class | tokens | $ | share of gate |
|---|---|---|---|
| cache writes (5-minute TTL, 1.25x input) | 3.66M | 45.73 | 49% |
| output (includes 225k thinking tokens, $11.25) | 0.74M | 37.15 | 40% |
| cache reads | 29.4M | 7.35 | 8% |
| uncached input | 23k | 0.23 | 0.2% |
**Prompt caching is already on and working.** 89% of input tokens are cache reads; without caching the same input would cost $331 per gate instead of $53. Inside sessions, cache misses (TTL expiry, prefix changes) re-write only 1.9% of cache-write tokens ($0.87 per gate). The cache-write bill is new content entering the context once: tool results (mostly the agent reading gstack skill files: SKILL.md, review-army.md, checklist.md, release-body.md, plan-completion.md...) and the session's first request.
The one cacheable share not yet shared: each session's first request writes a median 7.8k tokens ($11.03 per gate, 24% of cache writes) after reading an 8.7k-token prefix (tools and the static system prompt) that other sessions already cached. 13 of 93 sessions read 0 tokens on their first request (they started before any sibling had written the shared prefix). The written part is the per-session system prompt (working directory, date, git status) plus the skill listing and the case prompt. A byte-identical session environment could share part of it: **estimated $3 to $5 per gate (3 to 5%), to be measured before building.**
Output and thinking are how the model under test behaves for users, so they are not a lever. Trimming what the agent reads is trimming gstack's skill files themselves (a product change, out of scope here).
## 3. Per case and per trial
Every PR-gate case is kind `rule` with one trial; the 138 `behavior` cases are deferred out of the PR gate already ("deferred: 138 broad behaviors"). Judges are kind `judge` with 3 samples ($0.06 per judge in total). **Trial-count policy (EVAL_POLICY panel n=3, k=2) does not touch PR-gate spend**; it only prices the weekly census.
Top 20 cases (48% of a gate; full list in per-case.csv):
| # | case | $ | runner |
|---|---|---|---|
| 1 | shared-libs-review-revalidation | 5.02 | Agent SDK, 4 sessions |
| 2 | shared-libs-review-path-eligibility | 4.00 | Agent SDK, 3 sessions |
| 3 | ship-docsync-store | 3.62 | claude -p |
| 4 | ship-docsync-completion | 3.61 | claude -p |
| 5 | ship-docsync-current | 3.36 | claude -p |
| 6 | shared-libs-review-index-flags | 2.62 | Agent SDK, 2 sessions |
| 7 | shared-libs-review-prior-coverage | 2.60 | Agent SDK, 2 sessions |
| 8 | shared-libs-review-lifecycle | 2.44 | Agent SDK, 2 sessions |
| 9 | ship-exploratory-late-input | 2.31 | claude -p |
| 10 | plan-ceo-finding-floor | ~2.27 (est.) | PTY |
| 11 | plan-design-with-ui-scope | ~2.18 (est.) | PTY |
| 12 | ship-docsync-failure | 2.03 | claude -p |
| 13 | ship-exploratory-plan-checks | 2.00 | claude -p |
| 14 | review-exploratory-small-cli | 1.97 | claude -p |
| 15 | ship-exploratory-small-cli | 1.94 | claude -p |
| 16 | ship-exploratory-unavailable | 1.87 | claude -p |
| 17 | plan-mode-no-op | ~1.84 (est.) | PTY, 6 sessions |
| 18 | plan-devex-finding-floor | ~1.74 (est.) | PTY |
| 19 | document-release | 1.72 | claude -p |
| 20 | land-and-deploy-workflow | 1.71 | claude -p |
By file: ship-docsync $19.4, qa-callers $10.1, shared-libs-paths $9.2, shared-libs $8.8, deploy $5.5. Per-case cost is stable: the largest spread across nine gates is ±12%.
## 4. How many cases a PR's diff really touches
| selection for the 92 merged PRs (final squash diff) | mean cases | median cases | mean $ per gate | PRs selecting ≥90% of the gate |
|---|---|---|---|---|
| today's selector | 80 | 108 | 69 | 67 |
| direct dependencies only (no hub files) | 31.5 | 9.5 | 27 | 12 |
| direct + a $12 canary set whenever a hub file changed | | | 33.5 (median 15.6) | |
222 changed files in the window each select ≥90% of the gate by themselves; 643 tracked files do. The coupling is the derived reference walk in scripts/pr-dependencies.ts, which follows every textual mention transitively. Representative chains (`trace`):
- `cso/sections/audit-phases.md > scripts/gen-skill-docs.ts > lib/doctor-components.ts > lib/result-codes.ts` (the build-time generator's whole import closure becomes a runtime input of every case).
- `qa/SKILL.md.tmpl > bin/gstack-upgrade > bin/gstack-capy-install > bin/gstack-doctor-components.sh > bin/gstack-pregate > lib/pregate/workflows.ts > scripts/test-free-shards.ts > ...` (the preamble's bins mention other bins in help text and rare branches).
- `browse/src/terminal-agent.ts > browse/src/server.ts > bin/gstack-config > bin/gstack-models > lib/model-policy-cli.ts > bin/gstack-eval-plan > ...`.
- `test/skill-e2e-cso.test.ts > test/helpers/e2e-helpers.ts > scripts/test-paid-shards.ts > scripts/lib/paid-report.ts > scripts/eval-flake-rank.ts > scripts/test-free-shards.ts > scripts/lib/windows-curation.ts`.
Graph surgery does not fix it. Stopping the walk at gen-skill-docs.ts, at the harness's global set and at the 13 preamble bins together moved the mean from $78.5 to $77.0 (same scale), because the graph is one connected component and another path always remains. The exercised-command map (R4) is the precise fix but is not ready (GSTA-47: activation no earlier than about Oct 31). What does work is a depth limit: a PR tier that selects by a case's direct inputs and leaves the transitive closure to a full gate that runs per merge batch.
**Coverage check on history.** Of the 40 PR-caused red results in the window (bucket b), a direct-only tier selects 29. The 11 it misses are 7 judges (all 24 judges cost $1.44, so the plan always runs them) and 4 e2e results: plan-mode-no-op (#3059, #3017), office-hours-auto-mode (#3077) and design-review-plugin-handoff (#3121). Those four would be caught by the batch full gate. Of 76 outside-red results (flakes and main defects), the direct tier still selects 35, so PRs would see 54% fewer false reds.
## 5. Spend model (dollars a week)
Replayed over GSTA-47's 15.1-day window (91 merged PRs, 65 paid anything). "Gates per PR" counts one ready gate, plus one re-gate per PR-caused red, plus outside reds scaled by the tier's exposure and by #3260's fixes (60% of outside-red spend remains).
| step | $ / week | multiple of $9,950 |
|---|---|---|
| today | 9,950 | 1.0x |
| GSTA-48 in flight: R1 readiness-gated dispatch, R2 reuse across main moves and restamps | ~7,000 | 1.4x |
| + on-demand ready gate (paid gate only when the lane marks the PR ready; 1.58 gates per paying PR) | ~5,050 | 2.0x |
| + direct-dependency PR tier and a full gate per merge batch of 5 ($786) | ~3,400 | 2.9x |
| + per-case receipt reuse keyed on direct inputs (a re-gate reruns about 30% of the tier) | ~2,550 | 3.9x |
| + shared first-request prompt prefix (−4%) | ~2,450 | 4.1x |
Batch size moves the result by about $130 a week per step: a full gate per 4 merges is $983, per 6 merges $655, nightly-only $652. The honest range for the full combination is **3.6x to 4.1x**; it reaches 4x only with the ready-gate discipline and the receipt reuse both working.
## 6. Levers not recommended (with numbers)
- Cheaper judges: $1.15 per gate maximum, about $35 a week after this plan; a calibration panel would cost more than a year of savings.
- Cheaper fixture actors: none exist in the gate.
- Prompt caching "turned on": it is on; the residual is the first-request prefix above.
- Hub graph surgery: −2% measured; superseded by the depth-limited tier.
- Trial counts: 0 behavior cases in the PR gate.
- Model under test (data only; excluded by Garry's rule that it stays as users run it): Fable is 97% of the bill. The same gate on claude-opus-5-5 list prices would cost about $40 at the same token counts.
@@ -0,0 +1,101 @@
<!-- gstack-autoplan-scope: --no-ui --developer-tool -->
# Cut the paid PR gate bill about 4x (October 17, 2026)
## Implementation plan
### Context
Owner request (Garry, 2026-10-17): "is there any way we can reduce cost like 4x". Paid PR gates (E2E Evals, evals.yml: 108 rule cases and 24 judges, $93 recorded per full gate) cost about $9,950 a week (GSTA-47, reports/paid-gate-spend-2026-10.md). Garry pre-approved the direction the same day ("Do it all") for five levers: (1) PRs run only diff-relevant cases plus a full gate per merge batch or nightly on main; (2) a merge train with bisection on red; (3) decoupling the shared preamble so small edits stop selecting every case; (4) cheaper judges and fixture actors, proven by calibration, with the model under test unchanged; (5) prompt caching. Trial counts and case kinds are out of scope (data only). Coordinator: GStack Minder (GSTA-36) owns merges and routes file conflicts; PRs go READY unstamped (CREW.md 6).
Base: origin/main `fd13cf2` (v1.91.133.0). In flight and not touched by this plan: GSTA-48 (R1 readiness-gated dispatch, R2/R2a verdict reuse across main moves and restamps, and the receipt-store restore fix; it owns `.github/workflows/evals*.yml` and `scripts/gate-reuse.ts` until it merges). Any lane here that edits those files waits for GSTA-48's PR and is routed through GStack Minder first.
Measurement: docs/designs/gate-cost-4x-2026-10-17/MEASUREMENT.md (nine full gates' transcripts, a replay of all 92 merged PRs through today's selector, and the spend model). The findings that shape the plan:
1. **97% of a gate is the model under test (Fable 5.1).** Judges are $1.44 per gate, Claude Code's side calls $0.09, and there are no LLM fixture actors. Lever 4 is worth at most about $35 a week and is dropped.
2. **Prompt caching already works**: 89% of input tokens are cache reads, and in-session misses waste 1.9% of cache writes. The remaining cacheable share is each session's first request ($11 per gate written, of which an estimated $3 to $5 could be shared). Lever 5 is small and goes last, behind a measurement.
3. **The coupling is a transitive textual walk, not one preamble file.** 643 tracked files each select ≥90% of the gate through `scripts/pr-dependencies.ts`'s reference closure (skill text names a bin, the bin's help text names another bin, the build-time generator's whole import closure counts as a runtime input). Cutting the three biggest hubs (generator, harness, 13 preamble bins) moved the replay by 2%: another path always remains. A depth limit works: selecting by each case's direct inputs drops the mean merged PR from 80 cases ($69) to 31.5 cases ($27), median 9.5 cases.
4. **Frequency is the bigger lever.** After GSTA-48, a paying PR still runs about 2.4 full-gate equivalents; a ready-only gate with repair re-gates is 1.58.
Thesis: **every merged commit is still covered by the full gate, but a PR pays only for what it directly touches, only when it is ready, and only once per unchanged case.**
### Spend target and path
| step | $ / week | multiple |
|---|---|---|
| today | 9,950 | 1.0x |
| GSTA-48 (R1, R2) | ~7,000 | 1.4x |
| L1 ready-only paid gate | ~5,050 | 2.0x |
| L2 batch full gate on main + L3 direct-dependency PR tier | ~3,400 | 2.9x |
| L4 per-case reuse keyed on direct inputs | ~2,550 | 3.9x |
| L5 shared session prefix | ~2,450 | 4.1x |
Honest range 3.6x to 4.1x. Re-measured with GSTA-47's scripts after each landing and reported to GStack Minder against the $2,500 target.
### Release promises
- **P1. No commit reaches main's released state without the full gate.** Lowering PR coverage is paired with a full gate per merge batch that runs before the batch's PRs merge (L2), and a nightly full gate on main whenever something merged outside a batch.
- **P2. A PR's paid gate selects every case whose direct inputs the PR changed**, plus all 24 judges, plus a fixed canary set when a transitive-only or full-gate file changed. Every miss the batch gate finds is attributed and becomes a selection fix.
- **P3. A green per-case verdict is reused only when that case's direct inputs are byte-identical** and the batch gate still runs it.
- **P4. Nothing about how the model under test runs changes**: same model, effort, prompts, skills and assertions.
- **P5. Progress is measured, not asserted**: each lane's PR body states its replayed saving, and GStack Minder gets a re-measurement after each landing.
### Premise table
| premise | claim | receipt | verdict |
|---|---|---|---|
| P-a | a full gate records $90.93 to $94.70 | report-verdict of the nine runs in MEASUREMENT.md | true |
| P-b | Fable is 97.1% of recorded gate cost | `modelUsage` in every `result` event; fit reproduces lib/pricing.ts | true |
| P-c | judges cost $0.06 each, 24 per gate | eval JSON `skill-llm-eval` rows | true |
| P-d | PR gate has 0 behavior cases | trial-outcomes kinds: 108 rule, 24 judge; report "deferred: 138 broad behaviors" | true |
| P-e | 643 files reach ≥90% of cases | `derivedDependencies` on fd13cf2 | true |
| P-f | direct-only tier catches 29 of 40 PR-caused red results; 7 of the 11 misses are judges | per-run.csv bucket b vs replayed selection | true |
| P-g | receipt reuse fired on 3.9% of shards | GSTA-47 Q2.1 | true |
| P-h | the store never restores (Ubicloud cache vs GitHub cache) | GSTA-48 report 2026-10-18 03:54Z | reported, unverified here |
| P-i | unrecorded PTY cases cost about $12.9 per gate | session-seconds x measured $/s | estimate |
| P-j | 4.1x is reachable | spend model in MEASUREMENT.md §5 | estimate |
unverified=1, estimates=2
### Lanes (ordered by dollars saved per unit of risk)
**L1. Ready-only paid gate on PRs (~$1,950 a week; no coverage change).** Every merged head keeps the same selection; only the trigger changes. The paid lane starts on a PR head when GSTA-48's readiness conditions hold (free tests green, quiet window) AND the PR carries the `paid-gate` label. `/ship` and `gstack-ship-receipt` add the label when the lane declares READY; GStack Minder's slot call adds it too. Removing the label on a new push is not required: a newer head with the label simply gates when ready. Drafts never gate. Lanes keep using focused local paid cases (CREW.md 14) for iteration. Files: `.github/workflows/evals.yml` trigger and readiness job (GSTA-48 owns it: ship after its PR merges, or GSTA-48 folds it into R1; GStack Minder routes), `ship/SKILL.md.tmpl` label step, `lib/ship-receipt.ts`. Tests: workflow-shape test that an unlabeled PR head skips the paid slices with a visible "not requested" summary and a neutral check; that a labeled head runs; that the required-check name is unchanged so branch protection still requires a verdict before merge.
**L2. Batch full gate (+$650 to $980 a week; the safety net L3 needs).** `bin/gstack-train` (logic in `lib/train.ts`) for GStack Minder: `gstack-train build --prs 3301,3302,...` creates `train/<n>` by merging main and each READY PR in slot order (stops at the first conflict and names it), dispatches the full gate (`evals_all`) on the train head, and records `train.json` (PR heads, train tree, run id). `gstack-train verdict` reads the run: green means each listed head is mergeable as-is under R2's reuse rules (same own-diff), so GStack Minder merges them in order with no further paid gate. Red: `gstack-train bisect` reruns only the red cases (focused, `EVALS_ONLY`) on main alone and on main plus each PR prefix, and prints which PR introduced each red or that it is red on main (outside cause). At most 5 PRs per train (replay: about $786 a week). If anything merged outside a train in the last 24 hours, a nightly scheduled job runs the full gate on main HEAD; a red opens or updates one tracking issue and freezes merges that touch the red cases' inputs until a revert or fix-forward lands (GStack Minder's rule, written into CREW.md by GStack Minder). Files: new `lib/train.ts`, `bin/gstack-train`, new `.github/workflows/evals-nightly-main.yml` (routed through GStack Minder because it matches `evals*.yml`), docs/troubleshooting.md anchors, result codes `TRAIN_CONFLICT`, `TRAIN_RED`, `TRAIN_STALE`. Tests: train build on a fixture repo with a conflict and with a clean stack; verdict parsing of a recorded green and red report-verdict; bisect attribution on a synthetic red that one PR causes and one that main causes; a stale train (main moved after build) refuses `verdict` with `TRAIN_STALE`.
**L3. Direct-dependency PR tier (~$2,430 a week gross; ~$1,650 net of L2's cost).** A third selection mode `direct` in `scripts/lib/paid-select.ts` and `scripts/test-pr-profile.ts`, computed from `scripts/pr-dependencies.ts` with a depth rule: a case's direct inputs are its declared touchfiles; its owning test file and that file's static `test/helpers`/`test/fixtures` import closure (excluding the harness global set, which stays a global touchfile); the template, sections and generated SKILL.md of every skill the case invokes; every bin named in that skill's own non-preamble text or the test file, with those bins' static code-import closure; nothing reached only through another bin's text or through the build-time generator. A changed file that reaches a case only transitively, or a full-gate file (`setup`, `bun.lock`, Dockerfile.ci, the preamble resolvers, `scripts/gen-skill-docs.ts`), adds the canary set instead of every case. The canary set is a reviewed list in `test/helpers/touchfiles-data.ts` of about 10 cheap cases that cover the preamble, plan mode and the bins every skill runs (hermetic-canary, skillmd-setup-discovery, context-save-writes-file, ship-triage, tpa-absent-linux, plan-mode-no-op, office-hours-auto-mode, design-review-plugin-handoff and two more chosen from replay), about $12. All 24 judges always run ($1.44). Rollout: shadow first. Every PR gate computes and prints both selections (`coverage.direct` in the manifest summary) for 7 days or 10 batch gates, whichever is later; activation needs zero e2e cases that the batch gate found red, that a PR caused, and that the direct tier would have dropped. The replay over the window is a precondition too: 0 dropped e2e PR-caused reds once the canary set is in place. GStack Minder flips one flag (`.gstack/pr-tier.json`: `{"mode":"direct"}`); flipping it back restores today's selection. Tests: unit tests of the depth rule on fixture graphs (a bin's help text naming another bin does not propagate; a static import does; generator code does not reach a case, the generated SKILL.md does); the four historical misses selected; a preamble-resolver edit selects the canary set and all judges, not every case; the full-fallback list still forces the canary set; map-diff selection for touchfiles-data.ts unchanged.
**L4. Per-case receipt reuse keyed on direct inputs (~$850 a week; needs GSTA-48's store fix).** Today's per-shard receipts hash inputs that include the hub, so any change invalidates every receipt. Key each case's receipt on its L3 direct-input digest plus the harness global set, the case's own test file and HARNESS_VERSION. A re-gate on the same PR reruns only cases whose key changed and cases that were not green, and reports the rest as reused with the source run and head. Never reused: a case whose last result was red, partial or cancelled; any case when a full-gate file changed; judges (they are cheap). The batch gate (L2) never reuses (`evals_all` disables receipts), so every transitive input is still exercised before merge. Files: `scripts/e2e-shard-reuse.ts`, `scripts/eval-input-cache.ts` (routed through GStack Minder in case GSTA-48 owns them), tests with negative controls for each never-reused class.
**L5. Shared session prefix (~$100 a week; measured before built).** A preregistered panel (under $10, `bin/gstack-eval-plan` prereg) runs 6 short Fable sessions in two working-directory layouts and reads first-request `cache_read_input_tokens`. Build only if the shared prefix grows by at least 3k tokens per session: run each paid session from a fixed path through a per-session mount namespace (`unshare` bind mount, Linux CI only) and warm the shared prefix once per slice before parallel sessions start. Files: `test/helpers/hermetic-env.ts` (a global touchfile: one full gate when it lands). Abandon with a written reason otherwise.
### Not built (data only)
- Cheaper judges and fixture actors: at most $1.15 per gate; no LLM actors exist.
- Trial counts and case kinds: the PR gate runs no behavior cases.
- R4 (exercised-command map): stays in shadow under GSTA-45's criteria; once active it can replace L3's depth rule with observed commands.
- Model under test: Fable is 97% of the bill; Opus 5.5 at the same tokens would be about $40 a gate. Excluded by Garry's rule.
### Gate-integrity edits
L1 (when the paid gate runs), L2 (merge authority moves to the batch verdict), L3 (PR selection), L4 (verdict reuse). Each PR body lists its gate edits; /review lists them; all fall inside Garry's 2026-10-17 approval. Nothing lowers a threshold, changes a case's assertions, changes trials or kinds, or retries a paid case to change a verdict.
### Risks
- **A defect a PR causes in a transitively reached case is caught at the batch gate, not on the PR.** Replay: 4 of 33 e2e PR-caused reds, each covered by the canary set. Cost when it happens: one bisect (red cases only, a few dollars) and a train rebuild ($93 plus latency).
- **An outside red (flake or main defect) now blocks a whole train.** Bisect classifies it against main alone; a red on main alone follows the existing exception path through GStack Minder. Fewer outside reds reach PRs (the tier selects 46% of them).
- **Ready-only gating delays feedback.** Lanes lose the paid gate as a test loop; focused local paid runs replace it (CREW.md 14 already requires that).
- **The savings depend on discipline and on GSTA-48's store fix.** Without L4 the plan lands near 2.9x.
### Validation
Per AGENTS.md "Validation discipline": focused unit tests per lane during repair; `bun test test/parity-suite.test.ts` for the ship template edit; the SKILL.md size guard before any paid gate; one full `bun run test` (remote pregate, CREW.md 13) per PR on a frozen tree. Paid validation: each lane's own PR gate (pre-approved). L5's panel is preregistered and capped at $10; total paid calibration for this plan stays under $50 of the $300 approved.
### Binding decisions
- Garry, 2026-10-17: "Do it all" for levers 1 to 5; trial counts and kinds are data only; calibration panels up to $300 total, each preregistered.
- GStack Minder routes every edit to `.github/workflows/evals*.yml` and reuse logic through GSTA-48.
- Autoplan budget: at most 3 review rounds, then accepted-with-findings deferred to the build.
## Review record
(Filled in by the /autoplan run.)