mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
Merge remote-tracking branch 'origin/gsta-167-calls-1' into gsta-167-calls-2
This commit is contained in:
garrytan committed
commit
bdbe3a9f7e
95 files changed
+2975
-368
No files matched your search
@@ -79,7 +79,11 @@ already assume, and one optional field.
|
||||
`GSTACK_HOME`. Set both to `<work dir>/.gstack-home` for the agent, `setup` and `check`; `GSTACK_HOME` alone loses
|
||||
to a `GSTACK_STATE_ROOT` inherited from the caller's shell, and the agent then writes the caller's real state. A
|
||||
task's `seed.sh` creates `.gstack-home` and lists it in `.git/info/exclude`, so `git diff` checks ignore it and
|
||||
`check` reads state there (/learn's `add` and `prune-stale`, /office-hours' design-doc and mockup checks).
|
||||
`check` reads state there (/learn's `add` and `prune-stale`, /office-hours' design-doc and mockup checks). A
|
||||
`check` that reads `.gstack-home` first requires it to be a real directory, not a symlink, because `find` does not
|
||||
descend into a symlinked start point. /learn's checks read the one project folder named for the work dir, and
|
||||
/office-hours' no-code check lists untracked files against `.git/info/exclude` alone, so a `.gitignore` the agent
|
||||
writes hides nothing.
|
||||
- **`resumable`** (optional, in `verdict.json` and any error record): `true` when an INCOMPLETE run can be finished
|
||||
with `--resume <out>`, which runs only cells that never ran, under the same lock and trial count, so it adds no
|
||||
trials after a result; `false` for a terminal stop, which needs a fix and a fresh run. A missing field reads as
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# particle-1 on gstack's model panel (October 2026)
|
||||
|
||||
Measured October 30, 2026 at commit d209821, which preregistered the panel before any paid run. Run id `2026-10-30-0105`. Data: [rows.jsonl](particle-1-2026-10/rows.jsonl) (one row per cell) and [summary.json](particle-1-2026-10/summary.json).
|
||||
|
||||
## Result
|
||||
|
||||
| Model | Passed | 95% interval | INFRA cells | Cost per task |
|
||||
|---|---|---|---|---|
|
||||
| Opus 5.5 (`claude-opus-5-5`) | 15/15 | 80–100% | 0 | $0.088 |
|
||||
| Sonnet 5.5 (`claude-sonnet-5-5`) | 15/15 | 80–100% | 0 | $0.037 |
|
||||
| particle-1 (`particle-ai/particle-1`) | 15/15 | 80–100% | 0 | $0.007 |
|
||||
| DeepSeek V4.1 Flash (`deepseek/deepseek-v4.1-flash-1017`) | 13/14 | 69–99% | 1 | $0.008 |
|
||||
|
||||
The preregistered wording rule (`suggestionLine` in [test/helpers/model-panel-harness.ts](../../test/helpers/model-panel-harness.ts)) produced this line:
|
||||
|
||||
> particle-1 passed 15/15 tasks on gstack's model panel (Sonnet 5.5 15/15, Opus 5.5 15/15, DeepSeek V4.1 Flash 13/14) at $0.007 per task vs $0.037 for Sonnet 5.5, the cheapest of the four models measured (no measured difference from Sonnet 5.5 on this panel; measured 2026-10-30 in Claude Code via OpenRouter). Worth trying for low-stakes steps; keep blocking reviewers on your main model.
|
||||
|
||||
## How to read it
|
||||
|
||||
- **The panel is easy.** 58 of the 59 scored cells passed, across all four models. So the panel shows that particle-1 can do small gstack-shaped tasks at about one-fifth of Sonnet 5.5's cost per task. It can't tell the models apart on these tasks, so it says nothing about harder work. "No measured difference" means exactly that, not "as good as Sonnet".
|
||||
- **Cost is measured, not listed.** Each cell is priced from its own token counts (uncached input, cache reads, cache writes, output) at the rates in `lib/pricing.ts`, not from Claude Code's `total_cost_usd`, which prices gateway models at Anthropic rates. particle-1 and DeepSeek ran at OpenRouter's standard (peak) rates. Particle's own API is 50% cheaper from 8:30 pm to 4 am Eastern.
|
||||
- **Same harness for every model.** Every arm ran in Claude Code `2.1.312` with the same tools, turn cap (40), timeout (8 minutes), the extracted gstack build-discipline skill and a hidden check. Claude Code's default effort settings were used for every arm, and particle-1 used its default reasoning effort (high). Particle trained particle-1 in a different agent harness (Terminus 2), so this setup may understate it.
|
||||
- **No model mixing.** Every gateway cell billed only its own model. The harness pins every Claude Code model alias to the arm's model and would classify any other billed model as INFRA (MODEL_MIX). No gateway cell had an Anthropic credential in its environment.
|
||||
|
||||
## The panel
|
||||
|
||||
Five tasks, each scored by a hidden check that the selftest proves fails on the seed fixture and passes on a reference answer ([test/model-panel-selftest.test.ts](../../test/model-panel-selftest.test.ts)):
|
||||
|
||||
1. **bugfix:** fix a price-formatting bug without touching the tests (`arm-benchmark/bugfix-decoys`).
|
||||
2. **crud:** add `DELETE /notes/:id` with 204/404 and a test (`arm-benchmark/crud-endpoint`).
|
||||
3. **review:** pre-landing review of a new file with a planted SQL injection and a page-offset bug. REVIEW.md must name both, and the source must stay unchanged.
|
||||
4. **changelog:** write the 1.4.0 entry from a commit list, user-facing changes only.
|
||||
5. **investigate:** find the root cause of a timezone date bug, leaving the tests and a retry decoy untouched. The fix must also hold in a UTC+14 zone.
|
||||
|
||||
Four models × five tasks × three trials = 60 cells, at most four at a time, under a $15 spend ledger. The run spent $2.09, plus $0.15 still counted at its reservation for the one cell with no usage.
|
||||
|
||||
## Non-passing cells
|
||||
|
||||
- **DeepSeek, review, trial 1: INFRA.** The cell hit the 8-minute timeout. The CLI was killed before it wrote a result event, and `classifyCell` counts a missing result event as INFRA. That conflicts with the function's own comment, which says a timeout after work is scored by the check. The run kept the verdict as computed, because preregistered rules don't change after a result is seen. The fix (score a timed-out cell's working tree) is left for the next run. It affects only the DeepSeek count.
|
||||
- **DeepSeek, investigate, trial 2: fail.** It fixed the date bug, then deleted the retry decoy the ticket said to leave alone.
|
||||
|
||||
## Every cell
|
||||
|
||||
| Task | Arm | Trial | Verdict | Turns | Cost (USD) | Note |
|
||||
|---|---|---|---|---|---|---|
|
||||
| bugfix | opus | 1 | pass | 6 | 0.0724 | |
|
||||
| bugfix | opus | 2 | pass | 5 | 0.0673 | |
|
||||
| bugfix | opus | 3 | pass | 6 | 0.0726 | |
|
||||
| bugfix | sonnet | 1 | pass | 7 | 0.0346 | |
|
||||
| bugfix | sonnet | 2 | pass | 7 | 0.0336 | |
|
||||
| bugfix | sonnet | 3 | pass | 7 | 0.0335 | |
|
||||
| bugfix | particle | 1 | pass | 7 | 0.0058 | |
|
||||
| bugfix | particle | 2 | pass | 7 | 0.0063 | |
|
||||
| bugfix | particle | 3 | pass | 10 | 0.0080 | |
|
||||
| bugfix | deepseek | 1 | pass | 9 | 0.0055 | |
|
||||
| bugfix | deepseek | 2 | pass | 12 | 0.0036 | |
|
||||
| bugfix | deepseek | 3 | pass | 7 | 0.0035 | |
|
||||
| changelog | opus | 1 | pass | 5 | 0.0649 | |
|
||||
| changelog | opus | 2 | pass | 5 | 0.0638 | |
|
||||
| changelog | opus | 3 | pass | 5 | 0.0644 | |
|
||||
| changelog | sonnet | 1 | pass | 6 | 0.0294 | |
|
||||
| changelog | sonnet | 2 | pass | 6 | 0.0312 | |
|
||||
| changelog | sonnet | 3 | pass | 6 | 0.0314 | |
|
||||
| changelog | particle | 1 | pass | 6 | 0.0035 | |
|
||||
| changelog | particle | 2 | pass | 6 | 0.0034 | |
|
||||
| changelog | particle | 3 | pass | 7 | 0.0034 | |
|
||||
| changelog | deepseek | 1 | pass | 7 | 0.0052 | |
|
||||
| changelog | deepseek | 2 | pass | 6 | 0.0081 | |
|
||||
| changelog | deepseek | 3 | pass | 6 | 0.0076 | |
|
||||
| crud | opus | 1 | pass | 6 | 0.0905 | |
|
||||
| crud | opus | 2 | pass | 6 | 0.0899 | |
|
||||
| crud | opus | 3 | pass | 5 | 0.0832 | |
|
||||
| crud | sonnet | 1 | pass | 9 | 0.0401 | |
|
||||
| crud | sonnet | 2 | pass | 9 | 0.0408 | |
|
||||
| crud | sonnet | 3 | pass | 9 | 0.0411 | |
|
||||
| crud | particle | 1 | pass | 10 | 0.0110 | |
|
||||
| crud | particle | 2 | pass | 11 | 0.0071 | |
|
||||
| crud | particle | 3 | pass | 9 | 0.0077 | |
|
||||
| crud | deepseek | 1 | pass | 9 | 0.0061 | |
|
||||
| crud | deepseek | 2 | pass | 7 | 0.0061 | |
|
||||
| crud | deepseek | 3 | pass | 9 | 0.0046 | |
|
||||
| investigate | opus | 1 | pass | 8 | 0.1070 | |
|
||||
| investigate | opus | 2 | pass | 6 | 0.0949 | |
|
||||
| investigate | opus | 3 | pass | 6 | 0.0933 | |
|
||||
| investigate | sonnet | 1 | pass | 6 | 0.0358 | |
|
||||
| investigate | sonnet | 2 | pass | 6 | 0.0359 | |
|
||||
| investigate | sonnet | 3 | pass | 6 | 0.0352 | |
|
||||
| investigate | particle | 1 | pass | 8 | 0.0062 | |
|
||||
| investigate | particle | 2 | pass | 8 | 0.0064 | |
|
||||
| investigate | particle | 3 | pass | 8 | 0.0100 | |
|
||||
| investigate | deepseek | 1 | pass | 10 | 0.0070 | |
|
||||
| investigate | deepseek | 2 | fail | 12 | 0.0124 | FAIL: the retry decoy must not change |
|
||||
| investigate | deepseek | 3 | pass | 16 | 0.0056 | |
|
||||
| review | opus | 1 | pass | 6 | 0.1140 | |
|
||||
| review | opus | 2 | pass | 6 | 0.1214 | |
|
||||
| review | opus | 3 | pass | 6 | 0.1136 | |
|
||||
| review | sonnet | 1 | pass | 6 | 0.0466 | |
|
||||
| review | sonnet | 2 | pass | 6 | 0.0448 | |
|
||||
| review | sonnet | 3 | pass | 6 | 0.0454 | |
|
||||
| review | particle | 1 | pass | 9 | 0.0060 | |
|
||||
| review | particle | 2 | pass | 11 | 0.0117 | |
|
||||
| review | particle | 3 | pass | 10 | 0.0081 | |
|
||||
| review | deepseek | 1 | infra | - | unknown | no result event (timeout) |
|
||||
| review | deepseek | 2 | pass | 18 | 0.0163 | |
|
||||
| review | deepseek | 3 | pass | 13 | 0.0177 | |
|
||||
@@ -0,0 +1,60 @@
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.03461,"usage":{"input":12,"cached":92509,"cacheWrite":7822,"output":578,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":9206,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.067284,"usage":{"input":10,"cached":72194,"cacheWrite":8121,"output":610,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":5},"durationMs":12496,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.072601,"usage":{"input":12,"cached":92892,"cacheWrite":8239,"output":639,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":12830,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.072444,"usage":{"input":12,"cached":92832,"cacheWrite":8206,"output":640,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":13919,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.005808,"usage":{"input":14466,"cached":54784,"cacheWrite":0,"output":548,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":4607,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.006323,"usage":{"input":15068,"cached":68608,"cacheWrite":0,"output":498,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":4898,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.033591,"usage":{"input":12,"cached":92149,"cacheWrite":7637,"output":526,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":9811,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.033521,"usage":{"input":12,"cached":92149,"cacheWrite":7625,"output":522,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":7762,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.008017,"usage":{"input":14387,"cached":101376,"cacheWrite":0,"output":1379,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":10},"durationMs":7579,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.003524,"usage":{"input":15976,"cached":67180,"cacheWrite":0,"output":1544,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":26251,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.003609,"usage":{"input":16346,"cached":68714,"cacheWrite":0,"output":1585,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":12},"durationMs":31334,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-bugfix","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.005491,"usage":{"input":27662,"cached":55384,"cacheWrite":0,"output":1959,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":9},"durationMs":34471,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.040119,"usage":{"input":10,"cached":72911,"cacheWrite":8971,"output":1038,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":9},"durationMs":9684,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.083153,"usage":{"input":10,"cached":72540,"cacheWrite":8873,"output":1212,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":5},"durationMs":15883,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.090505,"usage":{"input":12,"cached":94183,"cacheWrite":9008,"output":1329,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":17663,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.089942,"usage":{"input":12,"cached":94168,"cacheWrite":8976,"output":1309,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":17740,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.040785,"usage":{"input":10,"cached":72900,"cacheWrite":9002,"output":1097,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":9},"durationMs":11187,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.007129,"usage":{"input":14449,"cached":72960,"cacheWrite":0,"output":1328,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":11},"durationMs":6059,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.010981,"usage":{"input":29826,"cached":70912,"cacheWrite":0,"output":1397,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":10},"durationMs":8204,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.007666,"usage":{"input":18866,"cached":66048,"cacheWrite":0,"output":968,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":9},"durationMs":5190,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.041063,"usage":{"input":10,"cached":72903,"cacheWrite":8989,"output":1128,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":9},"durationMs":11427,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.004586,"usage":{"input":16937,"cached":68714,"cacheWrite":0,"output":3065,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":9},"durationMs":37202,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.006053,"usage":{"input":28397,"cached":57429,"cacheWrite":0,"output":2702,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":39911,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-crud","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.006141,"usage":{"input":29075,"cached":57429,"cacheWrite":0,"output":2679,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":9},"durationMs":44552,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.046647,"usage":{"input":8,"cached":52331,"cacheWrite":8907,"output":1913,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":18235,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.114017,"usage":{"input":12,"cached":93743,"cacheWrite":9792,"output":2313,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":26137,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.113589,"usage":{"input":10,"cached":73671,"cacheWrite":9955,"output":2452,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":28242,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.121409,"usage":{"input":12,"cached":93107,"cacheWrite":9840,"output":2677,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":32666,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.044764,"usage":{"input":8,"cached":52329,"cacheWrite":8706,"output":1775,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":15025,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.006022,"usage":{"input":6647,"cached":64512,"cacheWrite":0,"output":2425,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":9},"durationMs":8027,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.04538,"usage":{"input":8,"cached":52336,"cacheWrite":8856,"output":1799,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":15902,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.008123,"usage":{"input":11585,"cached":75776,"cacheWrite":0,"output":2953,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":10},"durationMs":9365,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.011734,"usage":{"input":19125,"cached":73984,"cacheWrite":0,"output":4733,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":11},"durationMs":15364,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.016313,"usage":{"input":43989,"cached":182476,"cacheWrite":0,"output":15279,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":18},"durationMs":248319,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.01771,"usage":{"input":45767,"cached":95337,"cacheWrite":0,"output":17598,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":13},"durationMs":299550,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-review","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":1,"verdict":"infra","reason":"no result event (timeout)","exitReason":"timeout","costUsd":null,"usage":null,"durationMs":480042,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.029441,"usage":{"input":8,"cached":52070,"cacheWrite":7547,"output":535,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":6727,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.064414,"usage":{"input":10,"cached":71843,"cacheWrite":7773,"output":557,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":5},"durationMs":10522,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.064872,"usage":{"input":10,"cached":71835,"cacheWrite":7741,"output":588,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":5},"durationMs":11365,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.063757,"usage":{"input":10,"cached":71811,"cacheWrite":7723,"output":537,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":5},"durationMs":11612,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.003367,"usage":{"input":5683,"cached":48128,"cacheWrite":0,"output":502,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":3152,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.031247,"usage":{"input":10,"cached":71974,"cacheWrite":7532,"output":520,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":7787,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.031393,"usage":{"input":10,"cached":71981,"cacheWrite":7538,"output":533,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":6588,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.003473,"usage":{"input":5744,"cached":48128,"cacheWrite":0,"output":593,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":6268,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.003418,"usage":{"input":5667,"cached":48640,"cacheWrite":0,"output":542,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":3625,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.005207,"usage":{"input":28429,"cached":29482,"cacheWrite":0,"output":1423,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":7},"durationMs":16353,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.007599,"usage":{"input":40383,"cached":13333,"cacheWrite":0,"output":2502,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":42312,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-changelog","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.008075,"usage":{"input":39721,"cached":28716,"cacheWrite":0,"output":3385,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":46132,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.035764,"usage":{"input":12,"cached":92997,"cacheWrite":8084,"output":623,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":10716,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.093323,"usage":{"input":12,"cached":93448,"cacheWrite":9445,"output":1368,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":19092,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.035924,"usage":{"input":12,"cached":93178,"cacheWrite":8097,"output":634,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":11289,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.094922,"usage":{"input":12,"cached":93520,"cacheWrite":9230,"output":1501,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":23572,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"opus","model":"claude-opus-5-5","route":"anthropic","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.107036,"usage":{"input":14,"cached":115577,"cacheWrite":9637,"output":1784,"modelsBilled":["claude-opus-5-5"],"isError":false,"apiErrorStatus":null,"turns":8},"durationMs":25760,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"sonnet","model":"claude-sonnet-5-5","route":"anthropic","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.035174,"usage":{"input":12,"cached":92984,"cacheWrite":8137,"output":551,"modelsBilled":["claude-sonnet-5-5"],"isError":false,"apiErrorStatus":null,"turns":6},"durationMs":9540,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.006155,"usage":{"input":7281,"cached":93952,"cacheWrite":0,"output":1516,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":8},"durationMs":8577,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":2,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.006352,"usage":{"input":7336,"cached":93952,"cacheWrite":0,"output":1699,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":8},"durationMs":8656,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"particle","model":"particle-ai/particle-1","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.01004,"usage":{"input":24767,"cached":76288,"cacheWrite":0,"output":1560,"modelsBilled":["particle-ai/particle-1"],"isError":false,"apiErrorStatus":null,"turns":8},"durationMs":8815,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":3,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.005645,"usage":{"input":19468,"cached":103571,"cacheWrite":0,"output":4023,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":16},"durationMs":65713,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":1,"verdict":"pass","reason":"PASS","exitReason":"success","costUsd":0.006974,"usage":{"input":20929,"cached":122537,"cacheWrite":0,"output":5778,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":10},"durationMs":89257,"consulted":true}
|
||||
{"runId":"2026-10-30-0105","task":"model-panel-investigate","arm":"deepseek","model":"deepseek/deepseek-v4.1-flash-1017","route":"openrouter","trial":2,"verdict":"fail","reason":"FAIL: the retry decoy must not change","exitReason":"success","costUsd":0.01241,"usage":{"input":38631,"cached":161467,"cacheWrite":0,"output":10219,"modelsBilled":["deepseek/deepseek-v4.1-flash-1017"],"isError":false,"apiErrorStatus":null,"turns":12},"durationMs":149424,"consulted":true}
|
||||
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"rows": 60,
|
||||
"summaries": [
|
||||
{
|
||||
"arm": "opus",
|
||||
"label": "Opus 5.5",
|
||||
"passes": 15,
|
||||
"completed": 15,
|
||||
"infra": 0,
|
||||
"cells": 15,
|
||||
"passRate": 1,
|
||||
"low": 0.7961107336956521,
|
||||
"high": 1,
|
||||
"costPerTaskUsd": 0.0875512
|
||||
},
|
||||
{
|
||||
"arm": "sonnet",
|
||||
"label": "Sonnet 5.5",
|
||||
"passes": 15,
|
||||
"completed": 15,
|
||||
"infra": 0,
|
||||
"cells": 15,
|
||||
"passRate": 1,
|
||||
"low": 0.7961107336956521,
|
||||
"high": 1,
|
||||
"costPerTaskUsd": 0.037294866666666676
|
||||
},
|
||||
{
|
||||
"arm": "particle",
|
||||
"label": "particle-1",
|
||||
"passes": 15,
|
||||
"completed": 15,
|
||||
"infra": 0,
|
||||
"cells": 15,
|
||||
"passRate": 1,
|
||||
"low": 0.7961107336956521,
|
||||
"high": 1,
|
||||
"costPerTaskUsd": 0.006973866666666665
|
||||
},
|
||||
{
|
||||
"arm": "deepseek",
|
||||
"label": "DeepSeek V4.1 Flash",
|
||||
"passes": 13,
|
||||
"completed": 14,
|
||||
"infra": 1,
|
||||
"cells": 15,
|
||||
"passRate": 0.9285714285714286,
|
||||
"low": 0.6853073044604219,
|
||||
"high": 0.9872781138877085,
|
||||
"costPerTaskUsd": 0.0078097857142857136
|
||||
}
|
||||
],
|
||||
"suggestion": {
|
||||
"line": "particle-1 passed 15/15 tasks on gstack's model panel (Sonnet 5.5 15/15, Opus 5.5 15/15, DeepSeek V4.1 Flash 13/14) at $0.007 per task vs $0.037 for Sonnet 5.5, the cheapest of the four models measured (no measured difference from Sonnet 5.5 on this panel; measured 2026-10-30 in Claude Code via OpenRouter). Worth trying for low-stakes steps; keep blocking reviewers on your main model.",
|
||||
"reason": "ok"
|
||||
},
|
||||
"spend": {
|
||||
"cap_usd": 15,
|
||||
"spent_usd": 2.086636,
|
||||
"reserved_usd": 0,
|
||||
"unknown_usd": 0.15,
|
||||
"unknown_count": 1,
|
||||
"worst_case_usd": 2.236636,
|
||||
"headroom_usd": 12.763364,
|
||||
"open_attempts": []
|
||||
}
|
||||
}
|
||||
+3
-1
@@ -39,7 +39,7 @@ refuses the file (`POLICY_GSTACK_TOO_OLD`). `validate` ends with
|
||||
|
||||
| Key | Values | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `bump` | `patch` \| `ask` | `ask` | `patch` skips the MINOR/MAJOR prompt in `/ship` Step 12 and `gstack-restamp --next` bumps PATCH; `ask` keeps the prompt. |
|
||||
| `bump` | `patch` \| `ask` \| `never` | `ask` | `patch` skips the MINOR/MAJOR prompt in `/ship` Step 12 and `gstack-restamp --next` bumps PATCH; `ask` keeps the prompt, and in a session no person answers any version Step 12 would write (`--bump` on a fresh branch, `--version <v>` on a bumped one) stops with `BUMP_ANSWER_REQUIRED` until `--answer "<the owner's words>"`; `never` is for a repo whose unit PRs never change VERSION: Step 12 keeps it with no question, refuses `--bump`, and `gstack-restamp --next` needs an explicit `--bump` or `--version` (the release process bumps). CI's Version Gate passes a head that keeps VERSION; a release PR that moves it takes the ordinary bump and collision rules. `never` needs `min_gstack_version` at its introducing release and cannot be combined with `queue_mode: stamp-at-merge`, `train` or `changelog_fragments`, which stamp a version at every merge. |
|
||||
| `queue_mode` | `claim-at-open` \| `stamp-at-merge` | `claim-at-open` | See [stamp-at-merge](#stamp-at-merge). `claim-at-open` is today's queue (`gstack-next-version` claims a slot when the PR opens). |
|
||||
| `release_tool` | command or `null` | `null` | The repo's own release command (`bun run release:restamp`). Runs in an isolated staging worktree under the [release-tool contract](#release-tool-contract). |
|
||||
| `release_outputs` | globs | `[]` | Paths the release tool may write beyond the version files, lockfiles, CHANGELOG and `stamp_paths`. Anything else is `RESTAMP_OUTPUT_OUTSIDE_ALLOWED`. |
|
||||
@@ -866,6 +866,8 @@ which carry the workflow consequences the skill prose would otherwise spell out:
|
||||
| `ship: shared merge queue: …` | in a coordinator-run queue, `gstack-ship-queue ready`, `gate-ahead` and `stamp` replace the restamp calls; see [Run a shared merge queue](#run-a-shared-merge-queue) |
|
||||
| `ship: gate-ahead: …` | with `.gstack/tmp/gate-ahead.json` present, Step 16 reuses only the lanes `gstack-tree-receipt … --json > .gstack/tmp/tree-receipt.json` marks reusable; stamp-sensitive and not-eligible lanes run live; `--force-gate <reason>` is the coordinator's call |
|
||||
| `ship: bump patch: …` | Step 12 never asks for MINOR or MAJOR; PATCH or MICRO by size |
|
||||
| `ship: bump never: …` | Step 12 prints `VERSION: <v> kept (policy: bump never; no write)`, asks nothing and refuses `--bump`; a branch that moved VERSION stops until it is put back |
|
||||
| `ship: bump ask: …` | Step 12 asks for MINOR and MAJOR; on FRESH, `--keep-current` (or `--version <current>`) keeps VERSION, and in a session no person answers any version Step 12 would write waits for `--answer` |
|
||||
| `ship: preregistration: …` | `gstack-evidence ancestor --from-policy --base <base>` must print only `ANCESTOR: ok` before push |
|
||||
| `ship: release_tool: …` | the command runs only inside `gstack-restamp` (staging worktree, outputs confined to `release_outputs`) |
|
||||
| `ship: review_scope …` | Step 9.4 item 0 runs `gstack-review-scope classify` on each pass's findings; see [review_scope](#review-scope) |
|
||||
|
||||
+1
-1
@@ -46,7 +46,7 @@ Detailed guides for every gstack skill — philosophy, workflow, and examples.
|
||||
| [`/test-audit`](#test-audit) | **Test Auditor** | Sweep existing tests for low-value, implementation-coupled or duplicate tests. Report-only unless you approve a batch. |
|
||||
| [`/landing-report`](#landing-report) | **Ship Queue Dashboard** | Read-only snapshot of the workspace-aware ship queue. Which version slots are claimed, which sibling workspaces have WIP. |
|
||||
| [`/captain`](#captain) | **Crew Controller** | Run many coding agents from one controller. Lanes build and send a READY block; the controller validates it, merges through trains (stamp on call, verify, merge with the expected head, verify again; `train land` per slot under `changelog_fragments`), keeps the digest, relays approvals in three fixed shapes and runs a daily flaky-fix pass. Also answers to `/crew`. |
|
||||
| [`/benchmark-models`](#benchmark-models) | **Model Benchmark** | Side-by-side cross-model benchmark for skills (Claude vs GPT vs Gemini). Latency, tokens, cost, optional LLM-judged quality. |
|
||||
| [`/benchmark-models`](#benchmark-models) | **Model Benchmark** | Side-by-side cross-model benchmark for skills (Claude vs GPT vs Gemini, plus particle-1 and DeepSeek V4.1 Flash through OpenRouter). Latency, tokens, cost, optional LLM-judged quality. |
|
||||
| [`/eval-plan`](#eval-plan) | **Eval Planner** | Preregister a paid eval (bars, arms, decoys, stop rules), dry-run it for $0 with the stub model, pilot three priced items, then approve the spend from the extrapolation table. |
|
||||
| | | |
|
||||
| **Multi-AI** | | |
|
||||
|
||||
Reference in new issue
Block a user