mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
Garry Tanandcapy-ai[bot]
db745675bd
* refactor(eval-history): share pass-rate history loading as loadPassRateHistory() eval:pass-rates' main block loaded local dirs or the weekly trial-outcomes artifacts inline. Move that into an exported function returning records, scope and fetch errors, so the off-ship sweep reads the same history the same way. Behavior unchanged. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-measure): the measurement bar and the weekly off-ship sweep CLI Decide each diagnostic measurement with one bar (scripts/lib/measure-bar.ts): MEETS at 9/10; MEETS-qualified at 8/10 when every red is a mechanical provider error, judge noise or a cited model miss; EXTEND at 7/10 (or an unqualified 8/10) for one more batch on identical inputs, decided once on the pooled 20; BELOW otherwise and for any timeout, hang, regression or fixable cause. Behavior uses 12 trials (11/10/9; 22/20 of 24); judge counts median-panel outputs. A batch with more than 30% provider-evidence failures is void and redispatched once. Below-strict reds nobody classified return needs-classify. ship-measure gains classify (append-only records; provider refused without evidence), decide and extend (refuses a changed identity, names the field, never a third batch), exit codes 5 and 6, and a pooling identity (frozen tree, runner, CLI, model, CI image, Bun, policy and harness versions). ship-measure sweep ranks main's gate cases by their cost to the all-green probability from eval:pass-rates history, measures the top K with the same runner and bar under a weekly cap shared with the last 7 days of sweep reports (ship_measure_sweep_cases, ship_measure_sweep_budget_usd), and writes a report with the predicted all-green probability before and after. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship): the measure loop decides by the bar's exit codes Step 4 branches on MEETS/MEETS-qualified, needs-classify (with the qualifying and fix-round classes), EXTEND once and BELOW; the rest of the loop prose is tightened to offset most of the growth. ship's union size cap moves from 1.495 to 1.4975 (measured 281,058 / 187,706 = 1.4973). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci: weekly eval-sweep workflow on main Mondays 12:00 UTC and on dispatch (k, cap_usd, dry_run): runs ship-measure sweep in the published CI image by digest with the census's paid-eval setup, downloads the last 7 days of sweep reports first so the weekly cap holds across runs, uploads the report and captures, and writes the job summary. Read-only grants; it never pushes or opens a pull request. One sweep at a time repository-wide, never cancelled mid-spend. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the measurement bar table and the weekly sweep Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: v1.91.33.0 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation" This reverts commitf4c58f2869. * ci: classify eval-sweep.yml, the sweep CLI and actionlint labels as free-only PR inputs None is read by a PR-gate case, so a change to them selects no paid case instead of restoring the full gate (test/free-fixtures.test.ts workflow inventory). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): the selection preflight accepts name-mode cases A case registered beside others in its file lists its trial shard as <file>#<id>~t1, not <file> (trial 1), so the preflight refused every name-mode case (for example ship-exploratory-late-input) before any paid trial. Accept both forms. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation" This reverts commit9619734307. * test: record the test-pr-profile.ts change as non-behavioral for HARNESS_VERSION Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): the name-mode preflight fix Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): a partly funded sweep row shows the trials that ran Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(sweep): the fake single-case command is a Bun script, so it runs on Windows The Windows free lane runs test/ship-measure-sweep.test.ts; CreateProcess cannot exec a shebang shell stub, so every fake trial errored and the cap stopped the sweep after one trial. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(plan-mode-no-op): the CEO sub-test reviews a fixed pasted plan Without a plan, plan-ceo-review reviewed the checkout's branch diff, so its pre-question work (audit and landscape searches) grew with whatever PR the gate ran on. On this PR it timed out at 300s in 3 of 10 trials while still researching the diff (ship-measure baseline 7/10). With a fixed small plan, the same way the devex sub-test already runs, it measured 10/10 (CEO sub-test 83-255s). No budget or threshold changed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): each diagnostic trial gets its own EVALS_RUN_ID Evidence-retaining cases (functional QA, docs faults) refuse to start without EVALS_RUN_ID, which only CI sets, so measuring them locally ran 0/10 sessions. qa-functional-cli-report, red once in this PR's gate, measured 10/10 once the trials started ($11.13). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
65 lines
4.2 KiB
Cheetah
65 lines
4.2 KiB
Cheetah
## Measure-then-fix loop (a red paid eval or a failed free-suite shard)
|
|
|
|
Every rerun here is **diagnostic, not a verdict**: it never changes the recorded
|
|
verdict of the run that failed, its artifacts are labeled `diagnostic` under
|
|
`.context/ship-measure/` (never the eval history), and the lane verdict comes
|
|
only from the one full gate run at the end. Never raise a budget, lower a
|
|
threshold, add a verdict retry or skip a case to get a pass.
|
|
`ship-measure` below means `bun run ~/.claude/skills/gstack/scripts/ship-measure.ts`.
|
|
|
|
**Failed free-suite shard (Step 5).** Before calling the failure pre-existing or
|
|
pushing, rerun that shard's exact file list (in run order) with
|
|
`ship-measure free --files <f1,f2,...>` (gstack local runs: `--shard <i>`): 10
|
|
reruns, rerun 1 alone as the baseline and the rest labeled parallel, each with
|
|
its own HOME and state root, the flaky retry off, under a 10-minute cap.
|
|
`ship_rerun_backend=ubicloud` with `UBICLOUD_API_KEY` set runs them on one
|
|
Ubicloud VM. If the runner cannot take a file list, rerun the whole failing
|
|
command 10 times. Report the pass rate, the completed count (for example 6 of 10
|
|
completed at the cap) and one failure capture, then continue the triage.
|
|
|
|
**Red paid eval (Step 6).** For each red case:
|
|
|
|
1. **Classify.** Read the red line's cause, evidence and Expected/Received
|
|
(`failure_cause`, `failure_detail`; gstack: `bun run eval:pass-rates --run <run> --case <id>`).
|
|
Name it: product change from this branch, detector or actor defect, fixture
|
|
defect, provider event, or model miss. A detector fix gets a free replay test
|
|
before any paid trial. A red that is infrastructure (provider outage, runner
|
|
loss) may skip measurement: `ship-measure skip --case <id> --reason "<why>"`
|
|
labels it `unmeasured` in the PR body. It never counts as a pass.
|
|
2. **Show the plan.** Print `ship-measure table` (trials, bar and budget per
|
|
kind, from the `ship_measure_*` gstack config keys) and the estimate:
|
|
per-trial cost from pass-rate history times trials, or "estimate unknown".
|
|
3. **Measure.** `ship-measure measure --case <id> --round baseline [--cost-per-trial <usd>]`
|
|
runs the case alone on the current head at its kind's count, in parallel,
|
|
through the project's documented single-case eval command,
|
|
`--command '<command> {case}'` (found the way Step 4 finds test commands;
|
|
gstack needs none and accepts judge ids). With no documented command, ask
|
|
once and record the answer in the report. Exit 2 needs approval: ask once
|
|
with its reason, then rerun with `--approved`.
|
|
4. **Decide by the exit code** (MEETS 9/10, MEETS-qualified 8/10 with every
|
|
red qualifying, EXTEND 7/10; behavior 11/12, 10/12, 9/12):
|
|
- 0: MEETS or MEETS-qualified; done.
|
|
- 5: needs-classify. Record each listed red from its capture:
|
|
`ship-measure classify --case <id> --round <r> --trials <2,5> --class <class> --evidence "<path:line>"`.
|
|
Qualifying: `provider` (only on the trial's own provider error),
|
|
`judge-noise`, `model-miss` (cite the line). Fix round: `timeout`,
|
|
`hang`, `regression`, `fixable`. `unclassified`: reviewed, no evidence.
|
|
- 6: EXTEND. `ship-measure extend --case <id> --round <r>` once, on the
|
|
unchanged tree; never a third batch.
|
|
- 1: BELOW, a fix round.
|
|
5. **Fix at the cause.** Root-cause the failed trials from their captures in
|
|
`.context/ship-measure/<case>/<round>/tNN/` and change the cause.
|
|
6. **Re-measure.** `ship-measure measure --case <id> --round round-<n> --fix "<cause and change>"`,
|
|
then item 4. Call a case fixed only with a named causal change and every
|
|
failure in the closing measurement explained;
|
|
report "observed k/n after fix at <cause>", never a proven rate.
|
|
7. **Stop.** Exit 3 is a named red (round limit, budget, or two void
|
|
batches): stop, do not push, and report `ship-measure report` with every trial.
|
|
|
|
**Gate once.** When every red case measures at or above target, run the full
|
|
gate once (Step 6 item 2). When the project's CI runs that gate on push (gstack:
|
|
the pull request's `evals` workflow), the push is the gate run: push and read
|
|
that run, and do not also start the gate command (gstack's `bun run eval:bg:pr`
|
|
on a pushed head dispatches a second paid run). Put the `ship-measure report` table (estimated
|
|
and actual spend per case) in the PR body.
|