mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 14:06:50 +01:00
main
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Garry Tanandcapy-ai[bot]
|
db745675bd |
v1.91.33.0 feat(ship): measurement bar with explained-red and pooled decisions, weekly flaky-test sweep on main (#3059)
* refactor(eval-history): share pass-rate history loading as loadPassRateHistory() eval:pass-rates' main block loaded local dirs or the weekly trial-outcomes artifacts inline. Move that into an exported function returning records, scope and fetch errors, so the off-ship sweep reads the same history the same way. Behavior unchanged. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-measure): the measurement bar and the weekly off-ship sweep CLI Decide each diagnostic measurement with one bar (scripts/lib/measure-bar.ts): MEETS at 9/10; MEETS-qualified at 8/10 when every red is a mechanical provider error, judge noise or a cited model miss; EXTEND at 7/10 (or an unqualified 8/10) for one more batch on identical inputs, decided once on the pooled 20; BELOW otherwise and for any timeout, hang, regression or fixable cause. Behavior uses 12 trials (11/10/9; 22/20 of 24); judge counts median-panel outputs. A batch with more than 30% provider-evidence failures is void and redispatched once. Below-strict reds nobody classified return needs-classify. ship-measure gains classify (append-only records; provider refused without evidence), decide and extend (refuses a changed identity, names the field, never a third batch), exit codes 5 and 6, and a pooling identity (frozen tree, runner, CLI, model, CI image, Bun, policy and harness versions). ship-measure sweep ranks main's gate cases by their cost to the all-green probability from eval:pass-rates history, measures the top K with the same runner and bar under a weekly cap shared with the last 7 days of sweep reports (ship_measure_sweep_cases, ship_measure_sweep_budget_usd), and writes a report with the predicted all-green probability before and after. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship): the measure loop decides by the bar's exit codes Step 4 branches on MEETS/MEETS-qualified, needs-classify (with the qualifying and fix-round classes), EXTEND once and BELOW; the rest of the loop prose is tightened to offset most of the growth. ship's union size cap moves from 1.495 to 1.4975 (measured 281,058 / 187,706 = 1.4973). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci: weekly eval-sweep workflow on main Mondays 12:00 UTC and on dispatch (k, cap_usd, dry_run): runs ship-measure sweep in the published CI image by digest with the census's paid-eval setup, downloads the last 7 days of sweep reports first so the weekly cap holds across runs, uploads the report and captures, and writes the job summary. Read-only grants; it never pushes or opens a pull request. One sweep at a time repository-wide, never cancelled mid-spend. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the measurement bar table and the weekly sweep Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: v1.91.33.0 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation" This reverts commit |