0 Fork 0

mirror of https://github.com/garrytan/gstack.git synced 2026-11-02 15:04:37 +01:00

Files

T

Garry Tanandcapy-ai[bot] db745675bd v1.91.33.0 feat(ship): measurement bar with explained-red and pooled decisions, weekly flaky-test sweep on main (#3059 )

* refactor(eval-history): share pass-rate history loading as loadPassRateHistory()

eval:pass-rates' main block loaded local dirs or the weekly trial-outcomes
artifacts inline. Move that into an exported function returning records,
scope and fetch errors, so the off-ship sweep reads the same history the
same way. Behavior unchanged.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(ship-measure): the measurement bar and the weekly off-ship sweep CLI

Decide each diagnostic measurement with one bar (scripts/lib/measure-bar.ts):
MEETS at 9/10; MEETS-qualified at 8/10 when every red is a mechanical
provider error, judge noise or a cited model miss; EXTEND at 7/10 (or an
unqualified 8/10) for one more batch on identical inputs, decided once on
the pooled 20; BELOW otherwise and for any timeout, hang, regression or
fixable cause. Behavior uses 12 trials (11/10/9; 22/20 of 24); judge counts
median-panel outputs. A batch with more than 30% provider-evidence failures
is void and redispatched once. Below-strict reds nobody classified return
needs-classify.

ship-measure gains classify (append-only records; provider refused without
evidence), decide and extend (refuses a changed identity, names the field,
never a third batch), exit codes 5 and 6, and a pooling identity (frozen
tree, runner, CLI, model, CI image, Bun, policy and harness versions).

ship-measure sweep ranks main's gate cases by their cost to the all-green
probability from eval:pass-rates history, measures the top K with the same
runner and bar under a weekly cap shared with the last 7 days of sweep
reports (ship_measure_sweep_cases, ship_measure_sweep_budget_usd), and
writes a report with the predicted all-green probability before and after.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(ship): the measure loop decides by the bar's exit codes

Step 4 branches on MEETS/MEETS-qualified, needs-classify (with the
qualifying and fix-round classes), EXTEND once and BELOW; the rest of the
loop prose is tightened to offset most of the growth. ship's union size
cap moves from 1.495 to 1.4975 (measured 281,058 / 187,706 = 1.4973).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ci: weekly eval-sweep workflow on main

Mondays 12:00 UTC and on dispatch (k, cap_usd, dry_run): runs
ship-measure sweep in the published CI image by digest with the census's
paid-eval setup, downloads the last 7 days of sweep reports first so the
weekly cap holds across runs, uploads the report and captures, and writes
the job summary. Read-only grants; it never pushes or opens a pull request.
One sweep at a time repository-wide, never cancelled mid-spend.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: the measurement bar table and the weekly sweep

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: v1.91.33.0

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation"

This reverts commit f4c58f2869.

* ci: classify eval-sweep.yml, the sweep CLI and actionlint labels as free-only PR inputs

None is read by a PR-gate case, so a change to them selects no paid case
instead of restoring the full gate (test/free-fixtures.test.ts workflow
inventory).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ship-measure): the selection preflight accepts name-mode cases

A case registered beside others in its file lists its trial shard as
<file>#<id>~t1, not <file> (trial 1), so the preflight refused every
name-mode case (for example ship-exploratory-late-input) before any paid
trial. Accept both forms.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation"

This reverts commit 9619734307.

* test: record the test-pr-profile.ts change as non-behavioral for HARNESS_VERSION

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(changelog): the name-mode preflight fix

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ship-measure): a partly funded sweep row shows the trials that ran

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(sweep): the fake single-case command is a Bun script, so it runs on Windows

The Windows free lane runs test/ship-measure-sweep.test.ts; CreateProcess
cannot exec a shebang shell stub, so every fake trial errored and the cap
stopped the sweep after one trial.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(plan-mode-no-op): the CEO sub-test reviews a fixed pasted plan

Without a plan, plan-ceo-review reviewed the checkout's branch diff, so its
pre-question work (audit and landscape searches) grew with whatever PR the
gate ran on. On this PR it timed out at 300s in 3 of 10 trials while still
researching the diff (ship-measure baseline 7/10). With a fixed small plan,
the same way the devex sub-test already runs, it measured 10/10 (CEO
sub-test 83-255s). No budget or threshold changed.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ship-measure): each diagnostic trial gets its own EVALS_RUN_ID

Evidence-retaining cases (functional QA, docs faults) refuse to start
without EVALS_RUN_ID, which only CI sets, so measuring them locally ran
0/10 sessions. qa-functional-cli-report, red once in this PR's gate,
measured 10/10 once the trials started ($11.13).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

2026-10-07 14:00:06 +00:00

1.7 KiB

Raw Blame History

gstack digest v1.91.33.0 — regenerate/re-copy after upgrading gstack

Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, reviews, and evals on top of these rules.

Ethos

Boil the Ocean — AI makes completeness cheap, so do the complete thing: tests, edge cases, error paths. Shortcuts need an explicit, recorded decision.
Search Before Building — know what exists before deciding what to build. Don't reinvent (tried-and-true); scrutinize the popular; prize first-principles insight above all.
User Sovereignty — models recommend, the user decides. Cross-model agreement is signal, never permission. Ask before changing the user's stated direction.
Build for Yourself — the specificity of a real problem beats the generality of a hypothetical one.

The reuse ladder

Before writing new code, stop at the first rung that holds:

A helper, util, or pattern already in this repo.
The standard library.
A native platform feature (CSS over JS, DB constraint over app code).
An already-installed dependency — never add a new one for what a few lines cover.

Then build the complete version of what remains. Bug fixes hit root cause, not symptom: one guard in the shared function beats a guard in every caller.

Voice

Direct, concrete, builder-to-builder. Name the file, function, command, and user-visible impact. Short paragraphs; end with what to do. No filler, no corporate tone, no AI vocabulary.

Full gstack

Clone https://github.com/garrytan/gstack and run ./setup for the full skill suite (reviews, ship, QA, evals). This digest is generated — edit scripts/gen-agents-digest.ts, not this file.

1.7 KiB Raw Blame History