mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-02 15:04:37 +01:00
Garry Tanandcapy-ai[bot]
4c6fa89cdc
* security: isolate every gstack Bun entry point from the audited checkout's .env and bunfig.toml Bun auto-loads .env, .env.local and .env.<NODE_ENV> from its cwd and runs a cwd bunfig.toml preload before the program starts. gstack runs inside the repo it audits, so a committed .env could steer state roots, the claude binary or spend caps, and a bunfig preload ran arbitrary code in every bin and hook. Every bun shebang, shell and skill caller, TypeScript spawn and compiled build now disables both (--no-env-file --config=/dev/null, bunIsolationArgs(), --no-compile-autoload-dotenv/bunfig), with a ratchet and hostile-fixture regressions. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: name the directly executed scripts in the isolation ratchet Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: follow the isolation flags in touchfiles, copy lists, recipe assertions and the harness pin Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: keep browse/src/bun-child-config.ts unchanged so the native cookie qualification still binds Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(windows): the empty bunfig outlives its process so the timeline hook's detached repair child can read it Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.159.0 security: gstack's Bun entry points ignore the audited repo's .env and bunfig.toml (S1) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: plan-mode render pattern accepts the isolation flags Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
65 lines
4.3 KiB
Cheetah
65 lines
4.3 KiB
Cheetah
## Measure-then-fix loop (a red paid eval or a failed free-suite shard)
|
|
|
|
Every rerun here is **diagnostic, not a verdict**: it never changes the recorded
|
|
verdict of the run that failed, its artifacts are labeled `diagnostic` under
|
|
`.context/ship-measure/` (never the eval history), and the lane verdict comes
|
|
only from the one full gate run at the end. Never raise a budget, lower a
|
|
threshold, add a verdict retry or skip a case to get a pass.
|
|
`ship-measure` below means `bun --no-env-file --config=/dev/null run ~/.claude/skills/gstack/scripts/ship-measure.ts`.
|
|
|
|
**Failed free-suite shard (Step 5).** Before calling the failure pre-existing or
|
|
pushing, rerun that shard's exact file list (in run order) with
|
|
`ship-measure free --files <f1,f2,...>` (gstack local runs: `--shard <i>`): 10
|
|
reruns, rerun 1 alone as the baseline and the rest labeled parallel, each with
|
|
its own HOME and state root, the flaky retry off, under a 10-minute cap.
|
|
`ship_rerun_backend=ubicloud` with `UBICLOUD_API_KEY` set runs them on one
|
|
Ubicloud VM. If the runner cannot take a file list, rerun the whole failing
|
|
command 10 times. Report the pass rate, the completed count (for example 6 of 10
|
|
completed at the cap) and one failure capture, then continue the triage.
|
|
|
|
**Red paid eval (Step 6).** For each red case:
|
|
|
|
1. **Classify.** Read the red line's cause, evidence and Expected/Received
|
|
(`failure_cause`, `failure_detail`; gstack: `bun run eval:pass-rates --run <run> --case <id>`).
|
|
Name it: product change from this branch, detector or actor defect, fixture
|
|
defect, provider event, or model miss. A detector fix gets a free replay test
|
|
before any paid trial. A red that is infrastructure (provider outage, runner
|
|
loss) may skip measurement: `ship-measure skip --case <id> --reason "<why>"`
|
|
labels it `unmeasured` in the PR body. It never counts as a pass.
|
|
2. **Show the plan.** Print `ship-measure table` (trials, bar and budget per
|
|
kind, from the `ship_measure_*` gstack config keys) and the estimate:
|
|
per-trial cost from pass-rate history times trials, or "estimate unknown".
|
|
3. **Measure.** `ship-measure measure --case <id> --round baseline [--cost-per-trial <usd>]`
|
|
runs the case alone on the current head at its kind's count, in parallel,
|
|
through the project's documented single-case eval command,
|
|
`--command '<command> {case}'` (found the way Step 4 finds test commands;
|
|
gstack needs none and accepts judge ids). With no documented command, ask
|
|
once and record the answer in the report. Exit 2 needs approval: ask once
|
|
with its reason, then rerun with `--approved`.
|
|
4. **Decide by the exit code** (MEETS 9/10, MEETS-qualified 8/10 with every
|
|
red qualifying, EXTEND 7/10; behavior 11/12, 10/12, 9/12):
|
|
- 0: MEETS or MEETS-qualified; done.
|
|
- 5: needs-classify. Record each listed red from its capture:
|
|
`ship-measure classify --case <id> --round <r> --trials <2,5> --class <class> --evidence "<path:line>"`.
|
|
Qualifying: `provider` (only on the trial's own provider error),
|
|
`judge-noise`, `model-miss` (cite the line). Fix round: `timeout`,
|
|
`hang`, `regression`, `fixable`. `unclassified`: reviewed, no evidence.
|
|
- 6: EXTEND. `ship-measure extend --case <id> --round <r>` once, on the
|
|
unchanged tree; never a third batch.
|
|
- 1: BELOW, a fix round.
|
|
5. **Fix at the cause.** Root-cause the failed trials from their captures in
|
|
`.context/ship-measure/<case>/<round>/tNN/` and change the cause.
|
|
6. **Re-measure.** `ship-measure measure --case <id> --round round-<n> --fix "<cause and change>"`,
|
|
then item 4. Call a case fixed only with a named causal change and every
|
|
failure in the closing measurement explained;
|
|
report "observed k/n after fix at <cause>", never a proven rate.
|
|
7. **Stop.** Exit 3 is a named red (round limit, budget, or two void
|
|
batches): stop, do not push, and report `ship-measure report` with every trial.
|
|
|
|
**Gate once.** When every red case measures at or above target, run the full
|
|
gate once (Step 6 item 2). When the project's CI runs that gate on push (gstack:
|
|
the pull request's `evals` workflow), the push is the gate run: push and read
|
|
that run, and do not also start the gate command (gstack's `bun run eval:bg:pr`
|
|
on a pushed head dispatches a second paid run). Put the `ship-measure report` table (estimated
|
|
and actual spend per case) in the PR body.
|