mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
Garry Tanandcapy-ai[bot]
ab79e55119
* v1.91.70.0 feat(unattended): artifact contract, unattended session kind, findings-derived review log, gate list Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.72.0 feat(autoplan): unattended runner next|submit, pluggable outside voice, review checklists, spec-loop cap Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(b-runner): atomic lock creation in spend-ledger, judge bundle skips runner-only sections, fixture env spelling - lib/spend-ledger.ts: create the lock by linking a complete temp file so a contender never reads an empty lock and breaks a live owner (CI admitted 20 of a cap of 8 under contention); unreadable locks break only by mtime age. - test/helpers/workflow-judge-input.ts: exclude manifest runner_only sections (the B4 checklists) from the judge bundle; the judge rates the interactive workflow. test/workflow-judge-input.test.ts expects the same set. - test/helpers/autoplan-run-fixture.ts: spell the runner's session kind without the literal the free-suite environment scan rejects. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.74.0 feat(review): plan reality rows, sideways review and cso sweep, investigate recipes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: register the investigate carve case and map Reality rows to real headings in the review checklists Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: pin the corrected Reality rows obligation rows Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(investigate): use allowlisted placeholders in the flake and polluter fences; select the investigate carve case on PTY changes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
24 lines
1.3 KiB
Cheetah
24 lines
1.3 KiB
Cheetah
## Eval bisect recipe
|
|
|
|
A judged or scored metric regressed between two revisions. Bisect inputs, not
|
|
commits, and spend the paid judge last.
|
|
|
|
1. **Distinct revisions only.** `git log --oneline <good>..<bad> -- <prompt and
|
|
resolver paths>`; collapse commits that leave the expanded prompt bytes
|
|
identical (build the prompt with the actual builder at each revision and
|
|
compare hashes; a template edit that renders the same bytes is not a
|
|
candidate).
|
|
2. **A cached cheap scorer first.** Score every distinct revision with the
|
|
deterministic or cheap check (the structural judge, a token count, the
|
|
quality judge at its lowest tier) and cache results per
|
|
`(revision, case)` under `.gstack/tmp/eval-bisect/`. Bisect on the cheap
|
|
score; spend the paid judge only on the two revisions where the cheap scorer
|
|
disagrees with the reported regression.
|
|
3. **Detached runs.** Launch paid cases through the documented detached runner
|
|
(`bun run eval:bg:pr`), poll the log for `### gstack-detach EXIT=<code> ###`,
|
|
and keep the eval lock. Never retry a paid case; a failed launch is a launch
|
|
failure in the report, not a second sample.
|
|
4. **Report.** The first bad revision with its prompt diff, the cheap and paid
|
|
scores per revision (cached hits marked), the pre-registered panel size and
|
|
every trial, and the cases that were not run.
|