mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
v1.91.134.0 feat(evals): readiness-gated paid dispatch (R1), verdict reuse (R2a/R2), receipt store that restores (#3262)
* feat(evals): readiness-gated paid dispatch (R1), restamp-only gate reuse (R2a), and a receipt store that restores Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(evals): R2 verdict reuse across main moves; review fixes (newer-run classification, receipt parser, manual acceptance) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(evals): R2 own-diff signature covers release-output content and CHANGELOG deletions and preamble Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(evals): R2 normalises release outputs by stamp pattern only; compares stamp-file modes; unreadable blobs refuse Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: stamp v1.91.134.0 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(evals): autoplan-guard-pty actor continues through the phase boundary; name PHASE_BOUNDARY_STOP Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
fd13cf25e8
commit
a2a6cabfa6
21 files changed
+1758
-84
No files matched your search
@@ -1126,7 +1126,7 @@ and `test:health` both say so.
|
||||
through today's selector and its shadow, and exits 1 on any MISS: a failed
|
||||
gate case the shadow would not run. `test/exercised-map.test.ts` pins zero.
|
||||
|
||||
### PR receipts, recovery and debounce
|
||||
### PR receipts, recovery, readiness and verdict reuse
|
||||
|
||||
Passes and failures of single-trial shards are written as receipts at the
|
||||
execution boundary, so a cancelled run's finished work survives. The
|
||||
@@ -1139,13 +1139,65 @@ unresolved run. `plan-slices` turns reuse off when recovery degraded, the
|
||||
checkpoint is missing or a cancelled run is still unresolved; recovery itself
|
||||
never fails the run, and the job summary names the reason (rate limit with its
|
||||
reset time, `gh auth status`, the 60-second budget). The whole store is saved by
|
||||
`slices-report`.
|
||||
`slices-report`. Both the restore and the save run on GitHub-hosted runners: an
|
||||
Ubicloud runner sends `actions/cache` to Ubicloud's transparent cache, which a
|
||||
GitHub-hosted restore never sees (until GSTA-48 every restore missed).
|
||||
|
||||
On a `synchronize` push, when another run for the same PR started in the last 15
|
||||
minutes, the debounce job waits 90 seconds before planning; a run that outlives
|
||||
the wait reports "superseded by <sha>". PR labels: `evals-no-debounce` skips the
|
||||
wait and `evals-fresh` turns receipt reuse off. Labels apply from the next push,
|
||||
because adding a label does not start `evals.yml`.
|
||||
**Readiness (R1).** Paid slices for a PR head start only when the PR has been
|
||||
quiet for the window (repository variable `EVALS_QUIET_MINUTES`, default 10,
|
||||
measured from the run's creation) and Free Tests has concluded success on the
|
||||
same head SHA. The `readiness` job (`scripts/eval-readiness.ts`) sleeps out the
|
||||
window with no API calls, since a newer push cancels the run anyway, then polls
|
||||
Free Tests every minute for up to 80 minutes. Free Tests red on the head fails
|
||||
the run without paid work; re-run Free Tests, then re-run this workflow (a
|
||||
re-run keeps the creation time, so it does not wait out the window again). A
|
||||
head that is no longer the PR's head reports "superseded by <sha>" and plans
|
||||
nothing. Selection, trials and verdict rules do not change. Version Gate is not
|
||||
an input, so a stamp push waits only for its own Free Tests and always gets a
|
||||
gate. Dispatches (`eval:bg`, the census) never wait.
|
||||
|
||||
**Verdict reuse (R2a restamp-only, R2 main moved; a gate edit).** A PR head
|
||||
reuses the verdict of the newest executed gate of the same PR instead of
|
||||
running one only when all of these hold. Every newer run of the PR either bound
|
||||
a verdict or provably ran no paid slice (superseded, readiness-blocked, a reuse,
|
||||
or cancelled before a slice finished); a newer run that is still running,
|
||||
finished a slice, or holds a `paid-slice-*` artifact without a bound verdict
|
||||
blocks reuse, so a newer red or partial result is never skipped. The gate is
|
||||
GREEN with reconcile exit 0, a successful run and no manually accepted
|
||||
(unscored) case; it ran the PR profile on a different head. Then either:
|
||||
|
||||
- **R2a, restamp-only:** it ran on the same base (the tested merge commit's
|
||||
first parent), the tree receipt (`lib/tree-receipt.ts` under the base's ship
|
||||
policy) prints `TREE: same-modulo-stamps`, and every differing file is the
|
||||
version file, `package.json`, the CHANGELOG or a release output (the agents
|
||||
digest); or
|
||||
- **R2, main moved:** the base moved, and the PR's own diff from its
|
||||
merge-base, excluding stamp files, is byte-identical to the gated head's
|
||||
(`git diff --binary --full-index`, so pre-image blob ids are compared and a
|
||||
main change to a file the PR also changes is a difference; `package.json`
|
||||
compared without its version; release outputs with each revision's version
|
||||
replaced; the CHANGELOG by its preamble and the sections the PR added and
|
||||
removed, dropping only version or `[Unreleased]` headings).
|
||||
|
||||
Finally this head's own plan must have the same selection fingerprint (cases,
|
||||
statuses, trial plans, profile), so a main move that changes what the PR gate
|
||||
selects runs the full gate. `recover-receipts` proposes the candidate from
|
||||
base-ref code (`scripts/gate-reuse.ts find`), `plan-slices` confirms the
|
||||
selection, `eval-slices` is skipped, and the report and PR comment say
|
||||
`PASS (reused: restamp-only, R2a)` or `PASS (reused: main moved, R2)` with
|
||||
the reused run, its head, why, and the receipt digest or own-diff signature.
|
||||
Free Tests and every other check still run on the new head. Executed runs
|
||||
upload a `gate-binding-a<attempt>` artifact (base, verdict, reconcile exit,
|
||||
manual count, selection); reused runs bind nothing, so reuse never chains off a
|
||||
reuse. Any other case (a changed own diff, a non-stamp byte including CHANGELOG
|
||||
entry text or a file mode, a missing binding, receipt or merge-base, a gh or git
|
||||
error, `evals-fresh`) runs the full gate. For a restamp to qualify, write the
|
||||
CHANGELOG entry before the gated push (under `## [Unreleased]` or the
|
||||
tentative version) so the stamp only re-heads it.
|
||||
|
||||
PR labels: `evals-no-debounce` sets the quiet window to 0 (Free Tests is still
|
||||
required) and `evals-fresh` turns receipt reuse and verdict reuse off. Labels
|
||||
apply from the next push, because adding a label does not start `evals.yml`.
|
||||
|
||||
## Free suite runner, judge reuse and engine skips
|
||||
|
||||
|
||||
Reference in new issue
Block a user