v1.91.371.0 ci(evals): a pr-summary run skips the readiness wait it never used (#3609)

* ci(evals): a pr-summary run skips the readiness wait it never used

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: CHANGELOG [Unreleased] entry for the pr-summary readiness skip

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: move the CHANGELOG entry to changelog.d/capy-gsta149-pr-summary-wait.md (fragments policy since #3654)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-10-31 15:20:39 +00:00
1 parent cae783e6f4
commit 48599bb8ae
9 files changed
+31 -10

No files matched your search

+6 -2
View File
@@ -278,10 +278,14 @@ jobs:
# Free Tests red on this head fails the run without paid work (re-run this # Free Tests red on this head fails the run without paid work (re-run this
# workflow after a green Free Tests rerun). Dispatches never wait: they are # workflow after a green Free Tests rerun). Dispatches never wait: they are
# explicit (eval:bg, census). Version Gate is not an input, so a stamp push # explicit (eval:bg, census). Version Gate is not an input, so a stamp push
# waits for its own Free Tests only and always gets its gate. # waits for its own Free Tests only and always gets its gate. A pr-summary
# run (a repository with a train block) plans no paid slices, so it skips
# the wait: the train gate and the dispatched screen carry the paid work,
# and Free Tests stays the required free check.
readiness: readiness:
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
if: github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository needs: [route]
if: github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository && needs.route.outputs.mode != 'pr-summary'
timeout-minutes: 90 timeout-minutes: 90
permissions: permissions:
contents: read contents: read
+12
View File
@@ -1,5 +1,17 @@
# Changelog # Changelog
## [1.91.371.0] - 2026-10-31
**A PR push in a repository with a merge train no longer waits ten minutes for a check that does no work.** Under a train policy every PR push runs E2E Evals in `pr-summary` mode, which plans no paid slices, yet its `readiness` job still waited out the 600-second quiet window and green Free Tests before the run could finish. Now `readiness` waits for `route` and skips in `pr-summary` mode, so E2E Evals on a PR push finishes in about a minute (58 s on #3609) instead of after the 10-minute wait.
### Changed
- **`.github/workflows/evals.yml`**: `readiness` needs `route` and runs only when the mode is not `pr-summary`. In `pr` mode (a repository without a train) it still gates recovery and planning as before. In `pr-summary` mode a red Free Tests now shows on Free Tests alone, which stays the required free check; the train gate and the dispatched screen still carry the paid work.
- **`bun run test:health`**: the "PR lane push-to-verdict" rule text says `pr-summary` runs skip the wait since GSTA-149, so its p50 drops by about 10 minutes from this change on.
- **docs/TESTING_INTERNALS.md** describes the skip.
### Tests
- `test/evals-pr-lane-workflow.test.ts` pins `readiness.needs` and the full `if`. `test/evals-train-workflow.test.ts` checks that a `pr-summary` push runs only `build-image`, `route` and `screen-summary`, and that `pr` mode still runs `readiness`.
## [1.91.370.0] - 2026-10-31 ## [1.91.370.0] - 2026-10-31
**The free suite now runs on `ubicloud-standard-2` instead of `ubicloud-standard-8`, at the same 20 shards, for about a quarter of the runner cost.** Each shard runs its files one at a time, so it waits on disk and processes rather than CPU. In three paired rounds at one commit, the median per-shard time rose by 3 to 8 s, inside the round-to-round spread of −90 to +65 s, while runner-minutes stayed about equal (464 vs 475). Peak memory stayed under 2.5 GiB and free disk above 15 GiB. A 16-shard arm was slower beyond that spread, so the shard count stays at 20. **The free suite now runs on `ubicloud-standard-2` instead of `ubicloud-standard-8`, at the same 20 shards, for about a quarter of the runner cost.** Each shard runs its files one at a time, so it waits on disk and processes rather than CPU. In three paired rounds at one commit, the median per-shard time rose by 3 to 8 s, inside the round-to-round spread of −90 to +65 s, while runner-minutes stayed about equal (464 vs 475). Peak memory stayed under 2.5 GiB and free disk above 15 GiB. A 16-shard arm was slower beyond that spread, so the shard count stays at 20.
+1 -1
View File
@@ -1 +1 @@
1.91.370.0 1.91.371.0
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.91.370.0 — regenerate/re-copy after upgrading gstack # gstack digest v1.91.371.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows, for agent hosts without a full skill install. The full skills add workflows,
+4 -1
View File
@@ -1282,7 +1282,10 @@ re-run keeps the creation time, so it does not wait out the window again). A
head that is no longer the PR's head reports "superseded by <sha>" and plans head that is no longer the PR's head reports "superseded by <sha>" and plans
nothing. Selection, trials and verdict rules do not change. Version Gate is not nothing. Selection, trials and verdict rules do not change. Version Gate is not
an input, so a stamp push waits only for its own Free Tests and always gets a an input, so a stamp push waits only for its own Free Tests and always gets a
gate. Dispatches (`eval:bg`, the census) never wait. gate. Dispatches (`eval:bg`, the census) never wait. Neither does a
`pr-summary` run (a repository whose policy declares a train): it plans no paid
slices, so `readiness` is skipped, the run ends with the screening summary, and
a red Free Tests shows on Free Tests alone.
**Verdict reuse (R2a restamp-only, R2 main moved; a gate edit).** A PR head **Verdict reuse (R2a restamp-only, R2 main moved; a gate edit).** A PR head
reuses the verdict of the newest executed gate of the same PR instead of reuses the verdict of the newest executed gate of the same PR instead of
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "gstack", "name": "gstack",
"version": "1.91.370", "version": "1.91.371",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT", "license": "MIT",
"type": "module", "type": "module",
+1 -1
View File
@@ -277,7 +277,7 @@ export function collectHealth(inputs: HealthInputs): HealthReport {
return { value: `${(total / 60_000 * 7 / sinceDays).toFixed(0)} min/week (${cancelled.length} cancelled run(s))` }; return { value: `${(total / 60_000 * 7 / sinceDays).toFixed(0)} min/week (${cancelled.length} cancelled run(s))` };
})); }));
metrics.push(metric('push-to-verdict', 'PR lane push-to-verdict', 'created_at to updated_at of evals.yml pull_request runs that reached success or failure (includes queue and the readiness wait)', () => { metrics.push(metric('push-to-verdict', 'PR lane push-to-verdict', 'created_at to updated_at of evals.yml pull_request runs that reached success or failure (includes queue and the readiness wait, which pr-summary runs skip since GSTA-149, so their verdicts land about 10 min sooner)', () => {
const walls = need(prRuns).filter(run => run.conclusion === 'success' || run.conclusion === 'failure').map(run => span(run.createdAt, run.updatedAt)!); const walls = need(prRuns).filter(run => run.conclusion === 'success' || run.conclusion === 'failure').map(run => span(run.createdAt, run.updatedAt)!);
if (walls.length === 0) throw new Error('no PR run reached a verdict in the window'); if (walls.length === 0) throw new Error('no PR run reached a verdict in the window');
return { value: `p50 ${minutes(percentile(walls, 0.5))}, p95 ${minutes(percentile(walls, 0.95))} (${walls.length} runs)` }; return { value: `p50 ${minutes(percentile(walls, 0.5))}, p95 ${minutes(percentile(walls, 0.95))} (${walls.length} runs)` };
+2 -1
View File
@@ -83,7 +83,8 @@ describe('evals.yml readiness-gated dispatch (R1, GSTA-48)', () => {
test('a tiny PR-only job waits for a quiet PR and green Free Tests before recovery and planning', () => { test('a tiny PR-only job waits for a quiet PR and green Free Tests before recovery and planning', () => {
expect(readiness['runs-on']).toBe('ubuntu-24.04'); expect(readiness['runs-on']).toBe('ubuntu-24.04');
expect(readiness.if).toBe("github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository"); expect(readiness.if).toBe("github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository && needs.route.outputs.mode != 'pr-summary'");
expect(readiness.needs).toEqual(['route']);
expect(readiness.permissions).toEqual({ contents: 'read', actions: 'read', 'pull-requests': 'read' }); expect(readiness.permissions).toEqual({ contents: 'read', actions: 'read', 'pull-requests': 'read' });
expect(Number(readiness['timeout-minutes'])).toBeGreaterThan(80); expect(Number(readiness['timeout-minutes'])).toBeGreaterThan(80);
const wait = step('readiness', 'Wait for a quiet PR and green Free Tests on this head')!; const wait = step('readiness', 'Wait for a quiet PR and green Free Tests on this head')!;
+3 -2
View File
@@ -193,9 +193,10 @@ describe('without a train block nothing changes (the inert proof)', () => {
}); });
describe('under a train block (the gate edits, each only on its mode)', () => { describe('under a train block (the gate edits, each only on its mode)', () => {
test('a PR push starts no paid slice: route, readiness and the free screening summary only', () => { test('a PR push starts no paid slice and waits for nothing: route, the image and the free screening summary only', () => {
const sim = simulate(evals, { github: pr(), inputs: {}, outputs: { route: { ...ROUTE, ...train, mode: 'pr-summary' } } }); const sim = simulate(evals, { github: pr(), inputs: {}, outputs: { route: { ...ROUTE, ...train, mode: 'pr-summary' } } });
expect(sim.jobs).toEqual(['build-image', 'readiness', 'route', 'screen-summary']); expect(sim.jobs).toEqual(['build-image', 'route', 'screen-summary']);
expect(simulate(evals, { github: pr(), inputs: {}, outputs: { route: { ...ROUTE, mode: 'pr' } } }).jobs).toContain('readiness');
const summary = evals.jobs['screen-summary']!; const summary = evals.jobs['screen-summary']!;
expect(summary.permissions).toEqual({ contents: 'read' }); expect(summary.permissions).toEqual({ contents: 'read' });
expect(JSON.stringify(summary)).not.toMatch(/secrets\.|restore-deps|bun install/); expect(JSON.stringify(summary)).not.toMatch(/secrets\.|restore-deps|bun install/);