mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
v1.91.371.0 ci(evals): a pr-summary run skips the readiness wait it never used (#3609)
* ci(evals): a pr-summary run skips the readiness wait it never used Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: CHANGELOG [Unreleased] entry for the pr-summary readiness skip Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: move the CHANGELOG entry to changelog.d/capy-gsta149-pr-summary-wait.md (fragments policy since #3654) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
cae783e6f4
commit
48599bb8ae
9 files changed
+31
-10
No files matched your search
@@ -278,10 +278,14 @@ jobs:
|
||||
# Free Tests red on this head fails the run without paid work (re-run this
|
||||
# workflow after a green Free Tests rerun). Dispatches never wait: they are
|
||||
# explicit (eval:bg, census). Version Gate is not an input, so a stamp push
|
||||
# waits for its own Free Tests only and always gets its gate.
|
||||
# waits for its own Free Tests only and always gets its gate. A pr-summary
|
||||
# run (a repository with a train block) plans no paid slices, so it skips
|
||||
# the wait: the train gate and the dispatched screen carry the paid work,
|
||||
# and Free Tests stays the required free check.
|
||||
readiness:
|
||||
runs-on: ubuntu-24.04
|
||||
if: github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository
|
||||
needs: [route]
|
||||
if: github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository && needs.route.outputs.mode != 'pr-summary'
|
||||
timeout-minutes: 90
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
# Changelog
|
||||
|
||||
## [1.91.371.0] - 2026-10-31
|
||||
|
||||
**A PR push in a repository with a merge train no longer waits ten minutes for a check that does no work.** Under a train policy every PR push runs E2E Evals in `pr-summary` mode, which plans no paid slices, yet its `readiness` job still waited out the 600-second quiet window and green Free Tests before the run could finish. Now `readiness` waits for `route` and skips in `pr-summary` mode, so E2E Evals on a PR push finishes in about a minute (58 s on #3609) instead of after the 10-minute wait.
|
||||
|
||||
### Changed
|
||||
- **`.github/workflows/evals.yml`**: `readiness` needs `route` and runs only when the mode is not `pr-summary`. In `pr` mode (a repository without a train) it still gates recovery and planning as before. In `pr-summary` mode a red Free Tests now shows on Free Tests alone, which stays the required free check; the train gate and the dispatched screen still carry the paid work.
|
||||
- **`bun run test:health`**: the "PR lane push-to-verdict" rule text says `pr-summary` runs skip the wait since GSTA-149, so its p50 drops by about 10 minutes from this change on.
|
||||
- **docs/TESTING_INTERNALS.md** describes the skip.
|
||||
|
||||
### Tests
|
||||
- `test/evals-pr-lane-workflow.test.ts` pins `readiness.needs` and the full `if`. `test/evals-train-workflow.test.ts` checks that a `pr-summary` push runs only `build-image`, `route` and `screen-summary`, and that `pr` mode still runs `readiness`.
|
||||
|
||||
## [1.91.370.0] - 2026-10-31
|
||||
|
||||
**The free suite now runs on `ubicloud-standard-2` instead of `ubicloud-standard-8`, at the same 20 shards, for about a quarter of the runner cost.** Each shard runs its files one at a time, so it waits on disk and processes rather than CPU. In three paired rounds at one commit, the median per-shard time rose by 3 to 8 s, inside the round-to-round spread of −90 to +65 s, while runner-minutes stayed about equal (464 vs 475). Peak memory stayed under 2.5 GiB and free disk above 15 GiB. A 16-shard arm was slower beyond that spread, so the shard count stays at 20.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.91.370.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.91.371.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
@@ -1282,7 +1282,10 @@ re-run keeps the creation time, so it does not wait out the window again). A
|
||||
head that is no longer the PR's head reports "superseded by <sha>" and plans
|
||||
nothing. Selection, trials and verdict rules do not change. Version Gate is not
|
||||
an input, so a stamp push waits only for its own Free Tests and always gets a
|
||||
gate. Dispatches (`eval:bg`, the census) never wait.
|
||||
gate. Dispatches (`eval:bg`, the census) never wait. Neither does a
|
||||
`pr-summary` run (a repository whose policy declares a train): it plans no paid
|
||||
slices, so `readiness` is skipped, the run ends with the screening summary, and
|
||||
a red Free Tests shows on Free Tests alone.
|
||||
|
||||
**Verdict reuse (R2a restamp-only, R2 main moved; a gate edit).** A PR head
|
||||
reuses the verdict of the newest executed gate of the same PR instead of
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.91.370",
|
||||
"version": "1.91.371",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -277,7 +277,7 @@ export function collectHealth(inputs: HealthInputs): HealthReport {
|
||||
return { value: `${(total / 60_000 * 7 / sinceDays).toFixed(0)} min/week (${cancelled.length} cancelled run(s))` };
|
||||
}));
|
||||
|
||||
metrics.push(metric('push-to-verdict', 'PR lane push-to-verdict', 'created_at to updated_at of evals.yml pull_request runs that reached success or failure (includes queue and the readiness wait)', () => {
|
||||
metrics.push(metric('push-to-verdict', 'PR lane push-to-verdict', 'created_at to updated_at of evals.yml pull_request runs that reached success or failure (includes queue and the readiness wait, which pr-summary runs skip since GSTA-149, so their verdicts land about 10 min sooner)', () => {
|
||||
const walls = need(prRuns).filter(run => run.conclusion === 'success' || run.conclusion === 'failure').map(run => span(run.createdAt, run.updatedAt)!);
|
||||
if (walls.length === 0) throw new Error('no PR run reached a verdict in the window');
|
||||
return { value: `p50 ${minutes(percentile(walls, 0.5))}, p95 ${minutes(percentile(walls, 0.95))} (${walls.length} runs)` };
|
||||
|
||||
@@ -83,7 +83,8 @@ describe('evals.yml readiness-gated dispatch (R1, GSTA-48)', () => {
|
||||
|
||||
test('a tiny PR-only job waits for a quiet PR and green Free Tests before recovery and planning', () => {
|
||||
expect(readiness['runs-on']).toBe('ubuntu-24.04');
|
||||
expect(readiness.if).toBe("github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository");
|
||||
expect(readiness.if).toBe("github.event_name == 'pull_request' && github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository && needs.route.outputs.mode != 'pr-summary'");
|
||||
expect(readiness.needs).toEqual(['route']);
|
||||
expect(readiness.permissions).toEqual({ contents: 'read', actions: 'read', 'pull-requests': 'read' });
|
||||
expect(Number(readiness['timeout-minutes'])).toBeGreaterThan(80);
|
||||
const wait = step('readiness', 'Wait for a quiet PR and green Free Tests on this head')!;
|
||||
|
||||
@@ -193,9 +193,10 @@ describe('without a train block nothing changes (the inert proof)', () => {
|
||||
});
|
||||
|
||||
describe('under a train block (the gate edits, each only on its mode)', () => {
|
||||
test('a PR push starts no paid slice: route, readiness and the free screening summary only', () => {
|
||||
test('a PR push starts no paid slice and waits for nothing: route, the image and the free screening summary only', () => {
|
||||
const sim = simulate(evals, { github: pr(), inputs: {}, outputs: { route: { ...ROUTE, ...train, mode: 'pr-summary' } } });
|
||||
expect(sim.jobs).toEqual(['build-image', 'readiness', 'route', 'screen-summary']);
|
||||
expect(sim.jobs).toEqual(['build-image', 'route', 'screen-summary']);
|
||||
expect(simulate(evals, { github: pr(), inputs: {}, outputs: { route: { ...ROUTE, mode: 'pr' } } }).jobs).toContain('readiness');
|
||||
const summary = evals.jobs['screen-summary']!;
|
||||
expect(summary.permissions).toEqual({ contents: 'read' });
|
||||
expect(JSON.stringify(summary)).not.toMatch(/secrets\.|restore-deps|bun install/);
|
||||
|
||||
Reference in new issue
Block a user