diff --git a/CHANGELOG.md b/CHANGELOG.md
index e6017509e..285dcc834 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,17 @@
# Changelog
+## [1.92.59.0] - 2026-11-02
+
+**`test:health` and the factory scorecard leave reused main pushes out of their free-suite flake counts (#3725).** Since #3709, a main push whose merged pull request already passed on the same tree reuses that run. It finishes green, runs no shards and uploads nothing, so both readers counted it as a run with zero flakes. As reuse grows, that pulls real flake rates under the 5% bar, and `bun run test:health --enforce` stops flagging files without any error.
+
+### Fixed
+- **One rule for both readers** (`reuseChecker` in `scripts/lib/ci-history.ts`). A run with no artifacts and a green `reuse` job is a reused main push. `readCiFlakeLedgers` leaves it out of the window and its denominator and fills the window with older runs, so it still holds 20 runs that ran their shards. `readFreeRetries` in `lib/factory-scorecard.ts` leaves it out of the free-retry count. Both reports name how many runs they left out.
+- **Bounded and visible.** A run with any artifact costs no extra call. A run with none costs one job listing, at most 40 per read. Past that cap, or when a listing fails, the run counts as it did before and the report says so.
+
+### Tests
+- `test/test-health-report.test.ts`: reused runs among real flaky runs (2 flaky of 20 real runs, with 30 reused) failed on the old code with 2/45 = 4.4%, which passed the 5% bar; now 2/20 = 10% fails it. The lookup cap and a failed lookup both count the run as before, and the report says so.
+- `test/factory-scorecard.test.ts`: two reused runs in the window failed on the old code ("in 1 of 5 runs"); now the free-retry line matches the real runs only ("in 1 of 3 runs") and names the 2 runs it left out.
+
## [1.92.58.0] - 2026-11-02
**A parent agent can now run the CEO-only review through the driver: `gstack-autoplan next --ceo-only` (GST-537).** `/autoplan --ceo-only` only worked inside a Claude Code session, so a crew or eval harness driving `gstack-autoplan` from another host had to run the whole review and stop at the approval gate. Now the same review-only run works from the driver on any host.
diff --git a/VERSION b/VERSION
index 0922d3b4f..274d8da36 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-1.92.58.0
+1.92.59.0
diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md
index 2e1a5e229..0dc640d04 100644
--- a/agents-digest/gstack-AGENTS.md
+++ b/agents-digest/gstack-AGENTS.md
@@ -1,4 +1,4 @@
-# gstack digest v1.92.58.0 — regenerate/re-copy after upgrading gstack
+# gstack digest v1.92.59.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows,
diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md
index 8c913e020..ca7c3002c 100644
--- a/docs/TESTING_INTERNALS.md
+++ b/docs/TESTING_INTERNALS.md
@@ -1450,7 +1450,13 @@ Monday "Weekly test health" workflow (`bun run test:health --enforce`) reads
main's ledgers through `scripts/lib/ci-history.ts` and fails when a file flakes
in more than 5% of main free-tests runs over a window of at least 20 runs; a
window thinner than 20 runs in the last 7 days widens to the newest 20 runs, and
-fewer than 20 runs is report-only. `bun run test:health flake-summary
`
+fewer than 20 runs is report-only. A main push that reused its pull request's
+run ([CI reuse](pregate.md#ci-reuse)) ran no shards, so it stays out of the
+window and its denominator, and older runs take its place. Such a run has no
+artifacts and a green `reuse` job; checking that costs one job listing per
+run with no artifacts, at most 40 per read. Past that cap, or when a listing
+fails, the run counts as before and the report says so.
+`bun run test:health flake-summary `
prints a run's ledgers as a table. A flaky file is fixed at source; the retry
never masks it. `bun run eval:pass-rates` lists only the local ledger
(`flakeLedgerPath()`, overridden by `GSTACK_FLAKE_LEDGER`).
diff --git a/docs/recommended-harness.md b/docs/recommended-harness.md
index 9e16984c3..6da4b14c2 100644
--- a/docs/recommended-harness.md
+++ b/docs/recommended-harness.md
@@ -284,7 +284,7 @@ measured later does not count anything that happened after that week.
| Gate runs green / red / cancelled | completed runs of the gate workflow (`workflow_dispatch` by default; `--gate-event any`); cost from the run's `report-verdict-a*` artifact (`collector-outcomes.json` `.verdict.costUsd`), else the `**$N** rule/judge cost` or `cost=$N` line of the PR's verdict comment | green on 95%+ of runs, cancelled near 0, cost known for every run |
| Known gate cost per merged PR | known cost / merged PRs; unknown-cost runs make it a floor | known, stable or falling week over week |
| Red-train rate | runs whose head branch starts with `--train-prefix`, cancelled excluded; red = conclusion other than success; red backstops and screens shown beside it | under 5% (each red train is a bisect plus a rebuild, about two gates) |
-| Flaky events | `flake-records-a*` records of the newest gate runs (count, known rerun spend) and `flake-ledger-*` entries of free runs on the default branch; a run whose artifacts cannot be read is counted and named, never scored 0 | 0 repeat cases; under 1 flaky case per 10 gates; free retries under 1% of runs |
+| Flaky events | `flake-records-a*` records of the newest gate runs (count, known rerun spend) and `flake-ledger-*` entries of free runs on the default branch; a run whose artifacts cannot be read is counted and named, never scored 0, and a main push that reused its pull request's run (no artifacts, `reuse` job green) is left out and named | 0 repeat cases; under 1 flaky case per 10 gates; free retries under 1% of runs |
| Issues opened / closed | GitHub search, created and closed in the window | closed ≥ opened; nothing severe older than a week |
| Releases cut | releases published in the window, drafts excluded | at least one tagged release a week, on schedule |
diff --git a/lib/factory-scorecard.ts b/lib/factory-scorecard.ts
index deb64a92d..ebaac4c48 100644
--- a/lib/factory-scorecard.ts
+++ b/lib/factory-scorecard.ts
@@ -25,7 +25,8 @@
* flaky events flake-records-a artifacts of gate runs (paid, one record per case
* that failed once and passed its one rerun) and flake-ledger-* artifacts
* of runs on (free-suite serial retries); the same
- * two sources the daily flaky pass reads
+ * two sources the daily flaky pass reads; a main push that reused its PR's
+ * run (no artifacts, `reuse` job green; reuseChecker) is left out
* issues search/issues created / closed inside the window
* releases releases published inside the window (drafts excluded)
*
@@ -36,13 +37,13 @@ import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import {
- GH_CLIENT, createdFilter, downloadRunArtifacts, filesUnder, historyFetcher, listWorkflowRuns, readFlakeLedgerDir, unavailableReason,
+ GH_CLIENT, createdFilter, downloadRunArtifacts, filesUnder, historyFetcher, listWorkflowRuns, readFlakeLedgerDir, reuseChecker, unavailableReason,
type GhClient,
} from '../scripts/lib/ci-history';
export interface ScorecardSource {
getJson(apiPath: string): unknown;
- /** Downloaded, unzipped artifact directories of one run whose names match, keyed by artifact name; one listing per call. */
+ /** Downloaded, unzipped artifact directories of one run whose names match, keyed by artifact name; one listing per call, and `match` sees every name in it. */
artifactDirs(runId: number, match: (name: string) => boolean): Map;
}
@@ -93,6 +94,8 @@ export interface FreeRetryRun { id: number; conclusion: string | null; retries:
export interface Scorecard {
repo: string; base: string; windowStart: string; windowEnd: string;
prs: MergedPr[]; gates: GateRun[]; freeRuns: FreeRetryRun[]; artifactRunsRead: number; passCap: number;
+ /** Reused main pushes left out of freeRuns (reuseChecker), and the note naming runs it could not check. */
+ freeRunsSkipped: number; freeRunsNote: string | null;
/** Gate runs whose artifacts could not be read; they count without cost or flake data. */
artifactErrors: number;
issuesOpened: number | null; issuesClosed: number | null; releases: Array<{ tag: string; publishedAt: string }>;
@@ -308,14 +311,29 @@ export function readGateRuns(source: ScorecardSource, o: ScorecardOptions, w: Sc
return { gates, artifactRunsRead, artifactErrors };
}
-/** Free-suite runs on base in the window and the flaky-pass entries their flake-ledger-* artifacts hold. */
-export function readFreeRetries(source: ScorecardSource, o: ScorecardOptions, w: ScoreWindow): FreeRetryRun[] {
+/**
+ * Free-suite runs on base in the window and the flaky-pass entries their
+ * flake-ledger-* artifacts hold. A read run that reused its pull request's run
+ * (reuseChecker) is left out and counted in freeRunsSkipped; an unreadable run
+ * and a run past maxArtifactRuns are kept, as before.
+ */
+export function readFreeRetries(source: ScorecardSource, o: ScorecardOptions, w: ScoreWindow): Pick {
const client: GhClient = { getJson: p => source.getJson(p), downloadZip: () => { throw new Error('not used'); } };
const runs = listWorkflowRuns(client, o.repo, o.freeWorkflow, { since: w.start, until: w.end, branch: o.base });
- const retries = (runId: number): number | null => {
- try { return [...source.artifactDirs(runId, name => name.startsWith('flake-ledger-')).values()].flatMap(readFlakeLedgerDir).length; } catch { return null; }
- };
- return runs.map((run, index) => ({ id: run.id, conclusion: run.conclusion, retries: index < o.maxArtifactRuns ? retries(run.id) : 0 }));
+ const reuse = reuseChecker(client, o.repo);
+ const freeRuns: FreeRetryRun[] = [];
+ let freeRunsSkipped = 0;
+ runs.forEach((run, index) => {
+ if (index >= o.maxArtifactRuns) { freeRuns.push({ id: run.id, conclusion: run.conclusion, retries: 0 }); return; }
+ const names: string[] = [];
+ let retries: number | null;
+ try {
+ retries = [...source.artifactDirs(run.id, name => { names.push(name); return name.startsWith('flake-ledger-'); }).values()].flatMap(readFlakeLedgerDir).length;
+ } catch { retries = null; }
+ if (retries !== null && reuse.reused(run, names)) { freeRunsSkipped++; return; }
+ freeRuns.push({ id: run.id, conclusion: run.conclusion, retries });
+ });
+ return { freeRuns, freeRunsSkipped, freeRunsNote: reuse.note() ?? null };
}
function searchCount(source: ScorecardSource, q: string): number | null {
@@ -340,14 +358,14 @@ export function readReleases(source: ScorecardSource, repo: string, w: ScoreWind
export function collectScorecard(source: ScorecardSource, o: ScorecardOptions): Scorecard {
const w = scoreWindow(o);
const card: Scorecard = { repo: o.repo, base: o.base, windowStart: w.start, windowEnd: w.end,
- prs: [], gates: [], freeRuns: [], artifactRunsRead: 0, passCap: o.passCap, artifactErrors: 0, issuesOpened: null, issuesClosed: null, releases: [], unavailable: [] };
+ prs: [], gates: [], freeRuns: [], freeRunsSkipped: 0, freeRunsNote: null, artifactRunsRead: 0, passCap: o.passCap, artifactErrors: 0, issuesOpened: null, issuesClosed: null, releases: [], unavailable: [] };
const guard = (what: string, fn: () => void) => { try { fn(); } catch (e) { card.unavailable.push(`${what}: ${unavailableReason(e)}`); } };
guard('merged PRs', () => { card.prs = readMergedPrs(source, o, w); });
guard('gate runs', () => {
const prCost = new Map(card.prs.filter(p => p.commentCostUsd !== null).map(p => [p.number, p.commentCostUsd!] as const));
Object.assign(card, readGateRuns(source, o, w, prCost));
});
- guard('free retries', () => { card.freeRuns = readFreeRetries(source, o, w); });
+ guard('free retries', () => { Object.assign(card, readFreeRetries(source, o, w)); });
guard('issues', () => {
card.issuesOpened = searchCount(source, `repo:${o.repo} is:issue created:${searchRange(w)}`);
card.issuesClosed = searchCount(source, `repo:${o.repo} is:issue closed:${searchRange(w)}`);
@@ -412,7 +430,7 @@ export function scorecardRows(card: Scorecard): ScorecardRow[] {
return { id, metric, value: failed ? unavailableText(failed) : value(), benchmark: BENCHMARKS[id]!, rule };
};
const paidFlakes = () => `${flakes} paid case(s) passed on their one rerun (rerun spend ${usd(flakeUsd)})${card.artifactErrors ? `; artifacts unreadable on ${card.artifactErrors} run(s)` : ''}`;
- const freeRetries = () => `${retries} free-suite retr${retries === 1 ? 'y' : 'ies'} in ${retryRuns} of ${card.freeRuns.length} runs${unreadableFree ? ` (${unreadableFree} unreadable)` : ''}`;
+ const freeRetries = () => `${retries} free-suite retr${retries === 1 ? 'y' : 'ies'} in ${retryRuns} of ${card.freeRuns.length} runs${unreadableFree ? ` (${unreadableFree} unreadable)` : ''}${card.freeRunsSkipped ? `; ${card.freeRunsSkipped} reused main push(es) left out` : ''}${card.freeRunsNote ? `; ${card.freeRunsNote}` : ''}`;
return [
row('merged', 'Merged PRs', 'merged PRs', () => `${card.prs.length} (crew ${crewPrs.length}, outside ${card.prs.length - crewPrs.length})`,
`merged_at inside the window, base ${card.base}; crew by login list or author_association OWNER/MEMBER/COLLABORATOR`),
@@ -429,7 +447,7 @@ export function scorecardRows(card: Scorecard): ScorecardRow[] {
row('red_train', 'Red-train rate', 'gate runs', () => `${pct(trains.filter(t => t.outcome === 'red').length, trains.length)} (${trains.filter(t => t.outcome === 'red').length} red of ${trains.length} trains; ${by('red', 'backstop')} red backstop, ${by('red', 'screen')} red screens)`,
`runs whose head branch starts with the train prefix, cancelled runs excluded; red means the run's conclusion was not success`),
row('flaky', 'Flaky events', [], () => `${unavailable('gate runs') ? unavailableText('gate runs') : paidFlakes()}; ${unavailable('free retries') ? unavailableText('free retries') : freeRetries()}`,
- `flake-records artifacts of the newest ${card.artifactRunsRead} gate runs and flake-ledger artifacts of free runs on the base branch`),
+ `flake-records artifacts of the newest ${card.artifactRunsRead} gate runs and flake-ledger artifacts of free runs on the base branch, reused main pushes left out`),
row('issues', 'Issues opened / closed', 'issues', () => `${card.issuesOpened ?? 'n/a'} / ${card.issuesClosed ?? 'n/a'}`, 'GitHub search: created, closed inside the window'),
row('releases', 'Releases cut', 'releases', () => `${card.releases.length}${card.releases.length ? ` (${card.releases.map(r => r.tag).join(', ')})` : ''}`, 'releases published inside the window, drafts excluded'),
];
diff --git a/package.json b/package.json
index dbf39a65b..af7acc9c6 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "gstack",
- "version": "1.92.58",
+ "version": "1.92.59",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT",
"type": "module",
diff --git a/scripts/lib/ci-history.ts b/scripts/lib/ci-history.ts
index a7fe9833e..3ea51b75c 100644
--- a/scripts/lib/ci-history.ts
+++ b/scripts/lib/ci-history.ts
@@ -259,7 +259,38 @@ export function readFlakeLedgerDir(dir: string): FlakeLedgerEntry[] {
export const FLAKE_POLICY = { maxRate: 0.05, minRuns: 20 } as const;
export interface FlakyFile { file: string; flakyRuns: number; entries: number; rate: number; enforced: boolean; failing: boolean }
-export interface FlakeWindow { runs: number; flakyRuns: number; since: string; files: FlakyFile[]; enforceable: boolean }
+/** `skipped`: reused main pushes left out of the window (reuseChecker); `note`: runs that could not be checked. */
+export interface FlakeWindow { runs: number; flakyRuns: number; since: string; files: FlakyFile[]; enforceable: boolean; skipped?: number; note?: string }
+
+/** The most job listings one reuseChecker spends. */
+export const REUSE_LOOKUPS_MAX = 40;
+
+/**
+ * Finds runs that reused their merged pull request's verified run (#3709): such
+ * a run uploads no artifact and its `reuse` job succeeds, because every suite
+ * job was skipped. It holds no flake evidence, so flake counts leave it out of
+ * their runs and their denominator (#3725). A run with any artifact costs no
+ * call. A run with none costs one job listing, at most REUSE_LOOKUPS_MAX per
+ * checker; past the cap, or when the listing fails, the run counts as it did
+ * before #3725 and note() says so.
+ */
+export function reuseChecker(client: GhClient, repo: string) {
+ let lookups = 0;
+ let unchecked = 0;
+ let reason = '';
+ return {
+ reused(run: Pick, artifactNames: string[]): boolean {
+ if (artifactNames.length) return false;
+ if (lookups >= REUSE_LOOKUPS_MAX) { unchecked++; reason ||= `cap of ${REUSE_LOOKUPS_MAX} job listings reached`; return false; }
+ lookups++;
+ try {
+ return listRunJobs(client, repo, run.id, run.attempt).some(job => job.name === 'reuse' && job.conclusion === 'success');
+ } catch (error) { unchecked++; reason ||= unavailableReason(error); return false; }
+ },
+ note: (): string | undefined => (unchecked
+ ? `${unchecked} run(s) with no artifacts not checked for main-push reuse (${reason}); counted as before #3725` : undefined),
+ };
+}
/**
* Rank files by the share of runs in which they needed the retry pass. A file
@@ -286,9 +317,10 @@ export function rankFlakyFiles(runEntries: Map, runC
/**
* Read the flake ledgers of main's completed free-tests runs: the runs created
* in the last `sinceDays`, widened to the newest `minRuns` runs (within
- * `maxLookbackDays`) so the 5% rule has a window of at least 20 runs. A run
- * with no `flake-ledger-*` artifact had no flaky passes (the upload is skipped
- * when the ledger file does not exist).
+ * `maxLookbackDays`) so the 5% rule has a window of at least 20 runs. Reused
+ * main pushes (reuseChecker) are left out and older runs take their place. A
+ * run with no `flake-ledger-*` artifact had no flaky passes (the upload is
+ * skipped when the ledger file does not exist).
*/
export function readCiFlakeLedgers(opts: { client: GhClient; repo: string; sinceDays: number; now: number; cacheDir: string;
minRuns?: number; maxLookbackDays?: number; workflow?: string }): FlakeWindow {
@@ -297,16 +329,25 @@ export function readCiFlakeLedgers(opts: { client: GhClient; repo: string; since
const all = listWorkflowRuns(opts.client, opts.repo, opts.workflow ?? 'free-tests.yml',
{ since: iso(Math.max(opts.sinceDays, opts.maxLookbackDays ?? 60)), branch: 'main' })
.filter(run => run.conclusion === 'success' || run.conclusion === 'failure');
- const recent = all.filter(run => run.createdAt >= iso(opts.sinceDays));
- const window = recent.length >= minRuns ? recent : all.slice(0, minRuns);
+ const since = iso(opts.sinceDays);
+ const recent = all.filter(run => run.createdAt >= since).length;
const fetcher = historyFetcher(opts.client);
+ const reuse = reuseChecker(opts.client, opts.repo);
+ const window: CiRun[] = [];
+ let skipped = 0;
+ for (const [index, run] of all.entries()) {
+ if (index >= recent && window.length >= minRuns) break;
+ if (reuse.reused(run, fetcher.listArtifacts(opts.repo, run.id).map(artifact => artifact.name))) skipped++;
+ else window.push(run);
+ }
const runEntries = new Map();
for (const run of window) {
const dirs = downloadRunArtifacts({ repo: opts.repo, run, cacheDir: opts.cacheDir, fetcher, match: name => name.startsWith('flake-ledger-') });
const entries = dirs.flatMap(readFlakeLedgerDir);
if (entries.length) runEntries.set(run.id, entries);
}
- return rankFlakyFiles(runEntries, window.length, window[window.length - 1]?.createdAt ?? iso(opts.sinceDays));
+ const note = reuse.note();
+ return { ...rankFlakyFiles(runEntries, window.length, window[window.length - 1]?.createdAt ?? since), skipped, ...(note ? { note } : {}) };
}
/** Markdown table of one run's merged flake ledgers, for a job summary. */
diff --git a/scripts/test-health-report.ts b/scripts/test-health-report.ts
index 7d89e60c9..bbc48d17e 100644
--- a/scripts/test-health-report.ts
+++ b/scripts/test-health-report.ts
@@ -128,12 +128,18 @@ export function flakeCheck(window: FlakeWindow | Error): Check {
fix: 'Fix the race at source (docs/TESTING_INTERNALS.md#flake-ledger); never raise the timeout or add a retry. Reproduce with `bun test ` under load.' };
if (window instanceof Error) return { ...base, status: 'unavailable', detail: `unavailable: ${unavailableReason(window)}` };
const failing = window.files.filter(file => file.failing);
+ const left = flakeWindowNotes(window);
if (!window.enforceable) {
- return { ...base, status: 'pass', detail: `report-only: ${window.runs} main run(s) since ${window.since.slice(0, 10)} (< 20 needed to enforce)` };
+ return { ...base, status: 'pass', detail: `report-only: ${window.runs} main run(s) since ${window.since.slice(0, 10)} (< 20 needed to enforce)${left}` };
}
- if (failing.length === 0) return { ...base, status: 'pass', detail: `no file above 5% over ${window.runs} main runs` };
+ if (failing.length === 0) return { ...base, status: 'pass', detail: `no file above 5% over ${window.runs} main runs${left}` };
return { ...base, status: 'fail',
- detail: failing.map(file => `${file.file}: flaky in ${file.flakyRuns}/${window.runs} main runs (${pct(file.rate)} > 5%)`).join('; ') };
+ detail: failing.map(file => `${file.file}: flaky in ${file.flakyRuns}/${window.runs} main runs (${pct(file.rate)} > 5%)`).join('; ') + left };
+}
+
+/** The reused main pushes a flake window left out and the runs it could not check, or ''. */
+function flakeWindowNotes(window: FlakeWindow): string {
+ return `${window.skipped ? `; ${window.skipped} reused main push(es) left out` : ''}${window.note ? `; ${window.note}` : ''}`;
}
export function unseededCheck(root: string): Check {
@@ -286,9 +292,9 @@ export function collectHealth(inputs: HealthInputs): HealthReport {
metrics.push(...censusMetrics(inputs, artifactDirs));
const flakes = attempt(() => readCiFlakeLedgers({ client, repo, sinceDays, now, cacheDir }));
- metrics.push(metric('free-flaky-pass-rate', 'Free-suite flaky-pass rate', 'share of main free-tests runs (success or failure) with at least one flaky-pass ledger entry; per-file rate = runs with that file / runs', () => {
+ metrics.push(metric('free-flaky-pass-rate', 'Free-suite flaky-pass rate', 'share of main free-tests runs (success or failure) with at least one flaky-pass ledger entry; per-file rate = runs with that file / runs; a main push that reused its pull request\'s run (no artifacts, `reuse` job green) is left out', () => {
const window = need(flakes);
- return { value: `${pct(window.runs ? window.flakyRuns / window.runs : 0)} (${window.flakyRuns}/${window.runs} main runs since ${window.since.slice(0, 10)} needed the retry pass; ${window.files.length} file(s))`,
+ return { value: `${pct(window.runs ? window.flakyRuns / window.runs : 0)} (${window.flakyRuns}/${window.runs} main runs since ${window.since.slice(0, 10)} needed the retry pass; ${window.files.length} file(s)${flakeWindowNotes(window)})`,
detail: window.files.slice(0, 15).map(file => `${file.file}: ${file.flakyRuns}/${window.runs} runs (${pct(file.rate)})${file.failing ? ' FAIL > 5%' : ''}`) };
}));
diff --git a/test/factory-scorecard.test.ts b/test/factory-scorecard.test.ts
index 9e622bbcf..3794a333e 100644
--- a/test/factory-scorecard.test.ts
+++ b/test/factory-scorecard.test.ts
@@ -68,6 +68,24 @@ describe('factory-scorecard: recorded window', () => {
expect(values.releases).toBe('1 (v1.0.0)');
});
+ test('free runs that ran no shards (reused main pushes) stay out of the free-retry count and its denominator (#3725)', () => {
+ // Value: protects=the free-retry rate is read over runs that executed shards only; fails_when=a reused run (no artifacts, reuse job green) counts as a zero-retry run again; why_new=#3725, after #3709 a main push that reuses its PR's run uploads nothing; seam=fixtureSource; no_claim=runs past --max-artifact-runs, which are not read
+ const fx = fixture();
+ const freeRuns = fx.responses[Object.keys(fx.responses).find(k => k.includes('/free-tests.yml/runs?'))!] as { workflow_runs: any[] };
+ const real = freeRuns.workflow_runs[0];
+ freeRuns.workflow_runs.splice(0, 0, { ...real, id: 8005, created_at: '2026-10-25T18:00:00Z' });
+ freeRuns.workflow_runs.splice(2, 0, { ...real, id: 8004, created_at: '2026-10-24T18:00:00Z' });
+ for (const id of [8004, 8005]) fx.responses[`repos/acme/widgets/actions/runs/${id}/attempts/1/jobs?per_page=100&page=1`] = { jobs: [
+ { id: 1, name: 'reuse', conclusion: 'success' }, { id: 2, name: 'free-plan', conclusion: 'skipped' }, { id: 3, name: 'free-tests', conclusion: 'success' }] };
+ const card = collectScorecard(fixtureSource(fx, scratch()), options());
+ const clean = collectScorecard(fixtureSource(fixture(), scratch()), options());
+ expect(card.unavailable).toEqual([]);
+ expect(card.freeRuns).toEqual(clean.freeRuns);
+ expect(card.freeRuns.map(r => [r.id, r.retries])).toEqual([[8003, 0], [8002, 0], [8001, 2]]);
+ expect([card.freeRunsSkipped, clean.freeRunsSkipped]).toEqual([2, 0]);
+ expect(byId(scorecardRows(card)).flaky).toBe('2 paid case(s) passed on their one rerun (rerun spend $1.10); 2 free-suite retries in 1 of 3 runs; 2 reused main push(es) left out');
+ });
+
test('a crew login list overrides the author_association rule', () => {
const card = collectScorecard(fixtureSource(fixture(), scratch()), options({ crew: ['Alice'] }));
expect(byId(scorecardRows(card)).merged).toBe('3 (crew 1, outside 2)');
diff --git a/test/fixtures/factory-scorecard/recorded.json b/test/fixtures/factory-scorecard/recorded.json
index 8774118cb..01634e3cd 100644
--- a/test/fixtures/factory-scorecard/recorded.json
+++ b/test/fixtures/factory-scorecard/recorded.json
@@ -299,6 +299,12 @@
}
},
"artifacts": {
+ "8003": {
+ "free-result-shard-1": { "result.json": "{}" }
+ },
+ "8002": {
+ "free-result-shard-1": { "result.json": "{}" }
+ },
"8001": {
"flake-ledger-shard-1": {
"flake-ledger.jsonl": "{\"kind\":\"flaky-pass\",\"file\":\"test/a.test.ts\",\"shard\":\"shard-1\",\"attempts\":2}\n{\"kind\":\"flaky-pass\",\"file\":\"test/b.test.ts\",\"shard\":\"shard-1\",\"attempts\":2}\n{\"kind\":\"other\",\"file\":\"test/c.test.ts\"}\n"
diff --git a/test/test-health-report.test.ts b/test/test-health-report.test.ts
index 41f7d2736..3aca3162f 100644
--- a/test/test-health-report.test.ts
+++ b/test/test-health-report.test.ts
@@ -112,14 +112,60 @@ describe('W7a flake ledger reader', () => {
const client = fakeClient([
[/workflows\/free-tests\.yml\/runs/, () => ({ workflow_runs: runs })],
[/runs\/100\/artifacts/, () => ({ artifacts: [{ id: 1, name: 'flake-ledger-3', size_in_bytes: 10 }, { id: 2, name: 'free-result-3', size_in_bytes: 10 }] })],
- [/runs\/101\/artifacts/, () => ({ artifacts: [{ id: 3, name: 'flake-ledger-7', size_in_bytes: 10, expired: true }] })],
- [/runs\/\d+\/artifacts/, () => ({ artifacts: [] })],
+ [/runs\/101\/artifacts/, () => ({ artifacts: [{ id: 3, name: 'flake-ledger-7', size_in_bytes: 10, expired: true }, { id: 4, name: 'free-result-7', size_in_bytes: 10 }] })],
+ [/runs\/\d+\/artifacts/, () => ({ artifacts: [{ id: 5, name: 'free-result-1', size_in_bytes: 10 }] })],
], { 1: { 'flake-ledger.jsonl': ledger('test/a.test.ts') + ledger('test/a.test.ts') } });
const window = readCiFlakeLedgers({ client, repo: 'o/r', sinceDays: 7, now: NOW, cacheDir: tmp('health-cache-') });
expect(window).toMatchObject({ runs: 20, flakyRuns: 1, enforceable: true, files: [{ file: 'test/a.test.ts', flakyRuns: 1, entries: 2, rate: 0.05, failing: false }] });
expect(client.calls[0]).toContain('branch=main');
});
+ // A reused main push (#3709) uploads no artifact and its reuse job succeeds; a real run uploads free-plan and free-result-*.
+ const reuseClient = (runs: ReturnType[], kind: (id: number) => 'reused' | 'real' | 'flaky', jobs?: () => unknown) => fakeClient([
+ [/workflows\/free-tests\.yml\/runs/, () => ({ workflow_runs: runs })],
+ [/runs\/\d+\/artifacts/, apiPath => {
+ const k = kind(Number(/runs\/(\d+)\/artifacts/.exec(apiPath)![1]));
+ return { artifacts: k === 'reused' ? [] : [{ id: 8, name: 'free-plan', size_in_bytes: 10 },
+ ...(k === 'flaky' ? [{ id: 9, name: 'flake-ledger-3', size_in_bytes: 10 }] : [])] };
+ }],
+ [/runs\/\d+\/attempts\/1\/jobs/, jobs ?? (() => ({ jobs: [{ id: 1, name: 'reuse', conclusion: 'success' }, { id: 2, name: 'free-plan', conclusion: 'skipped' },
+ { id: 3, name: 'free-tests', conclusion: 'success' }] }))],
+ ], { 9: { 'flake-ledger.jsonl': ledger('test/a.test.ts') } });
+
+ test('reused main pushes that ran no shards stay out of the window and its denominator; the window refills with older runs that ran shards (#3725)', () => {
+ // Value: protects=test:health --enforce still flags a file flaky in >5% of runs that executed shards when main pushes reuse their PR's run; fails_when=a reused run (no artifacts, reuse job green) counts as a zero-flake run again; why_new=#3725, after #3709 reused runs diluted 2/20 (10%) to 2/45 (4.4%); seam=GhClient fake; no_claim=the live artifact retention
+ const recent = Array.from({ length: 45 }, (_, i) => run(200 + i, { created_at: day(1 + i * 0.1) }));
+ const older = Array.from({ length: 10 }, (_, i) => run(300 + i, { created_at: day(10 + i) }));
+ const client = reuseClient([...recent, ...older], id => (id === 200 || id === 203 ? 'flaky' : id < 300 && (id - 200) % 3 !== 0 ? 'reused' : 'real'));
+ const window = readCiFlakeLedgers({ client, repo: 'o/r', sinceDays: 7, now: NOW, cacheDir: tmp('health-cache-') });
+ expect(window).toMatchObject({ runs: 20, flakyRuns: 2, skipped: 30, enforceable: true, since: day(14),
+ files: [{ file: 'test/a.test.ts', flakyRuns: 2, rate: 0.1, failing: true }] });
+ expect(window.note).toBeUndefined();
+ expect(client.calls.filter(call => call.includes('/artifacts')).length).toBe(50);
+ expect(client.calls.filter(call => call.includes('/jobs')).length).toBe(30);
+ expect(flakeCheck(window)).toMatchObject({ status: 'fail', detail: 'test/a.test.ts: flaky in 2/20 main runs (10.0% > 5%); 30 reused main push(es) left out' });
+ });
+
+ test('the reuse check spends at most 40 job listings; past the cap or on a failed listing a run counts as before, and the report says so', () => {
+ // Value: protects=the reuse lookups stay bounded and fail toward the pre-#3725 count visibly; fails_when=lookups are uncapped, or a capped or failed lookup drops the run or stays silent; why_new=#3725 adds a per-run job listing; seam=GhClient fake; no_claim=none beyond protects
+ const runs = [...Array.from({ length: 20 }, (_, i) => run(400 + i, { created_at: day(1 + i * 0.1) })),
+ ...Array.from({ length: 60 }, (_, i) => run(500 + i, { created_at: day(8 + i * 0.5) }))];
+ const kind = (id: number) => (id === 400 ? 'flaky' : id === 401 ? 'real' : 'reused');
+ const client = reuseClient(runs, kind);
+ const window = readCiFlakeLedgers({ client, repo: 'o/r', sinceDays: 7, now: NOW, cacheDir: tmp('health-cache-') });
+ expect(window).toMatchObject({ runs: 20, flakyRuns: 1, skipped: 40, enforceable: true, since: day(8 + 39 * 0.5),
+ files: [{ file: 'test/a.test.ts', rate: 0.05, failing: false }] });
+ expect(window.note).toBe('18 run(s) with no artifacts not checked for main-push reuse (cap of 40 job listings reached); counted as before #3725');
+ expect(client.calls.filter(call => call.includes('/jobs')).length).toBe(40);
+ expect(client.calls.filter(call => call.includes('/artifacts')).length).toBe(60);
+ expect(flakeCheck(window).detail).toBe('no file above 5% over 20 main runs; 40 reused main push(es) left out; ' + window.note);
+
+ const failing = reuseClient(runs, kind, () => { throw new Error('gh api failed: HTTP 502: Bad Gateway'); });
+ const unchecked = readCiFlakeLedgers({ client: failing, repo: 'o/r', sinceDays: 7, now: NOW, cacheDir: tmp('health-cache-') });
+ expect(unchecked).toMatchObject({ runs: 20, skipped: 0, since: day(1 + 19 * 0.1), files: [{ file: 'test/a.test.ts', rate: 0.05 }] });
+ expect(unchecked.note).toStartWith('18 run(s) with no artifacts not checked for main-push reuse (GitHub API server error');
+ });
+
test('the job-summary table names files, counts and shards', () => {
expect(formatFlakeSummary([])).toContain('None: every file passed on its first run.');
const table = formatFlakeSummary(parseFlakeLedger(ledger('test/a.test.ts') + ledger('test/a.test.ts') + ledger('test/b.test.ts')));
@@ -188,7 +234,7 @@ describe('collectHealth over recorded fixture data', () => {
{ id: 4, name: 'redispatch', conclusion: 'skipped', started_at: null, completed_at: null }] })],
[/workflows\/free-tests\.yml\/runs/, () => ({ workflow_runs: [run(60), run(61, { conclusion: 'failure' })] })],
[/runs\/60\/artifacts/, () => ({ artifacts: [{ id: 61, name: 'flake-ledger-2', size_in_bytes: 10 }] })],
- [/runs\/61\/artifacts/, () => ({ artifacts: [] })],
+ [/runs\/61\/artifacts/, () => ({ artifacts: [{ id: 62, name: 'free-result-1', size_in_bytes: 10 }] })],
[/workflows\/windows-smoke\.yml\/runs/, () => ({ workflow_runs: windowsRuns })],
], {
11: { 'manifest.json': JSON.stringify({ prCoverage: { mode: 'full-fallback', e2e: ['gate-a', 'gate-b'], judges: ['judge-a'],