diff --git a/CHANGELOG.md b/CHANGELOG.md index e6017509e..285dcc834 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +## [1.92.59.0] - 2026-11-02 + +**`test:health` and the factory scorecard leave reused main pushes out of their free-suite flake counts (#3725).** Since #3709, a main push whose merged pull request already passed on the same tree reuses that run. It finishes green, runs no shards and uploads nothing, so both readers counted it as a run with zero flakes. As reuse grows, that pulls real flake rates under the 5% bar, and `bun run test:health --enforce` stops flagging files without any error. + +### Fixed +- **One rule for both readers** (`reuseChecker` in `scripts/lib/ci-history.ts`). A run with no artifacts and a green `reuse` job is a reused main push. `readCiFlakeLedgers` leaves it out of the window and its denominator and fills the window with older runs, so it still holds 20 runs that ran their shards. `readFreeRetries` in `lib/factory-scorecard.ts` leaves it out of the free-retry count. Both reports name how many runs they left out. +- **Bounded and visible.** A run with any artifact costs no extra call. A run with none costs one job listing, at most 40 per read. Past that cap, or when a listing fails, the run counts as it did before and the report says so. + +### Tests +- `test/test-health-report.test.ts`: reused runs among real flaky runs (2 flaky of 20 real runs, with 30 reused) failed on the old code with 2/45 = 4.4%, which passed the 5% bar; now 2/20 = 10% fails it. The lookup cap and a failed lookup both count the run as before, and the report says so. +- `test/factory-scorecard.test.ts`: two reused runs in the window failed on the old code ("in 1 of 5 runs"); now the free-retry line matches the real runs only ("in 1 of 3 runs") and names the 2 runs it left out. + ## [1.92.58.0] - 2026-11-02 **A parent agent can now run the CEO-only review through the driver: `gstack-autoplan next --ceo-only` (GST-537).** `/autoplan --ceo-only` only worked inside a Claude Code session, so a crew or eval harness driving `gstack-autoplan` from another host had to run the whole review and stop at the approval gate. Now the same review-only run works from the driver on any host. diff --git a/VERSION b/VERSION index 0922d3b4f..274d8da36 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.92.58.0 +1.92.59.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 2e1a5e229..0dc640d04 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.92.58.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.92.59.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 8c913e020..ca7c3002c 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1450,7 +1450,13 @@ Monday "Weekly test health" workflow (`bun run test:health --enforce`) reads main's ledgers through `scripts/lib/ci-history.ts` and fails when a file flakes in more than 5% of main free-tests runs over a window of at least 20 runs; a window thinner than 20 runs in the last 7 days widens to the newest 20 runs, and -fewer than 20 runs is report-only. `bun run test:health flake-summary ` +fewer than 20 runs is report-only. A main push that reused its pull request's +run ([CI reuse](pregate.md#ci-reuse)) ran no shards, so it stays out of the +window and its denominator, and older runs take its place. Such a run has no +artifacts and a green `reuse` job; checking that costs one job listing per +run with no artifacts, at most 40 per read. Past that cap, or when a listing +fails, the run counts as before and the report says so. +`bun run test:health flake-summary ` prints a run's ledgers as a table. A flaky file is fixed at source; the retry never masks it. `bun run eval:pass-rates` lists only the local ledger (`flakeLedgerPath()`, overridden by `GSTACK_FLAKE_LEDGER`). diff --git a/docs/recommended-harness.md b/docs/recommended-harness.md index 9e16984c3..6da4b14c2 100644 --- a/docs/recommended-harness.md +++ b/docs/recommended-harness.md @@ -284,7 +284,7 @@ measured later does not count anything that happened after that week. | Gate runs green / red / cancelled | completed runs of the gate workflow (`workflow_dispatch` by default; `--gate-event any`); cost from the run's `report-verdict-a*` artifact (`collector-outcomes.json` `.verdict.costUsd`), else the `**$N** rule/judge cost` or `cost=$N` line of the PR's verdict comment | green on 95%+ of runs, cancelled near 0, cost known for every run | | Known gate cost per merged PR | known cost / merged PRs; unknown-cost runs make it a floor | known, stable or falling week over week | | Red-train rate | runs whose head branch starts with `--train-prefix`, cancelled excluded; red = conclusion other than success; red backstops and screens shown beside it | under 5% (each red train is a bisect plus a rebuild, about two gates) | -| Flaky events | `flake-records-a*` records of the newest gate runs (count, known rerun spend) and `flake-ledger-*` entries of free runs on the default branch; a run whose artifacts cannot be read is counted and named, never scored 0 | 0 repeat cases; under 1 flaky case per 10 gates; free retries under 1% of runs | +| Flaky events | `flake-records-a*` records of the newest gate runs (count, known rerun spend) and `flake-ledger-*` entries of free runs on the default branch; a run whose artifacts cannot be read is counted and named, never scored 0, and a main push that reused its pull request's run (no artifacts, `reuse` job green) is left out and named | 0 repeat cases; under 1 flaky case per 10 gates; free retries under 1% of runs | | Issues opened / closed | GitHub search, created and closed in the window | closed ≥ opened; nothing severe older than a week | | Releases cut | releases published in the window, drafts excluded | at least one tagged release a week, on schedule | diff --git a/lib/factory-scorecard.ts b/lib/factory-scorecard.ts index deb64a92d..ebaac4c48 100644 --- a/lib/factory-scorecard.ts +++ b/lib/factory-scorecard.ts @@ -25,7 +25,8 @@ * flaky events flake-records-a artifacts of gate runs (paid, one record per case * that failed once and passed its one rerun) and flake-ledger-* artifacts * of runs on (free-suite serial retries); the same - * two sources the daily flaky pass reads + * two sources the daily flaky pass reads; a main push that reused its PR's + * run (no artifacts, `reuse` job green; reuseChecker) is left out * issues search/issues created / closed inside the window * releases releases published inside the window (drafts excluded) * @@ -36,13 +37,13 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { - GH_CLIENT, createdFilter, downloadRunArtifacts, filesUnder, historyFetcher, listWorkflowRuns, readFlakeLedgerDir, unavailableReason, + GH_CLIENT, createdFilter, downloadRunArtifacts, filesUnder, historyFetcher, listWorkflowRuns, readFlakeLedgerDir, reuseChecker, unavailableReason, type GhClient, } from '../scripts/lib/ci-history'; export interface ScorecardSource { getJson(apiPath: string): unknown; - /** Downloaded, unzipped artifact directories of one run whose names match, keyed by artifact name; one listing per call. */ + /** Downloaded, unzipped artifact directories of one run whose names match, keyed by artifact name; one listing per call, and `match` sees every name in it. */ artifactDirs(runId: number, match: (name: string) => boolean): Map; } @@ -93,6 +94,8 @@ export interface FreeRetryRun { id: number; conclusion: string | null; retries: export interface Scorecard { repo: string; base: string; windowStart: string; windowEnd: string; prs: MergedPr[]; gates: GateRun[]; freeRuns: FreeRetryRun[]; artifactRunsRead: number; passCap: number; + /** Reused main pushes left out of freeRuns (reuseChecker), and the note naming runs it could not check. */ + freeRunsSkipped: number; freeRunsNote: string | null; /** Gate runs whose artifacts could not be read; they count without cost or flake data. */ artifactErrors: number; issuesOpened: number | null; issuesClosed: number | null; releases: Array<{ tag: string; publishedAt: string }>; @@ -308,14 +311,29 @@ export function readGateRuns(source: ScorecardSource, o: ScorecardOptions, w: Sc return { gates, artifactRunsRead, artifactErrors }; } -/** Free-suite runs on base in the window and the flaky-pass entries their flake-ledger-* artifacts hold. */ -export function readFreeRetries(source: ScorecardSource, o: ScorecardOptions, w: ScoreWindow): FreeRetryRun[] { +/** + * Free-suite runs on base in the window and the flaky-pass entries their + * flake-ledger-* artifacts hold. A read run that reused its pull request's run + * (reuseChecker) is left out and counted in freeRunsSkipped; an unreadable run + * and a run past maxArtifactRuns are kept, as before. + */ +export function readFreeRetries(source: ScorecardSource, o: ScorecardOptions, w: ScoreWindow): Pick { const client: GhClient = { getJson: p => source.getJson(p), downloadZip: () => { throw new Error('not used'); } }; const runs = listWorkflowRuns(client, o.repo, o.freeWorkflow, { since: w.start, until: w.end, branch: o.base }); - const retries = (runId: number): number | null => { - try { return [...source.artifactDirs(runId, name => name.startsWith('flake-ledger-')).values()].flatMap(readFlakeLedgerDir).length; } catch { return null; } - }; - return runs.map((run, index) => ({ id: run.id, conclusion: run.conclusion, retries: index < o.maxArtifactRuns ? retries(run.id) : 0 })); + const reuse = reuseChecker(client, o.repo); + const freeRuns: FreeRetryRun[] = []; + let freeRunsSkipped = 0; + runs.forEach((run, index) => { + if (index >= o.maxArtifactRuns) { freeRuns.push({ id: run.id, conclusion: run.conclusion, retries: 0 }); return; } + const names: string[] = []; + let retries: number | null; + try { + retries = [...source.artifactDirs(run.id, name => { names.push(name); return name.startsWith('flake-ledger-'); }).values()].flatMap(readFlakeLedgerDir).length; + } catch { retries = null; } + if (retries !== null && reuse.reused(run, names)) { freeRunsSkipped++; return; } + freeRuns.push({ id: run.id, conclusion: run.conclusion, retries }); + }); + return { freeRuns, freeRunsSkipped, freeRunsNote: reuse.note() ?? null }; } function searchCount(source: ScorecardSource, q: string): number | null { @@ -340,14 +358,14 @@ export function readReleases(source: ScorecardSource, repo: string, w: ScoreWind export function collectScorecard(source: ScorecardSource, o: ScorecardOptions): Scorecard { const w = scoreWindow(o); const card: Scorecard = { repo: o.repo, base: o.base, windowStart: w.start, windowEnd: w.end, - prs: [], gates: [], freeRuns: [], artifactRunsRead: 0, passCap: o.passCap, artifactErrors: 0, issuesOpened: null, issuesClosed: null, releases: [], unavailable: [] }; + prs: [], gates: [], freeRuns: [], freeRunsSkipped: 0, freeRunsNote: null, artifactRunsRead: 0, passCap: o.passCap, artifactErrors: 0, issuesOpened: null, issuesClosed: null, releases: [], unavailable: [] }; const guard = (what: string, fn: () => void) => { try { fn(); } catch (e) { card.unavailable.push(`${what}: ${unavailableReason(e)}`); } }; guard('merged PRs', () => { card.prs = readMergedPrs(source, o, w); }); guard('gate runs', () => { const prCost = new Map(card.prs.filter(p => p.commentCostUsd !== null).map(p => [p.number, p.commentCostUsd!] as const)); Object.assign(card, readGateRuns(source, o, w, prCost)); }); - guard('free retries', () => { card.freeRuns = readFreeRetries(source, o, w); }); + guard('free retries', () => { Object.assign(card, readFreeRetries(source, o, w)); }); guard('issues', () => { card.issuesOpened = searchCount(source, `repo:${o.repo} is:issue created:${searchRange(w)}`); card.issuesClosed = searchCount(source, `repo:${o.repo} is:issue closed:${searchRange(w)}`); @@ -412,7 +430,7 @@ export function scorecardRows(card: Scorecard): ScorecardRow[] { return { id, metric, value: failed ? unavailableText(failed) : value(), benchmark: BENCHMARKS[id]!, rule }; }; const paidFlakes = () => `${flakes} paid case(s) passed on their one rerun (rerun spend ${usd(flakeUsd)})${card.artifactErrors ? `; artifacts unreadable on ${card.artifactErrors} run(s)` : ''}`; - const freeRetries = () => `${retries} free-suite retr${retries === 1 ? 'y' : 'ies'} in ${retryRuns} of ${card.freeRuns.length} runs${unreadableFree ? ` (${unreadableFree} unreadable)` : ''}`; + const freeRetries = () => `${retries} free-suite retr${retries === 1 ? 'y' : 'ies'} in ${retryRuns} of ${card.freeRuns.length} runs${unreadableFree ? ` (${unreadableFree} unreadable)` : ''}${card.freeRunsSkipped ? `; ${card.freeRunsSkipped} reused main push(es) left out` : ''}${card.freeRunsNote ? `; ${card.freeRunsNote}` : ''}`; return [ row('merged', 'Merged PRs', 'merged PRs', () => `${card.prs.length} (crew ${crewPrs.length}, outside ${card.prs.length - crewPrs.length})`, `merged_at inside the window, base ${card.base}; crew by login list or author_association OWNER/MEMBER/COLLABORATOR`), @@ -429,7 +447,7 @@ export function scorecardRows(card: Scorecard): ScorecardRow[] { row('red_train', 'Red-train rate', 'gate runs', () => `${pct(trains.filter(t => t.outcome === 'red').length, trains.length)} (${trains.filter(t => t.outcome === 'red').length} red of ${trains.length} trains; ${by('red', 'backstop')} red backstop, ${by('red', 'screen')} red screens)`, `runs whose head branch starts with the train prefix, cancelled runs excluded; red means the run's conclusion was not success`), row('flaky', 'Flaky events', [], () => `${unavailable('gate runs') ? unavailableText('gate runs') : paidFlakes()}; ${unavailable('free retries') ? unavailableText('free retries') : freeRetries()}`, - `flake-records artifacts of the newest ${card.artifactRunsRead} gate runs and flake-ledger artifacts of free runs on the base branch`), + `flake-records artifacts of the newest ${card.artifactRunsRead} gate runs and flake-ledger artifacts of free runs on the base branch, reused main pushes left out`), row('issues', 'Issues opened / closed', 'issues', () => `${card.issuesOpened ?? 'n/a'} / ${card.issuesClosed ?? 'n/a'}`, 'GitHub search: created, closed inside the window'), row('releases', 'Releases cut', 'releases', () => `${card.releases.length}${card.releases.length ? ` (${card.releases.map(r => r.tag).join(', ')})` : ''}`, 'releases published inside the window, drafts excluded'), ]; diff --git a/package.json b/package.json index dbf39a65b..af7acc9c6 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.92.58", + "version": "1.92.59", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/scripts/lib/ci-history.ts b/scripts/lib/ci-history.ts index a7fe9833e..3ea51b75c 100644 --- a/scripts/lib/ci-history.ts +++ b/scripts/lib/ci-history.ts @@ -259,7 +259,38 @@ export function readFlakeLedgerDir(dir: string): FlakeLedgerEntry[] { export const FLAKE_POLICY = { maxRate: 0.05, minRuns: 20 } as const; export interface FlakyFile { file: string; flakyRuns: number; entries: number; rate: number; enforced: boolean; failing: boolean } -export interface FlakeWindow { runs: number; flakyRuns: number; since: string; files: FlakyFile[]; enforceable: boolean } +/** `skipped`: reused main pushes left out of the window (reuseChecker); `note`: runs that could not be checked. */ +export interface FlakeWindow { runs: number; flakyRuns: number; since: string; files: FlakyFile[]; enforceable: boolean; skipped?: number; note?: string } + +/** The most job listings one reuseChecker spends. */ +export const REUSE_LOOKUPS_MAX = 40; + +/** + * Finds runs that reused their merged pull request's verified run (#3709): such + * a run uploads no artifact and its `reuse` job succeeds, because every suite + * job was skipped. It holds no flake evidence, so flake counts leave it out of + * their runs and their denominator (#3725). A run with any artifact costs no + * call. A run with none costs one job listing, at most REUSE_LOOKUPS_MAX per + * checker; past the cap, or when the listing fails, the run counts as it did + * before #3725 and note() says so. + */ +export function reuseChecker(client: GhClient, repo: string) { + let lookups = 0; + let unchecked = 0; + let reason = ''; + return { + reused(run: Pick, artifactNames: string[]): boolean { + if (artifactNames.length) return false; + if (lookups >= REUSE_LOOKUPS_MAX) { unchecked++; reason ||= `cap of ${REUSE_LOOKUPS_MAX} job listings reached`; return false; } + lookups++; + try { + return listRunJobs(client, repo, run.id, run.attempt).some(job => job.name === 'reuse' && job.conclusion === 'success'); + } catch (error) { unchecked++; reason ||= unavailableReason(error); return false; } + }, + note: (): string | undefined => (unchecked + ? `${unchecked} run(s) with no artifacts not checked for main-push reuse (${reason}); counted as before #3725` : undefined), + }; +} /** * Rank files by the share of runs in which they needed the retry pass. A file @@ -286,9 +317,10 @@ export function rankFlakyFiles(runEntries: Map, runC /** * Read the flake ledgers of main's completed free-tests runs: the runs created * in the last `sinceDays`, widened to the newest `minRuns` runs (within - * `maxLookbackDays`) so the 5% rule has a window of at least 20 runs. A run - * with no `flake-ledger-*` artifact had no flaky passes (the upload is skipped - * when the ledger file does not exist). + * `maxLookbackDays`) so the 5% rule has a window of at least 20 runs. Reused + * main pushes (reuseChecker) are left out and older runs take their place. A + * run with no `flake-ledger-*` artifact had no flaky passes (the upload is + * skipped when the ledger file does not exist). */ export function readCiFlakeLedgers(opts: { client: GhClient; repo: string; sinceDays: number; now: number; cacheDir: string; minRuns?: number; maxLookbackDays?: number; workflow?: string }): FlakeWindow { @@ -297,16 +329,25 @@ export function readCiFlakeLedgers(opts: { client: GhClient; repo: string; since const all = listWorkflowRuns(opts.client, opts.repo, opts.workflow ?? 'free-tests.yml', { since: iso(Math.max(opts.sinceDays, opts.maxLookbackDays ?? 60)), branch: 'main' }) .filter(run => run.conclusion === 'success' || run.conclusion === 'failure'); - const recent = all.filter(run => run.createdAt >= iso(opts.sinceDays)); - const window = recent.length >= minRuns ? recent : all.slice(0, minRuns); + const since = iso(opts.sinceDays); + const recent = all.filter(run => run.createdAt >= since).length; const fetcher = historyFetcher(opts.client); + const reuse = reuseChecker(opts.client, opts.repo); + const window: CiRun[] = []; + let skipped = 0; + for (const [index, run] of all.entries()) { + if (index >= recent && window.length >= minRuns) break; + if (reuse.reused(run, fetcher.listArtifacts(opts.repo, run.id).map(artifact => artifact.name))) skipped++; + else window.push(run); + } const runEntries = new Map(); for (const run of window) { const dirs = downloadRunArtifacts({ repo: opts.repo, run, cacheDir: opts.cacheDir, fetcher, match: name => name.startsWith('flake-ledger-') }); const entries = dirs.flatMap(readFlakeLedgerDir); if (entries.length) runEntries.set(run.id, entries); } - return rankFlakyFiles(runEntries, window.length, window[window.length - 1]?.createdAt ?? iso(opts.sinceDays)); + const note = reuse.note(); + return { ...rankFlakyFiles(runEntries, window.length, window[window.length - 1]?.createdAt ?? since), skipped, ...(note ? { note } : {}) }; } /** Markdown table of one run's merged flake ledgers, for a job summary. */ diff --git a/scripts/test-health-report.ts b/scripts/test-health-report.ts index 7d89e60c9..bbc48d17e 100644 --- a/scripts/test-health-report.ts +++ b/scripts/test-health-report.ts @@ -128,12 +128,18 @@ export function flakeCheck(window: FlakeWindow | Error): Check { fix: 'Fix the race at source (docs/TESTING_INTERNALS.md#flake-ledger); never raise the timeout or add a retry. Reproduce with `bun test ` under load.' }; if (window instanceof Error) return { ...base, status: 'unavailable', detail: `unavailable: ${unavailableReason(window)}` }; const failing = window.files.filter(file => file.failing); + const left = flakeWindowNotes(window); if (!window.enforceable) { - return { ...base, status: 'pass', detail: `report-only: ${window.runs} main run(s) since ${window.since.slice(0, 10)} (< 20 needed to enforce)` }; + return { ...base, status: 'pass', detail: `report-only: ${window.runs} main run(s) since ${window.since.slice(0, 10)} (< 20 needed to enforce)${left}` }; } - if (failing.length === 0) return { ...base, status: 'pass', detail: `no file above 5% over ${window.runs} main runs` }; + if (failing.length === 0) return { ...base, status: 'pass', detail: `no file above 5% over ${window.runs} main runs${left}` }; return { ...base, status: 'fail', - detail: failing.map(file => `${file.file}: flaky in ${file.flakyRuns}/${window.runs} main runs (${pct(file.rate)} > 5%)`).join('; ') }; + detail: failing.map(file => `${file.file}: flaky in ${file.flakyRuns}/${window.runs} main runs (${pct(file.rate)} > 5%)`).join('; ') + left }; +} + +/** The reused main pushes a flake window left out and the runs it could not check, or ''. */ +function flakeWindowNotes(window: FlakeWindow): string { + return `${window.skipped ? `; ${window.skipped} reused main push(es) left out` : ''}${window.note ? `; ${window.note}` : ''}`; } export function unseededCheck(root: string): Check { @@ -286,9 +292,9 @@ export function collectHealth(inputs: HealthInputs): HealthReport { metrics.push(...censusMetrics(inputs, artifactDirs)); const flakes = attempt(() => readCiFlakeLedgers({ client, repo, sinceDays, now, cacheDir })); - metrics.push(metric('free-flaky-pass-rate', 'Free-suite flaky-pass rate', 'share of main free-tests runs (success or failure) with at least one flaky-pass ledger entry; per-file rate = runs with that file / runs', () => { + metrics.push(metric('free-flaky-pass-rate', 'Free-suite flaky-pass rate', 'share of main free-tests runs (success or failure) with at least one flaky-pass ledger entry; per-file rate = runs with that file / runs; a main push that reused its pull request\'s run (no artifacts, `reuse` job green) is left out', () => { const window = need(flakes); - return { value: `${pct(window.runs ? window.flakyRuns / window.runs : 0)} (${window.flakyRuns}/${window.runs} main runs since ${window.since.slice(0, 10)} needed the retry pass; ${window.files.length} file(s))`, + return { value: `${pct(window.runs ? window.flakyRuns / window.runs : 0)} (${window.flakyRuns}/${window.runs} main runs since ${window.since.slice(0, 10)} needed the retry pass; ${window.files.length} file(s)${flakeWindowNotes(window)})`, detail: window.files.slice(0, 15).map(file => `${file.file}: ${file.flakyRuns}/${window.runs} runs (${pct(file.rate)})${file.failing ? ' FAIL > 5%' : ''}`) }; })); diff --git a/test/factory-scorecard.test.ts b/test/factory-scorecard.test.ts index 9e622bbcf..3794a333e 100644 --- a/test/factory-scorecard.test.ts +++ b/test/factory-scorecard.test.ts @@ -68,6 +68,24 @@ describe('factory-scorecard: recorded window', () => { expect(values.releases).toBe('1 (v1.0.0)'); }); + test('free runs that ran no shards (reused main pushes) stay out of the free-retry count and its denominator (#3725)', () => { + // Value: protects=the free-retry rate is read over runs that executed shards only; fails_when=a reused run (no artifacts, reuse job green) counts as a zero-retry run again; why_new=#3725, after #3709 a main push that reuses its PR's run uploads nothing; seam=fixtureSource; no_claim=runs past --max-artifact-runs, which are not read + const fx = fixture(); + const freeRuns = fx.responses[Object.keys(fx.responses).find(k => k.includes('/free-tests.yml/runs?'))!] as { workflow_runs: any[] }; + const real = freeRuns.workflow_runs[0]; + freeRuns.workflow_runs.splice(0, 0, { ...real, id: 8005, created_at: '2026-10-25T18:00:00Z' }); + freeRuns.workflow_runs.splice(2, 0, { ...real, id: 8004, created_at: '2026-10-24T18:00:00Z' }); + for (const id of [8004, 8005]) fx.responses[`repos/acme/widgets/actions/runs/${id}/attempts/1/jobs?per_page=100&page=1`] = { jobs: [ + { id: 1, name: 'reuse', conclusion: 'success' }, { id: 2, name: 'free-plan', conclusion: 'skipped' }, { id: 3, name: 'free-tests', conclusion: 'success' }] }; + const card = collectScorecard(fixtureSource(fx, scratch()), options()); + const clean = collectScorecard(fixtureSource(fixture(), scratch()), options()); + expect(card.unavailable).toEqual([]); + expect(card.freeRuns).toEqual(clean.freeRuns); + expect(card.freeRuns.map(r => [r.id, r.retries])).toEqual([[8003, 0], [8002, 0], [8001, 2]]); + expect([card.freeRunsSkipped, clean.freeRunsSkipped]).toEqual([2, 0]); + expect(byId(scorecardRows(card)).flaky).toBe('2 paid case(s) passed on their one rerun (rerun spend $1.10); 2 free-suite retries in 1 of 3 runs; 2 reused main push(es) left out'); + }); + test('a crew login list overrides the author_association rule', () => { const card = collectScorecard(fixtureSource(fixture(), scratch()), options({ crew: ['Alice'] })); expect(byId(scorecardRows(card)).merged).toBe('3 (crew 1, outside 2)'); diff --git a/test/fixtures/factory-scorecard/recorded.json b/test/fixtures/factory-scorecard/recorded.json index 8774118cb..01634e3cd 100644 --- a/test/fixtures/factory-scorecard/recorded.json +++ b/test/fixtures/factory-scorecard/recorded.json @@ -299,6 +299,12 @@ } }, "artifacts": { + "8003": { + "free-result-shard-1": { "result.json": "{}" } + }, + "8002": { + "free-result-shard-1": { "result.json": "{}" } + }, "8001": { "flake-ledger-shard-1": { "flake-ledger.jsonl": "{\"kind\":\"flaky-pass\",\"file\":\"test/a.test.ts\",\"shard\":\"shard-1\",\"attempts\":2}\n{\"kind\":\"flaky-pass\",\"file\":\"test/b.test.ts\",\"shard\":\"shard-1\",\"attempts\":2}\n{\"kind\":\"other\",\"file\":\"test/c.test.ts\"}\n" diff --git a/test/test-health-report.test.ts b/test/test-health-report.test.ts index 41f7d2736..3aca3162f 100644 --- a/test/test-health-report.test.ts +++ b/test/test-health-report.test.ts @@ -112,14 +112,60 @@ describe('W7a flake ledger reader', () => { const client = fakeClient([ [/workflows\/free-tests\.yml\/runs/, () => ({ workflow_runs: runs })], [/runs\/100\/artifacts/, () => ({ artifacts: [{ id: 1, name: 'flake-ledger-3', size_in_bytes: 10 }, { id: 2, name: 'free-result-3', size_in_bytes: 10 }] })], - [/runs\/101\/artifacts/, () => ({ artifacts: [{ id: 3, name: 'flake-ledger-7', size_in_bytes: 10, expired: true }] })], - [/runs\/\d+\/artifacts/, () => ({ artifacts: [] })], + [/runs\/101\/artifacts/, () => ({ artifacts: [{ id: 3, name: 'flake-ledger-7', size_in_bytes: 10, expired: true }, { id: 4, name: 'free-result-7', size_in_bytes: 10 }] })], + [/runs\/\d+\/artifacts/, () => ({ artifacts: [{ id: 5, name: 'free-result-1', size_in_bytes: 10 }] })], ], { 1: { 'flake-ledger.jsonl': ledger('test/a.test.ts') + ledger('test/a.test.ts') } }); const window = readCiFlakeLedgers({ client, repo: 'o/r', sinceDays: 7, now: NOW, cacheDir: tmp('health-cache-') }); expect(window).toMatchObject({ runs: 20, flakyRuns: 1, enforceable: true, files: [{ file: 'test/a.test.ts', flakyRuns: 1, entries: 2, rate: 0.05, failing: false }] }); expect(client.calls[0]).toContain('branch=main'); }); + // A reused main push (#3709) uploads no artifact and its reuse job succeeds; a real run uploads free-plan and free-result-*. + const reuseClient = (runs: ReturnType[], kind: (id: number) => 'reused' | 'real' | 'flaky', jobs?: () => unknown) => fakeClient([ + [/workflows\/free-tests\.yml\/runs/, () => ({ workflow_runs: runs })], + [/runs\/\d+\/artifacts/, apiPath => { + const k = kind(Number(/runs\/(\d+)\/artifacts/.exec(apiPath)![1])); + return { artifacts: k === 'reused' ? [] : [{ id: 8, name: 'free-plan', size_in_bytes: 10 }, + ...(k === 'flaky' ? [{ id: 9, name: 'flake-ledger-3', size_in_bytes: 10 }] : [])] }; + }], + [/runs\/\d+\/attempts\/1\/jobs/, jobs ?? (() => ({ jobs: [{ id: 1, name: 'reuse', conclusion: 'success' }, { id: 2, name: 'free-plan', conclusion: 'skipped' }, + { id: 3, name: 'free-tests', conclusion: 'success' }] }))], + ], { 9: { 'flake-ledger.jsonl': ledger('test/a.test.ts') } }); + + test('reused main pushes that ran no shards stay out of the window and its denominator; the window refills with older runs that ran shards (#3725)', () => { + // Value: protects=test:health --enforce still flags a file flaky in >5% of runs that executed shards when main pushes reuse their PR's run; fails_when=a reused run (no artifacts, reuse job green) counts as a zero-flake run again; why_new=#3725, after #3709 reused runs diluted 2/20 (10%) to 2/45 (4.4%); seam=GhClient fake; no_claim=the live artifact retention + const recent = Array.from({ length: 45 }, (_, i) => run(200 + i, { created_at: day(1 + i * 0.1) })); + const older = Array.from({ length: 10 }, (_, i) => run(300 + i, { created_at: day(10 + i) })); + const client = reuseClient([...recent, ...older], id => (id === 200 || id === 203 ? 'flaky' : id < 300 && (id - 200) % 3 !== 0 ? 'reused' : 'real')); + const window = readCiFlakeLedgers({ client, repo: 'o/r', sinceDays: 7, now: NOW, cacheDir: tmp('health-cache-') }); + expect(window).toMatchObject({ runs: 20, flakyRuns: 2, skipped: 30, enforceable: true, since: day(14), + files: [{ file: 'test/a.test.ts', flakyRuns: 2, rate: 0.1, failing: true }] }); + expect(window.note).toBeUndefined(); + expect(client.calls.filter(call => call.includes('/artifacts')).length).toBe(50); + expect(client.calls.filter(call => call.includes('/jobs')).length).toBe(30); + expect(flakeCheck(window)).toMatchObject({ status: 'fail', detail: 'test/a.test.ts: flaky in 2/20 main runs (10.0% > 5%); 30 reused main push(es) left out' }); + }); + + test('the reuse check spends at most 40 job listings; past the cap or on a failed listing a run counts as before, and the report says so', () => { + // Value: protects=the reuse lookups stay bounded and fail toward the pre-#3725 count visibly; fails_when=lookups are uncapped, or a capped or failed lookup drops the run or stays silent; why_new=#3725 adds a per-run job listing; seam=GhClient fake; no_claim=none beyond protects + const runs = [...Array.from({ length: 20 }, (_, i) => run(400 + i, { created_at: day(1 + i * 0.1) })), + ...Array.from({ length: 60 }, (_, i) => run(500 + i, { created_at: day(8 + i * 0.5) }))]; + const kind = (id: number) => (id === 400 ? 'flaky' : id === 401 ? 'real' : 'reused'); + const client = reuseClient(runs, kind); + const window = readCiFlakeLedgers({ client, repo: 'o/r', sinceDays: 7, now: NOW, cacheDir: tmp('health-cache-') }); + expect(window).toMatchObject({ runs: 20, flakyRuns: 1, skipped: 40, enforceable: true, since: day(8 + 39 * 0.5), + files: [{ file: 'test/a.test.ts', rate: 0.05, failing: false }] }); + expect(window.note).toBe('18 run(s) with no artifacts not checked for main-push reuse (cap of 40 job listings reached); counted as before #3725'); + expect(client.calls.filter(call => call.includes('/jobs')).length).toBe(40); + expect(client.calls.filter(call => call.includes('/artifacts')).length).toBe(60); + expect(flakeCheck(window).detail).toBe('no file above 5% over 20 main runs; 40 reused main push(es) left out; ' + window.note); + + const failing = reuseClient(runs, kind, () => { throw new Error('gh api failed: HTTP 502: Bad Gateway'); }); + const unchecked = readCiFlakeLedgers({ client: failing, repo: 'o/r', sinceDays: 7, now: NOW, cacheDir: tmp('health-cache-') }); + expect(unchecked).toMatchObject({ runs: 20, skipped: 0, since: day(1 + 19 * 0.1), files: [{ file: 'test/a.test.ts', rate: 0.05 }] }); + expect(unchecked.note).toStartWith('18 run(s) with no artifacts not checked for main-push reuse (GitHub API server error'); + }); + test('the job-summary table names files, counts and shards', () => { expect(formatFlakeSummary([])).toContain('None: every file passed on its first run.'); const table = formatFlakeSummary(parseFlakeLedger(ledger('test/a.test.ts') + ledger('test/a.test.ts') + ledger('test/b.test.ts'))); @@ -188,7 +234,7 @@ describe('collectHealth over recorded fixture data', () => { { id: 4, name: 'redispatch', conclusion: 'skipped', started_at: null, completed_at: null }] })], [/workflows\/free-tests\.yml\/runs/, () => ({ workflow_runs: [run(60), run(61, { conclusion: 'failure' })] })], [/runs\/60\/artifacts/, () => ({ artifacts: [{ id: 61, name: 'flake-ledger-2', size_in_bytes: 10 }] })], - [/runs\/61\/artifacts/, () => ({ artifacts: [] })], + [/runs\/61\/artifacts/, () => ({ artifacts: [{ id: 62, name: 'free-result-1', size_in_bytes: 10 }] })], [/workflows\/windows-smoke\.yml\/runs/, () => ({ workflow_runs: windowsRuns })], ], { 11: { 'manifest.json': JSON.stringify({ prCoverage: { mode: 'full-fallback', e2e: ['gate-a', 'gate-b'], judges: ['judge-a'],