CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-16 09:55:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(test-runner): flaky-retry vetoes on ANY unattributable failure evidence

Browse Source 
The gate equated 'some failure attributed' with 'all failures attributed': a
shard with one attributed failure plus a headerless failure, an unhandled
error between tests, or a truncated run (no terminal summary) qualified for
retry — re-running only failingFiles and masking the rest as FLAKY-PASS,
re-opening the silent-truncation hole the strict classifier closes.
FreeShardOutcome now carries unattributedFailures; nonzero vetoes the retry.
Pins: mixed shard, truncated-with-attributed shard, empty-shard field values.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-29 05:42:57 +00:00
co-authored by Claude Fable 5
parent a101bf4093
commit 7e1061c6dd
3 changed files with 55 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/test-free-shards-sandbox-knobs.test.ts
+30
View File
@@ -94,6 +94,23 @@ describe('test-free-shards: FreeShardOutcome.failingFiles (flaky-retry feed)', (
const outcome = await runFreeShard(['planted'], 1, 1, { commandFor, quiet: true, log: () => {} });
expect(outcome.status).toBe('failed');
expect(outcome.failingFiles).toEqual(['test/planted.test.ts']);
expect(outcome.unattributedFailures).toBe(0); // fully attributed — retry-eligible
});
test('a MIXED shard (one attributed + one headerless failure) is flagged unattributable — retry must not mask the headerless one', async () => {
// The retry gate must not equate "some failure attributed" with "all
// failures attributed": re-running only test/planted.test.ts and passing
// would report the suite green over the headerless failure.
const commandFor = commandPrinting([
failLine('headerless failure before any file chunk'),
'test/planted.test.ts:',
failLine('planted failure'),
summary(2, 1),
]);
const outcome = await runFreeShard(['mixed'], 1, 1, { commandFor, quiet: true, log: () => {} });
expect(outcome.status).toBe('failed');
expect(outcome.failingFiles).toEqual(['test/planted.test.ts']);
expect(outcome.unattributedFailures).toBeGreaterThan(0);
});
test('failures across two files attribute both; a crashed worker file joins the set deduped', async () => {
@@ -128,5 +145,18 @@ describe('test-free-shards: FreeShardOutcome.failingFiles (flaky-retry feed)', (
const outcome = await runFreeShard(['truncated'], 1, 1, { commandFor, quiet: true, log: () => {} });
expect(outcome.status).toBe('failed');
expect(outcome.failingFiles).toEqual([]);
expect(outcome.unattributedFailures).toBeGreaterThan(0); // truncation counts as unattributable evidence
});
test('a truncated run WITH an attributed failure is still unattributable — tests after the cut never ran', async () => {
const commandFor = commandPrinting([
'test/planted.test.ts:',
failLine('planted failure'),
// no terminal summary: the child died mid-suite
]);
const outcome = await runFreeShard(['planted', 'neverran'], 1, 1, { commandFor, quiet: true, log: () => {} });
expect(outcome.status).toBe('failed');
expect(outcome.failingFiles).toEqual(['test/planted.test.ts']);
expect(outcome.unattributedFailures).toBeGreaterThan(0);
});
});