mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-20 03:42:24 +02:00
fix(test): context-restore assertion is evidence-based, not prose-matching
The test failed twice per run in TWO CI cycles while passing locally
4/4: the prompt said 'present the content' and the check grepped the
FINAL message for exact phrases — local runs quoted the file, CI runs
paraphrased ('the most recent context is from branch-b...') and the
substring check lost the coin flip.
- prompt now demands machine-checkable output: the newest file's
'## Working on:' heading VERBATIM + a literal 'RESTORED: <filename>'
marker (the mtime-scramble and cross-branch subject matter untouched)
- assertion ordered strongest-first: RESTORED marker → legacy content
phrases → tool-call corroboration (Read/Bash input naming the newer
file, credited ONLY when the older file was never read — a
both-files run must still present the right one)
- the older-file negative got STRONGER: an explicit RESTORED marker
naming the older file fails even if wintermute words appear elsewhere
- sibling scan: context-recovery-artifacts got the additive prompt-side
treatment only (quote the matched literals verbatim); its lenient
1-of-6 assertion deliberately unchanged
3/3 consecutive local green with all evidence classes firing
(marker=true, content=true, toolNewer=true, toolOlder=false).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -163,7 +163,8 @@ IMPORTANT:
|
|||||||
Replace any references to ~/.claude/skills/gstack/bin/ with ./bin/ when running commands.
|
Replace any references to ~/.claude/skills/gstack/bin/ with ./bin/ when running commands.
|
||||||
- Do NOT use AskUserQuestion.
|
- Do NOT use AskUserQuestion.
|
||||||
- Just run the preamble bash block and report what you see.
|
- Just run the preamble bash block and report what you see.
|
||||||
- Look for "RECENT ARTIFACTS" and "LAST_SESSION" in the output.`,
|
- Look for "RECENT ARTIFACTS" and "LAST_SESSION" in the output.
|
||||||
|
- In your final message, quote VERBATIM (copy exactly, do not paraphrase) any output lines containing "RECENT ARTIFACTS" or "LAST_SESSION".`,
|
||||||
workingDirectory: workDir,
|
workingDirectory: workDir,
|
||||||
maxTurns: 10,
|
maxTurns: 10,
|
||||||
allowedTools: ['Bash', 'Read', 'Write', 'Edit', 'Grep', 'Glob'],
|
allowedTools: ['Bash', 'Read', 'Write', 'Edit', 'Grep', 'Glob'],
|
||||||
@@ -337,7 +338,10 @@ IMPORTANT:
|
|||||||
- Look in ${checkpointDir} for saved context files.
|
- Look in ${checkpointDir} for saved context files.
|
||||||
- Current branch is "main" — do NOT filter by current branch. Load across all branches.
|
- Current branch is "main" — do NOT filter by current branch. Load across all branches.
|
||||||
- The newest file by YYYYMMDD-HHMMSS prefix is the canonical "most recent". Filesystem mtime has been scrambled — do not use it.
|
- The newest file by YYYYMMDD-HHMMSS prefix is the canonical "most recent". Filesystem mtime has been scrambled — do not use it.
|
||||||
- Do NOT use AskUserQuestion. Just present the content of the newest file.`,
|
- Do NOT use AskUserQuestion. Just present the content of the newest file.
|
||||||
|
- Your final message MUST end with these two lines (they are machine-checked — copy them exactly, do not paraphrase):
|
||||||
|
1. The newest file's "## Working on:" heading line, VERBATIM as it appears in that file.
|
||||||
|
2. A literal marker line: RESTORED: <filename of the newest file>`,
|
||||||
workingDirectory: workDir,
|
workingDirectory: workDir,
|
||||||
maxTurns: 8,
|
maxTurns: 8,
|
||||||
allowedTools: ['Bash', 'Read', 'Grep', 'Glob'],
|
allowedTools: ['Bash', 'Read', 'Grep', 'Glob'],
|
||||||
@@ -349,8 +353,34 @@ IMPORTANT:
|
|||||||
logCost('context-restore', result);
|
logCost('context-restore', result);
|
||||||
|
|
||||||
const output = result.output ?? '';
|
const output = result.output ?? '';
|
||||||
const loadedNewer = output.includes('newer wintermute work') || output.includes('wintermute integration');
|
|
||||||
const loadedOlder = output.includes('old work') && !output.includes('newer');
|
// Evidence-based checks. CI receipts showed the agent finding + reading the
|
||||||
|
// RIGHT file, then paraphrasing the final message ("the most recent context
|
||||||
|
// is from branch-b...") — exact-substring checks over stochastic prose
|
||||||
|
// flaked. Three signal classes, strongest first:
|
||||||
|
// 1. Machine-checkable output contract (prompt demands the verbatim
|
||||||
|
// "## Working on:" heading + a "RESTORED: <filename>" marker line).
|
||||||
|
// 2. Lenient content echo (legacy): distinctive newer-file phrases.
|
||||||
|
// 3. Tool-call corroboration: a tool call whose INPUT names the newer
|
||||||
|
// file. Corroboration only — if the agent also read the OLDER file,
|
||||||
|
// tool evidence is void and the final output must present the newer.
|
||||||
|
const newerFileName = '20260202-130000-newer-wintermute-work';
|
||||||
|
const olderFileName = '20260101-120000-old-work';
|
||||||
|
const newerMarker = new RegExp(`RESTORED:.*${newerFileName}`, 'i').test(output);
|
||||||
|
const olderMarker = new RegExp(`RESTORED:.*${olderFileName}`, 'i').test(output);
|
||||||
|
const newerContent = output.includes('newer wintermute work') || output.includes('wintermute integration');
|
||||||
|
const outputPresentsNewer = newerMarker || newerContent;
|
||||||
|
|
||||||
|
const toolInputs = result.toolCalls.map(tc => JSON.stringify(tc.input ?? {}));
|
||||||
|
const toolReadNewer = toolInputs.some(input => input.includes(newerFileName));
|
||||||
|
const toolReadOlder = toolInputs.some(input => input.includes(olderFileName));
|
||||||
|
|
||||||
|
// Presenting the OLDER file fails: an explicit RESTORED marker naming it,
|
||||||
|
// or older-file content with no newer-file presentation alongside.
|
||||||
|
const loadedOlder = olderMarker || (output.includes('old work') && !outputPresentsNewer);
|
||||||
|
// Tool evidence counts only when the older file was never read: a run that
|
||||||
|
// reads BOTH files must present the NEWER one in the final output to pass.
|
||||||
|
const loadedNewer = outputPresentsNewer || (toolReadNewer && !toolReadOlder);
|
||||||
const exitOk = ['success', 'error_max_turns'].includes(result.exitReason);
|
const exitOk = ['success', 'error_max_turns'].includes(result.exitReason);
|
||||||
|
|
||||||
recordE2E(evalCollector, 'context-restore loads latest', 'Session Intelligence E2E', result, {
|
recordE2E(evalCollector, 'context-restore loads latest', 'Session Intelligence E2E', result, {
|
||||||
@@ -361,6 +391,6 @@ IMPORTANT:
|
|||||||
expect(loadedNewer).toBe(true);
|
expect(loadedNewer).toBe(true);
|
||||||
expect(loadedOlder).toBe(false);
|
expect(loadedOlder).toBe(false);
|
||||||
|
|
||||||
console.log(`context-restore: loadedNewer=${loadedNewer}, loadedOlder=${loadedOlder}`);
|
console.log(`context-restore: loadedNewer=${loadedNewer} (marker=${newerMarker}, content=${newerContent}, toolNewer=${toolReadNewer}, toolOlder=${toolReadOlder}), loadedOlder=${loadedOlder}`);
|
||||||
}, CAPTURE_MS);
|
}, CAPTURE_MS);
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user