mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-10 23:19:09 +02:00
test(evals): scope-gate assertion carries its evidence tail; file the detector-flake TODO
The plan-design-review member fails ONLY scopeGateQuestionObserved intermittently on unchanged code (PR #2593: red rounds 3/11 + rerun, green rounds 5/6 — every attempt terminal, no plan-mode leak), and a bare Expected-true/Received-false is undiagnosable from CI logs. The check now throws with the last-2KB visible evidence, so the next failure distinguishes a detector-sensitivity miss from a real silent bypass. TODO filed with the full receipt trail.
This commit is contained in:
@@ -99,7 +99,21 @@ describeE2E('plan-mode-info no-op outside plan mode (gate regression)', () => {
|
||||
// outcome === 'asked' would let a silent-bypass run that reaches
|
||||
// plan_ready (isPlanReadyVisible also matches common prose) sail
|
||||
// through — the exact regression this test exists to catch.
|
||||
expect(obs.scopeGateQuestionObserved ?? false).toBe(true);
|
||||
//
|
||||
// Throw WITH the evidence tail instead of a bare expect: this member
|
||||
// (plan-design-review especially) intermittently fails ONLY this
|
||||
// check on unchanged code (PR #2593 rounds 3/11/rerun, passing
|
||||
// rounds 5/6), and a bare Expected-true/Received-false in CI logs is
|
||||
// undiagnosable — we can't tell a detector-sensitivity miss (render
|
||||
// shape scrolled/rephrased) from a real silent bypass without seeing
|
||||
// what the screen held.
|
||||
if (!(obs.scopeGateQuestionObserved ?? false)) {
|
||||
throw new Error(
|
||||
`scope-gate question NOT observed (${skillName}): outcome=${obs.outcome}\n` +
|
||||
`elapsed: ${obs.elapsedMs}ms\n` +
|
||||
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}, 360_000);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user