From ac177337a16a954e55bcd1a07b1c4b29c1deb348 Mon Sep 17 00:00:00 2001 From: garrytan Date: Wed, 30 Sep 2026 19:29:54 +0000 Subject: [PATCH] fix(qa): after an input change, a probe is affected unless shown otherwise CI late-input run finished in time but revalidated only the happy probe after the locale input changed and reported the stale adverse probe green. The revalidation step now treats any probe not shown to be unaffected as affected. --- qa-only/sections/exploratory.md | 4 ++-- qa/sections/exploratory.md | 4 ++-- scripts/resolvers/qa.ts | 6 +++--- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/qa-only/sections/exploratory.md b/qa-only/sections/exploratory.md index ad03e1bd7..559586ecb 100644 --- a/qa-only/sections/exploratory.md +++ b/qa-only/sections/exploratory.md @@ -84,7 +84,7 @@ Never batch probes. to confirm it, then minimize via those gates. Expiry leaves confirmation/minimization incomplete. Another input or a regression test is not that replay. 5. If the user or another process changes source, commands or fixtures, review the affected - contracts and return to step 2 for each affected revalidation. Do not make product changes yourself. + contracts and return to step 2 for each affected revalidation (unproven=affected). Do not make product changes yourself. Keep the original limits/notes; update outcomes only from fresh evidence. ## 3. Parent handoff @@ -97,7 +97,7 @@ with their failing contract and expected assertion; never create tests or freeze Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. Write R/annotations.json: {evidence: [{capture, command, contract, expected, classification}], limits}. -Before Markdown, `bun Q materialize R annotations.json` builds evidence.json; Q fills observed, revision, runtime, cwd and learning and prints reportLinks to include; you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. +Before Markdown, `bun Q materialize R annotations.json` builds evidence.json; Q fills observed, revision, runtime, cwd and learning and prints reportLinks; you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. Evidence is invocation-local. Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder. diff --git a/qa/sections/exploratory.md b/qa/sections/exploratory.md index 760ee41c0..f571b07b8 100644 --- a/qa/sections/exploratory.md +++ b/qa/sections/exploratory.md @@ -66,7 +66,7 @@ Never batch probes. 4. Replay the exact failing command/request from the same initial fixture state via steps 2–3 (same native command, fresh capture ID) before repair, then minimize via those gates. Expiry leaves confirmation/minimization incomplete. Another input or a regression test is not that replay. -5. After source/commands/fixtures change, re-review and return to step 2 for each affected revalidation. Keep limits/notes; status requires fresh evidence. +5. After source/commands/fixtures change, re-review and return to step 2 for each affected revalidation (unproven=affected). Keep limits/notes; status requires fresh evidence. ## 3. Parent handoff @@ -82,7 +82,7 @@ Never freeze buggy output, weaken tests or delete valid red tests. Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. Write R/annotations.json: {evidence: [{capture, command, contract, expected, classification}], limits}. -Before Markdown, `bun Q materialize R annotations.json` builds evidence.json; Q fills observed, revision, runtime, cwd and learning and prints reportLinks to include; you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. +Before Markdown, `bun Q materialize R annotations.json` builds evidence.json; Q fills observed, revision, runtime, cwd and learning and prints reportLinks; you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. Evidence is invocation-local; /ship reruns once per invocation. Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder. diff --git a/scripts/resolvers/qa.ts b/scripts/resolvers/qa.ts index bcdc40a3d..3863904e1 100644 --- a/scripts/resolvers/qa.ts +++ b/scripts/resolvers/qa.ts @@ -125,8 +125,8 @@ ${reportOnly ? ` For guarded text, copy the complete span between the guard's ${reportOnly ? 'to confirm it' : 'before repair'}, then minimize via those gates. Expiry leaves confirmation/minimization incomplete. Another input or a regression test is not that replay. ${reportOnly ? `5. If the user or another process changes source, commands or fixtures, review the affected - contracts and return to step 2 for each affected revalidation. Do not make product changes yourself. - Keep the original limits/notes; update outcomes only from fresh evidence.` : `5. After source/commands/fixtures change, re-review and return to step 2 for each affected revalidation. Keep limits/notes; status requires fresh evidence.`} + contracts and return to step 2 for each affected revalidation (unproven=affected). Do not make product changes yourself. + Keep the original limits/notes; update outcomes only from fresh evidence.` : `5. After source/commands/fixtures change, re-review and return to step 2 for each affected revalidation (unproven=affected). Keep limits/notes; status requires fresh evidence.`} ## 3. Parent handoff @@ -144,7 +144,7 @@ Never freeze buggy output, weaken tests or delete valid red tests.`} Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. Write R/annotations.json: {evidence: [{capture, command, contract, expected, classification}], limits}. -Before Markdown, \`bun Q materialize R annotations.json\` builds evidence.json; Q fills observed, revision, runtime, cwd and learning and prints reportLinks to include; you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. +Before Markdown, \`bun Q materialize R annotations.json\` builds evidence.json; Q fills observed, revision, runtime, cwd and learning and prints reportLinks; you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. Evidence is invocation-local${reportOnly ? '.' : '; /ship reruns once per invocation.'} Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder.