From c2eb0355510b220ca153417a6b991125d40ae21b Mon Sep 17 00:00:00 2001 From: garrytan Date: Sat, 17 Oct 2026 04:20:03 +0000 Subject: [PATCH] fix(qa): an invalid-input probe breaks only the rule it tests The paid gate's qa-functional-cli-report failed again on #3236 (run 39001107574): the agent probed the digits-only amount rule with -5, which the positivity check rejected first, so the planted parseInt defect stayed hidden and it reported no defect. The contract map's invalid-input row now asks for each input to break only the rule it tests, so another check cannot reject it first. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- qa/sections/system-functional.md | 2 +- scripts/resolvers/qa.ts | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/qa/sections/system-functional.md b/qa/sections/system-functional.md index 3d809ecb0..3268189ba 100644 --- a/qa/sections/system-functional.md +++ b/qa/sections/system-functional.md @@ -30,7 +30,7 @@ pass/fail/blocked/not run/inconclusive/not applicable (reason). | Contract | Observe | |---|---| | Successful execution | Expected return/output and final business effect, not just launch/acceptance | -| Invalid/missing input | One probe per documented input rule (allowed characters, format, sign, range, arity): declared rejection, correct status and no forbidden state change | +| Invalid/missing input | One probe per documented input rule (allowed characters, format, sign, range, arity), each input breaking only that rule so no other check rejects it first: declared rejection, correct status and no forbidden state change | | Authentication/authorization | Valid identity, missing/invalid identity, wrong owner/role and durable no-effect boundary | | CLI process contract | Exact exit code, stdout and stderr separately; resulting file/state changes | | State transitions | Initial, intermediate and completed/failed states and their permitted transitions | diff --git a/scripts/resolvers/qa.ts b/scripts/resolvers/qa.ts index 8f1d3c39e..d4d8c7a5a 100644 --- a/scripts/resolvers/qa.ts +++ b/scripts/resolvers/qa.ts @@ -185,7 +185,7 @@ pass/fail/blocked/not run/inconclusive/not applicable (reason). | Contract | Observe | |---|---| | Successful execution | Expected return/output and final business effect, not just launch/acceptance | -| Invalid/missing input | One probe per documented input rule (allowed characters, format, sign, range, arity): declared rejection, correct status and no forbidden state change | +| Invalid/missing input | One probe per documented input rule (allowed characters, format, sign, range, arity), each input breaking only that rule so no other check rejects it first: declared rejection, correct status and no forbidden state change | | Authentication/authorization | Valid identity, missing/invalid identity, wrong owner/role and durable no-effect boundary | | CLI process contract | Exact exit code, stdout and stderr separately; resulting file/state changes | | State transitions | Initial, intermediate and completed/failed states and their permitted transitions |