From fc046b5c832d3e4ecaae79d2747ac1f435f643b7 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Tue, 20 Oct 2026 11:23:57 -0700 Subject: [PATCH] v1.91.168.0 fix(evals): QA deadline joins, QA-callers handoff order and plan-check probes (GSTA-57) (#3321) * fix(evals): QA deadline joins, QA-callers handoff order and plan-check probes (GSTA-57) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(fixtures): register the QA-callers handoff-order capture as a free fixture Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.168.0 stamp Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 10 + VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- package.json | 2 +- .../qa-caller-handoff-order-public.json | 301 ++++++++++++++++++ .../qa-only-bookkeeping-joins-public.json | 182 +++++++++++ test/helpers/free-fixtures-data.ts | 1 + test/helpers/qa-browser-deadline-evidence.ts | 36 ++- test/helpers/qa-callers-fixture.ts | 10 +- test/helpers/touchfiles-data.ts | 2 +- test/qa-browser-deadline-evidence.test.ts | 73 ++++- test/qa-exploratory-callers.test.ts | 76 ++++- 12 files changed, 685 insertions(+), 12 deletions(-) create mode 100644 test/fixtures/qa-caller-handoff-order-public.json create mode 100644 test/fixtures/qa-only-bookkeeping-joins-public.json diff --git a/CHANGELOG.md b/CHANGELOG.md index bf5fc11e3..19cdf82a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## [1.91.168.0] - 2026-10-20 + +**Four QA paid cases stop failing on valid or unclear agent behavior: `qa-only-no-fix` accepts reads joined with its exact bookkeeping calls, and the QA-callers fixture says where the handoff check goes and stops counting the explicit plan check as a wasted rerun.** Daily flaky-fix pass (GSTA-57), from the captured events of Opus gate runs 39313931713, 39332134395, 39336730429, 39320737890, 39181697029 and 39351737002. + +### Changed +- **Gate integrity, widening (`test/helpers/qa-browser-deadline-evidence.ts`):** `qa-only-no-fix` failed three times with `QA deadline: unsupported outer shell composition`. Each time the agent joined grammar reads with one of the fixture's exact bookkeeping calls outside the guard, for example `ls -la; git -C branch --show-current`. The deadline check now accepts a `;`, `&&` or `||` join in which every command is an exact listed bookkeeping call or a shared-grammar read checked from the fixture root, and the fixture prompt declares that join. Still rejected, each with a negative control: a joined guard call, a browser command before or after the join, a git call outside the exact list (`-c` options, other subcommands), `cd` inside a join, a pipe into `sh`, `&`, substitutions, newlines, escaped `;`, a dangling `&&`, a read outside the root, the readiness block with an appended command, and a `QA_DEADLINE ` receipt in the output. The existing "unbatched metadata" control now joins the metadata call with an unguarded browser command, since two exact calls joined are accepted. A free replay runs all 100 non-guard Bash commands from 11 captured attempts (3 failed, 8 passed) inside a real guarded attempt; the three failed attempts fail on the previous helper with the CI message. No threshold, kind, trial or quarantine change. The replay fixture `test/fixtures/qa-only-bookkeeping-joins-public.json` is registered in the `qa-only-no-fix` touchfiles. +- **Gate integrity, widening (`validateCallerEvidence`):** `ship-exploratory-plan-checks` failed with `duplicate unchanged passing probe` after the agent ran `bun scripts/probe.ts 9` as a guarded smoke probe and then as PLAN.md's release check with `--timeout-ms 10000`, the form the prompt reserves for an explicit plan check outside the smoke budget. One explicit plan check beside the smoke probe is no longer a duplicate. A guarded repeat of the same input, a second plan check, and ordinary input under the plan-check form still are. A free replay runs the captured capture sequence through the real helpers in a real fixture. +- **QA-callers prompt (`qaCallerSessionOptions`):** the handoff freshness check is one rule. Read HANDOFF.md and reports/HANDOFF.md and wait for the results in an earlier message than each of three things: `materialize` (it runs once per report root and freezes the verdict), every `completed:true` review record including `adversarial-review`, and the completion report. `review-exploratory-small-cli` failed twice when the agent wrote the adversarial-review record in the same response as its first HANDOFF Read (39320737890) or before any Read (39332134395); the detector was right, and a free replay of both captured event sets keeps them failing and passes them reordered. `ship-exploratory-late-input` failed twice when the agent materialized before reading the late reports/HANDOFF.md, so a refused second `materialize` left a stale `pass` verdict beside an honest `inconclusive` receipt. +- **Focused paid runs (Opus 5.5, gate tier, this tree):** `qa-only-no-fix`, `review-exploratory-small-cli`, `ship-exploratory-plan-checks` and `ship-exploratory-late-input` each passed 5 of 5 ($14.98 total). One `qa-only-no-fix` trial used two bookkeeping joins that the previous helper would have failed. + ## [1.91.167.0] - 2026-10-20 **Two paid detectors now accept the rewordings Opus 5.5 actually uses, with their negative controls kept (daily flaky-fix pass, GSTA-57).** `plan-design-with-ui-scope` went red three times on the ledger with "no acknowledged Design focus" while the agent had asked Step 0D and a numbered finding; `tpa-absent-darwin` went red once on a valid download pitch. diff --git a/VERSION b/VERSION index abf115b44..884aa495e 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.167.0 +1.91.168.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 0447401ab..205930435 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.167.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.168.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/package.json b/package.json index e073c3319..5d0773c88 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.167", + "version": "1.91.168", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/test/fixtures/qa-caller-handoff-order-public.json b/test/fixtures/qa-caller-handoff-order-public.json new file mode 100644 index 000000000..65ff82354 --- /dev/null +++ b/test/fixtures/qa-caller-handoff-order-public.json @@ -0,0 +1,301 @@ +{ + "note": "Public HANDOFF.md Reads and completed:true gstack-review-log records, with their results, from two failed review-exploratory-small-cli attempts; fixture paths are rewritten to /fixture-root.", + "attempts": [ + { + "source": "run 39320737890 paid-slice-8-a1 review-exploratory-small-cli", + "head": "f278f5b", + "sourceEventIndices": [ + 541, + 547, + 580, + 584, + 817, + 821 + ], + "events": [ + { + "type": "assistant", + "session_id": "0fa1b922-1e77-4aa4-a152-a33997782193", + "parent_tool_use_id": null, + "message": { + "id": "msg_011CgCFmhNcdng5NVrYMeEXc", + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "toolu_018VDo7yRPYpS71H6uZZQXfr", + "name": "Read", + "input": { + "file_path": "/fixture-root/product/HANDOFF.md" + }, + "caller": { + "type": "direct" + } + } + ] + } + }, + { + "type": "user", + "session_id": "0fa1b922-1e77-4aa4-a152-a33997782193", + "parent_tool_use_id": null, + "tool_use_result": { + "type": "text", + "file": { + "filePath": "/fixture-root/product/HANDOFF.md", + "content": "No concurrent input update.\n", + "numLines": 2, + "startLine": 1, + "totalLines": 2 + } + }, + "message": { + "role": "user", + "content": [ + { + "tool_use_id": "toolu_018VDo7yRPYpS71H6uZZQXfr", + "type": "tool_result", + "content": "1\tNo concurrent input update.\n2\t" + } + ] + } + }, + { + "type": "assistant", + "session_id": "0fa1b922-1e77-4aa4-a152-a33997782193", + "parent_tool_use_id": null, + "message": { + "id": "msg_011CgCFmhNcdng5NVrYMeEXc", + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "toolu_01RE7HZrWaWAoZyEhpSQthmp", + "name": "Bash", + "input": { + "command": "/fixture-root/host/runtime/bin/gstack-review-log '{\"skill\":\"adversarial-review\",\"timestamp\":\"'\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"'\",\"status\":\"issues_found\",\"source\":\"in-host\",\"host\":\"claude\",\"outside_provider\":\"codex\",\"outside_status\":\"unavailable\",\"phase\":\"adversarial\",\"tier\":\"always\",\"gate\":\"informational\",\"commit\":\"'\"$(git rev-parse --short HEAD)\"'\",\"completed\":true,\"converged\":true}' --finish be9a452e-5250-4734-8c5a-3cb4922ea744", + "description": "Persist native adversarial review record" + }, + "caller": { + "type": "direct" + } + } + ] + } + }, + { + "type": "user", + "session_id": "0fa1b922-1e77-4aa4-a152-a33997782193", + "parent_tool_use_id": null, + "tool_use_result": { + "stdout": "", + "stderr": "", + "interrupted": false, + "isImage": false, + "noOutputExpected": false + }, + "message": { + "role": "user", + "content": [ + { + "tool_use_id": "toolu_01RE7HZrWaWAoZyEhpSQthmp", + "type": "tool_result", + "content": "(Bash completed with no output)", + "is_error": false + } + ] + } + }, + { + "type": "assistant", + "session_id": "0fa1b922-1e77-4aa4-a152-a33997782193", + "parent_tool_use_id": null, + "message": { + "id": "msg_011CgCFoAqtJycy2UWEB8WG7", + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "toolu_011TvxdKFapV3hPc7FKpnW7G", + "name": "Read", + "input": { + "file_path": "/fixture-root/product/HANDOFF.md" + }, + "caller": { + "type": "direct" + } + } + ] + } + }, + { + "type": "user", + "session_id": "0fa1b922-1e77-4aa4-a152-a33997782193", + "parent_tool_use_id": null, + "tool_use_result": { + "type": "file_unchanged", + "file": { + "filePath": "/fixture-root/product/HANDOFF.md" + } + }, + "message": { + "role": "user", + "content": [ + { + "tool_use_id": "toolu_011TvxdKFapV3hPc7FKpnW7G", + "type": "tool_result", + "content": "Wasted call \u2014 file unchanged since your last Read. Refer to that earlier tool_result instead." + } + ] + } + } + ] + }, + { + "source": "run 39332134395 paid-slice-8-a2 review-exploratory-small-cli", + "head": "8802d99", + "sourceEventIndices": [ + 745, + 749, + 777, + 781, + 790, + 794 + ], + "events": [ + { + "type": "assistant", + "session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e", + "parent_tool_use_id": null, + "message": { + "id": "msg_011CgCP93PHjujPRobVJASRi", + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "toolu_018xntzUCrFxBhzJTuszB54U", + "name": "Bash", + "input": { + "command": "/fixture-root/host/runtime/bin/gstack-review-log '{\"skill\":\"adversarial-review\",\"timestamp\":\"'\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"'\",\"status\":\"issues_found\",\"source\":\"in-host\",\"host\":\"claude\",\"outside_provider\":\"codex\",\"outside_status\":\"unavailable\",\"phase\":\"adversarial\",\"tier\":\"always\",\"gate\":\"informational\",\"commit\":\"'\"$(git rev-parse --short HEAD)\"'\",\"completed\":true,\"converged\":true}' --finish 0fa0d55a-16d1-4fa5-a65c-98ba0d2adb3a", + "description": "Persist native adversarial review record" + }, + "caller": { + "type": "direct" + } + } + ] + } + }, + { + "type": "user", + "session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e", + "parent_tool_use_id": null, + "tool_use_result": { + "stdout": "", + "stderr": "", + "interrupted": false, + "isImage": false, + "noOutputExpected": false + }, + "message": { + "role": "user", + "content": [ + { + "tool_use_id": "toolu_018xntzUCrFxBhzJTuszB54U", + "type": "tool_result", + "content": "(Bash completed with no output)", + "is_error": false + } + ] + } + }, + { + "type": "assistant", + "session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e", + "parent_tool_use_id": null, + "message": { + "id": "msg_011CgCP9T1qta9KojKXBkn7S", + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "toolu_01KxpCiHqeq6EjStFc1eGKKm", + "name": "Read", + "input": { + "file_path": "/fixture-root/product/HANDOFF.md" + }, + "caller": { + "type": "direct" + } + } + ] + } + }, + { + "type": "user", + "session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e", + "parent_tool_use_id": null, + "tool_use_result": { + "type": "text", + "file": { + "filePath": "/fixture-root/product/HANDOFF.md", + "content": "No concurrent input update.\n", + "numLines": 2, + "startLine": 1, + "totalLines": 2 + } + }, + "message": { + "role": "user", + "content": [ + { + "tool_use_id": "toolu_01KxpCiHqeq6EjStFc1eGKKm", + "type": "tool_result", + "content": "1\tNo concurrent input update.\n2\t" + } + ] + } + }, + { + "type": "assistant", + "session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e", + "parent_tool_use_id": null, + "message": { + "id": "msg_011CgCP9T1qta9KojKXBkn7S", + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "toolu_01FbKjc6xXVDmtcU9Zd3GATA", + "name": "Read", + "input": { + "file_path": "/fixture-root/product/reports/HANDOFF.md" + }, + "caller": { + "type": "direct" + } + } + ] + } + }, + { + "type": "user", + "session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e", + "parent_tool_use_id": null, + "tool_use_result": "Error: File does not exist. Note: your current working directory is /fixture-root/product.", + "message": { + "role": "user", + "content": [ + { + "type": "tool_result", + "content": "File does not exist. Note: your current working directory is /fixture-root/product.", + "is_error": true, + "tool_use_id": "toolu_01FbKjc6xXVDmtcU9Zd3GATA" + } + ] + } + } + ] + } + ] +} \ No newline at end of file diff --git a/test/fixtures/qa-only-bookkeeping-joins-public.json b/test/fixtures/qa-only-bookkeeping-joins-public.json new file mode 100644 index 000000000..d2f5ecb97 --- /dev/null +++ b/test/fixtures/qa-only-bookkeeping-joins-public.json @@ -0,0 +1,182 @@ +{ + "note": "Every top-level non-guard Bash command from 11 captured qa-only-no-fix attempts; guard calls are omitted because their receipts belong to the original clock.", + "attempts": [ + { + "source": "run 39181697029 paid-slice-8-a1 qa-only-no-fix", + "passed": true, + "commands": [ + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "git -C '__QA_FIXTURE__' log -1 --format=%cI" + ], + "head": "f0722bc" + }, + { + "source": "run 39313931713 paid-slice-8-a1 qa-only-no-fix", + "passed": false, + "commands": [ + "wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80", + "sed -n 389,603p qa-only/SKILL.md", + "cat qa-only/sections/exploratory.md", + "cat qa/sections/browser-setup.md", + "cat qa/sections/scope.md; echo ======; cat qa/sections/qa-patterns.md", + "cat qa/sections/qa-patterns.md", + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "ls -la; cat TODOS.md 2>/dev/null; git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "date -u +%Y-%m-%dT%H:%M:%SZ" + ], + "head": "5186498" + }, + { + "source": "run 39313931713 paid-slice-8-a2 qa-only-no-fix", + "passed": true, + "commands": [ + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "ls -la '__QA_FIXTURE__/qa-reports/screenshots'", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "ls -la __QA_FIXTURE__" + ], + "head": "5186498" + }, + { + "source": "run 39320737890 paid-slice-8-a1 qa-only-no-fix", + "passed": true, + "commands": [ + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "ls -la '__QA_FIXTURE__/qa-reports/screenshots'", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "git -C '__QA_FIXTURE__' status --short", + "ls -la __QA_FIXTURE__", + "git -C '__QA_FIXTURE__' log -1 --format=%cI", + "grep -n \"baseline.json\" __QA_FIXTURE__/qa-reports/qa-only-report.md" + ], + "head": "f278f5b" + }, + { + "source": "run 39320737890 paid-slice-8-a2 qa-only-no-fix", + "passed": true, + "commands": [ + "wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80", + "sed -n 389,603p qa-only/SKILL.md", + "cat qa-only/sections/exploratory.md; ls qa/sections qa/templates qa-only/sections", + "cat qa/sections/scope.md", + "cat qa/sections/qa-patterns.md", + "cat qa/sections/browser-setup.md", + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "ls -la; find . -maxdepth 1 -iname 'TODOS.md' -o -maxdepth 1 -iname 'README*' -o -maxdepth 1 -name package.json", + "cat qa-only/sections/reporting.md; echo ======; cat qa/templates/qa-report-template.md" + ], + "head": "f278f5b" + }, + { + "source": "run 39320737890 paid-slice-8-a3 qa-only-no-fix", + "passed": true, + "commands": [ + "wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md", + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "git -C '__QA_FIXTURE__' log -1 --format=%cI", + "ls -la '__QA_FIXTURE__/qa-reports/screenshots'" + ], + "head": "f278f5b" + }, + { + "source": "run 39332134395 paid-slice-8-a1 qa-only-no-fix", + "passed": false, + "commands": [ + "wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80", + "sed -n 378,592p qa-only/SKILL.md", + "cat qa-only/sections/exploratory.md; echo ======; cat qa/sections/browser-setup.md", + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "ls -la '__QA_FIXTURE__/qa-reports/screenshots'", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "ls -la; git -C '__QA_FIXTURE__' branch --show-current" + ], + "head": "8802d99" + }, + { + "source": "run 39332134395 paid-slice-8-a2 qa-only-no-fix", + "passed": true, + "commands": [ + "wc -l qa-only/SKILL.md; ls qa-only qa qa-only/sections qa/sections qa/templates 2>/dev/null", + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "ls -la '__QA_FIXTURE__/qa-reports/screenshots'", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "git -C '__QA_FIXTURE__' branch --show-current", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "ls -la; find . -maxdepth 2 -not -path './.git*' -type f | sort | head -50", + "cat fixture-browser-probe.ts" + ], + "head": "8802d99" + }, + { + "source": "run 39332134395 paid-slice-8-a3 qa-only-no-fix", + "passed": true, + "commands": [ + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "ls -la '__QA_FIXTURE__/qa-reports'", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "date -u +%Y-%m-%dT%H:%M:%SZ" + ], + "head": "8802d99" + }, + { + "source": "run 39336730429 paid-slice-8-a1 qa-only-no-fix", + "passed": false, + "commands": [ + "wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80", + "cat qa-only/sections/exploratory.md", + "cat qa/sections/browser-setup.md; ls -la qa-reports qa-reports/screenshots; ls qa/sections qa-only/sections qa/templates", + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "ls -la; git -C '__QA_FIXTURE__' branch --show-current", + "git -C '__QA_FIXTURE__' log -1 --format=%cI" + ], + "head": "afa0bbf" + }, + { + "source": "run 39336730429 paid-slice-8-a2 qa-only-no-fix", + "passed": true, + "commands": [ + "wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80", + "sed -n 389,603p qa-only/SKILL.md", + "cat qa-only/sections/exploratory.md", + "cat qa/sections/browser-setup.md", + "cat qa/sections/scope.md; echo ======; cat qa/sections/qa-patterns.md", + "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi", + "ls -la '__QA_FIXTURE__/qa-reports'", + "[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP", + "date -u +%Y-%m-%dT%H:%M:%SZ", + "git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "ls; ls qa/templates; ls -a screenshots qa-reports/screenshots 2>/dev/null" + ], + "head": "afa0bbf" + } + ] +} \ No newline at end of file diff --git a/test/helpers/free-fixtures-data.ts b/test/helpers/free-fixtures-data.ts index 317e26b5d..fe508b1f1 100644 --- a/test/helpers/free-fixtures-data.ts +++ b/test/helpers/free-fixtures-data.ts @@ -159,6 +159,7 @@ export const FREE_FIXTURES: Readonly> = { 'test/fixtures/plan-ceo-floor-unasked-decisions-872e154.json': ['test/plan-ceo-floor-interactive-boundary.test.ts'], 'test/fixtures/plan-floor-edit-wrap-37176835584.json': ['test/plan-floor-edit-wrap.test.ts'], 'test/fixtures/plan-format-kind-notes.json': ['test/plan-format-kind-note.test.ts'], + 'test/fixtures/qa-caller-handoff-order-public.json': ['test/qa-exploratory-callers.test.ts'], 'test/fixtures/qa-caller-read-probe-mtb1007a.json': ['test/qa-exploratory-callers.test.ts'], 'test/fixtures/qa-caller-section-skip-ship-small-cli.json': ['test/qa-exploratory-callers.test.ts'], 'test/fixtures/plans/autoplan-dashboard.md': ['test/autoplan-init.test.ts', 'test/autoplan-preconfigured-onboarding-ar.test.ts', 'test/autoplan-snapshot.test.ts'], diff --git a/test/helpers/qa-browser-deadline-evidence.ts b/test/helpers/qa-browser-deadline-evidence.ts index 5d2b628da..cd09cbccc 100644 --- a/test/helpers/qa-browser-deadline-evidence.ts +++ b/test/helpers/qa-browser-deadline-evidence.ts @@ -23,7 +23,7 @@ export function qaDeadlineShellPolicy(directory: string, guard: string, browse: `git -C ${quote(directory)} log -1 --format=%cI`, `git -C ${quote(directory)} branch --show-current`]; const file = path.join(directory, 'qa-reports/deadline.json'); - return { allowed, file, prompt: `This native fixture accepts a deliberately narrow executable outer form. Run the BROWSER SETUP Aside readiness bash block verbatim, as its own Bash call (no appended commands). The fallback binary is already supplied. Other than that block, these standalone setup/bookkeeping Bash calls are allowed: + return { allowed, file, prompt: `This native fixture accepts a deliberately narrow executable outer form. Run the BROWSER SETUP Aside readiness bash block verbatim, as its own Bash call (no appended commands). The fallback binary is already supplied. Other than that block, these setup/bookkeeping Bash calls are allowed, each exactly as written. One Bash call may run one of them alone or join several of them and the shared-grammar reads below with ;, && or || (no cd in such a join): ${allowed.slice(1).join('\n')} ${readOnlyInterfaceText(directory)} Use Read/Glob for any other local inspection and Write only inside ${path.join(directory, 'qa-reports')}; the report and screenshots directories already exist. Memory files and learning stores outside that directory are not authorized. Write the initial charters and final report to the same caller-owned file ${path.join(directory, 'qa-reports/qa-only-report.md')}, not a separate charter file. @@ -47,6 +47,38 @@ function literalArgv(command: string): string[] { return words; } +/** + * A ;, && or || join of the fixture's exact bookkeeping calls with each other and with shared-grammar + * reads, as the fixture prompt declares. Each joined command must be an exact bookkeeping call or a + * grammar read checked from the fixture root, so cd is excluded; guard calls and browser work never qualify. + */ +function bookkeepingJoin(command: string, bookkeeping: string[], root: string): boolean { + const segments: Array<{ text: string; exact: boolean }> = []; + let rest = command.trim(); + while (true) { + let end = bookkeeping.find(entry => rest.startsWith(entry) && /^[ \t]*(?:;|&&|\|\||$)/.test(rest.slice(entry.length)))?.length; + const exact = end !== undefined; + if (end === undefined) { + let quoted = ''; + for (end = 0; end < rest.length; end++) { + const c = rest[end]; + if (quoted) { if (c === quoted) quoted = ''; else if (c === '\\' && quoted === '"') end++; } + else if (c === "'" || c === '"') quoted = c; + else if (c === '\\') end++; + else if (c === ';' || rest.startsWith('&&', end) || rest.startsWith('||', end)) break; + } + } + segments.push({ text: rest.slice(0, end).trim(), exact }); + rest = rest.slice(end).replace(/^[ \t]+/, ''); + if (!rest) break; + const operator = rest[0] === ';' ? 1 : 2; + rest = rest.slice(operator).replace(/^[ \t]+/, ''); + if (!rest) { if (operator === 1) break; return false; } + } + return segments.length > 1 && segments.some(segment => segment.exact) && segments.every(segment => segment.exact + || (!/^cd(?:[ \t]|$)/.test(segment.text) && readOnlyCommandAllowed(segment.text, { root }))); +} + function owned(file: string, root: string) { const resolved = path.resolve(root, file); if (resolved !== root && !resolved.startsWith(root + path.sep)) throw new Error('QA deadline: artifact outside owned directory'); @@ -215,7 +247,7 @@ export function assertQaBrowserDeadline(calls: Call[], options: Options & { expe if ((call.output ?? '').includes('QA_DEADLINE ')) fail('receipt outside trusted guard invocation'); continue; } - if (readOnlyCommandAllowed(command, { root: directory })) { + if (readOnlyCommandAllowed(command, { root: directory }) || bookkeepingJoin(command, allowed.slice(1), directory)) { if ((call.output ?? '').split('\n').some(line => line.startsWith('QA_DEADLINE '))) fail('receipt outside trusted guard invocation'); continue; } diff --git a/test/helpers/qa-callers-fixture.ts b/test/helpers/qa-callers-fixture.ts index f0437ed3d..9ec7a6cdd 100644 --- a/test/helpers/qa-callers-fixture.ts +++ b/test/helpers/qa-callers-fixture.ts @@ -347,9 +347,6 @@ export function validateCallerEvidence(input: { let guardedDiagnostics = 0; const checkpointProbes: Array<{ command: string; observed: CallerProbe; index: number }> = []; for (const probe of input.probes) { - const key = JSON.stringify([probe.charter, probe.input, probe.snapshot]); - if (seen.has(key) && probe.status === 'pass') errors.push(`duplicate unchanged passing probe: ${probe.id}`); - seen.add(key); const tool = tools.find(tool => tool.name === 'Bash' && (/^(?:\s*cd\s+[^\n&;]+\s*&&)?\s*(?:bun|[\w./-]+\/bun)\s+(?:run\s+)?(?:scripts\/probe\.ts|'scripts\/probe\.ts'|"scripts\/probe\.ts")(?:\s|$)/.test(String(tool.input.command)) || callerDeadlineCommand(String(tool.input.command), deadline)?.action === 'run' @@ -359,6 +356,11 @@ export function validateCallerEvidence(input: { : tool.output.split('\n').some(line => { try { return JSON.stringify(JSON.parse(line)) === JSON.stringify(probe); } catch { return false; } }))); + const planCheck = !!tool && probe.charter === 'plan:nine' && input.requiredCharters.includes('plan:nine') + && callerEvidenceCommand(String(tool.input.command), deadline)?.timeoutMs === 10000; + const key = JSON.stringify([planCheck, probe.charter, probe.input, probe.snapshot]); + if (seen.has(key) && probe.status === 'pass') errors.push(`duplicate unchanged passing probe: ${probe.id}`); + seen.add(key); if (!tool) errors.push(`probe missing native command/result: ${probe.id}`); else { checkpointProbes.push({ command: String(tool.input.command), observed: probe, index: tool.index }); @@ -697,7 +699,7 @@ export function callerReviewRecordTemplate(fixture: Pick[0] { return { - prompt: `Load gstack's /${fixture.caller} supplied parent phase from caller-${fixture.caller}.md and resume it on the selected working-tree diff against origin/main. This excerpt comes from ${fixture.runtime}/${fixture.caller}/SKILL.md; resolve installed-relative references there, not from the excerpt file or product directory. That path identifies the asset base, not another entrypoint: do not read or invoke the full parent SKILL.md or rerun its preamble. Earlier preamble/branch/base setup is complete; use the existing local origin/main ref without fetch. Earlier-phase asset locators are ${fixture.runtime}/review/checklist.md and ${fixture.runtime}/qa/templates/functional-report-template.md. Read those files directly when referenced; recursive Glob does not follow the installed asset symlinks. Cross-project learnings are configured off in this owned fixture. ${fixture.reviewStart ? `The actual review-start helper already captured REVIEW_START=${fixture.reviewStart} for this unchanged core pass; retain that token. ` : ''} This fixture evaluates only the supplied parent phase, not later publication stages. The fixture owner recorded these read-only observations when this phase began; they are current until an input changes, so use them instead of re-running those commands, and recheck freshness before completion outputs:\n${fixture.startObservations}\n Read README.md for the project contract and commands. Use diagnostic-client commands such as \`bun scripts/probe.ts \` for exploratory discoveries and their checkpoint evidence. A required \`bun run test\` is separate suite verification: report it as verification, never as a diagnostic observation or checkpoint anchor/target. Use the production evidence helper to publish each diagnostic checkpoint as \`reports/exploration-NNN.json\`, not inside a nested directory; do not transcribe its observed payload. ${fixture.caseId === 'ship-exploratory-plan-checks' ? 'The previously discovered plan is PLAN.md.' : 'No plan file was found.'} There is no remote service and no release publication is authorized. There is no interactive approver; do not invent answers or permission. Keep normal parent decision gates. Before the completion report and each completed:true review record, read HANDOFF.md and reports/HANDOFF.md if present for any concurrent collaborator update, await the results, and compare evidence with current inputs. If a probe's snapshot differs from an earlier probe's, read reports/HANDOFF.md first; it names any collaborator change. A gstack-review-log completed:true record is a completion, not preliminary bookkeeping; a later handoff read cannot validate an earlier completion.\n\nQA fixture bindings: the deadline guard, capture, checkpoint and materialize command forms are in the installed qa/sections/exploratory.md this phase requires you to Read; use those forms verbatim with these bindings substituted literally. DEADLINE_TOOL = ${fixture.runtime}/bin/gstack-qa-deadline. EVIDENCE_TOOL = ${fixture.runtime}/bin/gstack-qa-evidence. PROBE_DIR = ${fixture.cwd}/reports. DEADLINE_FILE = ${fixture.cwd}/reports/deadline.json. SECONDS must be positive and at most 300. EARLIER_UTC is the optional caller absolute deadline: use the section clock's Hard deadline UTC, never its Runner entry UTC, reserve-start time or a clock-read time. The only native COMMAND is the existing diagnostic client, bun scripts/probe.ts, with zero or one literal argument. Resolve these exact helper and state paths; do not use variables, another helper, another state file, nested wrappers, scripts, operators or substitutions. Its diagnostic receipts are declared public/synthetic, so --public is approved. Only an explicit plan check this phase requires may use --timeout-ms 10000 in place of the deadline; every other capture, including a rerun or revalidation of an earlier probe, uses the deadline. This does not reset or bypass the smoke deadline. Only the deadline helper may create or change reports/deadline.json and its .qa-deadline- temporary files; never use Write/Edit/MultiEdit on those paths. Record the full outer command in checkpoints and evidence; keep the unchanged child JSON as observed, separate from prefixed guard diagnostics. A completed expired guard-run is not a probe or a pass: retain its unused checkpoint, report not-run coverage and do not restart the deadline. Keep the 12-probe smoke limit. Required suites and explicit plan checks are outside the bounded smoke budget, not permission to reset it. Quote arguments literally. Sources stay inside reports. Decide to execute the next probe before publishing its checkpoint, then await successful publication and dispatch that exact probe. If you defer an optional idea or stop exploration, do not publish a checkpoint for it; describe it as not run in Markdown. An unused checkpoint requires an actual authenticated expired-capture result; nearing the deadline or choosing to stop is not enough. A complete capture can truthfully retain a nonzero domain exit (including expected rejection or unavailable dependency); it is not automatically a pass. Wrapper-failed, interrupted or incomplete captures are not observations or passes; an authenticated expired capture can retain an unused checkpoint as not-run evidence. No new native child command or shell authority is granted.\n\nThe supported Bash interface is one literal documented native command or one exact generated workflow shell block with main substituted for and PLAN.md for ; even read-only commands must not be chained except for the exact DIFF_BASE preface below. Native CLI commands accept no argument or one literal argument: an unquoted shell-safe word, single-quoted text, or double-quoted text without expansion or escapes; no multiline arguments or shell composition. Inventory commands are pwd, bun --version, and ls with an optional combined -l/-a flag, optional -- separator, and literal path operands using the same quoting rules; operands beginning with - require --. Listing never permits other options, glob expansion, substitution, redirection or composition. The supported Git forms are git status --short, git status --porcelain, git branch --show-current, git rev-parse HEAD, git rev-parse --short HEAD, git merge-base origin/main HEAD, the plan-completion section's git log origin/main..HEAD --oneline, git diff (optional origin/main base and at most one output mode: --stat, --numstat, --name-only, or --name-status, before or after the base; optional literal pathspec arguments after --), git ls-files, and git ls-files --others --exclude-standard. Diff pathspecs use the same literal argument syntax as the CLI; quote globs so Git, not the shell, interprets them. The only variable-base form is DIFF_BASE=$(git merge-base origin/main HEAD) && git diff with exactly the double-quoted "$DIFF_BASE" base, the same optional output mode and literal pathspecs. That exact preface recomputes the base in the same command; other assignments and variable expansions are forbidden; the only additional command substitutions are the two installed bookkeeping fields described below. No other Git options, configuration overrides, --output, --no-index, --ext-diff, --textconv, external helpers or mutations are authorized. Exact generated workflow blocks remain allowed. Review bookkeeping permits the installed gstack-review-log start commands and single-quoted JSON records for review/adversarial-review (the installed quote-spliced double-quoted $(date -u +%Y-%m-%dT%H:%M:%SZ) is permitted only as the complete top-level timestamp value, and $(git rev-parse --short HEAD) only as the complete top-level commit value; no other substitutions, field placements or shell evaluation), gstack-review-read, gstack-specialist-stats, plus date -u +%Y-%m-%dT%H:%M:%SZ; use an original start token for a started attempt and completed:false/converged:false without a token for an unstarted one. These installed helpers are shell scripts: run them by exact installed path, never through bun or another interpreter. A review record fills this installed template, keeping its keys and adding none: \`${callerReviewRecordTemplate(fixture)}\`. A review record's status is clean only for a completed review with zero unresolved defects, otherwise issues_found${fixture.caller === 'ship' ? ' (unavailable for missing dispatched reviewer output)' : ''}; a review stopped at a gate records completed:false, never a receipt status such as blocked. Bookkeeping has an owned state directory and private Git object store; the real product, index, refs and config must stay unchanged. Outside providers are unavailable in this isolated fixture; the native reviewer is still required.${fixture.child ? ` Subagent dispatch uses a deterministic child transport instead of Agent/Task: when the workflow dispatches ${fixture.caller === 'review' ? 'the native adversarial reviewer (Step 4.8)' : 'the plan completion audit (Step 8)'}, complete its dispatch prerequisites, do not invoke Agent or Task, and Read ${fixture.child} as that child's complete final output; apply the parent's normal handling to it.` : ''} Other interpreters, shell composition outside those generated blocks or the exact DIFF_BASE preface, global setup and external operations are not authorized. Use Read/Glob/Grep for discovery instead of cat/head/tail; use native Write/Edit tools for any authorized file output. Pass these same command and write boundaries to any child; generated shell fragments remain verbatim apart from the declared substitutions.\n\nWrite the phase report to reports/review.md. Keep it compact (aim under 300 words): retain each required heading and field, link to reports/evidence.json and the checkpoints for details already recorded there, and do not restate the diff, the evidence rows or the review record. Also save reports/receipt.json with this machine-readable result shape: {\"status\":\"pass|fail|blocked|inconclusive\",\"probes\":[\"observed diagnostic receipt ids\"],\"remaining\":[\"names of incomplete contracts\"]}. The status is the overall supplied phase gate, not whether some probes passed: copy it from the verdict.status that materialize prints and writes to reports/evidence.json, and if that verdict is not pass, rerun what it lists as open before finishing when a pass is required. The probes array contains the exact id values from the captured child JSON (probe-...), never the helper's three-digit capture IDs or QA_EVIDENCE.id. Capture IDs select stored observations for checkpoint/materialize; child diagnostic IDs identify the actual probes in this final receipt. Read the child JSON's id for each probe used as evidence; do not derive it from a filename or execution counter. remaining names unresolved required contracts or gates in this phase. Pass requires no remaining required contracts or gates. Optional unavailable providers and later stages outside this excerpt are not required remainder. This format does not establish that any work succeeded. Stop at the end of this phase, or at its first unresolved approval gate.`, + prompt: `Load gstack's /${fixture.caller} supplied parent phase from caller-${fixture.caller}.md and resume it on the selected working-tree diff against origin/main. This excerpt comes from ${fixture.runtime}/${fixture.caller}/SKILL.md; resolve installed-relative references there, not from the excerpt file or product directory. That path identifies the asset base, not another entrypoint: do not read or invoke the full parent SKILL.md or rerun its preamble. Earlier preamble/branch/base setup is complete; use the existing local origin/main ref without fetch. Earlier-phase asset locators are ${fixture.runtime}/review/checklist.md and ${fixture.runtime}/qa/templates/functional-report-template.md. Read those files directly when referenced; recursive Glob does not follow the installed asset symlinks. Cross-project learnings are configured off in this owned fixture. ${fixture.reviewStart ? `The actual review-start helper already captured REVIEW_START=${fixture.reviewStart} for this unchanged core pass; retain that token. ` : ''} This fixture evaluates only the supplied parent phase, not later publication stages. The fixture owner recorded these read-only observations when this phase began; they are current until an input changes, so use them instead of re-running those commands, and recheck freshness before completion outputs:\n${fixture.startObservations}\n Read README.md for the project contract and commands. Use diagnostic-client commands such as \`bun scripts/probe.ts \` for exploratory discoveries and their checkpoint evidence. A required \`bun run test\` is separate suite verification: report it as verification, never as a diagnostic observation or checkpoint anchor/target. Use the production evidence helper to publish each diagnostic checkpoint as \`reports/exploration-NNN.json\`, not inside a nested directory; do not transcribe its observed payload. ${fixture.caseId === 'ship-exploratory-plan-checks' ? 'The previously discovered plan is PLAN.md.' : 'No plan file was found.'} There is no remote service and no release publication is authorized. There is no interactive approver; do not invent answers or permission. Keep normal parent decision gates. Handoff freshness check: Read HANDOFF.md and reports/HANDOFF.md if present for any concurrent collaborator update, await those results, and compare evidence with current inputs; rerun any probe whose input changed. Run the check before each of these, and dispatch each only in a later message than the check's Read results: the evidence helper's materialize, which runs once per report root and freezes the verdict; every gstack-review-log completed:true record, adversarial-review included, since a record is a completion, not preliminary bookkeeping; and the completion report. A later handoff read cannot validate an earlier materialize or completion. If a probe's snapshot differs from an earlier probe's, read reports/HANDOFF.md first; it names any collaborator change.\n\nQA fixture bindings: the deadline guard, capture, checkpoint and materialize command forms are in the installed qa/sections/exploratory.md this phase requires you to Read; use those forms verbatim with these bindings substituted literally. DEADLINE_TOOL = ${fixture.runtime}/bin/gstack-qa-deadline. EVIDENCE_TOOL = ${fixture.runtime}/bin/gstack-qa-evidence. PROBE_DIR = ${fixture.cwd}/reports. DEADLINE_FILE = ${fixture.cwd}/reports/deadline.json. SECONDS must be positive and at most 300. EARLIER_UTC is the optional caller absolute deadline: use the section clock's Hard deadline UTC, never its Runner entry UTC, reserve-start time or a clock-read time. The only native COMMAND is the existing diagnostic client, bun scripts/probe.ts, with zero or one literal argument. Resolve these exact helper and state paths; do not use variables, another helper, another state file, nested wrappers, scripts, operators or substitutions. Its diagnostic receipts are declared public/synthetic, so --public is approved. Only an explicit plan check this phase requires may use --timeout-ms 10000 in place of the deadline; every other capture, including a rerun or revalidation of an earlier probe, uses the deadline. This does not reset or bypass the smoke deadline. Only the deadline helper may create or change reports/deadline.json and its .qa-deadline- temporary files; never use Write/Edit/MultiEdit on those paths. Record the full outer command in checkpoints and evidence; keep the unchanged child JSON as observed, separate from prefixed guard diagnostics. A completed expired guard-run is not a probe or a pass: retain its unused checkpoint, report not-run coverage and do not restart the deadline. Keep the 12-probe smoke limit. Required suites and explicit plan checks are outside the bounded smoke budget, not permission to reset it. Quote arguments literally. Sources stay inside reports. Decide to execute the next probe before publishing its checkpoint, then await successful publication and dispatch that exact probe. If you defer an optional idea or stop exploration, do not publish a checkpoint for it; describe it as not run in Markdown. An unused checkpoint requires an actual authenticated expired-capture result; nearing the deadline or choosing to stop is not enough. A complete capture can truthfully retain a nonzero domain exit (including expected rejection or unavailable dependency); it is not automatically a pass. Wrapper-failed, interrupted or incomplete captures are not observations or passes; an authenticated expired capture can retain an unused checkpoint as not-run evidence. No new native child command or shell authority is granted.\n\nThe supported Bash interface is one literal documented native command or one exact generated workflow shell block with main substituted for and PLAN.md for ; even read-only commands must not be chained except for the exact DIFF_BASE preface below. Native CLI commands accept no argument or one literal argument: an unquoted shell-safe word, single-quoted text, or double-quoted text without expansion or escapes; no multiline arguments or shell composition. Inventory commands are pwd, bun --version, and ls with an optional combined -l/-a flag, optional -- separator, and literal path operands using the same quoting rules; operands beginning with - require --. Listing never permits other options, glob expansion, substitution, redirection or composition. The supported Git forms are git status --short, git status --porcelain, git branch --show-current, git rev-parse HEAD, git rev-parse --short HEAD, git merge-base origin/main HEAD, the plan-completion section's git log origin/main..HEAD --oneline, git diff (optional origin/main base and at most one output mode: --stat, --numstat, --name-only, or --name-status, before or after the base; optional literal pathspec arguments after --), git ls-files, and git ls-files --others --exclude-standard. Diff pathspecs use the same literal argument syntax as the CLI; quote globs so Git, not the shell, interprets them. The only variable-base form is DIFF_BASE=$(git merge-base origin/main HEAD) && git diff with exactly the double-quoted "$DIFF_BASE" base, the same optional output mode and literal pathspecs. That exact preface recomputes the base in the same command; other assignments and variable expansions are forbidden; the only additional command substitutions are the two installed bookkeeping fields described below. No other Git options, configuration overrides, --output, --no-index, --ext-diff, --textconv, external helpers or mutations are authorized. Exact generated workflow blocks remain allowed. Review bookkeeping permits the installed gstack-review-log start commands and single-quoted JSON records for review/adversarial-review (the installed quote-spliced double-quoted $(date -u +%Y-%m-%dT%H:%M:%SZ) is permitted only as the complete top-level timestamp value, and $(git rev-parse --short HEAD) only as the complete top-level commit value; no other substitutions, field placements or shell evaluation), gstack-review-read, gstack-specialist-stats, plus date -u +%Y-%m-%dT%H:%M:%SZ; use an original start token for a started attempt and completed:false/converged:false without a token for an unstarted one. These installed helpers are shell scripts: run them by exact installed path, never through bun or another interpreter. A review record fills this installed template, keeping its keys and adding none: \`${callerReviewRecordTemplate(fixture)}\`. A review record's status is clean only for a completed review with zero unresolved defects, otherwise issues_found${fixture.caller === 'ship' ? ' (unavailable for missing dispatched reviewer output)' : ''}; a review stopped at a gate records completed:false, never a receipt status such as blocked. Bookkeeping has an owned state directory and private Git object store; the real product, index, refs and config must stay unchanged. Outside providers are unavailable in this isolated fixture; the native reviewer is still required.${fixture.child ? ` Subagent dispatch uses a deterministic child transport instead of Agent/Task: when the workflow dispatches ${fixture.caller === 'review' ? 'the native adversarial reviewer (Step 4.8)' : 'the plan completion audit (Step 8)'}, complete its dispatch prerequisites, do not invoke Agent or Task, and Read ${fixture.child} as that child's complete final output; apply the parent's normal handling to it.` : ''} Other interpreters, shell composition outside those generated blocks or the exact DIFF_BASE preface, global setup and external operations are not authorized. Use Read/Glob/Grep for discovery instead of cat/head/tail; use native Write/Edit tools for any authorized file output. Pass these same command and write boundaries to any child; generated shell fragments remain verbatim apart from the declared substitutions.\n\nWrite the phase report to reports/review.md. Keep it compact (aim under 300 words): retain each required heading and field, link to reports/evidence.json and the checkpoints for details already recorded there, and do not restate the diff, the evidence rows or the review record. Also save reports/receipt.json with this machine-readable result shape: {\"status\":\"pass|fail|blocked|inconclusive\",\"probes\":[\"observed diagnostic receipt ids\"],\"remaining\":[\"names of incomplete contracts\"]}. The status is the overall supplied phase gate, not whether some probes passed: copy it from the verdict.status that materialize prints and writes to reports/evidence.json, and if that verdict is not pass, rerun what it lists as open before finishing when a pass is required. The probes array contains the exact id values from the captured child JSON (probe-...), never the helper's three-digit capture IDs or QA_EVIDENCE.id. Capture IDs select stored observations for checkpoint/materialize; child diagnostic IDs identify the actual probes in this final receipt. Read the child JSON's id for each probe used as evidence; do not derive it from a filename or execution counter. remaining names unresolved required contracts or gates in this phase. Pass requires no remaining required contracts or gates. Optional unavailable providers and later stages outside this excerpt are not required remainder. This format does not establish that any work succeeded. Stop at the end of this phase, or at its first unresolved approval gate.`, appendSystemPrompt: `Caller execution scheduling (fixture contract): This session has at most 25 assistant turns, including required verification and final artifacts. The command boundary applies to each Bash call, not to the number of independent tool calls in an assistant turn. After required clock and approval prerequisites settle, issue independent source Reads and read-only discovery together as separate native tool calls once their paths and inputs are known. Read-only discovery is Read, Glob, Grep and the declared inventory, Git and review-read forms; a capture, a deadline guard start or run, or a diagnostic-client run is a probe, never discovery. For example, after the first clock read, one response can Read the caller excerpt, README.md, every product file and the referenced checklist and templates. Wait for their results before decisions that depend on them. diff --git a/test/helpers/touchfiles-data.ts b/test/helpers/touchfiles-data.ts index 09d22b5e6..004b48abb 100644 --- a/test/helpers/touchfiles-data.ts +++ b/test/helpers/touchfiles-data.ts @@ -110,7 +110,7 @@ export const E2E_TOUCHFILES: Record = { 'qa-b6-static': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'qa/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval.html', 'test/fixtures/qa-eval-ground-truth.json', 'test/skill-e2e-qa-bugs.test.ts', 'test/helpers/aside-available.ts'], 'qa-b7-spa': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'qa/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval-spa.html', 'test/fixtures/qa-eval-spa-ground-truth.json', 'test/skill-e2e-qa-bugs.test.ts', 'test/helpers/aside-available.ts'], 'qa-b8-checkout': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'qa/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval-checkout.html', 'test/fixtures/qa-eval-checkout-ground-truth.json', 'test/skill-e2e-qa-bugs.test.ts', 'test/helpers/aside-available.ts'], - 'qa-only-no-fix': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'test/helpers/qa-only-cleanup.ts', 'test/helpers/qa-checkpoint-evidence.ts', 'test/fixtures/qa-only-observation-public.json', 'test/fixtures/qa-only-stop-cleanup-public.json', 'test/fixtures/qa-only-charter-public.json', 'test/fixtures/qa-only-browser-probe.ts', 'browse/test/fixtures/qa-only.html', 'test/helpers/qa-browser-deadline-evidence.ts', 'test/helpers/read-only-commands.ts', 'bin/gstack-qa-deadline', 'lib/qa-deadline.ts', 'lib/bun-isolation.ts', 'lib/claude-code-windows-job.ts', 'qa-only/**', 'qa/sections/**', 'qa/templates/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/skill-e2e-qa-workflow.test.ts', 'test/helpers/aside-available.ts', 'test/helpers/bootstrap-retention.ts', 'test/helpers/qa-evidence-producer.ts'], + 'qa-only-no-fix': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'test/helpers/qa-only-cleanup.ts', 'test/helpers/qa-checkpoint-evidence.ts', 'test/fixtures/qa-only-observation-public.json', 'test/fixtures/qa-only-stop-cleanup-public.json', 'test/fixtures/qa-only-bookkeeping-joins-public.json', 'test/fixtures/qa-only-charter-public.json', 'test/fixtures/qa-only-browser-probe.ts', 'browse/test/fixtures/qa-only.html', 'test/helpers/qa-browser-deadline-evidence.ts', 'test/helpers/read-only-commands.ts', 'bin/gstack-qa-deadline', 'lib/qa-deadline.ts', 'lib/bun-isolation.ts', 'lib/claude-code-windows-job.ts', 'qa-only/**', 'qa/sections/**', 'qa/templates/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/skill-e2e-qa-workflow.test.ts', 'test/helpers/aside-available.ts', 'test/helpers/bootstrap-retention.ts', 'test/helpers/qa-evidence-producer.ts'], 'qa-fix-loop': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-qa-deadline', 'lib/qa-deadline.ts', 'lib/bun-isolation.ts', 'lib/claude-code-windows-job.ts', 'qa/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/skill-e2e-qa-workflow.test.ts', 'test/helpers/aside-available.ts', 'test/fixtures/qa-only-browser-probe.ts', 'test/helpers/bootstrap-retention.ts', 'test/helpers/qa-browser-deadline-evidence.ts', 'test/helpers/read-only-commands.ts', 'test/helpers/qa-checkpoint-evidence.ts', 'test/helpers/qa-evidence-producer.ts', 'test/helpers/qa-only-cleanup.ts'], 'qa-bootstrap': ['test/helpers/bootstrap-retention.ts', 'qa/**', 'ship/**', 'test/skill-e2e-qa-workflow.test.ts', diff --git a/test/qa-browser-deadline-evidence.test.ts b/test/qa-browser-deadline-evidence.test.ts index 87d3d36c2..e482d53c5 100644 --- a/test/qa-browser-deadline-evidence.test.ts +++ b/test/qa-browser-deadline-evidence.test.ts @@ -7,6 +7,7 @@ import { assertQaBrowserDeadline, assertQaBrowserPreparation, assertQaBrowserChe import capturedPreparation from './fixtures/qa-only-charter-public.json'; import capturedObservation from './fixtures/qa-only-observation-public.json'; import capturedStop from './fixtures/qa-only-stop-cleanup-public.json'; +import capturedBookkeeping from './fixtures/qa-only-bookkeeping-joins-public.json'; const ROOT = path.resolve(import.meta.dir, '..'); const guard = path.join(ROOT, 'bin/gstack-qa-deadline'); @@ -223,7 +224,7 @@ test.each(['extra-option', 'suffix-option', 'operator', 'newline', 'foreign-cwd' const commands = { 'extra-option': `git -C ${quote(f.directory)} -c color.ui=false rev-parse HEAD`, 'suffix-option': metadata + ' --git-dir', - operator: metadata + '; ' + metadata, + operator: metadata + '; ' + quote(browse) + ' --version', newline: metadata + '\n' + metadata, 'foreign-cwd': `git -C ${quote(path.dirname(f.directory))} rev-parse HEAD`, 'missing-cwd': 'git rev-parse HEAD', @@ -232,6 +233,76 @@ test.each(['extra-option', 'suffix-option', 'operator', 'newline', 'foreign-cwd' expect(() => f.check()).toThrow('QA deadline:'); }); +// Opus gates 39313931713, 39332134395 and 39336730429 (slice 8, attempt 1) failed qa-only-no-fix with +// "unsupported outer shell composition" after joining reads with an exact bookkeeping call, as in +// `ls -la; git -C branch --show-current`. Every non-guard Bash command from those three +// attempts and from the eight passing attempts beside them replays inside a real guarded attempt. +const bookkeepingReplay = (f: ReturnType, command: string) => command + .replaceAll('__QA_FIXTURE__', f.directory).replaceAll('__QA_GUARD__', guard).replaceAll('__QA_BROWSE__', browse); +test.each(capturedBookkeeping.attempts.map(attempt => [attempt.source, attempt.passed, attempt.commands] as const)) + ('captured non-guard Bash commands replay as bookkeeping: %s (passed %p)', (_, __, commands) => { + const f = fixture(); + const start = f.calls.shift()!; + for (const command of commands) f.calls.push({ tool: 'Bash', input: { command: bookkeepingReplay(f, command) }, output: 'bookkeeping' }); + f.calls.push(start); + f.run(); + expect(f.check()).toEqual({ launchedRuns: 1, completedRuns: 1, refusedRuns: 0, timedOutRuns: 0 }); + }); + +test('the captured joins are the three failing attempts\' only unaccepted commands', () => { + const failing = capturedBookkeeping.attempts.filter(attempt => !attempt.passed); + expect(failing.map(attempt => attempt.source.split(' ')[1])).toEqual(['39313931713', '39332134395', '39336730429']); + expect(failing.flatMap(attempt => attempt.commands.filter(command => /^ls -la; /.test(command)))).toEqual([ + "ls -la; cat TODOS.md 2>/dev/null; git -C '__QA_FIXTURE__' rev-parse --short HEAD", + "ls -la; git -C '__QA_FIXTURE__' branch --show-current", + "ls -la; git -C '__QA_FIXTURE__' branch --show-current", + ]); +}); + +test.each(['captured', 'two-bookkeeping', 'and-or', 'readiness-check-joined', 'trailing-semicolon', + 'browser-after', 'browser-before', 'guard-joined', 'guard-tail', 'git-option', 'git-other-subcommand', 'pipe-to-shell', 'background', + 'substitution', 'newline', 'cd-then-read', 'read-outside-root', 'dangling-and', 'escaped-semicolon', 'quoted-operator', 'receipt-in-output', 'readiness-block-joined']) + ('bookkeeping joins accept exact calls and grammar reads only: %s', scenario => { + const f = fixture(); + f.run(); + const status = `git -C ${quote(f.directory)} status --short`; + const branch = `git -C ${quote(f.directory)} branch --show-current`; + const guarded = ['bun', guard, 'run', f.policy.file, '--', browse, '--version'].map(quote).join(' '); + const commands: Record = { + captured: `ls -la; ${branch}`, + 'two-bookkeeping': `${status}; ${branch}`, + 'and-or': `ls -la qa-reports && ${status} || echo none`, + 'readiness-check-joined': `${f.policy.allowed[3]}; ${branch}`, + 'trailing-semicolon': `ls -la; ${branch};`, + 'browser-after': `ls -la; ${branch}; ${quote(browse)} goto http://127.0.0.1:1/`, + 'browser-before': `${quote(browse)} --version && ${status}`, + 'guard-joined': `${status} && ${guarded}`, + 'guard-tail': `${guarded}; ${status}`, + 'git-option': `ls -la; git -C ${quote(f.directory)} -c core.pager=cat log -1`, + 'git-other-subcommand': `ls -la; git -C ${quote(f.directory)} config --list`, + 'pipe-to-shell': `${status} | sh`, + background: `${status} & ${quote(browse)} --version`, + substitution: `ls -la "$(${quote(browse)} --version)"; ${status}`, + newline: `ls -la\n${status}`, + 'cd-then-read': `cd qa-reports; cat deadline.json; ${status}`, + 'read-outside-root': `cat /etc/hostname; ${status}`, + 'dangling-and': `ls -la; ${status} &&`, + 'escaped-semicolon': `ls -la \\; ${quote(browse)} --version; ${status}`, + 'quoted-operator': `echo 'a; b'; ${status}`, + 'receipt-in-output': `ls -la; ${status}`, + 'readiness-block-joined': `${f.policy.allowed[0]}; ${status}`, + }; + f.calls.push({ tool: 'Bash', input: { command: commands[scenario] }, output: scenario === 'receipt-in-output' ? 'QA_DEADLINE {"guard":"qa-deadline"}' : 'bookkeeping' }); + if (['captured', 'two-bookkeeping', 'and-or', 'readiness-check-joined', 'trailing-semicolon', 'quoted-operator'].includes(scenario)) expect(f.check()).toEqual({ launchedRuns: 1, completedRuns: 1, refusedRuns: 0, timedOutRuns: 0 }); + else expect(() => f.check()).toThrow(scenario === 'receipt-in-output' ? 'receipt outside trusted guard invocation' : 'QA deadline:'); + }); + +test('the deadline prompt declares the bookkeeping join it accepts', () => { + const { policy } = fixture(); + expect(policy.prompt).toContain('join several of them and the shared-grammar reads below with ;, && or || (no cd in such a join)'); + expect(policy.prompt).toContain('as its own Bash call (no appended commands)'); +}); + test('quoted question marks stay literal but bare glob expansion is rejected', () => { const f = fixture(); fs.writeFileSync(path.join(f.directory, 'x'), 'glob target'); diff --git a/test/qa-exploratory-callers.test.ts b/test/qa-exploratory-callers.test.ts index 24eb0b194..82e8cb953 100644 --- a/test/qa-exploratory-callers.test.ts +++ b/test/qa-exploratory-callers.test.ts @@ -16,6 +16,7 @@ import { readQACheckpointFiles } from './helpers/qa-checkpoint-evidence'; import { generateQAExploratory, generateQAResource, generateQAReview, generateQAReviewPreflight } from '../scripts/resolvers/qa'; import { HOST_PATHS } from '../scripts/resolvers/types'; import { qaProbeNames } from './helpers/qa-probe-names'; +import capturedHandoffOrder from './fixtures/qa-caller-handoff-order-public.json'; function nativeCall(id: string, name: string, input: object, output: string, parent: string | null = null, failed = false) { return [ @@ -246,6 +247,22 @@ describe('caller native-event observer controls', () => { expect(validateCallerEvidence(observed)).toContain('review completion preceded handoff freshness decision'); }); + // Opus gates 39320737890 attempt 1 and 39332134395 attempt 2 (slice 8, review-exploratory-small-cli) wrote the + // completed:true adversarial-review record in the same response as the first HANDOFF.md Read, or before any Read. + // The same public events pass once the record follows every handoff result in its own response. + test.each(capturedHandoffOrder.attempts.map(attempt => [attempt.source, attempt] as const)) + ('captured adversarial-review completion before its handoff result is flagged; reordered it passes: %s', (_, attempt) => { + const observed = evidence(); + const base = observed.result.transcript; + observed.result.transcript = [...base, ...structuredClone(attempt.events)]; + expect(validateCallerEvidence(observed)).toEqual(['review completion preceded handoff freshness decision']); + const ordered = structuredClone(attempt.events) as any[]; + const [use, result] = ordered.splice(ordered.findIndex(event => event.message.content[0].name === 'Bash'), 2); + use.message.id = 'record-after-handoff-results'; + observed.result.transcript = [...base, ...ordered, use, result]; + expect(validateCallerEvidence(observed)).toEqual([]); + }); + test('captured gate-census-5 review record: bun-wrapped helper and receipt status are outside the interface', () => { // ci-36629958451-1-gate-census-5 review-exploratory-small-cli, native events 68 and 70 (fixture paths shortened). const record = (status: string) => `/runtime/bin/gstack-review-log '{"skill":"review","timestamp":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'","commit":"'"$(git rev-parse --short HEAD)"'","branch":"caller-change","status":"${status}","completed":false,"converged":false,"critical":1,"informational":0,"findings":[{"path":"scale.ts","line":3,"category":"functional-contract","severity":"CRITICAL","fingerprint":"scale.ts:3:functional-contract","action":"ask-pending"}]}' --finish d45404cf-ba19-4bc6-a505-9801e9322f54`; @@ -892,6 +909,62 @@ describe('real caller-specific native fixture and capture boundary', () => { const probe = (fixture: QaCallerFixture, value: string) => spawnSync(process.execPath, ['scripts/probe.ts', value], { cwd: fixture.cwd, encoding: 'utf8', timeout: 5000 }); const dispose = async (fixture: QaCallerFixture) => { await fixture.close(); fs.rmSync(fixture.root, { recursive: true, force: true }); }; + // Opus gate 39320737890 attempt 2 slice 8 (ship-exploratory-plan-checks, head f278f5b): capture 001 was the + // guarded smoke probe of 9, 002 probed 10, and 003 ran PLAN.md's required release check of 9 with + // --timeout-ms 10000, the form the prompt reserves for an explicit plan check outside the smoke budget. + // The validator read 003 as a wasted rerun. One explicit plan check beside the smoke probe is verification; + // a second smoke or plan-check run of the same unchanged input is still a duplicate. + test.each([ + ['captured', ['deadline 9', 'deadline 10', 'timeout 9'], undefined], + ['plan check guarded like a smoke probe', ['deadline 9', 'deadline 10', 'deadline 9'], 2], + ['plan check repeated', ['deadline 9', 'deadline 10', 'timeout 9', 'timeout 9'], 3], + ['ordinary input under the plan-check form', ['deadline 3', 'deadline 10', 'timeout 3'], 2], + ] as const)('an explicit plan check is not a duplicate of the smoke probe: %s', async (_, steps, duplicate) => { + const fixture = await fixtureFor('ship-exploratory-plan-checks', true); + try { + const reportRoot = path.join(fixture.cwd, 'reports'); + const state = path.join(reportRoot, 'deadline.json'); + const evidenceTool = path.join(fixture.runtime, 'bin/gstack-qa-evidence'); + const result = await runQaCaller(fixture, 'free-plan-check-replay', async options => { + const transcript: unknown[] = [ + ...nativeCall('parent', 'Read', { file_path: `${fixture.cwd}/caller-ship.md` }, 'parent workflow'), + ...nativeCall('shared', 'Read', { file_path: `${fixture.runtime}/qa/sections/exploratory.md` }, 'shared method'), + ...nativeCall('functional', 'Read', { file_path: `${fixture.runtime}/qa/sections/system-functional.md` }, 'functional method'), + ...nativeCall('ship-army', 'Read', { file_path: `${fixture.runtime}/ship/sections/review-army.md` }, 'ship Step 9'), + ]; + const execute = (id: string, cmd: string) => { + expect(qaCallerCommandAllowed(cmd, fixture.workflowCommands, { runtime: fixture.runtime, fixtureRoot: fixture.cwd }), cmd).toBe(true); + const actual = spawnSync('bash', ['-c', cmd], { cwd: options.workingDirectory, env: { ...process.env, ...options.env }, encoding: 'utf8', timeout: 10_000 }); + expect(actual.error).toBeUndefined(); + transcript.push(...nativeCall(id, 'Bash', { command: cmd }, (actual.status ? `Exit code ${actual.status}\n` : '') + actual.stdout + actual.stderr, null, actual.status !== 0)); + }; + execute('start', `bun ${path.join(fixture.runtime, 'bin/gstack-qa-deadline')} start ${state} 300`); + for (const [index, step] of steps.entries()) { + const [lane, value] = step.split(' '); + const id = String(index + 1).padStart(3, '0'); + const bound = lane === 'deadline' ? `--deadline ${state}` : '--timeout-ms 10000'; + const after = index ? ` --after ${String(index).padStart(3, '0')} --hypothesis 'Input ${value} checks the next documented contract.'` : ''; + execute(`capture-${id}`, `bun ${evidenceTool} capture ${reportRoot} ${id} --public ${bound}${after} -- bun scripts/probe.ts ${value}`); + } + return { exitReason: 'success', transcript } as SkillTestResult; + }); + await fixture.close(); + const probes = fixture.probes(); + expect(probes).toHaveLength(steps.length); + const checkpointFiles = readQACheckpointFiles(reportRoot); + const errors = validateCallerEvidence({ + caller: fixture.caller, result, probes, + receipt: { status: 'pass', probes: probes.map(probe => probe.id), remaining: [] }, + currentSnapshot: fixture.snapshot(), requiredCharters: ['happy', 'adverse', 'plan:nine'], mutations: fixture.mutationEvents, + observerComplete: fixture.observation?.complete === true && !fixture.observerErrors.length, + fixtureRoot: fixture.cwd, runtime: fixture.runtime, reportRoot, requireGuardedSmoke: true, requireCapturedEvidence: true, + checkpointFiles, reportMarkdown: Object.keys(checkpointFiles).map(name => `[checkpoint](${name})`).join('\n'), + }); + if (duplicate === undefined) expect(errors).toEqual([]); + else expect(errors).toContain(`duplicate unchanged passing probe: ${probes[duplicate].id}`); + } finally { await dispose(fixture); } + }); + test('deadline authority is closed over the installed helper, owned state and literal probe child', async () => { const fixture = await fixtureFor('ship-exploratory-small-cli', true); try { @@ -1503,7 +1576,8 @@ describe('real caller-specific native fixture and capture boundary', () => { expect(options.appendSystemPrompt).not.toMatch(/bun scripts\/probe\.ts \d|invalid input|highest.risk/i); expect(options.prompt).toMatch(/keep normal parent decision gates/i); expect(options.prompt).toMatch(/Hard deadline UTC, never its Runner entry UTC/i); - expect(options.prompt).toMatch(/completed:true review record, read HANDOFF\.md/i); + expect(options.prompt).toMatch(/Handoff freshness check: Read HANDOFF\.md and reports\/HANDOFF\.md/); + expect(options.prompt).toMatch(/only in a later message than the check's Read results: the evidence helper's materialize, which runs once per report root and freezes the verdict; every gstack-review-log completed:true record, adversarial-review included/); expect(options.prompt).toMatch(/do not publish a checkpoint for it/i); expect(options.prompt).toMatch(/authenticated expired-capture result/i); expect(options.prompt).toContain('reports/review.md');