mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
v1.91.168.0 fix(evals): QA deadline joins, QA-callers handoff order and plan-check probes (GSTA-57) (#3321)
* fix(evals): QA deadline joins, QA-callers handoff order and plan-check probes (GSTA-57) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(fixtures): register the QA-callers handoff-order capture as a free fixture Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.168.0 stamp Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed
1 parent
6a1b3be141
commit
fc046b5c83
12 files changed
+685
-12
No files matched your search
@@ -1,5 +1,15 @@
|
||||
# Changelog
|
||||
|
||||
## [1.91.168.0] - 2026-10-20
|
||||
|
||||
**Four QA paid cases stop failing on valid or unclear agent behavior: `qa-only-no-fix` accepts reads joined with its exact bookkeeping calls, and the QA-callers fixture says where the handoff check goes and stops counting the explicit plan check as a wasted rerun.** Daily flaky-fix pass (GSTA-57), from the captured events of Opus gate runs 39313931713, 39332134395, 39336730429, 39320737890, 39181697029 and 39351737002.
|
||||
|
||||
### Changed
|
||||
- **Gate integrity, widening (`test/helpers/qa-browser-deadline-evidence.ts`):** `qa-only-no-fix` failed three times with `QA deadline: unsupported outer shell composition`. Each time the agent joined grammar reads with one of the fixture's exact bookkeeping calls outside the guard, for example `ls -la; git -C <fixture> branch --show-current`. The deadline check now accepts a `;`, `&&` or `||` join in which every command is an exact listed bookkeeping call or a shared-grammar read checked from the fixture root, and the fixture prompt declares that join. Still rejected, each with a negative control: a joined guard call, a browser command before or after the join, a git call outside the exact list (`-c` options, other subcommands), `cd` inside a join, a pipe into `sh`, `&`, substitutions, newlines, escaped `;`, a dangling `&&`, a read outside the root, the readiness block with an appended command, and a `QA_DEADLINE ` receipt in the output. The existing "unbatched metadata" control now joins the metadata call with an unguarded browser command, since two exact calls joined are accepted. A free replay runs all 100 non-guard Bash commands from 11 captured attempts (3 failed, 8 passed) inside a real guarded attempt; the three failed attempts fail on the previous helper with the CI message. No threshold, kind, trial or quarantine change. The replay fixture `test/fixtures/qa-only-bookkeeping-joins-public.json` is registered in the `qa-only-no-fix` touchfiles.
|
||||
- **Gate integrity, widening (`validateCallerEvidence`):** `ship-exploratory-plan-checks` failed with `duplicate unchanged passing probe` after the agent ran `bun scripts/probe.ts 9` as a guarded smoke probe and then as PLAN.md's release check with `--timeout-ms 10000`, the form the prompt reserves for an explicit plan check outside the smoke budget. One explicit plan check beside the smoke probe is no longer a duplicate. A guarded repeat of the same input, a second plan check, and ordinary input under the plan-check form still are. A free replay runs the captured capture sequence through the real helpers in a real fixture.
|
||||
- **QA-callers prompt (`qaCallerSessionOptions`):** the handoff freshness check is one rule. Read HANDOFF.md and reports/HANDOFF.md and wait for the results in an earlier message than each of three things: `materialize` (it runs once per report root and freezes the verdict), every `completed:true` review record including `adversarial-review`, and the completion report. `review-exploratory-small-cli` failed twice when the agent wrote the adversarial-review record in the same response as its first HANDOFF Read (39320737890) or before any Read (39332134395); the detector was right, and a free replay of both captured event sets keeps them failing and passes them reordered. `ship-exploratory-late-input` failed twice when the agent materialized before reading the late reports/HANDOFF.md, so a refused second `materialize` left a stale `pass` verdict beside an honest `inconclusive` receipt.
|
||||
- **Focused paid runs (Opus 5.5, gate tier, this tree):** `qa-only-no-fix`, `review-exploratory-small-cli`, `ship-exploratory-plan-checks` and `ship-exploratory-late-input` each passed 5 of 5 ($14.98 total). One `qa-only-no-fix` trial used two bookkeeping joins that the previous helper would have failed.
|
||||
|
||||
## [1.91.167.0] - 2026-10-20
|
||||
|
||||
**Two paid detectors now accept the rewordings Opus 5.5 actually uses, with their negative controls kept (daily flaky-fix pass, GSTA-57).** `plan-design-with-ui-scope` went red three times on the ledger with "no acknowledged Design focus" while the agent had asked Step 0D and a numbered finding; `tpa-absent-darwin` went red once on a valid download pitch.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.91.167.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.91.168.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.91.167",
|
||||
"version": "1.91.168",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
+301
@@ -0,0 +1,301 @@
|
||||
{
|
||||
"note": "Public HANDOFF.md Reads and completed:true gstack-review-log records, with their results, from two failed review-exploratory-small-cli attempts; fixture paths are rewritten to /fixture-root.",
|
||||
"attempts": [
|
||||
{
|
||||
"source": "run 39320737890 paid-slice-8-a1 review-exploratory-small-cli",
|
||||
"head": "f278f5b",
|
||||
"sourceEventIndices": [
|
||||
541,
|
||||
547,
|
||||
580,
|
||||
584,
|
||||
817,
|
||||
821
|
||||
],
|
||||
"events": [
|
||||
{
|
||||
"type": "assistant",
|
||||
"session_id": "0fa1b922-1e77-4aa4-a152-a33997782193",
|
||||
"parent_tool_use_id": null,
|
||||
"message": {
|
||||
"id": "msg_011CgCFmhNcdng5NVrYMeEXc",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_018VDo7yRPYpS71H6uZZQXfr",
|
||||
"name": "Read",
|
||||
"input": {
|
||||
"file_path": "/fixture-root/product/HANDOFF.md"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"session_id": "0fa1b922-1e77-4aa4-a152-a33997782193",
|
||||
"parent_tool_use_id": null,
|
||||
"tool_use_result": {
|
||||
"type": "text",
|
||||
"file": {
|
||||
"filePath": "/fixture-root/product/HANDOFF.md",
|
||||
"content": "No concurrent input update.\n",
|
||||
"numLines": 2,
|
||||
"startLine": 1,
|
||||
"totalLines": 2
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"tool_use_id": "toolu_018VDo7yRPYpS71H6uZZQXfr",
|
||||
"type": "tool_result",
|
||||
"content": "1\tNo concurrent input update.\n2\t"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"session_id": "0fa1b922-1e77-4aa4-a152-a33997782193",
|
||||
"parent_tool_use_id": null,
|
||||
"message": {
|
||||
"id": "msg_011CgCFmhNcdng5NVrYMeEXc",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_01RE7HZrWaWAoZyEhpSQthmp",
|
||||
"name": "Bash",
|
||||
"input": {
|
||||
"command": "/fixture-root/host/runtime/bin/gstack-review-log '{\"skill\":\"adversarial-review\",\"timestamp\":\"'\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"'\",\"status\":\"issues_found\",\"source\":\"in-host\",\"host\":\"claude\",\"outside_provider\":\"codex\",\"outside_status\":\"unavailable\",\"phase\":\"adversarial\",\"tier\":\"always\",\"gate\":\"informational\",\"commit\":\"'\"$(git rev-parse --short HEAD)\"'\",\"completed\":true,\"converged\":true}' --finish be9a452e-5250-4734-8c5a-3cb4922ea744",
|
||||
"description": "Persist native adversarial review record"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"session_id": "0fa1b922-1e77-4aa4-a152-a33997782193",
|
||||
"parent_tool_use_id": null,
|
||||
"tool_use_result": {
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
"interrupted": false,
|
||||
"isImage": false,
|
||||
"noOutputExpected": false
|
||||
},
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"tool_use_id": "toolu_01RE7HZrWaWAoZyEhpSQthmp",
|
||||
"type": "tool_result",
|
||||
"content": "(Bash completed with no output)",
|
||||
"is_error": false
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"session_id": "0fa1b922-1e77-4aa4-a152-a33997782193",
|
||||
"parent_tool_use_id": null,
|
||||
"message": {
|
||||
"id": "msg_011CgCFoAqtJycy2UWEB8WG7",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_011TvxdKFapV3hPc7FKpnW7G",
|
||||
"name": "Read",
|
||||
"input": {
|
||||
"file_path": "/fixture-root/product/HANDOFF.md"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"session_id": "0fa1b922-1e77-4aa4-a152-a33997782193",
|
||||
"parent_tool_use_id": null,
|
||||
"tool_use_result": {
|
||||
"type": "file_unchanged",
|
||||
"file": {
|
||||
"filePath": "/fixture-root/product/HANDOFF.md"
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"tool_use_id": "toolu_011TvxdKFapV3hPc7FKpnW7G",
|
||||
"type": "tool_result",
|
||||
"content": "Wasted call \u2014 file unchanged since your last Read. Refer to that earlier tool_result instead."
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "run 39332134395 paid-slice-8-a2 review-exploratory-small-cli",
|
||||
"head": "8802d99",
|
||||
"sourceEventIndices": [
|
||||
745,
|
||||
749,
|
||||
777,
|
||||
781,
|
||||
790,
|
||||
794
|
||||
],
|
||||
"events": [
|
||||
{
|
||||
"type": "assistant",
|
||||
"session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e",
|
||||
"parent_tool_use_id": null,
|
||||
"message": {
|
||||
"id": "msg_011CgCP93PHjujPRobVJASRi",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_018xntzUCrFxBhzJTuszB54U",
|
||||
"name": "Bash",
|
||||
"input": {
|
||||
"command": "/fixture-root/host/runtime/bin/gstack-review-log '{\"skill\":\"adversarial-review\",\"timestamp\":\"'\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"'\",\"status\":\"issues_found\",\"source\":\"in-host\",\"host\":\"claude\",\"outside_provider\":\"codex\",\"outside_status\":\"unavailable\",\"phase\":\"adversarial\",\"tier\":\"always\",\"gate\":\"informational\",\"commit\":\"'\"$(git rev-parse --short HEAD)\"'\",\"completed\":true,\"converged\":true}' --finish 0fa0d55a-16d1-4fa5-a65c-98ba0d2adb3a",
|
||||
"description": "Persist native adversarial review record"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e",
|
||||
"parent_tool_use_id": null,
|
||||
"tool_use_result": {
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
"interrupted": false,
|
||||
"isImage": false,
|
||||
"noOutputExpected": false
|
||||
},
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"tool_use_id": "toolu_018xntzUCrFxBhzJTuszB54U",
|
||||
"type": "tool_result",
|
||||
"content": "(Bash completed with no output)",
|
||||
"is_error": false
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e",
|
||||
"parent_tool_use_id": null,
|
||||
"message": {
|
||||
"id": "msg_011CgCP9T1qta9KojKXBkn7S",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_01KxpCiHqeq6EjStFc1eGKKm",
|
||||
"name": "Read",
|
||||
"input": {
|
||||
"file_path": "/fixture-root/product/HANDOFF.md"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e",
|
||||
"parent_tool_use_id": null,
|
||||
"tool_use_result": {
|
||||
"type": "text",
|
||||
"file": {
|
||||
"filePath": "/fixture-root/product/HANDOFF.md",
|
||||
"content": "No concurrent input update.\n",
|
||||
"numLines": 2,
|
||||
"startLine": 1,
|
||||
"totalLines": 2
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"tool_use_id": "toolu_01KxpCiHqeq6EjStFc1eGKKm",
|
||||
"type": "tool_result",
|
||||
"content": "1\tNo concurrent input update.\n2\t"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e",
|
||||
"parent_tool_use_id": null,
|
||||
"message": {
|
||||
"id": "msg_011CgCP9T1qta9KojKXBkn7S",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_01FbKjc6xXVDmtcU9Zd3GATA",
|
||||
"name": "Read",
|
||||
"input": {
|
||||
"file_path": "/fixture-root/product/reports/HANDOFF.md"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"session_id": "e05cf21a-eced-4ee5-a031-89a8c253c54e",
|
||||
"parent_tool_use_id": null,
|
||||
"tool_use_result": "Error: File does not exist. Note: your current working directory is /fixture-root/product.",
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"content": "File does not exist. Note: your current working directory is /fixture-root/product.",
|
||||
"is_error": true,
|
||||
"tool_use_id": "toolu_01FbKjc6xXVDmtcU9Zd3GATA"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
{
|
||||
"note": "Every top-level non-guard Bash command from 11 captured qa-only-no-fix attempts; guard calls are omitted because their receipts belong to the original clock.",
|
||||
"attempts": [
|
||||
{
|
||||
"source": "run 39181697029 paid-slice-8-a1 qa-only-no-fix",
|
||||
"passed": true,
|
||||
"commands": [
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"git -C '__QA_FIXTURE__' log -1 --format=%cI"
|
||||
],
|
||||
"head": "f0722bc"
|
||||
},
|
||||
{
|
||||
"source": "run 39313931713 paid-slice-8-a1 qa-only-no-fix",
|
||||
"passed": false,
|
||||
"commands": [
|
||||
"wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80",
|
||||
"sed -n 389,603p qa-only/SKILL.md",
|
||||
"cat qa-only/sections/exploratory.md",
|
||||
"cat qa/sections/browser-setup.md",
|
||||
"cat qa/sections/scope.md; echo ======; cat qa/sections/qa-patterns.md",
|
||||
"cat qa/sections/qa-patterns.md",
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"ls -la; cat TODOS.md 2>/dev/null; git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ"
|
||||
],
|
||||
"head": "5186498"
|
||||
},
|
||||
{
|
||||
"source": "run 39313931713 paid-slice-8-a2 qa-only-no-fix",
|
||||
"passed": true,
|
||||
"commands": [
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports/screenshots'",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"ls -la __QA_FIXTURE__"
|
||||
],
|
||||
"head": "5186498"
|
||||
},
|
||||
{
|
||||
"source": "run 39320737890 paid-slice-8-a1 qa-only-no-fix",
|
||||
"passed": true,
|
||||
"commands": [
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports/screenshots'",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"git -C '__QA_FIXTURE__' status --short",
|
||||
"ls -la __QA_FIXTURE__",
|
||||
"git -C '__QA_FIXTURE__' log -1 --format=%cI",
|
||||
"grep -n \"baseline.json\" __QA_FIXTURE__/qa-reports/qa-only-report.md"
|
||||
],
|
||||
"head": "f278f5b"
|
||||
},
|
||||
{
|
||||
"source": "run 39320737890 paid-slice-8-a2 qa-only-no-fix",
|
||||
"passed": true,
|
||||
"commands": [
|
||||
"wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80",
|
||||
"sed -n 389,603p qa-only/SKILL.md",
|
||||
"cat qa-only/sections/exploratory.md; ls qa/sections qa/templates qa-only/sections",
|
||||
"cat qa/sections/scope.md",
|
||||
"cat qa/sections/qa-patterns.md",
|
||||
"cat qa/sections/browser-setup.md",
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"ls -la; find . -maxdepth 1 -iname 'TODOS.md' -o -maxdepth 1 -iname 'README*' -o -maxdepth 1 -name package.json",
|
||||
"cat qa-only/sections/reporting.md; echo ======; cat qa/templates/qa-report-template.md"
|
||||
],
|
||||
"head": "f278f5b"
|
||||
},
|
||||
{
|
||||
"source": "run 39320737890 paid-slice-8-a3 qa-only-no-fix",
|
||||
"passed": true,
|
||||
"commands": [
|
||||
"wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md",
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"git -C '__QA_FIXTURE__' log -1 --format=%cI",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports/screenshots'"
|
||||
],
|
||||
"head": "f278f5b"
|
||||
},
|
||||
{
|
||||
"source": "run 39332134395 paid-slice-8-a1 qa-only-no-fix",
|
||||
"passed": false,
|
||||
"commands": [
|
||||
"wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80",
|
||||
"sed -n 378,592p qa-only/SKILL.md",
|
||||
"cat qa-only/sections/exploratory.md; echo ======; cat qa/sections/browser-setup.md",
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports/screenshots'",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"ls -la; git -C '__QA_FIXTURE__' branch --show-current"
|
||||
],
|
||||
"head": "8802d99"
|
||||
},
|
||||
{
|
||||
"source": "run 39332134395 paid-slice-8-a2 qa-only-no-fix",
|
||||
"passed": true,
|
||||
"commands": [
|
||||
"wc -l qa-only/SKILL.md; ls qa-only qa qa-only/sections qa/sections qa/templates 2>/dev/null",
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports/screenshots'",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"git -C '__QA_FIXTURE__' branch --show-current",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"ls -la; find . -maxdepth 2 -not -path './.git*' -type f | sort | head -50",
|
||||
"cat fixture-browser-probe.ts"
|
||||
],
|
||||
"head": "8802d99"
|
||||
},
|
||||
{
|
||||
"source": "run 39332134395 paid-slice-8-a3 qa-only-no-fix",
|
||||
"passed": true,
|
||||
"commands": [
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ"
|
||||
],
|
||||
"head": "8802d99"
|
||||
},
|
||||
{
|
||||
"source": "run 39336730429 paid-slice-8-a1 qa-only-no-fix",
|
||||
"passed": false,
|
||||
"commands": [
|
||||
"wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80",
|
||||
"cat qa-only/sections/exploratory.md",
|
||||
"cat qa/sections/browser-setup.md; ls -la qa-reports qa-reports/screenshots; ls qa/sections qa-only/sections qa/templates",
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"ls -la; git -C '__QA_FIXTURE__' branch --show-current",
|
||||
"git -C '__QA_FIXTURE__' log -1 --format=%cI"
|
||||
],
|
||||
"head": "afa0bbf"
|
||||
},
|
||||
{
|
||||
"source": "run 39336730429 paid-slice-8-a2 qa-only-no-fix",
|
||||
"passed": true,
|
||||
"commands": [
|
||||
"wc -l qa-only/SKILL.md; grep -n '^#' qa-only/SKILL.md | head -80",
|
||||
"sed -n 389,603p qa-only/SKILL.md",
|
||||
"cat qa-only/sections/exploratory.md",
|
||||
"cat qa/sections/browser-setup.md",
|
||||
"cat qa/sections/scope.md; echo ======; cat qa/sections/qa-patterns.md",
|
||||
"_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\n unset _o\nfi",
|
||||
"ls -la '__QA_FIXTURE__/qa-reports'",
|
||||
"[ -x '__QA_BROWSE__' ] && echo READY || echo NEEDS_SETUP",
|
||||
"date -u +%Y-%m-%dT%H:%M:%SZ",
|
||||
"git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"ls; ls qa/templates; ls -a screenshots qa-reports/screenshots 2>/dev/null"
|
||||
],
|
||||
"head": "afa0bbf"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -159,6 +159,7 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
|
||||
'test/fixtures/plan-ceo-floor-unasked-decisions-872e154.json': ['test/plan-ceo-floor-interactive-boundary.test.ts'],
|
||||
'test/fixtures/plan-floor-edit-wrap-37176835584.json': ['test/plan-floor-edit-wrap.test.ts'],
|
||||
'test/fixtures/plan-format-kind-notes.json': ['test/plan-format-kind-note.test.ts'],
|
||||
'test/fixtures/qa-caller-handoff-order-public.json': ['test/qa-exploratory-callers.test.ts'],
|
||||
'test/fixtures/qa-caller-read-probe-mtb1007a.json': ['test/qa-exploratory-callers.test.ts'],
|
||||
'test/fixtures/qa-caller-section-skip-ship-small-cli.json': ['test/qa-exploratory-callers.test.ts'],
|
||||
'test/fixtures/plans/autoplan-dashboard.md': ['test/autoplan-init.test.ts', 'test/autoplan-preconfigured-onboarding-ar.test.ts', 'test/autoplan-snapshot.test.ts'],
|
||||
|
||||
@@ -23,7 +23,7 @@ export function qaDeadlineShellPolicy(directory: string, guard: string, browse:
|
||||
`git -C ${quote(directory)} log -1 --format=%cI`,
|
||||
`git -C ${quote(directory)} branch --show-current`];
|
||||
const file = path.join(directory, 'qa-reports/deadline.json');
|
||||
return { allowed, file, prompt: `This native fixture accepts a deliberately narrow executable outer form. Run the BROWSER SETUP Aside readiness bash block verbatim, as its own Bash call (no appended commands). The fallback binary is already supplied. Other than that block, these standalone setup/bookkeeping Bash calls are allowed:
|
||||
return { allowed, file, prompt: `This native fixture accepts a deliberately narrow executable outer form. Run the BROWSER SETUP Aside readiness bash block verbatim, as its own Bash call (no appended commands). The fallback binary is already supplied. Other than that block, these setup/bookkeeping Bash calls are allowed, each exactly as written. One Bash call may run one of them alone or join several of them and the shared-grammar reads below with ;, && or || (no cd in such a join):
|
||||
${allowed.slice(1).join('\n')}
|
||||
${readOnlyInterfaceText(directory)}
|
||||
Use Read/Glob for any other local inspection and Write only inside ${path.join(directory, 'qa-reports')}; the report and screenshots directories already exist. Memory files and learning stores outside that directory are not authorized. Write the initial charters and final report to the same caller-owned file ${path.join(directory, 'qa-reports/qa-only-report.md')}, not a separate charter file.
|
||||
@@ -47,6 +47,38 @@ function literalArgv(command: string): string[] {
|
||||
return words;
|
||||
}
|
||||
|
||||
/**
|
||||
* A ;, && or || join of the fixture's exact bookkeeping calls with each other and with shared-grammar
|
||||
* reads, as the fixture prompt declares. Each joined command must be an exact bookkeeping call or a
|
||||
* grammar read checked from the fixture root, so cd is excluded; guard calls and browser work never qualify.
|
||||
*/
|
||||
function bookkeepingJoin(command: string, bookkeeping: string[], root: string): boolean {
|
||||
const segments: Array<{ text: string; exact: boolean }> = [];
|
||||
let rest = command.trim();
|
||||
while (true) {
|
||||
let end = bookkeeping.find(entry => rest.startsWith(entry) && /^[ \t]*(?:;|&&|\|\||$)/.test(rest.slice(entry.length)))?.length;
|
||||
const exact = end !== undefined;
|
||||
if (end === undefined) {
|
||||
let quoted = '';
|
||||
for (end = 0; end < rest.length; end++) {
|
||||
const c = rest[end];
|
||||
if (quoted) { if (c === quoted) quoted = ''; else if (c === '\\' && quoted === '"') end++; }
|
||||
else if (c === "'" || c === '"') quoted = c;
|
||||
else if (c === '\\') end++;
|
||||
else if (c === ';' || rest.startsWith('&&', end) || rest.startsWith('||', end)) break;
|
||||
}
|
||||
}
|
||||
segments.push({ text: rest.slice(0, end).trim(), exact });
|
||||
rest = rest.slice(end).replace(/^[ \t]+/, '');
|
||||
if (!rest) break;
|
||||
const operator = rest[0] === ';' ? 1 : 2;
|
||||
rest = rest.slice(operator).replace(/^[ \t]+/, '');
|
||||
if (!rest) { if (operator === 1) break; return false; }
|
||||
}
|
||||
return segments.length > 1 && segments.some(segment => segment.exact) && segments.every(segment => segment.exact
|
||||
|| (!/^cd(?:[ \t]|$)/.test(segment.text) && readOnlyCommandAllowed(segment.text, { root })));
|
||||
}
|
||||
|
||||
function owned(file: string, root: string) {
|
||||
const resolved = path.resolve(root, file);
|
||||
if (resolved !== root && !resolved.startsWith(root + path.sep)) throw new Error('QA deadline: artifact outside owned directory');
|
||||
@@ -215,7 +247,7 @@ export function assertQaBrowserDeadline(calls: Call[], options: Options & { expe
|
||||
if ((call.output ?? '').includes('QA_DEADLINE ')) fail('receipt outside trusted guard invocation');
|
||||
continue;
|
||||
}
|
||||
if (readOnlyCommandAllowed(command, { root: directory })) {
|
||||
if (readOnlyCommandAllowed(command, { root: directory }) || bookkeepingJoin(command, allowed.slice(1), directory)) {
|
||||
if ((call.output ?? '').split('\n').some(line => line.startsWith('QA_DEADLINE '))) fail('receipt outside trusted guard invocation');
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -347,9 +347,6 @@ export function validateCallerEvidence(input: {
|
||||
let guardedDiagnostics = 0;
|
||||
const checkpointProbes: Array<{ command: string; observed: CallerProbe; index: number }> = [];
|
||||
for (const probe of input.probes) {
|
||||
const key = JSON.stringify([probe.charter, probe.input, probe.snapshot]);
|
||||
if (seen.has(key) && probe.status === 'pass') errors.push(`duplicate unchanged passing probe: ${probe.id}`);
|
||||
seen.add(key);
|
||||
const tool = tools.find(tool => tool.name === 'Bash'
|
||||
&& (/^(?:\s*cd\s+[^\n&;]+\s*&&)?\s*(?:bun|[\w./-]+\/bun)\s+(?:run\s+)?(?:scripts\/probe\.ts|'scripts\/probe\.ts'|"scripts\/probe\.ts")(?:\s|$)/.test(String(tool.input.command))
|
||||
|| callerDeadlineCommand(String(tool.input.command), deadline)?.action === 'run'
|
||||
@@ -359,6 +356,11 @@ export function validateCallerEvidence(input: {
|
||||
: tool.output.split('\n').some(line => {
|
||||
try { return JSON.stringify(JSON.parse(line)) === JSON.stringify(probe); } catch { return false; }
|
||||
})));
|
||||
const planCheck = !!tool && probe.charter === 'plan:nine' && input.requiredCharters.includes('plan:nine')
|
||||
&& callerEvidenceCommand(String(tool.input.command), deadline)?.timeoutMs === 10000;
|
||||
const key = JSON.stringify([planCheck, probe.charter, probe.input, probe.snapshot]);
|
||||
if (seen.has(key) && probe.status === 'pass') errors.push(`duplicate unchanged passing probe: ${probe.id}`);
|
||||
seen.add(key);
|
||||
if (!tool) errors.push(`probe missing native command/result: ${probe.id}`);
|
||||
else {
|
||||
checkpointProbes.push({ command: String(tool.input.command), observed: probe, index: tool.index });
|
||||
@@ -697,7 +699,7 @@ export function callerReviewRecordTemplate(fixture: Pick<QaCallerFixture, 'calle
|
||||
|
||||
export function qaCallerSessionOptions(fixture: QaCallerFixture, runId: string): Parameters<typeof runSkillTest>[0] {
|
||||
return {
|
||||
prompt: `Load gstack's /${fixture.caller} supplied parent phase from caller-${fixture.caller}.md and resume it on the selected working-tree diff against origin/main. This excerpt comes from ${fixture.runtime}/${fixture.caller}/SKILL.md; resolve installed-relative references there, not from the excerpt file or product directory. That path identifies the asset base, not another entrypoint: do not read or invoke the full parent SKILL.md or rerun its preamble. Earlier preamble/branch/base setup is complete; use the existing local origin/main ref without fetch. Earlier-phase asset locators are ${fixture.runtime}/review/checklist.md and ${fixture.runtime}/qa/templates/functional-report-template.md. Read those files directly when referenced; recursive Glob does not follow the installed asset symlinks. Cross-project learnings are configured off in this owned fixture. ${fixture.reviewStart ? `The actual review-start helper already captured REVIEW_START=${fixture.reviewStart} for this unchanged core pass; retain that token. ` : ''} This fixture evaluates only the supplied parent phase, not later publication stages. The fixture owner recorded these read-only observations when this phase began; they are current until an input changes, so use them instead of re-running those commands, and recheck freshness before completion outputs:\n${fixture.startObservations}\n Read README.md for the project contract and commands. Use diagnostic-client commands such as \`bun scripts/probe.ts <literal>\` for exploratory discoveries and their checkpoint evidence. A required \`bun run test\` is separate suite verification: report it as verification, never as a diagnostic observation or checkpoint anchor/target. Use the production evidence helper to publish each diagnostic checkpoint as \`reports/exploration-NNN.json\`, not inside a nested directory; do not transcribe its observed payload. ${fixture.caseId === 'ship-exploratory-plan-checks' ? 'The previously discovered plan is PLAN.md.' : 'No plan file was found.'} There is no remote service and no release publication is authorized. There is no interactive approver; do not invent answers or permission. Keep normal parent decision gates. Before the completion report and each completed:true review record, read HANDOFF.md and reports/HANDOFF.md if present for any concurrent collaborator update, await the results, and compare evidence with current inputs. If a probe's snapshot differs from an earlier probe's, read reports/HANDOFF.md first; it names any collaborator change. A gstack-review-log completed:true record is a completion, not preliminary bookkeeping; a later handoff read cannot validate an earlier completion.\n\nQA fixture bindings: the deadline guard, capture, checkpoint and materialize command forms are in the installed qa/sections/exploratory.md this phase requires you to Read; use those forms verbatim with these bindings substituted literally. DEADLINE_TOOL = ${fixture.runtime}/bin/gstack-qa-deadline. EVIDENCE_TOOL = ${fixture.runtime}/bin/gstack-qa-evidence. PROBE_DIR = ${fixture.cwd}/reports. DEADLINE_FILE = ${fixture.cwd}/reports/deadline.json. SECONDS must be positive and at most 300. EARLIER_UTC is the optional caller absolute deadline: use the section clock's Hard deadline UTC, never its Runner entry UTC, reserve-start time or a clock-read time. The only native COMMAND is the existing diagnostic client, bun scripts/probe.ts, with zero or one literal argument. Resolve these exact helper and state paths; do not use variables, another helper, another state file, nested wrappers, scripts, operators or substitutions. Its diagnostic receipts are declared public/synthetic, so --public is approved. Only an explicit plan check this phase requires may use --timeout-ms 10000 in place of the deadline; every other capture, including a rerun or revalidation of an earlier probe, uses the deadline. This does not reset or bypass the smoke deadline. Only the deadline helper may create or change reports/deadline.json and its .qa-deadline- temporary files; never use Write/Edit/MultiEdit on those paths. Record the full outer command in checkpoints and evidence; keep the unchanged child JSON as observed, separate from prefixed guard diagnostics. A completed expired guard-run is not a probe or a pass: retain its unused checkpoint, report not-run coverage and do not restart the deadline. Keep the 12-probe smoke limit. Required suites and explicit plan checks are outside the bounded smoke budget, not permission to reset it. Quote arguments literally. Sources stay inside reports. Decide to execute the next probe before publishing its checkpoint, then await successful publication and dispatch that exact probe. If you defer an optional idea or stop exploration, do not publish a checkpoint for it; describe it as not run in Markdown. An unused checkpoint requires an actual authenticated expired-capture result; nearing the deadline or choosing to stop is not enough. A complete capture can truthfull
Line truncated
|
||||
prompt: `Load gstack's /${fixture.caller} supplied parent phase from caller-${fixture.caller}.md and resume it on the selected working-tree diff against origin/main. This excerpt comes from ${fixture.runtime}/${fixture.caller}/SKILL.md; resolve installed-relative references there, not from the excerpt file or product directory. That path identifies the asset base, not another entrypoint: do not read or invoke the full parent SKILL.md or rerun its preamble. Earlier preamble/branch/base setup is complete; use the existing local origin/main ref without fetch. Earlier-phase asset locators are ${fixture.runtime}/review/checklist.md and ${fixture.runtime}/qa/templates/functional-report-template.md. Read those files directly when referenced; recursive Glob does not follow the installed asset symlinks. Cross-project learnings are configured off in this owned fixture. ${fixture.reviewStart ? `The actual review-start helper already captured REVIEW_START=${fixture.reviewStart} for this unchanged core pass; retain that token. ` : ''} This fixture evaluates only the supplied parent phase, not later publication stages. The fixture owner recorded these read-only observations when this phase began; they are current until an input changes, so use them instead of re-running those commands, and recheck freshness before completion outputs:\n${fixture.startObservations}\n Read README.md for the project contract and commands. Use diagnostic-client commands such as \`bun scripts/probe.ts <literal>\` for exploratory discoveries and their checkpoint evidence. A required \`bun run test\` is separate suite verification: report it as verification, never as a diagnostic observation or checkpoint anchor/target. Use the production evidence helper to publish each diagnostic checkpoint as \`reports/exploration-NNN.json\`, not inside a nested directory; do not transcribe its observed payload. ${fixture.caseId === 'ship-exploratory-plan-checks' ? 'The previously discovered plan is PLAN.md.' : 'No plan file was found.'} There is no remote service and no release publication is authorized. There is no interactive approver; do not invent answers or permission. Keep normal parent decision gates. Handoff freshness check: Read HANDOFF.md and reports/HANDOFF.md if present for any concurrent collaborator update, await those results, and compare evidence with current inputs; rerun any probe whose input changed. Run the check before each of these, and dispatch each only in a later message than the check's Read results: the evidence helper's materialize, which runs once per report root and freezes the verdict; every gstack-review-log completed:true record, adversarial-review included, since a record is a completion, not preliminary bookkeeping; and the completion report. A later handoff read cannot validate an earlier materialize or completion. If a probe's snapshot differs from an earlier probe's, read reports/HANDOFF.md first; it names any collaborator change.\n\nQA fixture bindings: the deadline guard, capture, checkpoint and materialize command forms are in the installed qa/sections/exploratory.md this phase requires you to Read; use those forms verbatim with these bindings substituted literally. DEADLINE_TOOL = ${fixture.runtime}/bin/gstack-qa-deadline. EVIDENCE_TOOL = ${fixture.runtime}/bin/gstack-qa-evidence. PROBE_DIR = ${fixture.cwd}/reports. DEADLINE_FILE = ${fixture.cwd}/reports/deadline.json. SECONDS must be positive and at most 300. EARLIER_UTC is the optional caller absolute deadline: use the section clock's Hard deadline UTC, never its Runner entry UTC, reserve-start time or a clock-read time. The only native COMMAND is the existing diagnostic client, bun scripts/probe.ts, with zero or one literal argument. Resolve these exact helper and state paths; do not use variables, another helper, another state file, nested wrappers, scripts, operators or substitutions. Its diagnostic receipts are declared public/synthetic, so --public is approved. Only an explicit plan check this phase requires may use --timeout-ms 10000 in place of the deadline; every other capture, including a rerun or revalidation of an earlier probe, uses the deadline. This does not reset or bypass the smoke deadline. Only the deadline helper may create or change reports/deadline.json and its .qa-deadline- temporary files; never use Write/Edit/MultiEdit on those paths. Record the full outer command in checkpoints and evidence; keep the unchanged child JSON as observed, separate from prefixed guard diagnostics. A completed expired guard-run is not a probe or a pass: retain its unused checkpoint, report not-run coverage and do not restart the deadline. Keep the 12-probe smoke limit. Required suites and explicit plan checks are outside the bounded smoke budget, not permission to reset it. Quote arguments literally. Sources stay inside reports. Decide to execute the next probe before publishing its checkpoint, then await successful publication and dispatch that exact probe. I
Line truncated
|
||||
appendSystemPrompt: `Caller execution scheduling (fixture contract):
|
||||
This session has at most 25 assistant turns, including required verification and final artifacts. The command boundary applies to each Bash call, not to the number of independent tool calls in an assistant turn.
|
||||
After required clock and approval prerequisites settle, issue independent source Reads and read-only discovery together as separate native tool calls once their paths and inputs are known. Read-only discovery is Read, Glob, Grep and the declared inventory, Git and review-read forms; a capture, a deadline guard start or run, or a diagnostic-client run is a probe, never discovery. For example, after the first clock read, one response can Read the caller excerpt, README.md, every product file and the referenced checklist and templates. Wait for their results before decisions that depend on them.
|
||||
|
||||
@@ -110,7 +110,7 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
|
||||
'qa-b6-static': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'qa/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval.html', 'test/fixtures/qa-eval-ground-truth.json', 'test/skill-e2e-qa-bugs.test.ts', 'test/helpers/aside-available.ts'],
|
||||
'qa-b7-spa': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'qa/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval-spa.html', 'test/fixtures/qa-eval-spa-ground-truth.json', 'test/skill-e2e-qa-bugs.test.ts', 'test/helpers/aside-available.ts'],
|
||||
'qa-b8-checkout': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'qa/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval-checkout.html', 'test/fixtures/qa-eval-checkout-ground-truth.json', 'test/skill-e2e-qa-bugs.test.ts', 'test/helpers/aside-available.ts'],
|
||||
'qa-only-no-fix': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'test/helpers/qa-only-cleanup.ts', 'test/helpers/qa-checkpoint-evidence.ts', 'test/fixtures/qa-only-observation-public.json', 'test/fixtures/qa-only-stop-cleanup-public.json', 'test/fixtures/qa-only-charter-public.json', 'test/fixtures/qa-only-browser-probe.ts', 'browse/test/fixtures/qa-only.html', 'test/helpers/qa-browser-deadline-evidence.ts', 'test/helpers/read-only-commands.ts', 'bin/gstack-qa-deadline', 'lib/qa-deadline.ts', 'lib/bun-isolation.ts', 'lib/claude-code-windows-job.ts', 'qa-only/**', 'qa/sections/**', 'qa/templates/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/skill-e2e-qa-workflow.test.ts', 'test/helpers/aside-available.ts', 'test/helpers/bootstrap-retention.ts', 'test/helpers/qa-evidence-producer.ts'],
|
||||
'qa-only-no-fix': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'test/helpers/qa-only-cleanup.ts', 'test/helpers/qa-checkpoint-evidence.ts', 'test/fixtures/qa-only-observation-public.json', 'test/fixtures/qa-only-stop-cleanup-public.json', 'test/fixtures/qa-only-bookkeeping-joins-public.json', 'test/fixtures/qa-only-charter-public.json', 'test/fixtures/qa-only-browser-probe.ts', 'browse/test/fixtures/qa-only.html', 'test/helpers/qa-browser-deadline-evidence.ts', 'test/helpers/read-only-commands.ts', 'bin/gstack-qa-deadline', 'lib/qa-deadline.ts', 'lib/bun-isolation.ts', 'lib/claude-code-windows-job.ts', 'qa-only/**', 'qa/sections/**', 'qa/templates/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/skill-e2e-qa-workflow.test.ts', 'test/helpers/aside-available.ts', 'test/helpers/bootstrap-retention.ts', 'test/helpers/qa-evidence-producer.ts'],
|
||||
'qa-fix-loop': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-qa-deadline', 'lib/qa-deadline.ts', 'lib/bun-isolation.ts', 'lib/claude-code-windows-job.ts', 'qa/**', 'scripts/resolvers/aside.ts', 'browse/src/**', 'browse/test/test-server.ts', 'test/skill-e2e-qa-workflow.test.ts',
|
||||
'test/helpers/aside-available.ts', 'test/fixtures/qa-only-browser-probe.ts', 'test/helpers/bootstrap-retention.ts', 'test/helpers/qa-browser-deadline-evidence.ts', 'test/helpers/read-only-commands.ts', 'test/helpers/qa-checkpoint-evidence.ts', 'test/helpers/qa-evidence-producer.ts', 'test/helpers/qa-only-cleanup.ts'],
|
||||
'qa-bootstrap': ['test/helpers/bootstrap-retention.ts', 'qa/**', 'ship/**', 'test/skill-e2e-qa-workflow.test.ts',
|
||||
|
||||
@@ -7,6 +7,7 @@ import { assertQaBrowserDeadline, assertQaBrowserPreparation, assertQaBrowserChe
|
||||
import capturedPreparation from './fixtures/qa-only-charter-public.json';
|
||||
import capturedObservation from './fixtures/qa-only-observation-public.json';
|
||||
import capturedStop from './fixtures/qa-only-stop-cleanup-public.json';
|
||||
import capturedBookkeeping from './fixtures/qa-only-bookkeeping-joins-public.json';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const guard = path.join(ROOT, 'bin/gstack-qa-deadline');
|
||||
@@ -223,7 +224,7 @@ test.each(['extra-option', 'suffix-option', 'operator', 'newline', 'foreign-cwd'
|
||||
const commands = {
|
||||
'extra-option': `git -C ${quote(f.directory)} -c color.ui=false rev-parse HEAD`,
|
||||
'suffix-option': metadata + ' --git-dir',
|
||||
operator: metadata + '; ' + metadata,
|
||||
operator: metadata + '; ' + quote(browse) + ' --version',
|
||||
newline: metadata + '\n' + metadata,
|
||||
'foreign-cwd': `git -C ${quote(path.dirname(f.directory))} rev-parse HEAD`,
|
||||
'missing-cwd': 'git rev-parse HEAD',
|
||||
@@ -232,6 +233,76 @@ test.each(['extra-option', 'suffix-option', 'operator', 'newline', 'foreign-cwd'
|
||||
expect(() => f.check()).toThrow('QA deadline:');
|
||||
});
|
||||
|
||||
// Opus gates 39313931713, 39332134395 and 39336730429 (slice 8, attempt 1) failed qa-only-no-fix with
|
||||
// "unsupported outer shell composition" after joining reads with an exact bookkeeping call, as in
|
||||
// `ls -la; git -C <fixture> branch --show-current`. Every non-guard Bash command from those three
|
||||
// attempts and from the eight passing attempts beside them replays inside a real guarded attempt.
|
||||
const bookkeepingReplay = (f: ReturnType<typeof fixture>, command: string) => command
|
||||
.replaceAll('__QA_FIXTURE__', f.directory).replaceAll('__QA_GUARD__', guard).replaceAll('__QA_BROWSE__', browse);
|
||||
test.each(capturedBookkeeping.attempts.map(attempt => [attempt.source, attempt.passed, attempt.commands] as const))
|
||||
('captured non-guard Bash commands replay as bookkeeping: %s (passed %p)', (_, __, commands) => {
|
||||
const f = fixture();
|
||||
const start = f.calls.shift()!;
|
||||
for (const command of commands) f.calls.push({ tool: 'Bash', input: { command: bookkeepingReplay(f, command) }, output: 'bookkeeping' });
|
||||
f.calls.push(start);
|
||||
f.run();
|
||||
expect(f.check()).toEqual({ launchedRuns: 1, completedRuns: 1, refusedRuns: 0, timedOutRuns: 0 });
|
||||
});
|
||||
|
||||
test('the captured joins are the three failing attempts\' only unaccepted commands', () => {
|
||||
const failing = capturedBookkeeping.attempts.filter(attempt => !attempt.passed);
|
||||
expect(failing.map(attempt => attempt.source.split(' ')[1])).toEqual(['39313931713', '39332134395', '39336730429']);
|
||||
expect(failing.flatMap(attempt => attempt.commands.filter(command => /^ls -la; /.test(command)))).toEqual([
|
||||
"ls -la; cat TODOS.md 2>/dev/null; git -C '__QA_FIXTURE__' rev-parse --short HEAD",
|
||||
"ls -la; git -C '__QA_FIXTURE__' branch --show-current",
|
||||
"ls -la; git -C '__QA_FIXTURE__' branch --show-current",
|
||||
]);
|
||||
});
|
||||
|
||||
test.each(['captured', 'two-bookkeeping', 'and-or', 'readiness-check-joined', 'trailing-semicolon',
|
||||
'browser-after', 'browser-before', 'guard-joined', 'guard-tail', 'git-option', 'git-other-subcommand', 'pipe-to-shell', 'background',
|
||||
'substitution', 'newline', 'cd-then-read', 'read-outside-root', 'dangling-and', 'escaped-semicolon', 'quoted-operator', 'receipt-in-output', 'readiness-block-joined'])
|
||||
('bookkeeping joins accept exact calls and grammar reads only: %s', scenario => {
|
||||
const f = fixture();
|
||||
f.run();
|
||||
const status = `git -C ${quote(f.directory)} status --short`;
|
||||
const branch = `git -C ${quote(f.directory)} branch --show-current`;
|
||||
const guarded = ['bun', guard, 'run', f.policy.file, '--', browse, '--version'].map(quote).join(' ');
|
||||
const commands: Record<string, string> = {
|
||||
captured: `ls -la; ${branch}`,
|
||||
'two-bookkeeping': `${status}; ${branch}`,
|
||||
'and-or': `ls -la qa-reports && ${status} || echo none`,
|
||||
'readiness-check-joined': `${f.policy.allowed[3]}; ${branch}`,
|
||||
'trailing-semicolon': `ls -la; ${branch};`,
|
||||
'browser-after': `ls -la; ${branch}; ${quote(browse)} goto http://127.0.0.1:1/`,
|
||||
'browser-before': `${quote(browse)} --version && ${status}`,
|
||||
'guard-joined': `${status} && ${guarded}`,
|
||||
'guard-tail': `${guarded}; ${status}`,
|
||||
'git-option': `ls -la; git -C ${quote(f.directory)} -c core.pager=cat log -1`,
|
||||
'git-other-subcommand': `ls -la; git -C ${quote(f.directory)} config --list`,
|
||||
'pipe-to-shell': `${status} | sh`,
|
||||
background: `${status} & ${quote(browse)} --version`,
|
||||
substitution: `ls -la "$(${quote(browse)} --version)"; ${status}`,
|
||||
newline: `ls -la\n${status}`,
|
||||
'cd-then-read': `cd qa-reports; cat deadline.json; ${status}`,
|
||||
'read-outside-root': `cat /etc/hostname; ${status}`,
|
||||
'dangling-and': `ls -la; ${status} &&`,
|
||||
'escaped-semicolon': `ls -la \\; ${quote(browse)} --version; ${status}`,
|
||||
'quoted-operator': `echo 'a; b'; ${status}`,
|
||||
'receipt-in-output': `ls -la; ${status}`,
|
||||
'readiness-block-joined': `${f.policy.allowed[0]}; ${status}`,
|
||||
};
|
||||
f.calls.push({ tool: 'Bash', input: { command: commands[scenario] }, output: scenario === 'receipt-in-output' ? 'QA_DEADLINE {"guard":"qa-deadline"}' : 'bookkeeping' });
|
||||
if (['captured', 'two-bookkeeping', 'and-or', 'readiness-check-joined', 'trailing-semicolon', 'quoted-operator'].includes(scenario)) expect(f.check()).toEqual({ launchedRuns: 1, completedRuns: 1, refusedRuns: 0, timedOutRuns: 0 });
|
||||
else expect(() => f.check()).toThrow(scenario === 'receipt-in-output' ? 'receipt outside trusted guard invocation' : 'QA deadline:');
|
||||
});
|
||||
|
||||
test('the deadline prompt declares the bookkeeping join it accepts', () => {
|
||||
const { policy } = fixture();
|
||||
expect(policy.prompt).toContain('join several of them and the shared-grammar reads below with ;, && or || (no cd in such a join)');
|
||||
expect(policy.prompt).toContain('as its own Bash call (no appended commands)');
|
||||
});
|
||||
|
||||
test('quoted question marks stay literal but bare glob expansion is rejected', () => {
|
||||
const f = fixture();
|
||||
fs.writeFileSync(path.join(f.directory, 'x'), 'glob target');
|
||||
|
||||
@@ -16,6 +16,7 @@ import { readQACheckpointFiles } from './helpers/qa-checkpoint-evidence';
|
||||
import { generateQAExploratory, generateQAResource, generateQAReview, generateQAReviewPreflight } from '../scripts/resolvers/qa';
|
||||
import { HOST_PATHS } from '../scripts/resolvers/types';
|
||||
import { qaProbeNames } from './helpers/qa-probe-names';
|
||||
import capturedHandoffOrder from './fixtures/qa-caller-handoff-order-public.json';
|
||||
|
||||
function nativeCall(id: string, name: string, input: object, output: string, parent: string | null = null, failed = false) {
|
||||
return [
|
||||
@@ -246,6 +247,22 @@ describe('caller native-event observer controls', () => {
|
||||
expect(validateCallerEvidence(observed)).toContain('review completion preceded handoff freshness decision');
|
||||
});
|
||||
|
||||
// Opus gates 39320737890 attempt 1 and 39332134395 attempt 2 (slice 8, review-exploratory-small-cli) wrote the
|
||||
// completed:true adversarial-review record in the same response as the first HANDOFF.md Read, or before any Read.
|
||||
// The same public events pass once the record follows every handoff result in its own response.
|
||||
test.each(capturedHandoffOrder.attempts.map(attempt => [attempt.source, attempt] as const))
|
||||
('captured adversarial-review completion before its handoff result is flagged; reordered it passes: %s', (_, attempt) => {
|
||||
const observed = evidence();
|
||||
const base = observed.result.transcript;
|
||||
observed.result.transcript = [...base, ...structuredClone(attempt.events)];
|
||||
expect(validateCallerEvidence(observed)).toEqual(['review completion preceded handoff freshness decision']);
|
||||
const ordered = structuredClone(attempt.events) as any[];
|
||||
const [use, result] = ordered.splice(ordered.findIndex(event => event.message.content[0].name === 'Bash'), 2);
|
||||
use.message.id = 'record-after-handoff-results';
|
||||
observed.result.transcript = [...base, ...ordered, use, result];
|
||||
expect(validateCallerEvidence(observed)).toEqual([]);
|
||||
});
|
||||
|
||||
test('captured gate-census-5 review record: bun-wrapped helper and receipt status are outside the interface', () => {
|
||||
// ci-36629958451-1-gate-census-5 review-exploratory-small-cli, native events 68 and 70 (fixture paths shortened).
|
||||
const record = (status: string) => `/runtime/bin/gstack-review-log '{"skill":"review","timestamp":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'","commit":"'"$(git rev-parse --short HEAD)"'","branch":"caller-change","status":"${status}","completed":false,"converged":false,"critical":1,"informational":0,"findings":[{"path":"scale.ts","line":3,"category":"functional-contract","severity":"CRITICAL","fingerprint":"scale.ts:3:functional-contract","action":"ask-pending"}]}' --finish d45404cf-ba19-4bc6-a505-9801e9322f54`;
|
||||
@@ -892,6 +909,62 @@ describe('real caller-specific native fixture and capture boundary', () => {
|
||||
const probe = (fixture: QaCallerFixture, value: string) => spawnSync(process.execPath, ['scripts/probe.ts', value], { cwd: fixture.cwd, encoding: 'utf8', timeout: 5000 });
|
||||
const dispose = async (fixture: QaCallerFixture) => { await fixture.close(); fs.rmSync(fixture.root, { recursive: true, force: true }); };
|
||||
|
||||
// Opus gate 39320737890 attempt 2 slice 8 (ship-exploratory-plan-checks, head f278f5b): capture 001 was the
|
||||
// guarded smoke probe of 9, 002 probed 10, and 003 ran PLAN.md's required release check of 9 with
|
||||
// --timeout-ms 10000, the form the prompt reserves for an explicit plan check outside the smoke budget.
|
||||
// The validator read 003 as a wasted rerun. One explicit plan check beside the smoke probe is verification;
|
||||
// a second smoke or plan-check run of the same unchanged input is still a duplicate.
|
||||
test.each([
|
||||
['captured', ['deadline 9', 'deadline 10', 'timeout 9'], undefined],
|
||||
['plan check guarded like a smoke probe', ['deadline 9', 'deadline 10', 'deadline 9'], 2],
|
||||
['plan check repeated', ['deadline 9', 'deadline 10', 'timeout 9', 'timeout 9'], 3],
|
||||
['ordinary input under the plan-check form', ['deadline 3', 'deadline 10', 'timeout 3'], 2],
|
||||
] as const)('an explicit plan check is not a duplicate of the smoke probe: %s', async (_, steps, duplicate) => {
|
||||
const fixture = await fixtureFor('ship-exploratory-plan-checks', true);
|
||||
try {
|
||||
const reportRoot = path.join(fixture.cwd, 'reports');
|
||||
const state = path.join(reportRoot, 'deadline.json');
|
||||
const evidenceTool = path.join(fixture.runtime, 'bin/gstack-qa-evidence');
|
||||
const result = await runQaCaller(fixture, 'free-plan-check-replay', async options => {
|
||||
const transcript: unknown[] = [
|
||||
...nativeCall('parent', 'Read', { file_path: `${fixture.cwd}/caller-ship.md` }, 'parent workflow'),
|
||||
...nativeCall('shared', 'Read', { file_path: `${fixture.runtime}/qa/sections/exploratory.md` }, 'shared method'),
|
||||
...nativeCall('functional', 'Read', { file_path: `${fixture.runtime}/qa/sections/system-functional.md` }, 'functional method'),
|
||||
...nativeCall('ship-army', 'Read', { file_path: `${fixture.runtime}/ship/sections/review-army.md` }, 'ship Step 9'),
|
||||
];
|
||||
const execute = (id: string, cmd: string) => {
|
||||
expect(qaCallerCommandAllowed(cmd, fixture.workflowCommands, { runtime: fixture.runtime, fixtureRoot: fixture.cwd }), cmd).toBe(true);
|
||||
const actual = spawnSync('bash', ['-c', cmd], { cwd: options.workingDirectory, env: { ...process.env, ...options.env }, encoding: 'utf8', timeout: 10_000 });
|
||||
expect(actual.error).toBeUndefined();
|
||||
transcript.push(...nativeCall(id, 'Bash', { command: cmd }, (actual.status ? `Exit code ${actual.status}\n` : '') + actual.stdout + actual.stderr, null, actual.status !== 0));
|
||||
};
|
||||
execute('start', `bun ${path.join(fixture.runtime, 'bin/gstack-qa-deadline')} start ${state} 300`);
|
||||
for (const [index, step] of steps.entries()) {
|
||||
const [lane, value] = step.split(' ');
|
||||
const id = String(index + 1).padStart(3, '0');
|
||||
const bound = lane === 'deadline' ? `--deadline ${state}` : '--timeout-ms 10000';
|
||||
const after = index ? ` --after ${String(index).padStart(3, '0')} --hypothesis 'Input ${value} checks the next documented contract.'` : '';
|
||||
execute(`capture-${id}`, `bun ${evidenceTool} capture ${reportRoot} ${id} --public ${bound}${after} -- bun scripts/probe.ts ${value}`);
|
||||
}
|
||||
return { exitReason: 'success', transcript } as SkillTestResult;
|
||||
});
|
||||
await fixture.close();
|
||||
const probes = fixture.probes();
|
||||
expect(probes).toHaveLength(steps.length);
|
||||
const checkpointFiles = readQACheckpointFiles(reportRoot);
|
||||
const errors = validateCallerEvidence({
|
||||
caller: fixture.caller, result, probes,
|
||||
receipt: { status: 'pass', probes: probes.map(probe => probe.id), remaining: [] },
|
||||
currentSnapshot: fixture.snapshot(), requiredCharters: ['happy', 'adverse', 'plan:nine'], mutations: fixture.mutationEvents,
|
||||
observerComplete: fixture.observation?.complete === true && !fixture.observerErrors.length,
|
||||
fixtureRoot: fixture.cwd, runtime: fixture.runtime, reportRoot, requireGuardedSmoke: true, requireCapturedEvidence: true,
|
||||
checkpointFiles, reportMarkdown: Object.keys(checkpointFiles).map(name => `[checkpoint](${name})`).join('\n'),
|
||||
});
|
||||
if (duplicate === undefined) expect(errors).toEqual([]);
|
||||
else expect(errors).toContain(`duplicate unchanged passing probe: ${probes[duplicate].id}`);
|
||||
} finally { await dispose(fixture); }
|
||||
});
|
||||
|
||||
test('deadline authority is closed over the installed helper, owned state and literal probe child', async () => {
|
||||
const fixture = await fixtureFor('ship-exploratory-small-cli', true);
|
||||
try {
|
||||
@@ -1503,7 +1576,8 @@ describe('real caller-specific native fixture and capture boundary', () => {
|
||||
expect(options.appendSystemPrompt).not.toMatch(/bun scripts\/probe\.ts \d|invalid input|highest.risk/i);
|
||||
expect(options.prompt).toMatch(/keep normal parent decision gates/i);
|
||||
expect(options.prompt).toMatch(/Hard deadline UTC, never its Runner entry UTC/i);
|
||||
expect(options.prompt).toMatch(/completed:true review record, read HANDOFF\.md/i);
|
||||
expect(options.prompt).toMatch(/Handoff freshness check: Read HANDOFF\.md and reports\/HANDOFF\.md/);
|
||||
expect(options.prompt).toMatch(/only in a later message than the check's Read results: the evidence helper's materialize, which runs once per report root and freezes the verdict; every gstack-review-log completed:true record, adversarial-review included/);
|
||||
expect(options.prompt).toMatch(/do not publish a checkpoint for it/i);
|
||||
expect(options.prompt).toMatch(/authenticated expired-capture result/i);
|
||||
expect(options.prompt).toContain('reports/review.md');
|
||||
|
||||
Reference in new issue
Block a user