mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-05 08:23:59 +01:00
fix(evals): qa-only checkpoint rule counts guarded "$B" stop as browser cleanup, like closetab
PR #3272's Opus gate 39217387697 (slice 8) failed qa-only-no-fix when the agent stopped the browser inside the guard; the rule counted stop as a third probe that needed a checkpoint. The prompt now names both cleanup children. A free replay of the captured three guarded calls passes (and fails on the old rule); stop before a probe, a disguised probe, stop --all and stop outside the guard still fail. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed
1 parent
2699b45a86
commit
c0c199fb35
4 files changed
+393
-7
No files matched your search
+2
-1
@@ -2,12 +2,13 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
**The qa-only deadline fixture accepts pure local reads, so `qa-only-no-fix` no longer fails when the agent checks a skill file with `wc -l …; ls …`.** Opus 5.5's first Bash call in TM1's focused `qa-only-no-fix` run at e03bfeb was `wc -l qa-only/SKILL.md; ls qa-only qa-only/sections qa/sections qa/templates`. The fixture's narrow list rejected it with `QA deadline: unsupported outer shell composition`, though it only read inside the fixture. The fixture now opts into the shared read-only grammar from 1.91.143.0, as the docs-sync fixture did. A focused Opus run with the opt-in then failed on one more read, `ls; ls qa/templates; cat TODOS.md 2>/dev/null | head -20`, so the grammar now also lets a read discard or merge its stderr.
|
||||
**The qa-only deadline fixture accepts pure local reads, so `qa-only-no-fix` no longer fails when the agent checks a skill file with `wc -l …; ls …`.** Opus 5.5's first Bash call in TM1's focused `qa-only-no-fix` run at e03bfeb was `wc -l qa-only/SKILL.md; ls qa-only qa-only/sections qa/sections qa/templates`. The fixture's narrow list rejected it with `QA deadline: unsupported outer shell composition`, though it only read inside the fixture. The fixture now opts into the shared read-only grammar from 1.91.143.0, as the docs-sync fixture did. A focused Opus run with the opt-in then failed on one more read, `ls; ls qa/templates; cat TODOS.md 2>/dev/null | head -20`, so the grammar now also lets a read discard or merge its stderr. PR #3272's Opus gate (run 39217387697) then failed the case after the agent stopped the browser inside the guard with `"$B" stop`, which the checkpoint rule counted as a third probe.
|
||||
|
||||
### Changed
|
||||
- **`test/helpers/qa-browser-deadline-evidence.ts`:** the deadline check also accepts any command the shared grammar allows inside the fixture directory, and the fixture prompt declares the grammar instead of "no other ls, wc, cat or find Bash calls". Receipts are still trusted only from the guard: a grammar read whose output has a line starting with `QA_DEADLINE ` fails as `receipt outside trusted guard invocation`. A grammar read of skill prose that only mentions QA_DEADLINE receipts mid-line passes, just as the same file read with Read does; the second focused Opus run tripped on `cat qa-only/sections/exploratory.md`. The fixture's existing allowed commands keep the stricter any-occurrence check.
|
||||
- **Gate integrity, widening:** pure reads inside the fixture root, such as listing `qa/sections` or `ls -laR` of the screenshots directory, now pass where they used to fail. A free replay runs the captured slip inside a real guarded attempt. Reads that leave the root (`…/../../..`, `/etc`), joins with a non-read tool, redirects, `| tee`, `awk` and `sed -i` still fail. No threshold, kind, trial or behavior assertion changes. The `qa-quick`, `qa-only-no-fix`, `qa-fix-loop` and `qa-bootstrap` touchfiles add `test/helpers/read-only-commands.ts`.
|
||||
- **Gate integrity, widening (shared grammar, GStack Minder-approved):** a read may end with exactly `2>/dev/null` or `2>&1`. Both fixtures that use the grammar accept it: qa-only and docs-sync. Every other redirect still fails, each with its own negative control: `2>` to a file, `2> /dev/null` with a space, `>` and `>>` on stdout, `&>`, `1>/dev/null`, `3>/dev/null`, `>&` and `2>&` to a file, `2>&2`, `2>/dev/tcp/…`, a leading `2>/dev/null`, a quoted `'2'>`, and `| tee`. The main control `ls docs 2>/dev/null` became `ls docs 2>docs/err.txt`. Two existing docs-sync controls only failed because of their stderr redirect, and each got a replacement that still fails: `ls 2>&1 | head` became `ls 2>&1 | sh`. The stored miss `cat {other}/bash-placeholder 2>/dev/null; cat {home}/state/config.yaml` moved to known_good_posix, and its `2>` to a file and `1>/dev/null` variants stay known_bad. The docs-sync prompt carries the updated grammar text, so the next PR gate selects the docs-sync cases.
|
||||
- **Gate integrity, widening (checkpoint rule):** a guarded child that is exactly `"$B" stop`, `$B stop` or `<browse> stop` now counts as browser cleanup, the same as `closetab`. Cleanup needs no checkpoint and may come only after the last probe. The fixture prompt names both cleanup commands. A free replay of the three guarded calls captured in run 39217387697 (two probes with one checkpoint, then `stop`) passes, and fails on the previous rule with `follow-up lacks one acknowledged preceding checkpoint`. These still fail: `stop` before a probe (`browser cleanup preceded a probe`), a probe disguised as cleanup (`"$B" goto …; "$B" stop`), `stop --all`, and `stop` outside the guard (direct, or through a shell variable).
|
||||
|
||||
## [1.91.144.0] - 2026-10-18
|
||||
|
||||
|
||||
+340
@@ -0,0 +1,340 @@
|
||||
{
|
||||
"sourceArchiveSha256": "33b7cd6a929fe2a0ab827dd0dd77aea9261357491f15220c0ef1359287b9d8d3",
|
||||
"sourceRun": "39217387697 slice 8 qa-only-no-fix (PR #3272 head 3986ac5)",
|
||||
"sourceEventIndices": [
|
||||
466,
|
||||
470,
|
||||
656,
|
||||
660,
|
||||
701,
|
||||
705,
|
||||
772,
|
||||
778
|
||||
],
|
||||
"events": [
|
||||
{
|
||||
"type": "assistant",
|
||||
"message": {
|
||||
"model": "claude-opus-5-5",
|
||||
"id": "msg_011CgBEbc3VPt2BCbX7eGKc6",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_01L7wBef1fN7eNy9mwQRni6M",
|
||||
"name": "Bash",
|
||||
"input": {
|
||||
"command": "bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \"__QA_FIXTURE__/fixture-browser-probe.ts\" \"__QA_BROWSE__\" \"http://127.0.0.1:39269/qa-only.html\" \"__QA_FIXTURE__/qa-reports/screenshots/initial.png\"'",
|
||||
"description": "Run baseline browser probe under deadline guard"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
],
|
||||
"container": null,
|
||||
"stop_reason": null,
|
||||
"stop_sequence": null,
|
||||
"stop_details": null,
|
||||
"usage": {
|
||||
"input_tokens": 2,
|
||||
"cache_creation_input_tokens": 329,
|
||||
"cache_read_input_tokens": 45732,
|
||||
"cache_creation": {
|
||||
"ephemeral_5m_input_tokens": 329,
|
||||
"ephemeral_1h_input_tokens": 0
|
||||
},
|
||||
"output_tokens": 16,
|
||||
"service_tier": "standard",
|
||||
"inference_geo": "global",
|
||||
"speed": "standard"
|
||||
},
|
||||
"input_transformations": [],
|
||||
"diagnostics": null,
|
||||
"context_management": null
|
||||
},
|
||||
"parent_tool_use_id": null,
|
||||
"session_id": "58ff97a2-be5b-40db-a3ce-fb390aad7c88",
|
||||
"uuid": "35b66938-677e-4421-b459-c0a4c376b84d",
|
||||
"timestamp": "2026-10-19T07:54:11.131Z",
|
||||
"request_id": "req_011CgBEbbm8oyfaUy4RiMpug",
|
||||
"wire_tool_inputs": {
|
||||
"toolu_01L7wBef1fN7eNy9mwQRni6M": {
|
||||
"command": "bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \"__QA_FIXTURE__/fixture-browser-probe.ts\" \"__QA_BROWSE__\" \"http://127.0.0.1:39269/qa-only.html\" \"__QA_FIXTURE__/qa-reports/screenshots/initial.png\"'",
|
||||
"description": "Run baseline browser probe under deadline guard"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"tool_use_id": "toolu_01L7wBef1fN7eNy9mwQRni6M",
|
||||
"type": "tool_result",
|
||||
"content": "QA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"started\",\"version\":1,\"startedAt\":\"2026-10-19T07:54:07.184Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"budgetMs\":30000,\"observedAt\":\"2026-10-19T07:54:11.155Z\",\"remainingMs\":26029,\"expired\":false}\n{\"navigation\":{\"stdout\":\"Navigated to http://127.0.0.1:39269/qa-only.html (200)\\n\",\"stderr\":\"[browse] Starting server...\\n\",\"exitCode\":0},\"console\":{\"stdout\":\"--- BEGIN UNTRUSTED EXTERNAL CONTENT (source: http://127.0.0.1:39269/qa-only.html) ---\\n[2026-10-19T07:54:11.698Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n--- END UNTRUSTED EXTERNAL CONTENT ---\\n\",\"stderr\":\"\",\"exitCode\":0},\"screenshot\":{\"stdout\":\"Screenshot saved: __QA_FIXTURE__/qa-reports/screenshots/initial.png\\n\",\"stderr\":\"\",\"exitCode\":0}}\n\nQA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"finished\",\"observedAt\":\"2026-10-19T07:54:11.808Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"timedOut\":false,\"exitCode\":0}",
|
||||
"is_error": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"parent_tool_use_id": null,
|
||||
"session_id": "58ff97a2-be5b-40db-a3ce-fb390aad7c88",
|
||||
"uuid": "a48d1ba7-2580-412a-9c36-a7e92de980d5",
|
||||
"timestamp": "2026-10-19T07:54:11.818Z",
|
||||
"tool_use_result": {
|
||||
"stdout": "QA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"started\",\"version\":1,\"startedAt\":\"2026-10-19T07:54:07.184Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"budgetMs\":30000,\"observedAt\":\"2026-10-19T07:54:11.155Z\",\"remainingMs\":26029,\"expired\":false}\n{\"navigation\":{\"stdout\":\"Navigated to http://127.0.0.1:39269/qa-only.html (200)\\n\",\"stderr\":\"[browse] Starting server...\\n\",\"exitCode\":0},\"console\":{\"stdout\":\"--- BEGIN UNTRUSTED EXTERNAL CONTENT (source: http://127.0.0.1:39269/qa-only.html) ---\\n[2026-10-19T07:54:11.698Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n--- END UNTRUSTED EXTERNAL CONTENT ---\\n\",\"stderr\":\"\",\"exitCode\":0},\"screenshot\":{\"stdout\":\"Screenshot saved: __QA_FIXTURE__/qa-reports/screenshots/initial.png\\n\",\"stderr\":\"\",\"exitCode\":0}}\n\nQA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"finished\",\"observedAt\":\"2026-10-19T07:54:11.808Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"timedOut\":false,\"exitCode\":0}",
|
||||
"stderr": "",
|
||||
"interrupted": false,
|
||||
"isImage": false,
|
||||
"noOutputExpected": false,
|
||||
"bashEditDiff": {
|
||||
"files": [],
|
||||
"moreFiles": 1,
|
||||
"changedFiles": [
|
||||
"__QA_FIXTURE__/qa-reports/screenshots/initial.png"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"message": {
|
||||
"model": "claude-opus-5-5",
|
||||
"id": "msg_011CgBEc8zLFuDZaD1fV1X27",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_01B5gouuWiBq47RwAQzdjaW6",
|
||||
"name": "Write",
|
||||
"input": {
|
||||
"file_path": "__QA_FIXTURE__/qa-reports/exploration-001.json",
|
||||
"content": "{\n \"observationCommand\": \"bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \\\"__QA_FIXTURE__/fixture-browser-probe.ts\\\" \\\"__QA_BROWSE__\\\" \\\"http://127.0.0.1:39269/qa-only.html\\\" \\\"__QA_FIXTURE__/qa-reports/screenshots/initial.png\\\"'\",\n \"observed\": {\"navigation\":{\"stdout\":\"Navigated to http://127.0.0.1:39269/qa-only.html (200)\\n\",\"stderr\":\"[browse] Starting server...\\n\",\"exitCode\":0},\"console\":{\"stdout\":\"--- BEGIN UNTRUSTED EXTERNAL CONTENT (source: http://127.0.0.1:39269/qa-only.html) ---\\n[2026-10-19T07:54:11.698Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n--- END UNTRUSTED EXTERNAL CONTENT ---\\n\",\"stderr\":\"\",\"exitCode\":0},\"screenshot\":{\"stdout\":\"Screenshot saved: __QA_FIXTURE__/qa-reports/screenshots/initial.png\\n\",\"stderr\":\"\",\"exitCode\":0}},\n \"hypothesis\": \"Baseline shows one console TypeError on load; exact replay of the same probe with a fresh screenshot confirms it is reproducible.\",\n \"nextCommand\": \"bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \\\"__QA_FIXTURE__/fixture-browser-probe.ts\\\" \\\"__QA_BROWSE__\\\" \\\"http://127.0.0.1:39269/qa-only.html\\\" \\\"__QA_FIXTURE__/qa-reports/screenshots/replay-001.png\\\"'\"\n}\n"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
],
|
||||
"container": null,
|
||||
"stop_reason": null,
|
||||
"stop_sequence": null,
|
||||
"stop_details": null,
|
||||
"usage": {
|
||||
"input_tokens": 2,
|
||||
"cache_creation_input_tokens": 354,
|
||||
"cache_read_input_tokens": 46991,
|
||||
"cache_creation": {
|
||||
"ephemeral_5m_input_tokens": 354,
|
||||
"ephemeral_1h_input_tokens": 0
|
||||
},
|
||||
"output_tokens": 8,
|
||||
"service_tier": "standard",
|
||||
"inference_geo": "global",
|
||||
"speed": "standard"
|
||||
},
|
||||
"input_transformations": [],
|
||||
"diagnostics": null,
|
||||
"context_management": null
|
||||
},
|
||||
"parent_tool_use_id": null,
|
||||
"session_id": "58ff97a2-be5b-40db-a3ce-fb390aad7c88",
|
||||
"uuid": "542289f5-d545-463a-a8b3-4789658d86d1",
|
||||
"timestamp": "2026-10-19T07:54:23.625Z",
|
||||
"request_id": "req_011CgBEc8jht4KHH1idTVyEj",
|
||||
"wire_tool_inputs": {
|
||||
"toolu_01B5gouuWiBq47RwAQzdjaW6": {
|
||||
"file_path": "__QA_FIXTURE__/qa-reports/exploration-001.json",
|
||||
"content": "{\n \"observationCommand\": \"bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \\\"__QA_FIXTURE__/fixture-browser-probe.ts\\\" \\\"__QA_BROWSE__\\\" \\\"http://127.0.0.1:39269/qa-only.html\\\" \\\"__QA_FIXTURE__/qa-reports/screenshots/initial.png\\\"'\",\n \"observed\": {\"navigation\":{\"stdout\":\"Navigated to http://127.0.0.1:39269/qa-only.html (200)\\n\",\"stderr\":\"[browse] Starting server...\\n\",\"exitCode\":0},\"console\":{\"stdout\":\"--- BEGIN UNTRUSTED EXTERNAL CONTENT (source: http://127.0.0.1:39269/qa-only.html) ---\\n[2026-10-19T07:54:11.698Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n--- END UNTRUSTED EXTERNAL CONTENT ---\\n\",\"stderr\":\"\",\"exitCode\":0},\"screenshot\":{\"stdout\":\"Screenshot saved: __QA_FIXTURE__/qa-reports/screenshots/initial.png\\n\",\"stderr\":\"\",\"exitCode\":0}},\n \"hypothesis\": \"Baseline shows one console TypeError on load; exact replay of the same probe with a fresh screenshot confirms it is reproducible.\",\n \"nextCommand\": \"bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \\\"__QA_FIXTURE__/fixture-browser-probe.ts\\\" \\\"__QA_BROWSE__\\\" \\\"http://127.0.0.1:39269/qa-only.html\\\" \\\"__QA_FIXTURE__/qa-reports/screenshots/replay-001.png\\\"'\"\n}\n"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"tool_use_id": "toolu_01B5gouuWiBq47RwAQzdjaW6",
|
||||
"type": "tool_result",
|
||||
"content": "File created successfully at: __QA_FIXTURE__/qa-reports/exploration-001.json (file state is current in your context \u2014 no need to Read it back)"
|
||||
}
|
||||
]
|
||||
},
|
||||
"parent_tool_use_id": null,
|
||||
"session_id": "58ff97a2-be5b-40db-a3ce-fb390aad7c88",
|
||||
"uuid": "f4b34e9f-8772-491e-96cb-5029c43001c9",
|
||||
"timestamp": "2026-10-19T07:54:23.636Z",
|
||||
"tool_use_result": {
|
||||
"type": "create",
|
||||
"filePath": "__QA_FIXTURE__/qa-reports/exploration-001.json",
|
||||
"content": "{\n \"observationCommand\": \"bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \\\"__QA_FIXTURE__/fixture-browser-probe.ts\\\" \\\"__QA_BROWSE__\\\" \\\"http://127.0.0.1:39269/qa-only.html\\\" \\\"__QA_FIXTURE__/qa-reports/screenshots/initial.png\\\"'\",\n \"observed\": {\"navigation\":{\"stdout\":\"Navigated to http://127.0.0.1:39269/qa-only.html (200)\\n\",\"stderr\":\"[browse] Starting server...\\n\",\"exitCode\":0},\"console\":{\"stdout\":\"--- BEGIN UNTRUSTED EXTERNAL CONTENT (source: http://127.0.0.1:39269/qa-only.html) ---\\n[2026-10-19T07:54:11.698Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n--- END UNTRUSTED EXTERNAL CONTENT ---\\n\",\"stderr\":\"\",\"exitCode\":0},\"screenshot\":{\"stdout\":\"Screenshot saved: __QA_FIXTURE__/qa-reports/screenshots/initial.png\\n\",\"stderr\":\"\",\"exitCode\":0}},\n \"hypothesis\": \"Baseline shows one console TypeError on load; exact replay of the same probe with a fresh screenshot confirms it is reproducible.\",\n \"nextCommand\": \"bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \\\"__QA_FIXTURE__/fixture-browser-probe.ts\\\" \\\"__QA_BROWSE__\\\" \\\"http://127.0.0.1:39269/qa-only.html\\\" \\\"__QA_FIXTURE__/qa-reports/screenshots/replay-001.png\\\"'\"\n}\n",
|
||||
"structuredPatch": [],
|
||||
"originalFile": null,
|
||||
"userModified": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"message": {
|
||||
"model": "claude-opus-5-5",
|
||||
"id": "msg_011CgBEc8zLFuDZaD1fV1X27",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_015WHowRypEs12DpL8sq2iBD",
|
||||
"name": "Bash",
|
||||
"input": {
|
||||
"command": "bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \"__QA_FIXTURE__/fixture-browser-probe.ts\" \"__QA_BROWSE__\" \"http://127.0.0.1:39269/qa-only.html\" \"__QA_FIXTURE__/qa-reports/screenshots/replay-001.png\"'",
|
||||
"description": "Replay browser probe with fresh screenshot"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
],
|
||||
"container": null,
|
||||
"stop_reason": null,
|
||||
"stop_sequence": null,
|
||||
"stop_details": null,
|
||||
"usage": {
|
||||
"input_tokens": 2,
|
||||
"cache_creation_input_tokens": 354,
|
||||
"cache_read_input_tokens": 46991,
|
||||
"cache_creation": {
|
||||
"ephemeral_5m_input_tokens": 354,
|
||||
"ephemeral_1h_input_tokens": 0
|
||||
},
|
||||
"output_tokens": 8,
|
||||
"service_tier": "standard",
|
||||
"inference_geo": "global",
|
||||
"speed": "standard"
|
||||
},
|
||||
"input_transformations": [],
|
||||
"diagnostics": null,
|
||||
"context_management": null
|
||||
},
|
||||
"parent_tool_use_id": null,
|
||||
"session_id": "58ff97a2-be5b-40db-a3ce-fb390aad7c88",
|
||||
"uuid": "7ca220bf-403d-483d-bef3-9ec8cde921a0",
|
||||
"timestamp": "2026-10-19T07:54:25.902Z",
|
||||
"request_id": "req_011CgBEc8jht4KHH1idTVyEj",
|
||||
"wire_tool_inputs": {
|
||||
"toolu_015WHowRypEs12DpL8sq2iBD": {
|
||||
"command": "bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'bun \"__QA_FIXTURE__/fixture-browser-probe.ts\" \"__QA_BROWSE__\" \"http://127.0.0.1:39269/qa-only.html\" \"__QA_FIXTURE__/qa-reports/screenshots/replay-001.png\"'",
|
||||
"description": "Replay browser probe with fresh screenshot"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"tool_use_id": "toolu_015WHowRypEs12DpL8sq2iBD",
|
||||
"type": "tool_result",
|
||||
"content": "QA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"started\",\"version\":1,\"startedAt\":\"2026-10-19T07:54:07.184Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"budgetMs\":30000,\"observedAt\":\"2026-10-19T07:54:25.941Z\",\"remainingMs\":11243,\"expired\":false}\n{\"navigation\":{\"stdout\":\"Navigated to http://127.0.0.1:39269/qa-only.html (200)\\n\",\"stderr\":\"\",\"exitCode\":0},\"console\":{\"stdout\":\"--- BEGIN UNTRUSTED EXTERNAL CONTENT (source: http://127.0.0.1:39269/qa-only.html) ---\\n[2026-10-19T07:54:11.698Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n[2026-10-19T07:54:25.967Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n--- END UNTRUSTED EXTERNAL CONTENT ---\\n\",\"stderr\":\"\",\"exitCode\":0},\"screenshot\":{\"stdout\":\"Screenshot saved: __QA_FIXTURE__/qa-reports/screenshots/replay-001.png\\n\",\"stderr\":\"\",\"exitCode\":0}}\n\nQA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"finished\",\"observedAt\":\"2026-10-19T07:54:26.040Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"timedOut\":false,\"exitCode\":0}",
|
||||
"is_error": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"parent_tool_use_id": null,
|
||||
"session_id": "58ff97a2-be5b-40db-a3ce-fb390aad7c88",
|
||||
"uuid": "4cc8ce35-f0d7-4076-991b-451564cf0144",
|
||||
"timestamp": "2026-10-19T07:54:26.050Z",
|
||||
"tool_use_result": {
|
||||
"stdout": "QA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"started\",\"version\":1,\"startedAt\":\"2026-10-19T07:54:07.184Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"budgetMs\":30000,\"observedAt\":\"2026-10-19T07:54:25.941Z\",\"remainingMs\":11243,\"expired\":false}\n{\"navigation\":{\"stdout\":\"Navigated to http://127.0.0.1:39269/qa-only.html (200)\\n\",\"stderr\":\"\",\"exitCode\":0},\"console\":{\"stdout\":\"--- BEGIN UNTRUSTED EXTERNAL CONTENT (source: http://127.0.0.1:39269/qa-only.html) ---\\n[2026-10-19T07:54:11.698Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n[2026-10-19T07:54:25.967Z] [error] TypeError: Cannot read properties of undefined (reading 'map')\\n--- END UNTRUSTED EXTERNAL CONTENT ---\\n\",\"stderr\":\"\",\"exitCode\":0},\"screenshot\":{\"stdout\":\"Screenshot saved: __QA_FIXTURE__/qa-reports/screenshots/replay-001.png\\n\",\"stderr\":\"\",\"exitCode\":0}}\n\nQA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"finished\",\"observedAt\":\"2026-10-19T07:54:26.040Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"timedOut\":false,\"exitCode\":0}",
|
||||
"stderr": "",
|
||||
"interrupted": false,
|
||||
"isImage": false,
|
||||
"noOutputExpected": false,
|
||||
"bashEditDiff": {
|
||||
"files": [],
|
||||
"moreFiles": 1,
|
||||
"changedFiles": [
|
||||
"__QA_FIXTURE__/qa-reports/screenshots/replay-001.png"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"message": {
|
||||
"model": "claude-opus-5-5",
|
||||
"id": "msg_011CgBEdLuDaVg7evd7cJGEr",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_01Fy5esoJHRgnfH9ts4HSMbR",
|
||||
"name": "Bash",
|
||||
"input": {
|
||||
"command": "bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'B=\"__QA_BROWSE__\"; \"$B\" stop'",
|
||||
"description": "Stop owned headless browser daemon under guard"
|
||||
},
|
||||
"caller": {
|
||||
"type": "direct"
|
||||
}
|
||||
}
|
||||
],
|
||||
"container": null,
|
||||
"stop_reason": null,
|
||||
"stop_sequence": null,
|
||||
"stop_details": null,
|
||||
"usage": {
|
||||
"input_tokens": 2,
|
||||
"cache_creation_input_tokens": 567,
|
||||
"cache_read_input_tokens": 49621,
|
||||
"cache_creation": {
|
||||
"ephemeral_5m_input_tokens": 567,
|
||||
"ephemeral_1h_input_tokens": 0
|
||||
},
|
||||
"output_tokens": 8,
|
||||
"service_tier": "standard",
|
||||
"inference_geo": "global",
|
||||
"speed": "standard"
|
||||
},
|
||||
"input_transformations": [],
|
||||
"diagnostics": null,
|
||||
"context_management": null
|
||||
},
|
||||
"parent_tool_use_id": null,
|
||||
"session_id": "58ff97a2-be5b-40db-a3ce-fb390aad7c88",
|
||||
"uuid": "b83d8680-da53-458a-bc98-72e2b46ff137",
|
||||
"timestamp": "2026-10-19T07:54:33.997Z",
|
||||
"request_id": "req_011CgBEdLeavMxE7VgpzTQCw",
|
||||
"wire_tool_inputs": {
|
||||
"toolu_01Fy5esoJHRgnfH9ts4HSMbR": {
|
||||
"command": "bun '__QA_GUARD__' run '__QA_FIXTURE__/qa-reports/deadline.json' -- bash -c 'B=\"__QA_BROWSE__\"; \"$B\" stop'",
|
||||
"description": "Stop owned headless browser daemon under guard"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"content": "Exit code 124\n\nQA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"started\",\"version\":1,\"startedAt\":\"2026-10-19T07:54:07.184Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"budgetMs\":30000,\"observedAt\":\"2026-10-19T07:54:34.020Z\",\"remainingMs\":3164,\"expired\":false}\nServer stopped\n\nQA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"finished\",\"observedAt\":\"2026-10-19T07:54:37.185Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"timedOut\":true,\"exitCode\":124}",
|
||||
"is_error": true,
|
||||
"tool_use_id": "toolu_01Fy5esoJHRgnfH9ts4HSMbR"
|
||||
}
|
||||
]
|
||||
},
|
||||
"parent_tool_use_id": null,
|
||||
"session_id": "58ff97a2-be5b-40db-a3ce-fb390aad7c88",
|
||||
"uuid": "074875af-080b-4620-80fc-5430f1aaf123",
|
||||
"timestamp": "2026-10-19T07:54:37.190Z",
|
||||
"tool_use_result": "Error: Exit code 124\n\nQA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"started\",\"version\":1,\"startedAt\":\"2026-10-19T07:54:07.184Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"budgetMs\":30000,\"observedAt\":\"2026-10-19T07:54:34.020Z\",\"remainingMs\":3164,\"expired\":false}\nServer stopped\n\nQA_DEADLINE {\"guard\":\"qa-deadline\",\"event\":\"finished\",\"observedAt\":\"2026-10-19T07:54:37.185Z\",\"deadlineAt\":\"2026-10-19T07:54:37.184Z\",\"timedOut\":true,\"exitCode\":124}"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -31,7 +31,7 @@ Every other Bash call must be exactly one invocation of bun ${quote(guard)}, wit
|
||||
Start exactly once, and only after you have read the workflow sections you need and your Write of the initial charter to ${path.join(directory, 'qa-reports/qa-only-report.md')} has completed; the 30-second deadline begins at start: bun ${quote(guard)} start ${quote(file)} 30
|
||||
Then use bun ${quote(guard)} status ${quote(file)} or bun ${quote(guard)} run ${quote(file)} -- COMMAND ARGS. Use the absolute guard path directly, not $G.
|
||||
For browser scripts, put all assignments, pipelines and scripts INSIDE the child: bun ${quote(guard)} run ${quote(file)} -- bash -c 'B="${browse}"; "$B" goto URL; "$B" snapshot -i'. Literal argv and single/double quoted literal arguments are accepted; expansions are only allowed inside the single-quoted child script.
|
||||
Do not write, reset, replace, chmod or remove deadline.json, invoke the guard recursively, or print QA_DEADLINE receipts yourself. Browser cleanup also goes inside the guard; after expiry only local report bookkeeping is allowed. Fallback screenshots must be saved directly in the owned screenshots directory; retain them and write the report after expiry without new browser calls. This fixture does not accept a separate post-expiry shell copy from Aside's session directory; mark that artifact unavailable rather than launching new browser work.
|
||||
Do not write, reset, replace, chmod or remove deadline.json, invoke the guard recursively, or print QA_DEADLINE receipts yourself. Browser cleanup also goes inside the guard, as a whole child that is exactly "$B" closetab or "$B" stop (B set to the supplied browse binary); any other child counts as a probe; after expiry only local report bookkeeping is allowed. Fallback screenshots must be saved directly in the owned screenshots directory; retain them and write the report after expiry without new browser calls. This fixture does not accept a separate post-expiry shell copy from Aside's session directory; mark that artifact unavailable rather than launching new browser work.
|
||||
An expired refusal is not an executed probe or a pass. Report unfinished coverage honestly; do not try to finish every page after expiry.` };
|
||||
}
|
||||
|
||||
@@ -110,13 +110,13 @@ export function assertQaBrowserPreparation(transcript: unknown[], options: Pick<
|
||||
|
||||
/**
|
||||
* The fixture prompt routes browser cleanup through the guard, and a probe
|
||||
* excludes bookkeeping. Only closing the tab with the supplied browse binary,
|
||||
* as the whole child, is cleanup; anything else in the child is a probe.
|
||||
* excludes bookkeeping. Only closing the tab or stopping the supplied browse
|
||||
* binary, as the whole child, is cleanup; anything else in the child is a probe.
|
||||
*/
|
||||
function browserCleanupChild(child: string[], browse: string): boolean {
|
||||
if (child.length === 2) return child[0] === browse && child[1] === 'closetab';
|
||||
return child.length === 3 && child[0] === 'bash' && child[1] === '-c'
|
||||
&& [`B="${browse}"; "$B" closetab`, `"${browse}" closetab`].includes(child[2].trim());
|
||||
if (child.length === 2) return child[0] === browse && ['closetab', 'stop'].includes(child[1]);
|
||||
return child.length === 3 && child[0] === 'bash' && child[1] === '-c' && ['closetab', 'stop'].some(verb =>
|
||||
[`B="${browse}"; "$B" ${verb}`, `B="${browse}"; $B ${verb}`, `"${browse}" ${verb}`].includes(child[2].trim()));
|
||||
}
|
||||
|
||||
export function assertQaBrowserCheckpoints(transcript: unknown[], options: Pick<Options, 'directory' | 'guard' | 'browse'>) {
|
||||
|
||||
@@ -6,6 +6,7 @@ import { spawnSync } from 'node:child_process';
|
||||
import { assertQaBrowserDeadline, assertQaBrowserPreparation, assertQaBrowserCheckpoints, qaDeadlineShellPolicy } from './helpers/qa-browser-deadline-evidence';
|
||||
import capturedPreparation from './fixtures/qa-only-charter-public.json';
|
||||
import capturedObservation from './fixtures/qa-only-observation-public.json';
|
||||
import capturedStop from './fixtures/qa-only-stop-cleanup-public.json';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const guard = path.join(ROOT, 'bin/gstack-qa-deadline');
|
||||
@@ -89,6 +90,50 @@ test.each(['cleanup-after-last-probe', 'cleanup-direct-argv', 'probe-after-clean
|
||||
else expect(check).toThrow(scenario === 'probe-after-cleanup' ? 'browser cleanup preceded a probe' : 'follow-up lacks one acknowledged preceding checkpoint');
|
||||
});
|
||||
|
||||
// PR #3272 Opus gate 39217387697 slice 8 (qa-only-no-fix, head 3986ac5): two guarded probes with one
|
||||
// checkpoint, then the browser stopped inside the guard. stop is cleanup like closetab; a disguised probe is not.
|
||||
test.each(['captured', 'unquoted-variable', 'direct-argv', 'closetab-then-stop', 'stop-before-probe', 'disguised-probe', 'other-verb'])
|
||||
('guarded "$B" stop after the last probe is cleanup: %s', scenario => {
|
||||
const f = fixture();
|
||||
const events = JSON.parse(JSON.stringify(capturedStop.events)
|
||||
.replaceAll('__QA_FIXTURE__', f.directory).replaceAll('__QA_GUARD__', guard).replaceAll('__QA_BROWSE__', browse));
|
||||
const write = events[2].message.content[0];
|
||||
fs.writeFileSync(write.input.file_path, write.input.content);
|
||||
const stop = events[6].message.content[0];
|
||||
const prefix = `bun ${quote(guard)} run ${quote(path.join(f.directory, 'qa-reports/deadline.json'))} -- `;
|
||||
expect(stop.input.command).toBe(prefix + `bash -c 'B="${browse}"; "$B" stop'`);
|
||||
const child = ({
|
||||
'unquoted-variable': `bash -c 'B="${browse}"; $B stop'`,
|
||||
'direct-argv': quote(browse) + ' stop',
|
||||
'disguised-probe': `bash -c 'B="${browse}"; "$B" goto http://127.0.0.1:1/; "$B" stop'`,
|
||||
'other-verb': `bash -c 'B="${browse}"; "$B" stop --all'`,
|
||||
} as Record<string, string>)[scenario];
|
||||
if (child) stop.input.command = prefix + child;
|
||||
if (scenario === 'closetab-then-stop') {
|
||||
const closetab = structuredClone(events.slice(6, 8));
|
||||
closetab[0].message.content[0].id = closetab[1].message.content[0].tool_use_id = 'toolu_closetab';
|
||||
closetab[0].message.content[0].input.command = prefix + `bash -c 'B="${browse}"; "$B" closetab'`;
|
||||
events.splice(6, 0, ...closetab);
|
||||
}
|
||||
if (scenario === 'stop-before-probe') events.splice(4, 0, ...events.splice(6, 2));
|
||||
const check = () => assertQaBrowserCheckpoints(events, { directory: f.directory, guard, browse });
|
||||
if (['captured', 'unquoted-variable', 'direct-argv', 'closetab-then-stop'].includes(scenario)) expect(check).not.toThrow();
|
||||
else expect(check).toThrow(scenario === 'stop-before-probe' ? 'browser cleanup preceded a probe' : 'follow-up lacks one acknowledged preceding checkpoint');
|
||||
});
|
||||
|
||||
test.each([['direct', (b: string) => `${quote(b)} stop`], ['shell variable', (b: string) => `B=${quote(b)}; "$B" stop`]])
|
||||
('browser stop outside the guard is not cleanup: %s', (_, command) => {
|
||||
const f = fixture();
|
||||
f.run();
|
||||
f.calls.push({ tool: 'Bash', input: { command: command(browse) }, output: '' });
|
||||
expect(() => f.check()).toThrow('QA deadline: ');
|
||||
});
|
||||
|
||||
test('the deadline prompt names both accepted cleanup children', () => {
|
||||
const { policy } = fixture();
|
||||
expect(policy.prompt).toContain('exactly "$B" closetab or "$B" stop');
|
||||
});
|
||||
|
||||
test('R70 retained public ordering has no completed report Write before its clock/baseline', () => {
|
||||
const f = fixture();
|
||||
const events = JSON.parse(JSON.stringify(capturedPreparation.events)
|
||||
|
||||
Reference in new issue
Block a user