From ddbf1c9a471dbce28ed4ce7340b8af4dd3272957 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 7 Nov 2026 10:43:11 -0800 Subject: [PATCH] fix: /autoplan gate makes each phase publish in order after its report (GST-1814) (#4135) * test: failing-first replays of the three E2 autoplan gate sequences (GST-1814) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: /autoplan gate refuses a phase publish out of order or before its report; per-step Stop budget that never exhausts before the first report (GST-1814) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes Before scope is known, CEO publishes first; an exhausted or released run no longer refuses publishes. Review-Finding: p1/red-team#1 reader=red-team Review-Finding: p1/red-team#3 reader=red-team Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: the publish check reads the transcript only up to the call it decides Review-Finding: p1v/outside-voice#1 reader=outside-voice Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: changelog fragment notes that a released or exhausted run no longer refuses publishes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- bin/gstack-settings-hook | 3 +- bin/gstack-skill-gates | 2 +- .../capy-gst-1814-autoplan-gate-order.md | 1 + docs/skill-gates.md | 24 +-- hosts/claude/hooks/autoplan-stop-gate-publish | 14 ++ hosts/claude/hooks/autoplan-stop-gate.ts | 101 ++++++++--- lib/autoplan-stop-gate.ts | 76 +++++++-- lib/skill-gates.ts | 27 +++ scripts/skill-gates-smoke.ts | 12 +- setup | 7 +- test/autoplan-stop-gate.test.ts | 159 ++++++++++++++++++ test/setup-skill-hooks-gate.test.ts | 20 ++- test/skill-gates.test.ts | 18 +- 13 files changed, 398 insertions(+), 66 deletions(-) create mode 100644 changelog.d/capy-gst-1814-autoplan-gate-order.md create mode 100755 hosts/claude/hooks/autoplan-stop-gate-publish diff --git a/bin/gstack-settings-hook b/bin/gstack-settings-hook index 2f71edb17..f7005384c 100755 --- a/bin/gstack-settings-hook +++ b/bin/gstack-settings-hook @@ -136,7 +136,8 @@ var KNOWN_HOOKS = { "investigate-gate-stop": { source: "gstack-investigate-gate", event: "Stop", matcher: "", relpath: "hosts/claude/hooks/investigate-gate-stop" }, "ceo-review-gate-stop": { source: "gstack-ceo-review-gate", event: "Stop", matcher: "", relpath: "hosts/claude/hooks/ceo-review-gate-stop" }, "ceo-review-gate-subagent-stop": { source: "gstack-ceo-review-gate", event: "SubagentStop", matcher: "", relpath: "hosts/claude/hooks/ceo-review-gate-subagent-stop" }, - "autoplan-stop-gate-stop": { source: "gstack-autoplan-gate", event: "Stop", matcher: "", relpath: "hosts/claude/hooks/autoplan-stop-gate-stop" } + "autoplan-stop-gate-stop": { source: "gstack-autoplan-gate", event: "Stop", matcher: "", relpath: "hosts/claude/hooks/autoplan-stop-gate-stop" }, + "autoplan-stop-gate-publish": { source: "gstack-autoplan-gate", event: "PreToolUse", matcher: "Bash", relpath: "hosts/claude/hooks/autoplan-stop-gate-publish" } }; function gsHadBashPrefix(c) { return String(c == null ? "" : c).trim().indexOf("bash ") === 0; } function gsStripWrap(c) { diff --git a/bin/gstack-skill-gates b/bin/gstack-skill-gates index f9cd27eb0..5bacff5ae 100755 --- a/bin/gstack-skill-gates +++ b/bin/gstack-skill-gates @@ -36,7 +36,7 @@ const HOOKS: Record` is now refused before it runs unless that phase is the next one due and its report was sent since the previous publish; the refusal starts `Next due: the Phase N () report.` A publish with no report or out of order never counts at Stop either, so a later report can't rescue it. Stop blocks now count against the step they name (3 per step, reset as the run advances), blocks before the run's first phase report don't count, and 12 blocks end any run; once a run is released or exhausted, its publishes are no longer refused. Setup registers one more hook for the gate (PreToolUse on Bash; its shim starts bun only for the publish command). Still off by default, and nothing changes with the switch off. Docs: docs/skill-gates.md#the-autoplan-gate. diff --git a/docs/skill-gates.md b/docs/skill-gates.md index c6d872b23..687be7154 100644 --- a/docs/skill-gates.md +++ b/docs/skill-gates.md @@ -5,7 +5,7 @@ gstack's advisory skills say what a careful agent does; a gate makes one rule ho | Gate | Switch | What it holds | |---|---|---| | /plan-ceo-review | `ceo_review_gate` | After /plan-ceo-review loads, the final stop (of the session or of a review subagent) waits until the review's output file ranks the top risks | -| /autoplan | `autoplan_gate` | After /autoplan loads, the session's final stop waits until every applicable phase has published its report and the Final Approval Gate is written (GST-1666) | +| /autoplan | `autoplan_gate` | After /autoplan loads, each phase publishes only in phase order after its own report (GST-1814), and the session's final stop waits until every applicable phase has published and the Final Approval Gate is written (GST-1666) | ## Turn a gate on @@ -16,7 +16,7 @@ The switch resolves like the contract hook's, and the first one set wins: 3. Config: `gstack-config set ceo_review_gate on|off` or `gstack-config set autoplan_gate on|off`. 4. The default, `SWITCH_DEFAULT` in `lib/skill-route.ts`: off for both. -Setup registers a gate's shims in `~/.claude/settings.json` when its switch is on (Stop and SubagentStop for /plan-ceo-review; Stop for /autoplan) and removes them when it is off. The hooks also read the switch on every call, so `gstack-config set ceo_review_gate off` takes effect on the next stop. +Setup registers a gate's shims in `~/.claude/settings.json` when its switch is on (Stop and SubagentStop for /plan-ceo-review; Stop, and PreToolUse on Bash for the publish check, for /autoplan) and removes them when it is off. The hooks also read the switch on every call, so `gstack-config set ceo_review_gate off` takes effect on the next stop. ### Headless runs @@ -66,18 +66,20 @@ Once a session loads /autoplan (the Skill tool, a typed `/autoplan`, or a Read o **What meets it.** One run is the transcript from the load to the next /autoplan load. A full run is complete when: 1. Phase 0 ran scope: a successful `gstack-autoplan-snapshot init` (its `scope` field) or `scope` result with `uiRequired` and `dxRequired`. It decides the applicable phases: CEO and Eng always, Design when `uiRequired`, DX when `dxRequired`. -2. Every applicable phase published, as `sections/phase-close.md` step 6 says: assistant text with its `Phase N complete` line, then a successful Bash no-op `true autoplan-published `. A no-op with no report since the previous one, or a failed one, does not count. +2. Every applicable phase published, in phase order, as `sections/phase-close.md` step 6 says: assistant text with its `Phase N complete` line, then a successful Bash no-op `true autoplan-published `. A no-op counts only when that report came after the previous no-op and every earlier applicable phase already counted. A no-op with no report, one for a phase that is not due yet, or a failed one does not count, and a later report does not make it count: the phase publishes again after its report. 3. The Phase 4 Final Approval Gate is written: a successful `gstack-gate render` call, or a Write or Edit of a `gate.json`. A `--ceo-only` run needs Phase 1 published and the `GSTACK_RESULT: skill=autoplan status=reviewed` end line from `sections/ceo-only.md`. Claude Code can fire Stop before it writes the turn's last lines to the transcript (the smoke run saw a scope call and a phase publication land about 200 ms late), so the hook re-reads the transcript until it holds the stop's `last_assistant_message`, for at most 2 seconds or half its deadline, and judges what is there then. +**The publish check (GST-1814).** In GEVA-2's E2 eval (3 told Opus 5.5 sessions, 1 exposed) every failure was in the report step: sessions ran the publish no-ops with no reports, published Eng's report before CEO's, and wrote reports only after Stop blocks. So the gate also checks each `true autoplan-published ` before it runs (PreToolUse on Bash; the `autoplan-stop-gate-publish` shim hands only that command to bun). It refuses the call, with a `blocked via=publish phase= why=no-report|out-of-order missing=` marker as the reason's first line, when the phase is not the next applicable phase to publish, or when no report for it was sent since the previous publish no-op (the report in the same message counts). The reason's next line is `Next due: the Phase N () report.`, then what to send. A phase that already counted, a phase scope makes inapplicable, a runner-mode run and a subagent's call are never refused; before scope is known CEO's must come first (it always applies), then only the report and forward order are checked; after the run is released or exhausted no publish is refused. Claude Code can run the hook before it writes the call to the transcript, so the hook waits up to 2 seconds for the call's own record; if it never appears the call runs with a `skipped reason=unsettled` marker and the Stop check still applies. + **What it never holds.** A stop while a native reviewer launched in the background has not reported back (autoplan's "Waiting for " barrier); the next final stop after the notification is checked. A run that drives the `gstack-autoplan` runner (`next`, `submit`, `answer` or `export`), which keeps its own record (docs/unattended.md): it gets one `skipped reason=runner` marker and is never blocked. -**The block.** Its reason starts with the `blocked via=stop missing= published=` marker and names the next step: run Phase 0 (`missing=scope`), continue at a phase's driver Read when it has not started, finish a started phase's close and publish its report (`missing=ceo|design|dx|eng`), write and render the gate (`missing=gate`), or end a `--ceo-only` run (`missing=end`). It lists the applicable and published phases and never names the off switch. +**The block.** Its reason starts with the `blocked via=stop missing= published=` marker; for a phase step the next line is `Next due: the Phase N () report.` Then it names the next step: run Phase 0 (`missing=scope`), continue at a phase's driver Read when it has not started, finish a started phase's close and publish its report (`missing=ceo|design|dx|eng`), write and render the gate (`missing=gate`), or end a `--ceo-only` run (`missing=end`). It lists publish no-ops that did not count and why (`no report before it`, `out of order: was due`), the applicable and published phases, and never names the off switch. -**The cap and the record.** As for /plan-ceo-review: at most 3 blocks per run, then the next final stop goes through with `exhausted missing=`; `stop_hook_active` does not skip the check. The session record keeps each run's `phases` and `published` as of its last final stop, so `gstack-skill-gates report --session --expect armed --json` says per session whether /autoplan armed, how often it was held, and whether it was released or exhausted. An eval can read exposure from that record instead of from the model's wording. +**The cap and the record.** A Stop block counts against the step it names, at most 3 per step; the count resets when the step advances, and blocks before the run's first phase report count against no step (E2 spent all 3 blocks of the old per-run cap on `missing=ceo` before any work, GST-1814). After 3 blocks at one step, or 12 blocks in the run whatever the step, the next final stop goes through with `exhausted missing=`; `stop_hook_active` does not skip the check. The session record keeps each run's `phases`, `published`, `step` and `step_blocks` as of its last final stop and its `denied` publish count, so `gstack-skill-gates report --session --expect armed --json` says per session whether /autoplan armed, how often it was held, and whether it was released or exhausted. An eval can read exposure from that record instead of from the model's wording. ## Markers @@ -87,16 +89,16 @@ Every decision is one marker line: | Event | When | Fields | |---|---|---| -| `fired` | the gate armed for a skill run (once per run; for /plan-ceo-review and /autoplan, its first final stop) | `skill`, `on` (the hook event) | -| `blocked` | each refusal | `skill`, `via=stop`, `files` (for /plan-ceo-review's `via=stop`, the candidate files checked, or `none`), `missing` and `published` (/autoplan: the next missing step and the phases published so far), `n` (blocks so far in this run) | +| `fired` | the gate armed for a skill run (once per run; for /plan-ceo-review, its first final stop; for /autoplan, its first final stop or refused publish) | `skill`, `on` (the hook event) | +| `blocked` | each refusal | `skill`, `via=stop` or `via=publish` (/autoplan's publish check), `files` (for /plan-ceo-review's `via=stop`, the candidate files checked, or `none`), `missing` and `published` (/autoplan: the next missing step and the phases published so far), `phase` and `why` (`via=publish`: the refused phase and `no-report` or `out-of-order`), `n` (stop blocks, or refused publishes for `via=publish`, so far in this run) | | `released` | an output file ranks the top risks, or the /autoplan run is complete (once per run) | `skill`, `file` (the output file's name) or `phases` (the phases /autoplan published), `blocked` | -| `exhausted` | /plan-ceo-review and /autoplan: a fourth final stop after 3 blocks, which goes through | `skill`, `missing` (/autoplan), `blocked` | -| `skipped` | the gate could not arm (once per session and reason), or an /autoplan run drives the gstack-autoplan runner (once per run) | `skill`, `reason=no-git|no-head|runner` | +| `exhausted` | /plan-ceo-review: a fourth final stop after 3 blocks; /autoplan: a final stop after 3 blocks at one step once a report exists, or after 12 in the run; it goes through | `skill`, `missing` (/autoplan), `blocked` | +| `skipped` | the gate could not arm (once per session and reason), or an /autoplan run drives the gstack-autoplan runner (once per run) | `skill`, `reason=no-git|no-head|runner|unsettled` (`unsettled`: /autoplan's publish check never saw its call in the transcript, so the call ran) | | `error` | a hook error; the call or stop went through | `code`, `why` | Where each one lands: -- **Session record** (the count source): `$GSTACK_STATE_ROOT/skill-gates//record.json`, mode 0600, written atomically under the session lock. Per gate and skill run it holds `fired_at`, `released_at`, `file`, `blocked`, `exhausted_at`, `agent` (a subagent's run), `phases` and `published` (/autoplan), plus the gate's `errors` and `skipped` reasons. `gstack-skill-gates report` reads these. +- **Session record** (the count source): `$GSTACK_STATE_ROOT/skill-gates//record.json`, mode 0600, written atomically under the session lock. Per gate and skill run it holds `fired_at`, `released_at`, `file`, `blocked`, `exhausted_at`, `agent` (a subagent's run), `phases`, `published`, `step`, `step_blocks` and `denied` (/autoplan), plus the gate's `errors` and `skipped` reasons. `gstack-skill-gates report` reads these. - **Trace log**: `$GSTACK_STATE_ROOT/hooks/ceo-review-gate.log` and `hooks/autoplan-gate.log`, one line per marker in the contract hook's shape (`