v1.27.1.0 fix: anti-shortcut clause + gate-tier AskUserQuestion floor tests for all plan-* skills (#1354)

* feat(test/helpers): runPlanSkillFloorCheck — minimal AskUserQuestion-floor observer

Adds a focused PTY observer that exits at the first non-permission
numbered-option render. Catches the May 2026 transcript-bug class
(model wrote plan + ExitPlanMode without firing any AUQ) without
needing to fingerprint or navigate past the AUQ.

Why separate from runPlanSkillCounting: plan-mode AUQs render every
option on a single logical line via cursor-positioning escapes that
stripAnsi can't simulate, so parseNumberedOptions returns < 2 options
and never records a fingerprint. Counting tests work on 25-min budgets
because eventually one frame parses cleanly; gate-tier floor tests
need to exit early on the first observation. Trades fingerprint
precision for early-exit reliability.

Also drops COMPLETION_SUMMARY_RE check from this helper — it matches
"GSTACK REVIEW REPORT" anywhere in the buffer including when the
agent does recon by reading existing plan files. plan_ready
(claude's actual "Ready to execute" confirmation) is the reliable
terminal signal for "agent finished without asking."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(resolvers): generateAntiShortcutClause shared resolver

Adds {{ANTI_SHORTCUT_CLAUSE}} placeholder backed by a single resolver
function in scripts/resolvers/review.ts. Plan-* review skills can now
include the clause via one placeholder line in their .tmpl rather than
cloning the paragraph four times. Future tightening edits one resolver,
all four skills update on next gen-skill-docs.

Wired into the existing RESOLVERS map alongside generateReviewDashboard
and generatePlanFileReviewReport — no gen-skill-docs.ts change needed
because the generator already does generic placeholder substitution
against that map.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(plan-*-review): anti-shortcut clause in all four review skills

Inserts {{ANTI_SHORTCUT_CLAUSE}} placeholder immediately after the
**Anti-skip rule:** paragraph in plan-{eng,ceo,design,devex}-review
SKILL.md.tmpl. The four templates use different surrounding section
headers (eng "Review Sections (after scope is agreed)" vs ceo/design/devex
variants), so anchoring on the paragraph rather than the heading works
across all four.

Closes the May 2026 transcript-bug loophole: existing STOP gates name
forbidden actions only AFTER a per-section finding is identified. The
anti-shortcut clause adds the pre-emptive rule — "the plan file is the
OUTPUT of the interactive review, not a substitute for it" — covering
the case the transcript exhibited (skip per-section walk, dump every
finding into one plan write, call ExitPlanMode).

Regenerated SKILL.md for all hosts via bun run gen:skill-docs --host all.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: gate-tier AskUserQuestion floor tests for all plan-* review skills

Adds 4 finding-floor tests (one per plan-* skill) that catch the May
2026 transcript-bug class — model wrote a plan and called ExitPlanMode
without firing any review-phase AskUserQuestion. Asserts via
runPlanSkillFloorCheck that ANY non-permission AUQ render fires before
the agent reaches plan_ready.

Verified:
- Eng floor: passed in 59s
- CEO floor: passed in 197s
- Design floor: passed
- Devex floor: passed
- Total ~$2-6 per CI run; only triggers on diff against the 4 plan-*
  templates, the shared resolver review.ts, the seeds fixture, or the
  PTY runner helper.

Fixtures live in test/fixtures/forcing-finding-seeds.ts, one constant
per skill. Each seed is engineered to force at least one obvious
finding under that skill's review focus (architectural smell for eng,
scope-creep for ceo, UI-slop for design, painful onboarding for devex).

Touchfiles wiring:
- E2E_TOUCHFILES: 4 plan-*-finding-floor entries with deps on the
  matching skill template, the shared resolver, the seeds fixture,
  and the PTY runner helper
- E2E_TIERS: all 4 entries marked 'gate'
- touchfiles.test.ts: count assertion bumped 21→22 with explicit
  plan-ceo-finding-floor containment check

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v1.27.1.0)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-06 20:27:20 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent f44de365c5
commit 7b4738bca0
21 changed files with 532 additions and 5 deletions
+80
View File
@@ -1,5 +1,85 @@
# Changelog
## [1.27.1.0] - 2026-05-06
## **Plan-mode reviews now refuse to dump findings without asking. Four gate-tier tests catch the regression on every PR.**
The four `/plan-*-review` skills (eng, ceo, design, devex) gain an
anti-shortcut clause baked in via a single shared resolver. The clause
names the May 2026 transcript-bug failure mode directly: model explores,
finds issues, dumps every finding into one plan write, calls
ExitPlanMode without firing AskUserQuestion. The new clause closes that
loophole: "the plan file is the OUTPUT of the interactive review, not a
substitute for it." Future tightening edits one resolver, all four
skills update on the next gen-skill-docs.
Four gate-tier E2E tests catch the regression class on every PR that
touches the four templates, the shared resolver, or the seeds fixture.
Each test drives the matching skill against a small "forcing finding"
seed and asserts the agent fires at least one AskUserQuestion before
reaching plan_ready. ~1-3 min wall time per test, ~$2-6 total per CI
hit. Eng floor: 59s. CEO floor: 197s. All four pass against the new
template.
### The numbers that matter
Verified end-to-end via live PTY runs against `claude` plan mode:
| Surface | Before | After | Δ |
|---|---|---|---|
| Plan-mode reviews with anti-shortcut clause | 0/4 | 4/4 | full coverage of plan-* family |
| Gate-tier regression tests for the transcript-bug class | 0 | 4 | one per skill |
| Wall time per floor test (typical) | n/a | 30s-3m | early exit on first AUQ render |
| Cost per gate run (when triggered) | n/a | ~$2-6 | diff-gated; only fires on relevant edits |
| Lines added / deleted | — | +450 / 3 | additive; no breaking changes |
The floor tests use a focused observer (`runPlanSkillFloorCheck`) that
exits at the first non-permission numbered-option render. Existing
periodic finding-count tests use `runPlanSkillCounting` for full
fingerprint analysis on a 25-min budget; the floor variant trades
fingerprint precision for early-exit reliability so it fits gate-tier
constraints. Both helpers live side-by-side in
`test/helpers/claude-pty-runner.ts`.
### What this means for the four review skills
Every plan-* review now has a structural rule against the precise
failure mode the transcript exhibited. The anti-shortcut clause
appears in the rendered prompt right after the existing Anti-skip
rule, so it's read alongside the per-section STOP gates v1.26.2.0
already added. If a future model regression revives the bug, the
gate-tier floor test fires with full PTY evidence on the next PR.
### Itemized changes
#### Added
- **`generateAntiShortcutClause` resolver** in `scripts/resolvers/review.ts`,
registered as `{{ANTI_SHORTCUT_CLAUSE}}` in the `RESOLVERS` map.
Plan-* SKILL.md.tmpl files include it via one placeholder line.
- **`runPlanSkillFloorCheck` PTY helper** in
`test/helpers/claude-pty-runner.ts` — minimal "did the agent fire ANY
AskUserQuestion?" observer with early exit on first non-permission
numbered-option render.
- **Four gate-tier finding-floor E2E tests** in
`test/skill-e2e-plan-{eng,ceo,design,devex}-finding-floor.test.ts`,
each using the shared `runPlanSkillFloorCheck` helper.
- **Four forcing-finding seeds** in `test/fixtures/forcing-finding-seeds.ts`,
one per skill, each engineered to surface at least one finding under
that skill's review focus.
#### Changed
- **All four `plan-*-review` SKILL.md** files now include the
anti-shortcut clause immediately after the `**Anti-skip rule:**`
paragraph. Anchored on the paragraph (not the surrounding heading)
so the same insertion works across all four templates regardless of
their differing section labels.
- **`test/helpers/touchfiles.ts`** adds 4 entries to `E2E_TOUCHFILES`
and `E2E_TIERS=gate`. The new entries depend on the matching skill
template, the shared resolver, the seeds fixture, and the PTY
runner helper.
- **`test/touchfiles.test.ts`** count assertion bumped 21→22 with
explicit `plan-ceo-finding-floor` containment.
## [1.27.0.0] - 2026-05-06
## **`/setup-gbrain` connects to a remote brain in one paste. Brain repo renamed to gstack-artifacts.**