mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-23 13:20:48 +02:00
v1.27.1.0 fix: anti-shortcut clause + gate-tier AskUserQuestion floor tests for all plan-* skills (#1354)
* feat(test/helpers): runPlanSkillFloorCheck — minimal AskUserQuestion-floor observer Adds a focused PTY observer that exits at the first non-permission numbered-option render. Catches the May 2026 transcript-bug class (model wrote plan + ExitPlanMode without firing any AUQ) without needing to fingerprint or navigate past the AUQ. Why separate from runPlanSkillCounting: plan-mode AUQs render every option on a single logical line via cursor-positioning escapes that stripAnsi can't simulate, so parseNumberedOptions returns < 2 options and never records a fingerprint. Counting tests work on 25-min budgets because eventually one frame parses cleanly; gate-tier floor tests need to exit early on the first observation. Trades fingerprint precision for early-exit reliability. Also drops COMPLETION_SUMMARY_RE check from this helper — it matches "GSTACK REVIEW REPORT" anywhere in the buffer including when the agent does recon by reading existing plan files. plan_ready (claude's actual "Ready to execute" confirmation) is the reliable terminal signal for "agent finished without asking." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(resolvers): generateAntiShortcutClause shared resolver Adds {{ANTI_SHORTCUT_CLAUSE}} placeholder backed by a single resolver function in scripts/resolvers/review.ts. Plan-* review skills can now include the clause via one placeholder line in their .tmpl rather than cloning the paragraph four times. Future tightening edits one resolver, all four skills update on next gen-skill-docs. Wired into the existing RESOLVERS map alongside generateReviewDashboard and generatePlanFileReviewReport — no gen-skill-docs.ts change needed because the generator already does generic placeholder substitution against that map. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(plan-*-review): anti-shortcut clause in all four review skills Inserts {{ANTI_SHORTCUT_CLAUSE}} placeholder immediately after the **Anti-skip rule:** paragraph in plan-{eng,ceo,design,devex}-review SKILL.md.tmpl. The four templates use different surrounding section headers (eng "Review Sections (after scope is agreed)" vs ceo/design/devex variants), so anchoring on the paragraph rather than the heading works across all four. Closes the May 2026 transcript-bug loophole: existing STOP gates name forbidden actions only AFTER a per-section finding is identified. The anti-shortcut clause adds the pre-emptive rule — "the plan file is the OUTPUT of the interactive review, not a substitute for it" — covering the case the transcript exhibited (skip per-section walk, dump every finding into one plan write, call ExitPlanMode). Regenerated SKILL.md for all hosts via bun run gen:skill-docs --host all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: gate-tier AskUserQuestion floor tests for all plan-* review skills Adds 4 finding-floor tests (one per plan-* skill) that catch the May 2026 transcript-bug class — model wrote a plan and called ExitPlanMode without firing any review-phase AskUserQuestion. Asserts via runPlanSkillFloorCheck that ANY non-permission AUQ render fires before the agent reaches plan_ready. Verified: - Eng floor: passed in 59s - CEO floor: passed in 197s - Design floor: passed - Devex floor: passed - Total ~$2-6 per CI run; only triggers on diff against the 4 plan-* templates, the shared resolver review.ts, the seeds fixture, or the PTY runner helper. Fixtures live in test/fixtures/forcing-finding-seeds.ts, one constant per skill. Each seed is engineered to force at least one obvious finding under that skill's review focus (architectural smell for eng, scope-creep for ceo, UI-slop for design, painful onboarding for devex). Touchfiles wiring: - E2E_TOUCHFILES: 4 plan-*-finding-floor entries with deps on the matching skill template, the shared resolver, the seeds fixture, and the PTY runner helper - E2E_TIERS: all 4 entries marked 'gate' - touchfiles.test.ts: count assertion bumped 21→22 with explicit plan-ceo-finding-floor containment check Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v1.27.1.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
f44de365c5
commit
7b4738bca0
@@ -1,5 +1,85 @@
|
||||
# Changelog
|
||||
|
||||
## [1.27.1.0] - 2026-05-06
|
||||
|
||||
## **Plan-mode reviews now refuse to dump findings without asking. Four gate-tier tests catch the regression on every PR.**
|
||||
|
||||
The four `/plan-*-review` skills (eng, ceo, design, devex) gain an
|
||||
anti-shortcut clause baked in via a single shared resolver. The clause
|
||||
names the May 2026 transcript-bug failure mode directly: model explores,
|
||||
finds issues, dumps every finding into one plan write, calls
|
||||
ExitPlanMode without firing AskUserQuestion. The new clause closes that
|
||||
loophole: "the plan file is the OUTPUT of the interactive review, not a
|
||||
substitute for it." Future tightening edits one resolver, all four
|
||||
skills update on the next gen-skill-docs.
|
||||
|
||||
Four gate-tier E2E tests catch the regression class on every PR that
|
||||
touches the four templates, the shared resolver, or the seeds fixture.
|
||||
Each test drives the matching skill against a small "forcing finding"
|
||||
seed and asserts the agent fires at least one AskUserQuestion before
|
||||
reaching plan_ready. ~1-3 min wall time per test, ~$2-6 total per CI
|
||||
hit. Eng floor: 59s. CEO floor: 197s. All four pass against the new
|
||||
template.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Verified end-to-end via live PTY runs against `claude` plan mode:
|
||||
|
||||
| Surface | Before | After | Δ |
|
||||
|---|---|---|---|
|
||||
| Plan-mode reviews with anti-shortcut clause | 0/4 | 4/4 | full coverage of plan-* family |
|
||||
| Gate-tier regression tests for the transcript-bug class | 0 | 4 | one per skill |
|
||||
| Wall time per floor test (typical) | n/a | 30s-3m | early exit on first AUQ render |
|
||||
| Cost per gate run (when triggered) | n/a | ~$2-6 | diff-gated; only fires on relevant edits |
|
||||
| Lines added / deleted | — | +450 / −3 | additive; no breaking changes |
|
||||
|
||||
The floor tests use a focused observer (`runPlanSkillFloorCheck`) that
|
||||
exits at the first non-permission numbered-option render. Existing
|
||||
periodic finding-count tests use `runPlanSkillCounting` for full
|
||||
fingerprint analysis on a 25-min budget; the floor variant trades
|
||||
fingerprint precision for early-exit reliability so it fits gate-tier
|
||||
constraints. Both helpers live side-by-side in
|
||||
`test/helpers/claude-pty-runner.ts`.
|
||||
|
||||
### What this means for the four review skills
|
||||
|
||||
Every plan-* review now has a structural rule against the precise
|
||||
failure mode the transcript exhibited. The anti-shortcut clause
|
||||
appears in the rendered prompt right after the existing Anti-skip
|
||||
rule, so it's read alongside the per-section STOP gates v1.26.2.0
|
||||
already added. If a future model regression revives the bug, the
|
||||
gate-tier floor test fires with full PTY evidence on the next PR.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
- **`generateAntiShortcutClause` resolver** in `scripts/resolvers/review.ts`,
|
||||
registered as `{{ANTI_SHORTCUT_CLAUSE}}` in the `RESOLVERS` map.
|
||||
Plan-* SKILL.md.tmpl files include it via one placeholder line.
|
||||
- **`runPlanSkillFloorCheck` PTY helper** in
|
||||
`test/helpers/claude-pty-runner.ts` — minimal "did the agent fire ANY
|
||||
AskUserQuestion?" observer with early exit on first non-permission
|
||||
numbered-option render.
|
||||
- **Four gate-tier finding-floor E2E tests** in
|
||||
`test/skill-e2e-plan-{eng,ceo,design,devex}-finding-floor.test.ts`,
|
||||
each using the shared `runPlanSkillFloorCheck` helper.
|
||||
- **Four forcing-finding seeds** in `test/fixtures/forcing-finding-seeds.ts`,
|
||||
one per skill, each engineered to surface at least one finding under
|
||||
that skill's review focus.
|
||||
|
||||
#### Changed
|
||||
- **All four `plan-*-review` SKILL.md** files now include the
|
||||
anti-shortcut clause immediately after the `**Anti-skip rule:**`
|
||||
paragraph. Anchored on the paragraph (not the surrounding heading)
|
||||
so the same insertion works across all four templates regardless of
|
||||
their differing section labels.
|
||||
- **`test/helpers/touchfiles.ts`** adds 4 entries to `E2E_TOUCHFILES`
|
||||
and `E2E_TIERS=gate`. The new entries depend on the matching skill
|
||||
template, the shared resolver, the seeds fixture, and the PTY
|
||||
runner helper.
|
||||
- **`test/touchfiles.test.ts`** count assertion bumped 21→22 with
|
||||
explicit `plan-ceo-finding-floor` containment.
|
||||
|
||||
## [1.27.0.0] - 2026-05-06
|
||||
|
||||
## **`/setup-gbrain` connects to a remote brain in one paste. Brain repo renamed to gstack-artifacts.**
|
||||
|
||||
Reference in New Issue
Block a user