CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-05-02 03:35:09 +02:00
Code Issues Packages Projects Releases Wiki Activity
260 Commits 233 Branches 0 Tags
9ef34603dfc9e55cf351ea547c0e68a1dc19faa4
Commit Graph

8 Commits

Author SHA1 Message Date
Garry Tan 9ef34603df test(harness): require ## Decisions section under --disallowedTools plan_ready 
Adversarial review (during /ship Step 11) found that the previous gate-test
envelope ['asked', 'plan_ready'] for the AskUserQuestion-blocked regression
cases accepted the bug they exist to catch: a model that silently skips
Step 0 entirely (writes a plan with no questions, no `## Decisions to
confirm` section, just ExitPlanModes) reaches plan_ready and passes.

The fix tightens the contract in two layers:

1. Harness: PlanSkillObservation gains a `planFile?: string` field
   populated when outcome is plan_ready. extractPlanFilePath() walks the
   visible TTY buffer for "Plan saved to:", "Plan file:", or
   ".claude/plans/<name>.md" patterns and resolves tilde to absolute.
   planFileHasDecisionsSection() reads the resolved file and returns true
   if it contains a `## Decisions` heading (any form: "to confirm",
   "needed", etc.).

2. Tests: 5 of 6 regression cases now require, when outcome is plan_ready,
   that obs.planFile is set AND planFileHasDecisionsSection returns true.
   Otherwise the test fails with a "Step 0 was silently skipped" diagnosis.
   plan-design-review remains the sole exception — it legitimately
   short-circuits to plan_ready on no-UI-scope branches and we have no
   deterministic way to distinguish that from a silent skip.

This closes the loophole the adversarial review identified. The fix
preamble flow already tells the model to write `## Decisions to confirm`
when neither AUQ variant is callable — now the test verifies the model
actually did it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 07:10:37 -07:00
Garry Tan d3ce41b2a6 merge: resolve conflicts with origin/main, bump to v1.22.0.0 
Main shipped v1.21.0.0 (browser-skills) and v1.21.1.0 (plan-ceo smoke
tightening + classifyVisible extraction) while this branch was in flight.
Resolves all four merge conflicts:

- VERSION 1.22.0.0 (was claiming v1.21.0.0; queue advanced past v1.21.1.0;
  MINOR scale on top per scale-aware bumps)
- package.json synced to 1.22.0.0
- claude-pty-runner.ts: integrated 'auto_decided' outcome into main's new
  classifyVisible() pure function. Branch order is now silent_write →
  auto_decided → plan_ready → asked. Kept BOTH new options on the runner
  (`extraArgs` from this branch, `env` from main). isPlanReadyVisible and
  isAutoDecidedVisible remain whitespace-tolerant.
- touchfiles.test.ts: count of plan-ceo-review/SKILL.md selections updates
  to 21 (main added plan-ceo-finding-count; this branch added
  autoplan-auto-mode + auto-decide-preserved).
- CHANGELOG.md: dropped this branch's stale v1.21.0.0 draft, kept main's
  v1.21.1.0 entry intact, wrote v1.22.0.0 entry on top covering the
  AskUserQuestion-blocked regression fix.

plan-ceo-plan-mode test auto-merged: main's stricter baseline ('asked'
only) + this branch's AskUserQuestion-blocked regression case both
present and run sequentially. SKILL.md goldens regenerated via
`bun run gen:skill-docs --host all` and the 3 host-ship goldens copied
into test/fixtures/golden/.

Free test suite: 470/470 pass post-merge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 23:10:21 -07:00
Garry Tan 78e4b770fa test(harness): fix detection order + whitespace-tolerant pattern matching 
Two bugs surfaced when validating the v1.21 fix end-to-end:

1. PlanSkillObservation outcome detection ran 'asked' (any numbered
   options list) BEFORE 'plan_ready'. Plan-mode's "Ready to execute?"
   confirmation IS a numbered options list (1=auto, 2=manual, ...), so
   any skill that successfully reached the native confirmation got
   misclassified as 'asked'. Reorder: 'auto_decided' (most specific,
   requires AUTO_DECIDE annotation) > 'plan_ready' (next, requires the
   "ready to execute" stem) > 'asked' (any remaining numbered list).

2. isPlanReadyVisible and isAutoDecidedVisible regexes only matched
   spaced forms ("ready to execute", "(your preference)"). stripAnsi
   removes cursor-positioning escapes (`\x1b[40C`) entirely instead of
   replacing them with spaces, so the same text can render as
   "readytoexecute" or "(yourpreference)". Both detectors now test the
   spaced form first, fall through to a whitespace-collapsed comparison.
   Inline unit smoke confirms both forms match.

Updates to the 5 strict 'asked' regression test cases (plan-ceo,
plan-eng, plan-devex, autoplan, office-hours): with the detection order
corrected, the model's plan-file fallback flow legitimately lands at
'plan_ready' instead of 'asked'. Pass envelope expanded to ['asked',
'plan_ready'] (matching plan-design-review's existing pattern). Failure
signals tightened to include 'auto_decided' (catches AUTO_DECIDE without
opt-in) plus the standard silent_write/exited/timeout. plan-design was
already on this contract from v1.21's first commit, no change needed.

The expanded envelope is correct: under --disallowedTools AskUserQuestion
the Tool resolution preamble routes the question through plan-mode's
native "Ready to execute?" surface — the user still sees the decision,
just via the plan-file flow rather than a numbered prompt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 22:27:33 -07:00
Garry Tan aaf2668103 test(e2e): add AskUserQuestion-blocked regression cases for 6 plan-mode skills 
Conductor launches Claude Code with --disallowedTools AskUserQuestion
--permission-mode default --permission-prompt-tool stdio (verified by
inspecting the live conductor claude process via ps -p ... -o args=).
Native AskUserQuestion is removed from the model's tool registry; without
fallback guidance the plan-mode skills (plan-ceo-review, plan-eng-review,
plan-design-review, plan-devex-review, autoplan, office-hours) silently
proceed and never surface decisions to the user.

Adds 6 gate-tier real-PTY regression cases:

  - 4 inline test cases inside the existing plan-X-review-plan-mode.test
    files, each exercising the same skill with extraArgs ['--disallowedTools',
    'AskUserQuestion'] and asserting outcome === 'asked'. plan-design-review
    keeps the ['asked', 'plan_ready'] envelope (legitimate short-circuit on
    no-UI-scope) but explicitly fails on 'auto_decided'.
  - 2 standalone test files for autoplan + office-hours (which had no prior
    plan-mode test). autoplan asserts the FIRST non-auto-decided gate fires
    (Phase 1 premise confirmation) — autoplan auto-decides intermediate
    questions BY DESIGN.

Touchfile entries:
  - autoplan-auto-mode + office-hours-auto-mode added to E2E_TOUCHFILES +
    E2E_TIERS (gate)
  - existing plan-X-review-plan-mode entries gain question-tuning.ts and
    generate-ask-user-format.ts touchfile deps so AUTO_DECIDE-related
    resolver changes correctly invalidate the regression tests
  - touchfiles.test.ts count updated 18 -> 19 to cover the autoplan
    touchfile dependency on plan-ceo-review/**

Filenames retain `auto-mode` for branch-history continuity. Auto-mode (the
AUTO_DECIDE preamble path when QUESTION_TUNING=true) is a related but
distinct silencing mechanism; both share the same fix surface in the
preamble.

These tests are expected to FAIL on this branch until the fix lands. The
failure is the receipt for the regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 21:13:48 -07:00
Garry Tan 454423aeb3 v1.21.1.0 test: tighten plan-ceo-review smoke (Step 0 must fire) (#1255) 
* test: extract classifyVisible() + permission-dialog filter in PTY runner

Pure classifier extracted from runPlanSkillObservation's polling loop so
unit tests can exercise the actual branch order with synthetic input
strings. Runner gains:

- env? passthrough on runPlanSkillObservation (forwarded to launchClaudePty).
  gstack-config does not yet honor env overrides; plumbing is in place for a
  future change to make tests hermetic.
- TAIL_SCAN_BYTES = 1500 exported constant. Replaces a duplicated magic
  number in test/skill-e2e-plan-ceo-mode-routing.test.ts so tuning stays
  in sync.
- isPermissionDialogVisible: the bare phrase "Do you want to proceed?" now
  requires a file-edit context co-trigger. Other clauses unchanged. Skill
  questions that contain the bare phrase are no longer mis-classified.
- classifyVisible(visible): pure function. Branch order silent_write →
  plan_ready → asked → null. Permission dialogs filtered out of the
  'asked' classification so a permission prompt cannot pose as a Step 0
  skill question.

Adds 24 unit tests covering all classifier branches, edge cases, and the
co-trigger contract.

* test: tighten plan-ceo-review smoke to require Step 0 fires first

Assertion narrows from ['asked', 'plan_ready'] to 'asked' only. Reaching
plan_ready first means the agent skipped Step 0 entirely and went
straight to ExitPlanMode — the regression we want to catch.

Why plan-ceo is special: unlike plan-eng / plan-design / plan-devex
(whose smokes legitimately reach plan_ready on certain branches without
asking), plan-ceo-review's template mandates Step 0A premise challenge
plus Step 0F mode selection BEFORE any plan write. There is no
legitimate path to plan_ready that does not first emit a skill-question
numbered prompt.

Failure message now branches on outcome (plan_ready vs timeout vs
silent_write) with a tailored diagnosis line per case. References the
skill template by section name ("Step 0 STOP rules", "One issue = one
AskUserQuestion call") instead of line numbers, so it survives template
edits.

Passes env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' }
through the runner. Today this is advisory — gstack-config reads only
~/.gstack/config.yaml, not env vars — but the wiring is in place for a
future change. Documented honestly in the docstring.

Verified across 4 PTY runs: 3 pre-refactor + 1 post-refactor, all PASS.

* chore: capture v1.21.1.0 follow-ups in TODOS.md

- P2: per-finding AskUserQuestion count assertion (V2)
- P3: honor env vars in gstack-config so test isolation env actually works
- P3: path-confusion hardening on SANCTIONED_WRITE_SUBSTRINGS

All three surfaced during the v1.21.1.0 plan-eng-review and adversarial
review passes. Captured here so the design intent persists.

* chore: bump version and changelog (v1.21.1.0)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: extract MODE_RE + optionsSignature into PTY runner exports

Refactor prep for the upcoming per-finding AskUserQuestion count test
across plan-{ceo,eng,design,devex}-review. Both new tests and the existing
mode-routing test need the same mode regex and the same option-list
fingerprint dedupe — pulling them into one source of truth in
test/helpers/claude-pty-runner.ts so a fifth mode (or a tweak to the
fingerprint shape) updates everywhere instead of drifting per-test.

Mechanical: no behavior change in the mode-routing test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: add per-finding count primitives + unit tests

Pure helpers landing ahead of runPlanSkillCounting:

  - parseQuestionPrompt(visible) — extract the 1-3 line prompt above
    the latest "❯ 1." cursor, normalize to a 240-char snippet
  - auqFingerprint(prompt, opts) — Bun.hash of normalized prompt + sorted
    options signature; distinct prompts with shared option labels
    (the generic A/B/C TODO menu) get distinct fingerprints
  - COMPLETION_SUMMARY_RE — terminal-signal regex matching all four
    plan-review skills' completion / verdict markers
  - assertReviewReportAtBottom(content) — checks "## GSTACK REVIEW
    REPORT" is present and is the last "## " heading in a plan file
  - Step0BoundaryPredicate type + four per-skill predicates
    (ceo / eng / design / devex) — fire on the answered AUQ's
    fingerprint, marking the end of Step 0 deterministically
    (event-based, not content-based, per Codex F7)

Plus 37 deterministic unit tests covering option-label collision
regression, prompt extraction edge cases, predicate positive AND
negative cases, and review-report-at-bottom triple-check
(missing / mid-file / multiple trailing).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: add runPlanSkillCounting PTY helper

Drives a plan-* skill end-to-end and counts distinct review-phase
AskUserQuestions. Composes the primitives from the previous commit:

  - Boot + auto-trust handler (existing launchClaudePty)
  - Send slash command alone, sleep 3s, send plan content as follow-up
    message (proven pattern from skill-e2e-plan-design-with-ui)
  - Poll loop with permission-dialog auto-grant, same-redraw skip,
    empty-prompt re-poll
  - Event-based Step-0 boundary via isLastStep0AUQ predicate fired on
    the answered AUQ's fingerprint (Codex F7 — boundary is observed
    event, not later rendered content)
  - Multi-signal terminals: hard ceiling, COMPLETION_SUMMARY_RE,
    plan_ready, silent_write, exited, timeout

Empty-prompt fingerprints are skipped per the contract documented in
auqFingerprint's unit tests — fingerprinting them would re-introduce
the option-label collision regression Codex F1 caught.

No E2E tests yet — those land in commit 5 with the four skill fixtures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: register four finding-count tests in touchfiles + tier map

Each new test depends on its skill template, the runner, and three
preamble resolvers (preamble.ts, generate-ask-user-format.ts,
generate-completion-status.ts) — those affect question cadence and
completion rendering, which is exactly what the test asserts on.

All four classified periodic. Sequential execution during calibration;
opt-in to concurrent only after measured comparison agrees (plan §D15).

Updated touchfiles.test.ts: plan-ceo-review/** now selects 19 tests
(was 18) because plan-ceo-finding-count joins the family.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: add four per-finding count E2E tests (plan-ceo + eng + design + devex)

Each test drives its plan-* skill through Step 0 then asserts the
review-phase AskUserQuestion count falls in [N-1, N+2] for an N=5
seeded plan, plus D19: produced plan file ends with
"## GSTACK REVIEW REPORT" as its last "## " heading.

plan-ceo also runs a paired-finding positive control: 2 deliberately
related findings should still produce 2 distinct AUQs, not 1 batched.

Periodic-tier (gate-skipped without EVALS=1, EVALS_TIER=periodic).
Sequential execution by plan §D15. Each fixture is inline TypeScript
content delivered as a follow-up message after the slash command, per
the proven pattern at skill-e2e-plan-design-with-ui.test.ts.

Calibration loop (5 runs per skill) and the manual pre-merge negative
check (D7 + D12) are required before merge per plan §Verification.
NOT yet run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: fix parseNumberedOptions for inline-cursor box-layout AUQs

Calibration run 1 timed out with step0=0 review=0 because the parser
could not find the cursor in /plan-ceo-review's scope-selection AUQ.
The TTY's box-layout rendering inlines divider + header + prompt +
"1." onto one logical line — cursor escapes get stripped, leaving
text crushed onto a single line.

Cursor anchor regex changed from anchored to unanchored so it matches
mid-line. Cursor-line option extraction uses a non-anchored regex;
subsequent options stay with the original start-of-line parser.

parseQuestionPrompt picks up the inline prompt text BEFORE the cursor
on the cursor line (after stripping box-drawing chars + sigil) and
appends it after any walked-up multi-line prompt above.

Three new unit tests: clean-cursor still works, inline-cursor
extracts all 7 options, prompt extraction strips box chars.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: add firstAUQPick + plan-ceo skip-interview routing

Calibration run 1 surfaced a second issue beyond the parser bug: the
default pick of 1 on /plan-ceo-review's scope-selection AUQ routes
the agent to "branch diff vs main" — so it reviews the gstack PR
itself (recursive!) instead of the seeded fixture plan we sent.

Added firstAUQPick callback to runPlanSkillCounting. Override applies
only to the FIRST AUQ; subsequent presses keep using defaultPick.

ceoStep0Boundary now fires on either the mode-pick AUQ (existing path)
or any AUQ containing "Skip interview and plan immediately" — which
is the scope-selection AUQ. Picking that option bypasses Step 0 and
routes straight to review-phase using the chat-paste plan as context.

Plan-ceo test wires firstAUQPick = pickSkipInterview which finds the
"Skip interview" option by label. Falls back to "describe inline" if
the option labels change.

Two new unit tests: ceoStep0Boundary fires on the scope-selection
fixture; existing mode-pick fixture still fires.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 02:50:09 -07:00
Garry Tan dde55103fc v1.15.0.0 feat: slim preamble + real-PTY plan-mode E2E harness (#1215) 
* chore: add gstack skill routing rules to CLAUDE.md

Per routing-injection preamble — once-per-project addition that lets
agents auto-invoke the right gstack skill instead of answering generically.

* refactor: slim preamble resolvers + sidecar-symlink helper

Compress prose across 18 preamble resolvers — Voice, Writing Style,
AskUserQuestion Format, Completeness Principle, Confusion Protocol,
Context Health, Context Recovery, Continuous Checkpoint, Lake Intro,
Proactive Prompt, Routing Injection, Telemetry Prompt, Upgrade Check,
Vendoring Deprecation, Writing Style Migration, Brain Sync Block,
Completion Status, and Question Tuning. Same semantic contract, ~half
the bytes. Restored "Treat the skill file as executable instructions"
phrase in the plan-mode info section after diagnosing it as load-bearing.
Restored "Effort both-scales" rule in AskUserQuestion format.

Bonus: scripts/skill-check.ts gains isRepoRootSymlink() so dev installs
that mount the repo root at host/skills/gstack as a runtime sidecar
(e.g., codex's .agents/skills/gstack) get skipped instead of double-counted.

opus-4-7 model overlay gets a Fan-Out directive — explicit instruction
to launch parallel reads/checks before synthesis.

Net token impact across all generated SKILL.md files: ~140K tokens
removed across 47 outputs. Plan-* skills retain full preamble surface
(Brain Sync, Context Recovery, Routing Injection) — load-bearing
functionality that early slim attempts incorrectly cut.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate SKILL.md outputs after preamble slim

bun run gen:skill-docs --host all output. Mirrors the resolver changes
in the previous commit. 47 generated SKILL.md files plus 3 ship-skill
golden fixtures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): real-PTY harness for plan-mode E2E tests

Adds test/helpers/claude-pty-runner.ts. Spawns the actual claude binary
via Bun.spawn({terminal:}) (Bun 1.3.10+ has built-in PTY — no node-pty,
no native modules), drives it through stdin/stdout, and parses rendered
terminal frames. Pattern adapted from the cc-pty-import branch's
terminal-agent.ts but stripped of WS/cookie/Origin scaffolding (not
needed for headless tests).

Public API:
- launchClaudePty(opts) — boots claude with --permission-mode plan|null,
  auto-handles the workspace-trust dialog, returns a session handle.
- session.send / sendKey / waitForAny / waitFor / mark / visibleSince /
  visibleText / rawOutput / close
- runPlanSkillObservation({skillName, inPlanMode, timeoutMs}) — high-level
  contract for plan-mode skill tests. Returns { outcome, summary, evidence,
  elapsedMs }. outcome ∈ {asked, plan_ready, silent_write, exited, timeout}.

Replaces the SDK-based runPlanModeSkillTest from plan-mode-helpers.ts
which never worked. Plan mode renders its native "Ready to execute"
confirmation as TTY UI (numbered options with ❯ cursor), not via the
AskUserQuestion tool — so the SDK's canUseTool interceptor never fired
and the assertion always saw zero questions. Real PTY observes the
rendered output directly.

Deletes test/helpers/plan-mode-helpers.ts. No production callers remained.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: rewrite 5 plan-mode E2E tests on the real-PTY harness

Replaces SDK-based assertions with runPlanSkillObservation contract. Each
test launches real claude --permission-mode plan, invokes the skill, and
asserts the outcome reaches 'asked' or 'plan_ready' within a 300s budget
(no silent Write/Edit, no crash, no timeout).

Affected:
- test/skill-e2e-plan-ceo-plan-mode.test.ts
- test/skill-e2e-plan-eng-plan-mode.test.ts
- test/skill-e2e-plan-design-plan-mode.test.ts
- test/skill-e2e-plan-devex-plan-mode.test.ts
- test/skill-e2e-plan-mode-no-op.test.ts (inPlanMode: false; tests the
  preamble plan-mode-info no-op path)

test/e2e-harness-audit.test.ts — recognize runPlanSkillObservation as a
valid coverage path alongside the legacy canUseTool / runPlanModeSkillTest.

test/helpers/touchfiles.ts — point the 5 plan-mode test selections and
the e2e-harness-audit selection at test/helpers/claude-pty-runner.ts
instead of the deleted plan-mode-helpers.ts.

Proof: bun test EVALS=1 EVALS_TIER=gate on these 5 files runs sequentially
in 790s and passes 5/5. Same tests were 0/5 on origin/main, on v1.0.0.0,
and on this branch with the SDK harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: align unit tests with slim resolvers + exempt 27MB security fixture

- test/skill-validation.test.ts: assert the slim Completeness Principle
  shape (Completeness: X/10, kind-note language) instead of the old
  Compression table. Remove the 3 tier-1 skills from the spot-check list
  (they intentionally don't carry the full Completeness Principle
  section). Exempt browse/test/fixtures/security-bench-haiku-responses.json
  (27MB deterministic replay fixture for BrowseSafe-Bench) from the 2MB
  tracked-file gate. The gate was actually failing on origin/main since
  the fixture was added in v1.6.4.0 — this is a side-fix to a real
  regression.

- test/brain-sync.test.ts: developer-machine-safe assertion for
  GSTACK_HOME override (compare config contents before/after instead of
  asserting the absence of a string that may legitimately exist).

- test/gen-skill-docs.test.ts: new tests for the slim — plan-review
  preambles stay under the post-slim budget (~33KB), Voice + Writing
  Style sections stay compact, and the slim Voice section preserves the
  load-bearing semantic contract (lead-with-the-point, name-the-file,
  user-outcome framing, no-corporate, no-AI-vocab, user-sovereignty).
  Update path-leakage scan to allow repo-root sidecar symlinks.

- test/writing-style-resolver.test.ts: assert the compact contract
  (gloss-on-first-use, outcome-framing, user-impact, terse-mode override)
  instead of the old 6-numbered-rules shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v1.13.1.0)

Slim preamble work + real-PTY plan-mode E2E harness on top of v1.13.0.0.
SKILL.md corpus -25.5% (3.08 MB → 2.30 MB, ~196K tokens). 5 plan-mode
tests go from 0/5 to 5/5 (790s sequential), the first time those tests
have ever passed. Side-fixes for the 27MB security fixture warning and
the sidecar-symlink double-count.

Reverts the Fan-Out directive accidentally restored to opus-4-7.md —
v1.10.1.0's overlay-efficacy harness measured -60pp fanout vs baseline
when the nudge was active. The intentional removal stays.

TODOS:
- Pre-existing test failures from v1.12.0.0 ship: RESOLVED on main + this branch
- security-bench-haiku-responses.json size gate: RESOLVED via warn-only + exemption

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): harness primitives — parseNumberedOptions + budget regression utils

claude-pty-runner.ts:
- parseNumberedOptions(visible) anchors on the latest "❯ 1." cursor and
  returns {index, label}[]; tests that route on option labels can find
  indices without hard-coding positions
- isPermissionDialogVisible(visible) detects file-grant + workspace-trust
  + bash-permission shapes (multiple regex variants)
- isNumberedOptionListVisible: replaced \b2\. word-boundary regex with
  [^0-9]2\. — stripAnsi removes TTY cursor-positioning escapes that
  collapse "Option 2." to "Option2.", and \b fails on word-to-word

eval-store.ts:
- findBudgetRegressions(comparison, opts?) — pure function returning
  tests where tools or turns grew >cap× vs prior run; floors at 5 prior
  tools / 3 prior turns to avoid noise on tiny numbers
- assertNoBudgetRegression() — wrapper that throws with full violation
  list. Env override GSTACK_BUDGET_RATIO

helpers-unit.test.ts: 23 unit tests covering empty/sparse/wrap-around
buffers for parseNumberedOptions, plus regression-floor + env-override
cases for findBudgetRegressions/assertNoBudgetRegression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: register 6 real-PTY E2E touchfiles + UI-heavy plan fixture

touchfiles.ts:
- 6 new entries in E2E_TOUCHFILES keyed to the new test files
- 6 matching E2E_TIERS classifications: 3 gate (auq-format-pty,
  plan-design-with-ui-scope, budget-regression-pty), 3 periodic
  (plan-ceo-mode-routing, ship-idempotency-pty, autoplan-chain-pty)
- gate ones are cheap/deterministic; periodic ones run weekly

touchfiles.test.ts:
- update the "skill-specific change selects only that skill" count
  from 15 → 18 (plan-ceo-review/SKILL.md change now also selects
  auq-format-pty, plan-ceo-mode-routing, autoplan-chain-pty)

test/fixtures/plans/ui-heavy-feature.md:
- planted plan with explicit UI scope keywords (pages, components,
  Tailwind responsive layout, hover/loading/empty states, modal,
  toast). Used by plan-design-with-ui-scope and autoplan-chain tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): 3 gate-tier real-PTY E2E tests

skill-e2e-auq-format-compliance.test.ts (~$0.50/run, 90-130s):
- Asserts /plan-ceo-review's first AUQ contains all 7 mandated format
  elements (ELI10, Recommendation, Pros/Cons with ✅/❌, Net,
  (recommended) label). Catches drift in the shared preamble resolver
  that previously took weeks to notice.
- Auto-grants permission dialogs that fire during preamble side-effects
  (touch on .feature-prompted markers in fresh user environments).
- Verified PASS in 126s.

skill-e2e-plan-design-with-ui.test.ts (~$0.80/run, 50-90s):
- Counterpart to the existing no-UI early-exit test. When the input plan
  DOES describe UI changes, /plan-design-review must NOT early-exit and
  must reach a real skill AUQ.
- Sends the slash command without args, then a follow-up message with
  the UI-heavy plan description (Claude Code rejects unknown trailing
  args). Asserts evidence does NOT contain "no UI scope".
- Verified PASS in 54s.

skill-budget-regression.test.ts (free, gate):
- Library-only assertion. Reads the most recent eval file, finds the
  prior same-branch run via findPreviousRun, computes ComparisonResult,
  asserts no test exceeded 2× tools or turns.
- Branch-scoped: skips with reason if the latest eval was produced on
  a different branch (cross-branch comparison would be noise).
- First-run grace (vacuous pass) when no prior data exists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): 3 periodic-tier real-PTY E2E tests

skill-e2e-plan-ceo-mode-routing.test.ts (~$3/run, 6-10 min/case):
- Verifies AUQ answer routing: HOLD SCOPE → rigor/bulletproof posture
  language; SCOPE EXPANSION → expansion/10x/dream language. Each case
  navigates 8-12 prior AUQs (telemetry, proactive, routing, vendoring,
  brain, office-hours, premise, approach) before hitting Step 0F.
- Periodic, not gate: navigation phase too slow for PR-blocking.
  V2 expansion to 4 modes (SELECTIVE + REDUCTION) when nav is faster.

skill-e2e-ship-idempotency.test.ts (~$3/run, 5-10 min):
- Builds a real git fixture with VERSION 0.0.2 already bumped, matching
  package.json, CHANGELOG entry, pushed to a local bare remote. Runs
  /ship in plan mode and asserts STATE: ALREADY_BUMPED echoes from the
  Step 12 idempotency check, OR plan_ready terminates without mutation.
- Snapshots VERSION + package.json + CHANGELOG entry count + commit
  count + branch HEAD before/after; fails if any changed.

skill-e2e-autoplan-chain.test.ts (~$8/run, 12-18 min):
- Asserts /autoplan phases run sequentially: tees timestamps as each
  "**Phase N complete.**" marker first appears. Phase 1 (CEO) must
  precede Phase 3 (Eng); Phase 2 (Design) is optional but if it
  appears, must sit between 1 and 3.
- Auto-grants permission dialogs that fire during phase transitions.

All three auto-handle permission dialogs (preamble side-effects on
fresh user envs without .feature-prompted-* markers).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: spell out AskUserQuestion everywhere instead of AUQ

Per user feedback: don't shorten AskUserQuestion to AUQ — the
abbreviation reads as cryptic. Apply across all the new code from this
branch:

- Rename test/skill-e2e-auq-format-compliance.test.ts →
  test/skill-e2e-ask-user-question-format-compliance.test.ts
- Touchfile entry auq-format-pty → ask-user-question-format-pty
  (touchfiles.ts + matching assertion in touchfiles.test.ts)
- Function rename navigateToModeAuq → navigateToModeAskUserQuestion
- Variable auqVisible → askUserQuestionVisible
- Outcome literal 'real_auq' → 'real_question'
- All comments + JSDoc + CHANGELOG entry write AskUserQuestion in full
- "AUQs" plural → "AskUserQuestions"

No behavior change. 49/49 free tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: harden v1.15.0.0 CHANGELOG entry against hostile readers

Per Garry: write the entry assuming a critic will screencap one line
and try to use it as ammunition.

Reframed the v1.15.0.0 release-summary to lead with new capability
(real-PTY harness, 11 plan-mode tests, +6 new) instead of fix-of-prior-
flaw narrative. Removed phrases that critics could weaponize:

- "0/5 → 5/5 passing", "finally pass", "∞ (never green)" — drop
- "Skill prompts get a 25% haircut" — implied self-inflicted bloat
- "770K → 574K tokens" — absolute number lets critics quote "still 574K
  of bloat"; replaced with relative "−196K tokens per invocation"
- "5 plan-mode E2E tests turned out to have never actually passed" —
  literal admission of long-term breakage; cut entirely
- Itemized "Fixed: tests finally pass" entry — moved to Changed with
  neutral "rewritten on the new harness" framing
- "Removed: harness with the runPlanModeSkillTest API that never
  worked" — replaced with "superseded by claude-pty-runner.ts"

Added concrete code receipts to pre-empt "it's just markdown":

- Net branch size: −11,609 lines (89 files, +7,240 / −18,849)
- 654 lines of TypeScript in test/helpers/claude-pty-runner.ts
- 8 new test files, ~1,453 lines of new TS code
- 23 helper unit tests + 6 new gate/periodic E2E tests

The deletion-heavy net diff (−11.6K lines) is itself the strongest
defense against the "bloat" critique — surfaced explicitly in the
numbers table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 13:55:13 -07:00
Garry Tan aeea57f96a v1.12.1.0 fix: remove vestigial plan-mode handshake (#1185) 
* refactor: remove vestigial plan-mode handshake resolver

Delete scripts/resolvers/preamble/generate-plan-mode-handshake.ts and
its four question-registry entries. Split the authoritative
"Plan Mode Safe Operations" and "Skill Invocation During Plan Mode"
sections out of generate-completion-status.ts into a sibling
generatePlanModeInfo() export in the same module, wired at preamble
position 1 where the handshake used to live. Same text, new position.

The vestigial handshake told interactive review skills to emit an
A=exit-and-rerun / C=cancel AskUserQuestion before running their
interactive STOP-Ask workflow. That contradicted the authoritative
rule at the tail of completion-status.ts saying AskUserQuestion
satisfies plan mode's end-of-turn requirement. Skills now run
directly when invoked in plan mode, with each finding gated by
AskUserQuestion just like outside plan mode.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: rename plan-mode-handshake-helpers to plan-mode-helpers, strengthen smokes

Rename test/helpers/plan-mode-handshake-helpers.ts to
test/helpers/plan-mode-helpers.ts. Keep the write-guard helper that
asserts no Write/Edit tool call before the first AskUserQuestion
(this is what catches silent-bypass regressions the textual smoke
can't see). Rename the API: runPlanModeHandshakeTest to
runPlanModeSkillTest, assertHandshakeShape to assertNotHandshakeShape.
Extend the capture struct with exitPlanModeBeforeAsk.

Rewrite the four per-skill E2E tests (plan-ceo, plan-eng, plan-design,
plan-devex) as smoke tests that assert the skill's Step 0 question
fires first, not an A/C handshake. Each test picks a cheap first
answer (HOLD, TRIAGE, numeric score) so the run terminates quickly.

Keep test/skill-e2e-plan-mode-no-op.test.ts as the outside-plan-mode
non-interference regression, per codex outside-voice review: deleting
it would lose coverage for "the hoisted section stays quiet when plan
mode is absent."

Replace the gen-skill-docs.test.ts handshake describe block (lines
2778+) with a plan-mode-info describe block that:
- scans every generated SKILL.md under the repo root + every host
  subdir (.agents, .openclaw, .opencode, .factory, .hermes, .kiro,
  .cursor, .slate) and asserts "## Plan Mode Handshake" is absent
- asserts "## Skill Invocation During Plan Mode" lands in the first
  15KB of each of the four review skills' generated SKILL.md

Both assertions run on every bun test. A PR that re-introduces the
handshake resolver fails CI immediately.

Update test/e2e-harness-audit.test.ts to reference the renamed
runPlanModeSkillTest. Update test/helpers/touchfiles.ts entries to
point at the new resolver owner (generate-completion-status.ts) and
the renamed helper, and align per-skill touchfile keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate SKILL.md across all hosts + refresh golden fixtures

Run bun run gen:skill-docs for every host to flush the vestigial
"## Plan Mode Handshake" section from every generated SKILL.md and
emit the hoisted "## Skill Invocation During Plan Mode" section at
preamble position 1 instead. Refresh the three golden-fixture
snapshots (claude, codex, factory) to match the new position.

No behavior change beyond the resolver swap in the prior commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v1.12.1.0)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-24 02:11:24 -07:00
Garry Tan 9e244c0bed v1.11.1.0 fix: plan-mode handshake + canUseTool test harness (#1182) 
* feat: plan-mode handshake for interactive review skills

Add a preamble-level STOP-Ask handshake that fires when the user invokes any
of the 4 interactive review skills (plan-ceo-review, plan-eng-review,
plan-design-review, plan-devex-review) while their Claude Code session is
in plan mode. Without this gate, plan mode's "this supercedes any other
instructions" system-reminder outranked the skills' interactive STOP gates
and the skills silently wrote plan files without any per-finding AskUserQuestion.

The handshake offers 2 options (exit-and-rerun, cancel) — the original
third "stay and batch" option was dropped after two independent reviewers
flagged it as a silent bypass of the skills' anti-skip rule.

Architecture decisions (CEO+Eng review):
- Preamble-level resolver, not per-template injection (Codex finding #2)
- Position 1 in preamble composition: after bash block (_SESSION_ID live),
  before onboarding AskUserQuestion gates (so fresh-install users see the
  handshake first, not drowned in telemetry/proactive/routing prompts)
- Generator-only `interactive: true` frontmatter flag, following the
  `preamble-tier` precedent (no host-config frontmatter allowlist edits)
- Host-scoped to Claude via `ctx.host === 'claude'` check inside the
  resolver (simpler than `suppressedResolvers` which only gates `{{}}`
  placeholders)
- One-way-door classification in scripts/question-registry.ts for all 4
  skills so question-tuning `never-ask` preferences can't suppress the gate
- Synchronous telemetry write to ~/.gstack/analytics/skill-usage.jsonl on
  handshake fire (captures A-exit and C-cancel outcomes that terminate the
  skill before end-of-run telemetry runs)

Also adds an explicit STOP block to plan-ceo-review Step 0C-bis so the
approach-selection question can't silently skip to mode selection.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat: extend agent-sdk-runner with canUseTool for AskUserQuestion interception

Test harness at test/helpers/agent-sdk-runner.ts gains an optional
`canUseTool` callback parameter. When a test supplies it, the harness
flips `permissionMode` from `bypassPermissions` (overlay-harness default)
to `default` so the SDK actually invokes the callback on every tool use,
and auto-adds `AskUserQuestion` to `allowedTools` so Claude can fire it
at all.

Exports a `passThroughNonAskUserQuestion` helper so tests that only want
to intercept AskUserQuestion can auto-allow every other tool with one
line: `return passThroughNonAskUserQuestion(toolName, input)`.

This is the foundation for D14 — every future interactive-skill E2E test
can now assert on AskUserQuestion shape and routing. Previous E2E tests
at `test/skill-e2e.test.ts` explicitly instructed the model to skip
AskUserQuestion ("non-interactive run") which meant no test could actually
verify the question content or routing.

6 new unit tests in test/agent-sdk-runner.test.ts cover:
- permissionMode flips to 'default' when canUseTool supplied
- permissionMode stays 'bypassPermissions' when canUseTool absent
- canUseTool callback reaches the SDK options
- AskUserQuestion auto-added to allowedTools when canUseTool supplied
- AskUserQuestion NOT added when canUseTool absent
- passThroughNonAskUserQuestion helper returns allow+updatedInput

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test: plan-mode handshake E2E coverage and unit assertions

Adds 6 E2E test files and 8 new unit assertions to verify the plan-mode
handshake works end-to-end and stays correct under regeneration.

E2E tests (gate-tier, paid, EVALS=1 EVALS_TIER=gate):
- test/skill-e2e-plan-ceo-plan-mode.test.ts — handshake fires before any
  Write/Edit when plan-mode distinctive phrase is present; 2-option shape
  (Exit/Cancel); option A routes to ExitPlanMode cleanly
- test/skill-e2e-plan-eng-plan-mode.test.ts — same contract for plan-eng
- test/skill-e2e-plan-design-plan-mode.test.ts — same contract for
  plan-design; exercises C-cancel branch instead of A-exit
- test/skill-e2e-plan-devex-plan-mode.test.ts — same contract for plan-devex
- test/skill-e2e-plan-mode-no-op.test.ts — negative regression: handshake
  must NOT fire when distinctive phrase is absent; skill proceeds normally
  through Step 0 (REGRESSION RULE guardrail against breaking existing
  interactive-review sessions)
- test/e2e-harness-audit.test.ts — free unit test asserting every
  `interactive: true` skill has at least one canUseTool-using test file
  (prevents future drift where a skill opts in without coverage)

Shared helper test/helpers/plan-mode-handshake-helpers.ts centralizes the
canUseTool interceptor + distinctive-phrase injection so the 4 sibling
E2E tests are thin wiring (~20 LOC each) and can't drift out of sync.

Unit assertions added to test/gen-skill-docs.test.ts:
- handshake section present in all 4 Claude-generated SKILL.md files
- handshake section absent from non-interactive Claude skills (ship,
  review, qa, office-hours, codex, retro, cso)
- handshake section absent from non-Claude host outputs (.agents, etc.)
- 0C-bis STOP block present in plan-ceo-review/SKILL.md at correct
  position (between the "Present these approach options" line and
  "### 0D-prelude" header)
- handshake resolver wired BEFORE generateUpgradeCheck in preamble
  composition order

6 new gate-tier entries added to test/helpers/touchfiles.ts so any change
to the handshake resolver, preamble composition, skill templates, question
registry, one-way-door classifier, or agent-sdk-runner fires the relevant
E2E tests. test/touchfiles.test.ts updated for the new selection count
(plan-ceo-review/** now triggers 15 tests, up from 8).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(v1.11.1.0): VERSION bump + CHANGELOG entry + TODOS follow-ups

Bumps from main's v1.11.0.0 to v1.11.1.0 (PATCH — bug-fix release, no new
user-facing artifacts). CHANGELOG entry covers the plan-mode handshake,
agent-sdk-runner canUseTool extension, and the 2 follow-up TODOs.

CHANGELOG order: v1.11.1.0 (this) → v1.11.0.0 (workspace-aware ship,
merged from main) → v1.10.1.0 (overlay efficacy harness). No duplicate
headers.

Syncs package.json version to match VERSION per the Step 12 idempotency
invariant (both files must agree or /ship halts).

TODOS.md:
- Preserves the Testing/security-bench-haiku-responses P1 added on main
- Adds P1 "Structural STOP-Ask forcing function" — broader class of the
  bug this release fixes
- Adds P2 "Apply interactive: true to non-review skills (office-hours,
  codex, investigate, qa, retro, cso)"

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-24 00:04:53 -07:00