test: apply ship review-army findings — helper extract, slice SKILL.md, defensive judge

Five categories of fixes surfaced by the /ship pre-landing reviews
(testing + maintainability + security + performance + adversarial Claude),
applied as one review-iteration commit.

Refactor — collapse 5x duplicated judge-assertion block:
- Add assertRecommendationQuality() + RECOMMENDATION_SUBSTANCE_THRESHOLD
  constant to test/helpers/e2e-helpers.ts.
- Plan-format (4 cases) and Phase 4 (1 case) collapse from ~22 lines each
  to a single helper call. Future rubric tweaks land in one place instead
  of five.

Performance — extract Phase 4 slice instead of copying full SKILL.md:
- Phase 4 test fixture now reads office-hours/SKILL.md and writes only the
  AskUserQuestion Format section + Phase 4 section to the tmpdir, per
  CLAUDE.md "extract, don't copy" rule. Verified locally: cost dropped
  from $0.51 → $0.36/run, turn count 8 → 4, latency 50s → 36s. Reduces
  Opus context bloat without weakening the regression check.
- Add `if (!workDir) return` guard to Phase 4 afterAll cleanup so a
  skipped describe block doesn't silently fs.rmSync(undefined) under the
  empty catch.

Defense — judge prompt + output:
- Wrap captured AskUserQuestion text in clearly delimited UNTRUSTED_CONTEXT
  block with explicit instruction to treat its content as data, not commands.
  Cheap defense against the (unlikely but real) injection vector where a
  captured AskUserQuestion contains "Ignore previous instructions" text.
- Bump captured-text budget from 4000 → 8000 chars; real plan-format menus
  with 4 options × ~800 chars exceed 4000 and were silently truncating
  Haiku context mid-option.

Cleanup — abbreviation rule + dead imports + touchfile consistency:
- AUQ → AskUserQuestion in 3 sites (office-hours/SKILL.md.tmpl Phase 4
  footer, two test comments) per the always-write-in-full memory rule.
  Regenerated office-hours/SKILL.md.
- Drop unused `describe`/`test` imports in 2 new test files (only
  describeIfSelected/testConcurrentIfSelected wrappers are used).
- Add `test/skill-e2e-office-hours-phase4.test.ts` to its own touchfile
  entry for consistency with other entries that include their test file.
- Fix misleading comment in fixture test about LLM short-circuiting (it's
  has_because, not commits, that skips the API call).

Verified: build clean, free `bun test` exits 0, fixture test 30/30
expect() calls pass, Phase 4 paid eval passes substance 5 in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-01 18:40:01 -07:00
co-authored by Claude Opus 4.7
parent 7658179879
commit 9a424a9f55
8 changed files with 133 additions and 105 deletions
+1 -1
View File
@@ -1250,7 +1250,7 @@ Rules:
**RECOMMENDATION:** Choose [X] because [one-line reason mapped to the founder's stated goal].
Emit ONE AskUserQuestion that lists every alternative (A/B and optionally C) as numbered options, using the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — write the question text and call the tool. If no AUQ variant is callable in this session (per the preamble's "Tool resolution" rules), fall back to writing `## Decisions to confirm` into the plan file and ExitPlanMode. Never silently auto-decide.
Emit ONE AskUserQuestion that lists every alternative (A/B and optionally C) as numbered options, using the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — write the question text and call the tool. If no AskUserQuestion variant is callable in this session (per the preamble's "Tool resolution" rules), fall back to writing `## Decisions to confirm` into the plan file and ExitPlanMode. Never silently auto-decide.
**STOP.** Do NOT proceed to Phase 4.5 (Founder Signal Synthesis), Phase 5 (Design Doc), Phase 6 (Closing), or any design-doc generation until the user responds. A "clearly winning approach" is still an approach decision and still needs explicit user approval before it lands in the design doc. Writing the recommendation in chat prose and continuing forward is the failure mode this gate exists to prevent.
+1 -1
View File
@@ -413,7 +413,7 @@ Rules:
**RECOMMENDATION:** Choose [X] because [one-line reason mapped to the founder's stated goal].
Emit ONE AskUserQuestion that lists every alternative (A/B and optionally C) as numbered options, using the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — write the question text and call the tool. If no AUQ variant is callable in this session (per the preamble's "Tool resolution" rules), fall back to writing `## Decisions to confirm` into the plan file and ExitPlanMode. Never silently auto-decide.
Emit ONE AskUserQuestion that lists every alternative (A/B and optionally C) as numbered options, using the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — write the question text and call the tool. If no AskUserQuestion variant is callable in this session (per the preamble's "Tool resolution" rules), fall back to writing `## Decisions to confirm` into the plan file and ExitPlanMode. Never silently auto-decide.
**STOP.** Do NOT proceed to Phase 4.5 (Founder Signal Synthesis), Phase 5 (Design Doc), Phase 6 (Closing), or any design-doc generation until the user responds. A "clearly winning approach" is still an approach decision and still needs explicit user approval before it lands in the design doc. Writing the recommendation in chat prose and continuing forward is the failure mode this gate exists to prevent.