The prompt was pinned three ways (sha256, ~20 sentences, an order check).
Kept: the skill and lazy-section paths, the plan path, the ordered
read -> review -> edit -> Write -> read-back -> summary steps, and the
no-browsing rule (case-insensitive). Dropped the hash and sentence pins.
Negative control: removing the read-back step turns it red.
reviewRevalidationPrompt and reviewPrompt are test-owned harness prompts
(non-test callers: 0); the paid shared-libs E2Es measure whether they work.
Kept: the three exact receipt commands, base-prompt prefix and resumed suffix
identity, branch selection (actor / default / resumed coverage), the state
path and turn-budget interpolations, read-record-before-diff and
first-response-before-start ordering, no embedded token, no supplied Skip
answer, reusable:true before reusable:false, the captured R59 stage boundaries
in the exclusion list, asset roots and interfaces, and the stale step-6
reference ban. The review template's Step 4.7/4.8 completion rule is matched
by step number on one line, not by sentence. Negative control: reordering the
diff-read instruction turns the test red.
docsNativeInterface is the test helper's own prompt, so its sentence pins
only checked that the helper says what it says. Kept: owned skill path, the
two extracted lifecycle commands (exact), the scripts-list interpolation, the
repo path and Git read examples, and every docsCommandAllowed /
docsToolFailures behavior check that enforces the interface.
Drops 13 phrase pins on the judge prompt. Kept: both inputs carried as JSON,
the response schema (wire format), and the separate untrusted-DATA test. The
validator behavior tests already use stub judges. Negative control: changing
the schema string in the prompt builder turns the test red.
Asserts the absolute skill path is the only SKILL.md named, the scenario,
AskUserQuestion unavailable, the section Read rule (case-insensitive, also at
:308), no git/gh/mutation, and the REPORT.md path. argv/tool-list checks are
unchanged. Negative control: dropping the no-mutation rule turns it red.
The ship capture prompt test asserts its purpose (no CEO-only writing
guidance) plus the functional rules: one allowed skill path, AskUserQuestion
unavailable, the section Read rule, no git/gh/mutation, and the REPORT.md
path. The two 'MUST actually Read' pins match the Read rule case-insensitively.
Negative control: dropping the no-mutation rule from the harness prompt turns
the test red.
- plan-eng-review-artifact: actor, observer and cleanup use one isolated
GSTACK_HOME (the hermetic run's, or a seeded temporary one when hermetic
mode is off) and the slug gstack-slug resolves for the fixture, instead of
os.homedir()/.gstack/projects/test-project. A guard refuses a state root
under the real home, and cleanup removes only the fixture's project
directory. The 'IMPORTANT … you MUST write the test-plan artifact' line is
replaced by the environment fact (state root and slug). The soft warning
becomes an assertion: exactly one new *eng-review-test-plan-*.md about the
reviewed dashboard change.
- plan-review-report: removes 'CRITICAL REQUIREMENT … MOST IMPORTANT
deliverable' and the restated report contract; keeps 'plan.md is the plan
file'. The existing assertions (report present, last, rows, unresolved
status) now measure the skill's own report prose.
The kind contract is already asserted: every live case has one kind, kinds
are rule/behavior/judge, every LLM judge is judge-kind, every behavior case
has a literal registration and a BEHAVIOR_WHY, and the named contract-shaped
cases stay rule. The census count only broke on adding or retiring a case.
expect(regressions).toEqual([]) compared a noisy three-sample mean against
eval-baselines.json (clarity 4), silently raising the stated clarity floor of
3. The comparison stays in judge_reasoning; pass and assertions use
BROWSE_JUDGE_FLOORS (clarity 3, completeness 4, actionability 4). Judge
prompt and schema are unchanged. Stored panel means: three known-good (two the
ratchet rejected) pass; three below-floor panels fail.
- Seeded eng/design plan-mode cases keep 'no scope menu' and replace the
exact 'Scope gate: plan mode — auto-selected B' requirement with
seededPlanTargeted(): the announcement or native seeded-plan selection, or
seed-only plan names appearing in the review output after the slash command
(trackTokens). The announcement is logged.
- The AskUserQuestion format forbids writing a decision to the plan file as a
substitute for asking. assertNoPlanFileDecisions() fails any plan_ready run
whose plan carries ## Decisions (eng, design, devex, office-hours
auto-mode). The AUQ-blocked cases (eng seeded, devex, office-hours) now
require outcome 'asked' (AskUserQuestion variant or prose fallback).
- plan-mode-evidence registration controls keep their verdicts; the
missing-decisions control now fails on 'without a question'.
- plan-mode-targeting.test.ts stores known-good (announcement; paraphrase plus
seed content) and known-bad (neither; an untracked token) observations and
plan files with and without ## Decisions.
The outcome, no fabricated Completeness: N/10, stays asserted. The required
note now matches any brief wording of the same fact (KIND_NOTE_RE in
helpers/plan-format-kind-note.ts). The format-compliance suite that pins the
declared kind-note string is unchanged. Stored lines: four known-good pass,
three known-bad (a completeness score, a recommendation, 'kind to the user')
fail.
recordsNonTestCallerSearch keeps the value-card contract (non_test_callers
recorded) and accepts git grep, rg, grep or the Grep tool as the search.
Severity checks and the golden exclusion are unchanged. Stored evidence:
four known-good (three the old check rejected) pass; three known-bad (no
search, no caller count, prose only) fail.
prCoverageRequestViolations keeps the skill's stated older-open-PR budget
(five metadata pages, fifty file-list pages plus recent PR 7) and the coverage
outcome (PR 42's file list read from page 1 through page 2). It drops the PR 42
three/four-page and single-first-page ceilings, which were tuned to one
observed capture with a truncated first view. The judge criteria are
unchanged. Stored endpoint lists: three known-good (including a re-read first
page the old ceilings rejected) pass; four known-bad plus an over-budget
file-list run fail. incompleteFirstFileView is no longer used by the paid case.
The regression is a candidate silently dropped to fit the four-option cap.
The scope validator already requires an offered Include/Defer/Cut disposition
for all five candidates; the extra N-1 call floor failed agents that batch
compatible candidates, which the test header calls legitimate. FLOOR now comes
from CEO_SPLIT_CALL_FLOOR = 2. Stored shapes (ceo-split-floor.test.ts): five
sequential, 2-call and 3-call batches pass; a dropped candidate fails with
'missing target decisions'. The registration controls keep their verdicts;
the 3-call plan_ready scenario still fails, now on missing decisions.
Completion now needs at least one allowed Edit, the correction in place, at
least one verifier run whose final status is 0, and no edit after the final
verification. The ending-error case still requires a single verifier run
(status 69), because that fixture's error tells the actor to stop without
retrying. Ownership acquisition/release checks are unchanged. New free
protocol controls: a passing re-verification passes (the old grader failed
it); a failing final re-verification and a retry after the ending error both
fail, as they did before.
The false-completion oracle kept being patched to tell historical review-log
mentions from current claims. The E2E prompt now asks for a closing JSON line
{"outside_review_this_run": ...}. When present, the coverage claim is that
report ('completed' fails) plus any undenied 'both reviewers agree';
'disabled' also satisfies attribution. Runs without it keep the existing
prose oracle, so all stored captures and mutation rows keep their verdicts.
Dispatch, CLI execution, preflight and persisted-disabled checks are
unchanged. Stored outputs: three known-good pass (the old grader rejected
the table form); four known-bad fail.
The diagram grammar grew through four waves of capture patches to accept
correct-but-different diagrams. The E2E prompt now asks for one closing JSON
line {"tested": [...], "untested": [...]}; coverageAuditVerdict passes the
outcome when processPayment is only in tested and refundPayment only in
untested (last summary wins; quoted or example summaries ignored). The legacy
grammar stays as a fallback, so every stored capture keeps its verdict
(all existing rows unchanged). Read evidence and capture success are still
required. Stored summaries: three known-good (which the old grader rejected)
pass; seven known-bad fail.
auqMachineFormatProblems() fails a first-question run when the
Recommendation: line is missing or empty, or when the brief does not label
exactly one option (recommended); recommendation substance >= 4 is unchanged.
ELI10, Pros / cons, ✅/❌ and Net: are still scored and logged but no longer
fail a paid case. scoreAuqFormat and the consistency suite are unchanged.
Stored briefs: three known-good (including one without emoji or Net) pass;
four known-bad (no or empty Recommendation, zero or two (recommended)) fail.
docsCommandAllowed now unwraps, on read-only commands only (pwd, ls, cat,
stat, sha256sum, Git reads): a leading 'cd <fixture repo> &&', '2>/dev/null',
a trailing '|| true', '--no-pager', '-C <fixture repo>' and '-c' with
core.pager=cat / core.quotepath / color.* settings. Lifecycle and script calls
stay literal so their failures cannot be suppressed. Writes, other repos,
--git-dir/--work-tree, config that can execute code, real composition and
redirects stay rejected.
Stored commands (fixtures/docsync-observer-commands.json): 11 known-good read
forms, which the old observer rejected, now pass; all 17 known-bad forms still
fail under the new and old observers. docsync-command-grammar moves its three
harmless rows to an accept test and keeps unsafe variants of them as rejects.
Drops the exact 'Keep <x> in scope' / 'Defer <x> to TODOS' labels, the
'D<n> — Keep … in scope, or defer' title, the '<plan>.md lists' phrase and the
2-of-5 keyword quota. Keeps the evidence bindings (native answer, session,
timestamps, source read, mode context, later withdrawal) and now requires: one
keep option and one defer option, the answer selects keep, a word of the kept
label names a listed plan requirement, no expansion language, and an ELI10 +
Stakes rationale of at least 20 words (so a bare mode label still fails).
Stored captures: both retained known-good runs pass; every existing known-bad
mutation still fails; four rewordings the old grader rejected now pass; three
new known-bad cases fail. The Net/Note single-brief checks (t02 #3) are
unchanged: the 'second sentence after Net' row adds scope ('Also add shared
views'), so it stays a known-bad.
Keeps the engine-routing checks (Aside driven, retired binary unused) and adds
the outcome: design-audit.md exists, names a seeded defect, and index.html or
style.css differs from the initial fixture commit. The style(design) commit
prefix is not required. This case still cannot gate the design-review
stop-condition rewrite (it self-skips without Aside and caps fixes in its
prompt), per the Eng item.
The old check passed if the words 'in-branch', 'pre-existing', 'math' and
'string' appeared anywhere in the final message, in any pairing. The prompt
now asks for a closing JSON map; helpers/ship-triage-labels.ts reads it (or
unambiguous per-failure lines) and the test requires truncate/string ->
in-branch and divide/math -> pre-existing, plus a Bash call that ran the
suite. Stored outputs show swapped, words-only and one-label runs fail.
Drops the untyped recordE2E extras (typecheck baseline shrinks by one).
The workflow, first-run and review-gate prompts drop the 'IMPORTANT … simulate:
1. Detect … 2. Infer … 3. Note …' scripts and the answers they supplied
(platform, app, URL, merge method, 'Eng Review should be NOT RUN'). They keep
the facts the fixture cannot provide (no remote, no PR, gh/fly unavailable,
gstack-review-read output, report directory). Assertions now check the outcome:
the artifacts name Fly and test-app, and the readiness report records the
missing review. Canary, benchmark and setup-deploy prompts are unchanged.
- CLAUDE.md fixture: the shipped ## Skill routing section replaces 'ALWAYS
invoke it via the Skill tool as your FIRST action' and its hand-written rules.
- scrape-match-path: prompt is 'Run /scrape latest hacker news stories'; the
'You MUST follow … 1. Run skill list … Do NOT enter the prototype phase'
script is gone. Asserts the outcome: the bundled skill ran, no goto
prototyping, no new skill or skillify staging written. Listing is logged.
- scrape-prototype-path: drops the five-step script and the self-fulfilling
'Suggest /skillify'; keeps the user's JSON shape request.
- skillify-provenance-refusal: prompt no longer dictates the refusal.
- Narration grader moves to helpers/skill-body-narration.ts: line-start
'I '/'I'll '/'Let me ' plus mid-sentence 'let me <verb>', so 'let me know'
is legal. Stored known-good/known-bad bodies show it still rejects every
known-bad case the old /Let me /i ban caught.
Replaces the shouted 'ALWAYS invoke it … FIRST action' fixture and its
hand-written context-save/restore rules with the section gstack appends to
CLAUDE.md, which already routes save/resume to those skills.
readShippedSkillRouting() parses the '## Skill routing' section from the
routing-injection heredoc in bin/gstack-skill-start at test time. The journey
fixture now carries that instruction instead of 'ALWAYS invoke … FIRST action.
Do NOT answer directly', and still withholds the per-skill rule list so a
regressed description can fail. bin/gstack-skill-start and the helper join the
journey, context-skills and skillify touchfiles. A free check catches heredoc
drift. Runs on unchanged templates set the new baseline.
Drops the IMPORTANT block at skill-e2e-workflow.test.ts:76-83, including
'auto-approve everything' and the CHANGELOG/VERSION instructions; keeps only
the environment facts (non-interactive, no remote). Asserts the outcome: the
original CHANGELOG entries remain, VERSION still reads 1.1.1, no commit after
the fixture head touches VERSION, and README mentions Feature C.
Every ON trial must pass the fixture's verify (already a valid-measurement
requirement) and lowerIsBetter20Pct must hold; when the OFF median is 0, every
ON trial must also be 0. assessOverlayArms enforces both through a new
fixture.gate. Overlay contract bumps to v3 so v2 records are never
reinterpreted. overlay-measurement gains rows for correct+worse (fails),
20%+incorrect (fails) and the zero baseline (passes only at 0). The prose
comment follows the s01 M1 claude.md wording.
The end marker now accepts a RegExp. ASK_QUESTIONS_HEADING matches the heading
with or without its 'CRITICAL RULE — ' prefix, so the excerpt is byte-identical
on the current templates and survives the s04 F9 / s05 L3 rename.
- M2/M3: '(#1946)' and '(#1079)' removed from the credential guard and REST
fallback headings; the rules are unchanged.
- M4: the audit-failure fallback states the reason (a skipped audit lets
unverified plan items ship as done) instead of 'the failure shape that
VAS-449 surfaced'.
- M7: 'Continue silently — just note the counts briefly' -> report the pass
counts in one line and continue to Step 6 (Step 16 cites these counts).
Not changed: s06 M1, the Step 16 IRON LAW (UC2 gate keeps its wording).
W7a pin: test/ship-plan-completion-invariants.test.ts:127 pinned the VAS-449
sentence; it now checks that the audit-failure fallback still forbids a
silent fail-open (case-insensitive). Negative control: deleting that
sentence turns it red. expected red until the regeneration commit
The squash-merge cleanup example created 'codex/post-merge-pr-N' branches on
every host. The example is now 'post-merge/pr-N', naming Codex Desktop's
codex/ prefix as the host-specific case and keeping the detached-HEAD reason
and the no-force-push rule. No test pins the removed text.
expected red until the regeneration commit
/ship restated the checklist's Pass 1 as two of its five CRITICAL categories
(missing Race Conditions, Shell Injection, Enum Completeness). Per the W5.3
decision it now names 'the checklist's Pass 1 categories', matching /review.
No test pins the removed text. expected red until the regeneration commit
- H6 (manual merge): TEST_COMMAND comes from CLAUDE.md or AGENTS.md; if
neither documents one, ask the user instead of defaulting to 'bun test'
(wrong for non-Bun projects and a bare suite run in gstack). The audit
hunk's 'persist the answer to CLAUDE.md' write is not adopted: it adds a
write the readiness gate never had.
- H7: E2E and judge results are searched in the eval store the collector
writes (~/.gstack/projects/<slug>/evals/ and its shards/*/ directories),
then the legacy ~/.gstack-dev/evals/ fallback; the gate no longer reports
'no E2E tests run today' when results exist.
test/state-root-ratchet.allowlist.json: allowlist the EVAL_DIR line with its
reason (test/helpers/eval-store.ts writes under os.homedir()/.gstack, so the
reader must use the same path, not $GSTACK_STATE_ROOT).
Parity cap (test/helpers/carve-guards.ts): land-and-deploy maxSizeRatio
1.10 -> 1.108 (measured 1.1076 at the branch tip, including the W7f marker).
expected red until the regeneration commit
- H1: TODOS-format cites /ship Step 14, not 5.5.
- H2: the land readiness review is the checklist /ship applies in Step 9 (there
is no Step 3.5).
- H3: the Apple adapter names the no-credential-talk rule by location instead
of 'line 14' (it renders at line 16).
- H4: greptile triage diffs against origin/<base> from the caller's detected
base, not origin/main.
- H5: the land Important Rules recap matches the steps (4-minute CI rounds in
Step 3; 30 seconds for merge readback and deploy status).
- H8: landing-report calls the util once per bump level (four), and drops the
unsupported 'cached by bun' claim.
- M6: landing-report says what its base detection does instead of claiming
it matches other skills.
No test pins the removed text. expected red until the regeneration commit
The Mode Quick Reference let TRIAGE skip the benchmark, the magical moment
and passes 2/4-8, contradicting the anti-skip rule, the 0C target gate and
the 'executed all 8 DX passes' self-check. Per the W5.5 decision the TRIAGE
cells now read 'Target only' / 'Verify exists' / 'All 8, blockers only',
matching the Rating Method's 'flag adoption blockers; skip nice-to-haves'.
No test pins the removed cells. expected red until the regeneration commit
- M1: 'YOUR PRIMARY TOOL' / 'The rule is simple ... Don't ask permission ...
Mockups ARE the plan' -> generate mockups without asking when the plan has
UI and the designer is available, with the reason; skip only under the
Step 0.5 conditions (which include the user's opt-out).
- M3: 'be MORE aggressive' on previously flagged areas -> check whether the
earlier issue recurs or was only partly fixed and say so.
- M4: drop the unverifiable 'Tier 2+ (15+ RPM)' rate-tier aside; keep the
one-at-a-time rule and the /design-shotgun pointer.
(s05 M2 landed with s04 F1.) No test pins the removed text; the remaining
failing case is the plan-scope-selection PTY case that fails under Bun 1.3.14
and passes under Bun 1.4.0. expected red until the regeneration commit
The scope gate said 'audit → mockups → Step 0' while the body runs Step 0
(0A-0D) first and 0D announces 'I'll generate visual mockups next'. Per the
W5.4 decision, the three gate phrases now read audit → Step 0 → Step 0.5
mockups. 'Reply with A, B, or C' is unchanged.
W7a pins:
- test/plan-scope-selection.test.ts:616 pinned the old order sentence; it now
checks the order of the 'After resolution:' steps (preamble, base branch,
audit, Step 0, mockups). Negative control: the old order turns it red.
- test/gen-skill-docs.test.ts:4737 normalizer strips the design-only
'Then run the pre-review audit, ... against that plan.' sentence by pattern.
expected red until the regeneration commit
- H1/H2: the skip list names the current headings ('Step 0: Detect platform
and base branch', 'Design Outside Voices (independent)'); the old names
never matched, risking a duplicate base-branch detection or outside-voice pass.
- H3: the DX journey map is the 6-stage table from Step 0F, not 9 stages.
- H4: the DX checklist names Step 0 'DX Investigation' and scores only after
the evidence exists, as plan-devex-review requires.
- H5 (manual merge): the eng test-plan artifact uses the eng skill's
{user}-{branch}-eng-review-test-plan-{datetime}.md name under
$GSTACK_STATE_ROOT/projects/$SLUG (consistent with s04 F3).
- M5: the Decision Audit Trail separator has 7 columns to match its header.
No test pins the removed text. Three autoplan PTY cases fail under Bun 1.3.14
and pass under Bun 1.4.0. expected red until the regeneration commit
The bodies are plain procedure; the caps label added nothing and competed
with the CRITICAL GAP output label. Applied in plan-ceo-review,
plan-design-review and plan-devex-review sections (s05 L3 promoted by the
plan).
W7a: test/devex-finding-fixture.test.ts:83-84 now finds the section by a
heading regex ending in 'How to ask questions'; the skill-e2e-plan-ceo-plan-mode
failure diagnosis names the new heading.
Judge anchors NOT changed here (builder E moves them to structural anchors):
skill-ceo-section-ordering.test.ts (old :938), workflow-excerpt.test.ts:109,296,
skill-llm-eval.test.ts:638, workflow-judge-input.test.ts:366,373; the mirror
in touchfiles.test.ts:219 also uses '## CRITICAL RULE'. These stay red until
E's anchor commit lands. expected red until the regeneration commit
garrytan