W7c (t06 #12). One test file. Each removed pin was the text fix for a past
`review/SKILL.md workflow` judge finding; that judge (gate lane) still
grades the prose, so the free test keeps only structure, tokens and gates.
Removed assertion groups (invariant -> remaining protection):
- Plan-audit sentences ("retain the exact command ...", "A mixed item
contributes to both lists", "Keep external-state ...", "If only behavioral
checks remain ...", "File-existence checks ... are static audit checks")
-> audited deliverables vs deferred behavioral checks: kept free checks on
the separation, never DONE from a diff, zero-deliverable no-waive,
UNVERIFIABLE -> Step 4.7, read-only audit (no apps/APIs/state) and inspect
-before-invoke order; gate E2E review-exploratory-small-cli (gate).
- QA loop/report duplicates of qa-probe-gates / qa-caller-authority (probe
loop sentences, shared contracts, report layout prose, "Only the parent
runs report-only discovery", "Read agent/user updates ...", "Prepare one
provisional QA section", smoke composition) -> owned there (free, every
host); here only `5 minutes/12 probes`, headings, link-every-checkpoint,
no second report, never overwrite another run, freshness vs current inputs.
- Persistence/finalization prose (~45: "final zero-edit pass, reconcile",
"by structural identity", "without requiring deleted pre-extraction
blocks", "raw bytes equal the bound snapshot blobs", "Keep both through
re-reviews", "Earlier core tokens remain unused", "Re-read its final-
snapshot supporting source ...", "Fixed advice needs no skip coverage",
"The required in-host adversarial result controls native completion",
"Optional outside attempts keep their own incomplete records ...", etc.)
-> kept free token list (REVIEW_START/PASS_START, COMPLETED, CONVERGED,
CYCLES=0 once, `review_binding`, `snapshot_covered_paths`,
sharedLibsFingerprint, `completed:false`, evidence_paths/helper_target),
cycle limit (at most 3, Below 3 repeat, At 3 persist `converged:false`),
section order, and the integrity gates below; judge `review/SKILL.md
workflow` (gate lane); gate E2Es review-army-delivery-audit,
review-army-json-findings, review-army-quality-score (gate).
- Confidence-calibration prose (data loss/outage examples, missing field /
nullable / race quoting, metaclass/ORM sentence, missing-literal sentence,
learning-log sentence) -> kept the score table as a routing table (each
band's action), `[CRITICAL]` / `[CRITICAL|INFORMATIONAL]`, no P0-2,
`file:line`, force 4-5, never invent 7+, unverified-without-quote, gate
before table; judge `review/SKILL.md workflow` (gate lane).
- Shared-code reuse safeguard enumeration (assume-unchanged/skip-worktree,
sparse index, symlinks, submodules, Git filters, encodings, fsmonitor,
textconv, provenance, binding wording) -> enforced in code and covered by
free tests test/shared-libs-snapshot-check.test.ts,
test/shared-libs-evidence.test.ts, test/shared-libs-checker-interface-
evidence.test.ts; kept the checker command/heredoc/JSON tokens and the
fail-closed lines.
- Misc definitions ("A probe is affected when ...", "An invocation is this
/review run ...", "Step 1.5's provisional notes ...", "Otherwise record
each considered specialist", "Core findings use the confidence gates
below", "highest confidence", "distinct specialists", "all source names",
"active or unknown reader/writer", Greptile "Before edits, inspect ...")
-> judge `review/SKILL.md workflow` (gate lane).
Kept as case-insensitive meaning checks (stop/consent/approval/safety):
no edits before Step 5, readers returned or confirmed stopped, no edits in
Step 4.8, stop without edits on unconfirmed readers (review and ship),
stop at 3 cycles with no clean summary or fourth pass, failed/blocked
probes mean incomplete, ship named-risk cannot complete review, native vs
outside cannot substitute, structured-review gate, never carry coverage
forward, never invent a binding, skipped advisory cannot suppress a defect,
user-skip-only suppression, Greptile reply != code approval and wait for
approval before editing, HIGH-impact gate options (A ends, read-only audit,
no shipping authorization), never downgrade severity, never trust a supplied
hash, missing metadata cannot dedupe, test creation needs approval,
report-only review never runs setup/installs/cookie import.
Kept unchanged: all marker-order arrays, manifest order, Pre-Landing summary
after the logger, finish-token count and negatives, stale-wording bans
(incl. `git diff origin/main`), ship/review routing negatives.
Cosmetic pins dropped: STOP / NON / COUNT capitals; the test titled "resolves
the judged ... ambiguities" is renamed to say what it keeps.
Mutation log: .context/mutation-review-workflow-clarity.md (23 mutations,
0 unexpected: 3 reorders, 4 markers, 11 safety red; 5 rewordings green).
W7c (t04 F5, F6, F7, F15; F1/F2 already landed with W7a). One test file.
This file guards judge-input fidelity; the judges that read these excerpts
grade the prose itself.
Removed assertion groups (invariant -> remaining protection):
- Ship rule sentences ("Use the documented selector and pre-merge command.",
"During pre-flight, read the existing review log", "Tests must pass or
retain approval ...", "For another repair, repeat rule 2 ...", "The
unchanged release steps follow", "Use this example only after confirming
...", "Retain queued fixes and restore coverage", "Each commit must work
independently", "Keep actual outcomes and incomplete flags", the long
"Continue to Step 9.2 with the core/design-lite findings ..." form, the
Y'd/D'd evidence sentence) -> judge `ship/SKILL.md workflow` (gate lane),
which reads this exact excerpt; kept free checks on step routing (`Step 4`,
`Step 8.1`, 9.2 -> 9.2.1 QA -> 9.3, `Insert Step 5, affected Steps 6–8 and
all of Step 9 before the pending Step 10`, `5 → 6 → … → 11.5`,
`continue to Step 16`), tokens (`BASE_VERSION`, `VERIFY_RESULT stays
fail`, `"partial":N,"not_done":N`, "intentionally dropped"), and the review
cycle order (9.1 < 9.2 < 9.2.1 < 9.3 < 9.4) as a structural check.
- Outside-review routing sentences ("Other preflight mode, including harness
mismatch", "Outside execution or output validation fails", "Retain its
output and diagnosis ...", "No prompt, outside process or native
replacement", "Other preflight failures retain their printed diagnosis",
the full fallback-list regex) -> disabled means no outside call: kept free
order of disabled/prompt/ready/fallback blocks, `outside_status` tokens,
`_DISABLED_REVIEW_MODE`, disabled recheck before dispatch, "disabled means
no replacement", fallback-not-completion; judges
`document-release/SKILL.md workflow`, `plan-eng-review/SKILL.md sections`
(gate lane).
- Eng/design prose ("quote the motivating plan requirement", "Use a scoped
Edit to save this record ... Leave other choices unchanged" full form,
"edit the plan and re-rate") -> judge `plan-eng-review/SKILL.md sections`
and `plan-design-review/SKILL.md passes` (gate lane); approval meaning
checks kept.
- Retro prose ("does not require saved history", "PRs referenced", "not
newly added test cases", "is the live **team** streak", "draft the
tweetable summary using the format in Step 14, then save") -> judge
`retro/SKILL.md instructions` (gate lane); kept headings, `prs_merged:
null`, `streak_days`, the one-second compare window, Capture Learnings
after Step 14, no `$(date`, two `today="<today>"`.
Loosened to case-insensitive meaning checks (stop/consent/approval and
safety lines, always kept): STOP for each pending decision (now a regex
anchor between Step 5 and the Test Plan Artifact), STOP until the actual
answer arrives, no remedy/next section/ExitPlanMode while a choice awaits an
answer, evidence-not-approval, reopened choices need answers, wait for
approval (design), AskUserQuestion once (t04 F6), per-finding approvals,
dispatched-reviewer-missing -> stop, CYCLES >= 3 -> stop with
`converged:false`, no fourth cycle, required-probe gate before Step 10,
failed probes block continuation, coverage-gate not bypassed, failed/unrun
probes not passed, three snapshots match, document-release VERSION-absent
rule, never create an empty commit, `read any other SKILL.md` (t04 F15),
AskUserQuestion-and-wait in the capture prompt.
Kept unchanged: heading/step order checks, `## Review Sections` single-copy
check, fail-closed marker tests, redact-before-edit ordering, approval gates
outside subagent prompts, plan-tune auto-decide output lines, CEO capture
harness contract, deploy formula execution, all stale-string bans.
Cosmetic pins dropped: STOP / ONCE / NOT / OTHER capitals.
Mutation log: .context/mutation-workflow-excerpt.md (21 mutations, 0
unexpected: 3 reorders, 3 markers, 8 safety red; 7 rewordings green).
W7c (t10 F12, F13). One test file; still renders every host x caller.
Removed assertion groups (invariant -> remaining protection):
- `STOP and wait` / `STOP for the answer` / `AskUserQuestion and WAIT`
capitals (t10 F12) -> approval before `./setup` or install: now an order
check (explicit approval < "only after approval" < `cd <SKILL_DIR> &&
./setup`; OK to proceed? < setup; AskUserQuestion-and-wait < install only
the actual choice), plus case-insensitive "stop for the answer". Stronger:
the mutation that moves setup ahead of the approval ask now goes red (the
old test stayed green). New negative control removes the approval ask.
- Report-layout prose for /review and /ship (keep metadata/outcome tables,
demote headings one/two levels, fields as subsections, keep scores
separate, save baseline normally) -> one final linked report section:
kept free checks on the `## Exploratory QA and Verification Results` /
`### Browser results` / `## Exploratory QA` headings, link every checkpoint
and no second report; gate E2Es review-exploratory-small-cli,
ship-exploratory-small-cli (gate).
- Smoke-pacing prose ("one success and the riskiest changed failure/edge",
"Required even for small diffs ...", "Required: plan commands/assertions,
listed separately", "Other ideas are optional, untested") and duplicate
report-status pins -> kept the numeric `5 minutes/12 probes` bound and the
report-clean-only-when-all-required-pass gate; gate E2Es
ship-exploratory-small-cli, ship-exploratory-plan-checks (gate).
- Dedup prose ("Combine saved `findings` ...", "If both history ... classify
normally", "Report the suppressed count once if nonzero", "If N > 0, print
once:", "Otherwise skip the summary") -> only user `skipped` findings are
suppressed: kept free order check (read decisions before matching) and the
`skipped` / never `fixed` / `auto-fixed` lines; gate E2E
ship-skipped-queued-finding (gate).
- Preparation sentence pins duplicated from qa-probe-gates (Reads in order,
no repeated Reads, "in full and select the surfaces", "Write a **charter**",
"charter** per behavior", "announce finite command timeouts") -> owned by
qa-probe-gates (free, every host); this file keeps scope < methods <
charter structural order for every caller.
- Learnings prose ("Otherwise skip this optional lookup", "Reading old notes
never requires writing new ones") -> report-only lookup never writes: kept
free checks (directory already known, caller permits, do not run the
search helper, no bash/gstack-config/AskUserQuestion, injection throw); the
query interpolation stays checked via `"webhook retries"`.
- Misc prose ("Re-run affected contracts and adjacent happy paths ...", "Run
only for selected browser surfaces", "Confirm each issue by retrying once",
"parent applies queued fixes", "No test framework detected", "Report QA
setup blockers", "required QA") -> judge `qa/SKILL.md workflow` and
`review/SKILL.md workflow` (gate lane); the adjacent kept gates remain free.
Kept as case-insensitive meaning checks (authority, consent, safety): probe-
local blocker rule, report-only never bootstraps/invokes skills, unknown
caller -> report-only, declined -> blocked, credentials/passwords/other tabs,
local-target ownership, no unit-test/curl substitute, functional-only reads
nothing browser-side, never-pass list, named-risk acceptance, noninteractive
-> blocked, test_stub ASK approval, ship waiver cannot complete review,
caller-required rechecks, blocked rechecks never verify repairs, no edits
during Step 4.8, no inner repair loop, REVIEW_START/token-once, native pass
required for 5.8, coverage-cap stop lines, report-only never commits or
dispatches, CI/test-weakening ban, never delete a valid red regression.
Kept unchanged: machine tokens (`bun G start D`, `$B snapshot -i`,
`DIFF_START`, `CONSOLE_ERRORS=`, `$B handoff/resume`), resource paths, the
`If A and allowance remains:` count, `Minimum = 60%, Target = 80%`, all
`.not.toContain` stale-workflow bans, existing negative controls.
Cosmetic pins dropped: STOP / WAIT / LOCAL / Never capitals.
Mutation log: .context/mutation-qa-caller-authority.md (23 mutations, 0
unexpected: 3 reorders, 3 markers, 11 safety red; 6 rewordings green).
W7c (t07 F6, F7, F8). One test file. The eval prompt itself is unchanged.
Removed assertion groups (invariant -> remaining protection):
- ~40 driver-prompt sentence pins in the disclosure, artifact and fix-budget
tests ("All four reads are required before probing ...", "command is the
exact full outer capture invocation, including ...", "different later
command", "not the required same-command replay", "Only the helper writes
observed fields", "Link the evidence and checkpoint files ...", "Evidence
rows contain ONLY complete JSON ...", "aim under 400 words", "one causal
sentence per checkpoint hypothesis ...", "a reproduced in-tier defect
requires ...", duplicates of "retain its headings ..." etc.) -> the run
produces verifiable evidence and a linked report: free validator tests in
this file (qaFunctionalVerdict replay, validateQACheckpoints on CI, R29 and
R85 captures) and gate E2Es qa-functional-cli-report,
qa-functional-webhook-report, qa-functional-cli-fix,
qa-functional-webhook-fix (gate). Kept the machine-read contract: read
list, `"command":"<exact full outer capture invocation>"`,
`annotations.learning`, the three copied field names, the >20-character
limit, evidence.json before report.md.
- `maxTurns: 40` / `completionReserveMs: timeout / 4` exact source pins ->
replaced by a `maxTurns >= 40` floor and presence of `completionReserveMs`.
- Fix-stage provenance sentences ("most recent completed native probe",
"Tests, source edits and clock reads ...", "write no checkpoint when there
is no next probe") -> test output never becomes `observed`: kept one free
meaning check plus the R29 validator test (free).
- Duplicate loop-order checks on qa-only/sections/exploratory.md and grep of
`scripts/resolvers/qa.ts` source (t07 F6, F8) -> loop ordering has one
owner, qa-probe-gates (free, every host). Here the source grep becomes a
check on `generateQAExploratory` output (one probe per native command,
never batch probes), and the caller test keeps resource paths, field names
and structural stage anchors.
Loosened to case-insensitive meaning checks (kept): never synthesize JSON,
never shorten native JSON, existing tests read-only, freeze tests after red,
repair scope `src/<family>.ts`, green suite does not replace native probes,
report incomplete honestly, report-only proposes test stubs, completion
reserve does not end coverage, all eight scenarios required, never backfill
notes, owned probe directory, only complete emitted JSON, wrapper/entrypoint
boundary lines.
Kept unchanged: paid-case selection, CI/R29/R85 validator replays, actor
boundary and command allowlist, fix-stage order, verification probe list,
scenario lists.
Cosmetic pins dropped: ONLY / NOT / Never capitals.
Mutation log: .context/mutation-qa-functional-prompt.md (19 mutations, 0
unexpected: 3 reorders, 4 markers, 7 safety red; 5 rewordings green).
W7c (t07 F3, F4, F5). One test file. This file stays the single owner of
shared QA loop ordering (qa-functional-prompt defers to it).
Removed assertion groups (invariant -> remaining protection):
- `> **STOP.** Before any probe ...` prefix pin (t07 F3, retire) -> parent
QA loads methods before any probe: kept free meaning check on "before any
probe, including plan checks, complete the ordered scope/method Reads" and
its order before the required-checks step.
- Preparation prose in the stage list ("in full and select the surfaces",
"2. Read the selected surface methods below in full", "Write a **charter**",
"Start once before baseline:") -> stage order scope -> methods -> charter ->
deadline start -> loop: kept as structural anchors (section Reads,
surface labels, `## 1. Charter and preflight`, `bun G start D SECONDS`,
step 1 title); added a reorder control moving a method Read after the charter.
- Bounds/layout sentences ("an unlimited mode uses the caller's bound",
"announce finite command timeouts", "EARLIER_UTC = caller's absolute
deadline, if set", "in the probe directory, beside its deadline if
bounded", duplicate SECONDS pins in the mode-bounds test, "charters as
Markdown in the report", the full "observed: its exact decoded child JSON
..." sentence) -> numeric bounds and field names are kept free checks;
the observed-field shape is enforced by the native checkpoint validator
(qa-functional-prompt validator tests, free) and gate E2Es
qa-functional-cli-report, qa-functional-webhook-report (gate).
- Publication prose ("No drafts/placeholders", "and replay limits in the
report", "An absolute state path is not itself a secret", "If capture is
incomplete, report that limit instead of reconstructing it") -> notes are
immutable and unsafe values withheld: kept free checks on "corrections
cannot repair published notes", never invent a substitute, withhold
secrets/unsafe values and stop the chain; validator-backed in
qa-functional-prompt; gate E2Es review-exploratory-small-cli,
ship-exploratory-small-cli (gate).
- Loop-step prose ("Guard if bounded; await completion", "Run that exact
probe; G enforces ..." duplicate, "then minimize via those gates", "Expiry
leaves confirmation/minimization incomplete", "for each affected
revalidation", "If bounded, run", "Reuse Step 4's ... completed Reads",
"Finish missing methods before charters") -> loop has 5 steps, guarded by
G, replays via steps 2–3, revalidation returns to step 2: kept free checks on
step count, `G enforces the deadline`, `bun G status D`, `via steps 2–3`,
`return to step 2`, refusals as not-run, do not repeat completed Reads.
- CI-fixture clause "description, section index or remembered method is not
a completed instruction Read" -> covered by the kept CI-proven clauses
(`## 0. Preparation gate`, await each Read, complete missing Reads before
Charter) and gate E2E ship-exploratory-small-cli (gate; the CI case).
Kept (meaning checks, case-insensitive where prose): CI-proven "await their
results before the first probe" rule (comment + mutation kept), Reads in
order, no repeated Reads, SECONDS=30/900, shorter mode/caller limit, no D
without a total limit, finite timeouts, REPORT_DIR layout and caller paths
win, plan checks after smoke with no smoke guard and no clock reset, caller
cap, not-run on expiry, await clock results, never reset D/bypass G,
invalid/missing D stops probes, expiry -> write the report, Step 4
read-only, isolation/permission preflight before setup, field names,
QA_DEADLINE receipts, all semantic-flip mutation controls, method/report
resource paths, helper commands.
Cosmetic pins dropped: STOP / AND capitals, bold STOP prefix.
Mutation log: .context/mutation-qa-probe-gates.md (22 mutations, 0
unexpected: 3 reorders, 4 markers, 8 safety red; 7 rewordings green).
W7c (t06 #9, #10, #11; #7/#8 already landed with W7a). One test file.
Removed assertion groups (invariant -> remaining protection):
- Execution-only extraction sentences ("implementation and test-creation
requirements", "retain its command, expected outcome and source verbatim",
"not EXTERNAL-STATE merely because ...", "Keep genuine external-state ...")
-> execution-only checks route to Step 8.1/9 outside counts and are never
DONE statically: kept free checks `Step 8.1/9`, outside implementation
counts, `never DONE from static inspection`, do-not-waive; gate E2E
ship-exploratory-plan-checks (gate).
- Plan-verification prep sentences (~17: "Clarify unknown outcomes", browser
vs functional items, "An API URL is not automatically a page", "Only browser
evidence needs screenshots", smoke-budget and command-limit sentences, risk
acceptance wording) -> plan checks are collected now, executed by the Step
9.2.1 owner before Fix-First, and never silently waived: kept free checks on
the heading list, `Collect now; execute in Step 9`, `Step 8.2`, `Handoff to
Step 9.2.1` order, `before Fix-First`, never silently waive, noninteractive
-> blocked, the `VERIFY_RESULT=` line and `## Verification Results`; gate
E2Es ship-exploratory-plan-checks, ship-exploratory-small-cli (gate).
- Parent-processing duplicates on the generated section (same pins as the
template test) -> kept on the template test; the generated copy keeps
section order + seven-field anchor; freshness via gen-skill-docs.
- "Check the task's terminal status", "recording the reason in the PR body and
Step 20 metrics" -> failed audits use the fallback: kept free LAST-line JSON
-> audit-failure fallback check and `(recommended/default)` marker.
- Step 16 evidence sentences ("Independently prove ...", "exact command, exit,
timestamp and log", "This recovery also applies ... stage 5") -> storage
failure is not stale input: kept free table-row structure, storage column
must not rerun `gstack-evidence run`, STALE/MISSING without proof, ledger
unavailable never FRESH; judge `ship/SKILL.md workflow` (gate lane).
- "Each commit must work independently" -> bisectable commits: kept
`bisectable commits`, `continue to Step 16`, no empty commit, history-
rewrite bans; judge `ship/SKILL.md workflow` (gate lane).
Loosened to case-insensitive meaning checks (kept): per-item confirmation,
blanket-confirm ban, classified DONE/NOT DONE by `[ -f` (MUST dropped), stop
publication on new/changed/unwaived test failure + `Run Steps 5–15`, waiver
reuse limited to verified pre-existing failures, fetch/push failure -> stop,
never rewrite history, only a successful push, stop live child before inline
audit, never race a late result, AskUserQuestion on second failure.
Kept unchanged: seven-field JSON contract parse, review-mode negatives,
CONTENT-SHAPE/validator tokens, Step 14 deferral routing, CHANGELOG negatives,
the executed push-idempotency shell test.
Cosmetic: MUST / mandatory / Do NOT capitals; "VAS-449" dropped from the
describe title.
Mutation log: .context/mutation-ship-plan-completion-invariants.md (19
mutations, 0 unexpected: 3 reorders, 4 markers, 7 safety red; 5 rewordings green).
W7c (t08 F3). One test file.
Removed assertion groups (invariant -> remaining protection):
- "Retain queued fixes and restore coverage" -> a missing reviewer keeps
queued fixes: gate E2E ship-skipped-queued-finding (gate) and judge
`ship/SKILL.md workflow` (gate lane); the stop itself stays a free check.
- qa decline print line ("Test bootstrap previously declined — skipping.")
-> qa skips bootstrap after a saved decline: kept free meaning check
`/BOOTSTRAP_DECLINED ... skip the rest of bootstrap/i`.
- "with the user's free-text evidence" -> UNVERIFIABLE items are confirmed
one by one: kept free `/per-item confirmation/i` here plus the
blanket-confirm check in ship-plan-completion-invariants.
- Docs reentry sentences ("On reentry, reuse only ...", its duplicate, "base/
input hashes still match", "retain its actual status and scope", "Otherwise
use Blocked recovery ...") -> reentry reuses only this invocation's
validated evidence: kept free checks "never reuse an audit across
invocations" and "reentry never resets the count or authorizes a launch";
gate E2Es ship-docsync-stale-before, ship-docsync-stale-after,
ship-docsync-recovery, ship-docsync-late-result (all gate).
- "left uncommitted after Step 15" -> late commits take only remaining
changes: kept free "never create an empty commit" and "preserve unrelated
user files"; gate E2E ship-local-workflow (gate).
Loosened to case-insensitive meaning checks (stop/consent/approval lines,
always kept): STOP blocks advancement, answer each AskUserQuestion, routine
authorization never waives gates, dispatched-reviewer-missing -> stop,
CYCLES >= 3 -> stop, Any N -> stop as NOT DONE, resume only after verified,
no second deferral, per-item confirmation, explicit Add-tests choice overrides
the decline for this invocation only, framework approval, first entry
launches the audit, no third attempt, unwaivable redaction/security gates.
Kept unchanged: bootstrap branch order, no `rm` in the decline branch, the
Step 14 -> 14.5 routing line, stale-string bans, `NEW_TITLE` field and
`--title "$NEW_TITLE"`.
Cosmetic pins dropped: STOP capitals, "mandatory".
Mutation log: .context/mutation-ship-reentry-gates.md (17 mutations, 0
unexpected: 1 reorder, 2 marker and 9 safety mutations red; 5 rewordings green).
W7c (t06 #19). One test file.
Removed assertion groups:
- "A genuine blocker may require a safety or named documentation-risk
decision" and "routine interactions and blocking decisions above" sentence
pins. Invariant: routine release authorization never waives a blocking
safety/docs decision. Remaining protection: kept free meaning check
`/stop for that decision rather than treating release authorization as a waiver/i`
plus `/named documentation-risk exception before distribution/i`.
- Full-sentence "in `read-only` mode against the selected release source".
Invariant: the docs preflight is read-only. Remaining protection: kept free
check on "`read-only` mode".
Loosened (case-insensitive, kept): apply-only-to-repository-landing scope
line, CLASSIFY-before-credentials (credential boundary), never abort a store
release over branch topology (demonstrated failure), two routine interactions
(numeric limit), stop-for-decision (consent/stop line).
Kept unchanged: manifest trigger routing, Apple read before the branch gate,
byte-exact branch gate appearing once, docs preflight before the
authorization moment, fastlane/iris/appPriceSchedules command anchors,
stale-wording bans. No gate-tier E2E covers the Apple adapter, so every
invariant keeps a free check here.
Cosmetic pins dropped: capitals on ONLY / CLASSIFY / STOP.
Mutation log: .context/mutation-ship-apple-gate.md (14 mutations, 0
unexpected: 2 reorders, 1 marker and 6 safety mutations red; 5 rewordings green).
W7h (t09 F12, F13). One test file.
Removed assertion groups:
- `cli/cli#3442` / `cli/cli#13380` pins (t09 F12). Invariant: none of its
own (upstream archaeology; the IDs stay in the template, only the pin goes).
The behavior they decorated, "after any non-zero `gh pr merge` exit, query
authoritative state before retrying", is now a kept free meaning check.
- `Universal invariant` label pin. Invariant: query-before-retry. Remaining
protection: the new case-insensitive check
`/non-zero exit from `gh pr merge`, query authoritative PR state before retrying/i`
(stronger: the mutation that inverts the rule now goes red; the old test
stayed green).
Loosened to case-insensitive meaning checks (kept, W7 keep list, data-loss
and destructive-op guards): STOP worktree cleanup without removing, no
`--force`, primary working tree never removed, CLOSED -> stop, never replay a
merge after MERGED, one direct fallback, no fallback from a direct attempt,
the OPEN/no-auto/no-queue fallback precondition, branch-absent vs check-failed
messages. Gate-tier E2E also covering the flow: land-and-deploy-workflow
(gate), judge `land-and-deploy/SKILL.md workflow` (gate lane).
Cosmetic pins dropped (no control needed): capitals on STOP / Do NOT.
Mutation log: .context/mutation-land-and-deploy-postfail.md (12 mutations,
0 unexpected: reorder, field delete and 7 safety deletes red; 3 rewordings green).
The prompt was pinned three ways (sha256, ~20 sentences, an order check).
Kept: the skill and lazy-section paths, the plan path, the ordered
read -> review -> edit -> Write -> read-back -> summary steps, and the
no-browsing rule (case-insensitive). Dropped the hash and sentence pins.
Negative control: removing the read-back step turns it red.
reviewRevalidationPrompt and reviewPrompt are test-owned harness prompts
(non-test callers: 0); the paid shared-libs E2Es measure whether they work.
Kept: the three exact receipt commands, base-prompt prefix and resumed suffix
identity, branch selection (actor / default / resumed coverage), the state
path and turn-budget interpolations, read-record-before-diff and
first-response-before-start ordering, no embedded token, no supplied Skip
answer, reusable:true before reusable:false, the captured R59 stage boundaries
in the exclusion list, asset roots and interfaces, and the stale step-6
reference ban. The review template's Step 4.7/4.8 completion rule is matched
by step number on one line, not by sentence. Negative control: reordering the
diff-read instruction turns the test red.
docsNativeInterface is the test helper's own prompt, so its sentence pins
only checked that the helper says what it says. Kept: owned skill path, the
two extracted lifecycle commands (exact), the scripts-list interpolation, the
repo path and Git read examples, and every docsCommandAllowed /
docsToolFailures behavior check that enforces the interface.
Drops 13 phrase pins on the judge prompt. Kept: both inputs carried as JSON,
the response schema (wire format), and the separate untrusted-DATA test. The
validator behavior tests already use stub judges. Negative control: changing
the schema string in the prompt builder turns the test red.
Asserts the absolute skill path is the only SKILL.md named, the scenario,
AskUserQuestion unavailable, the section Read rule (case-insensitive, also at
:308), no git/gh/mutation, and the REPORT.md path. argv/tool-list checks are
unchanged. Negative control: dropping the no-mutation rule turns it red.
The ship capture prompt test asserts its purpose (no CEO-only writing
guidance) plus the functional rules: one allowed skill path, AskUserQuestion
unavailable, the section Read rule, no git/gh/mutation, and the REPORT.md
path. The two 'MUST actually Read' pins match the Read rule case-insensitively.
Negative control: dropping the no-mutation rule from the harness prompt turns
the test red.
- plan-eng-review-artifact: actor, observer and cleanup use one isolated
GSTACK_HOME (the hermetic run's, or a seeded temporary one when hermetic
mode is off) and the slug gstack-slug resolves for the fixture, instead of
os.homedir()/.gstack/projects/test-project. A guard refuses a state root
under the real home, and cleanup removes only the fixture's project
directory. The 'IMPORTANT … you MUST write the test-plan artifact' line is
replaced by the environment fact (state root and slug). The soft warning
becomes an assertion: exactly one new *eng-review-test-plan-*.md about the
reviewed dashboard change.
- plan-review-report: removes 'CRITICAL REQUIREMENT … MOST IMPORTANT
deliverable' and the restated report contract; keeps 'plan.md is the plan
file'. The existing assertions (report present, last, rows, unresolved
status) now measure the skill's own report prose.
The kind contract is already asserted: every live case has one kind, kinds
are rule/behavior/judge, every LLM judge is judge-kind, every behavior case
has a literal registration and a BEHAVIOR_WHY, and the named contract-shaped
cases stay rule. The census count only broke on adding or retiring a case.
expect(regressions).toEqual([]) compared a noisy three-sample mean against
eval-baselines.json (clarity 4), silently raising the stated clarity floor of
3. The comparison stays in judge_reasoning; pass and assertions use
BROWSE_JUDGE_FLOORS (clarity 3, completeness 4, actionability 4). Judge
prompt and schema are unchanged. Stored panel means: three known-good (two the
ratchet rejected) pass; three below-floor panels fail.
- Seeded eng/design plan-mode cases keep 'no scope menu' and replace the
exact 'Scope gate: plan mode — auto-selected B' requirement with
seededPlanTargeted(): the announcement or native seeded-plan selection, or
seed-only plan names appearing in the review output after the slash command
(trackTokens). The announcement is logged.
- The AskUserQuestion format forbids writing a decision to the plan file as a
substitute for asking. assertNoPlanFileDecisions() fails any plan_ready run
whose plan carries ## Decisions (eng, design, devex, office-hours
auto-mode). The AUQ-blocked cases (eng seeded, devex, office-hours) now
require outcome 'asked' (AskUserQuestion variant or prose fallback).
- plan-mode-evidence registration controls keep their verdicts; the
missing-decisions control now fails on 'without a question'.
- plan-mode-targeting.test.ts stores known-good (announcement; paraphrase plus
seed content) and known-bad (neither; an untracked token) observations and
plan files with and without ## Decisions.
The outcome, no fabricated Completeness: N/10, stays asserted. The required
note now matches any brief wording of the same fact (KIND_NOTE_RE in
helpers/plan-format-kind-note.ts). The format-compliance suite that pins the
declared kind-note string is unchanged. Stored lines: four known-good pass,
three known-bad (a completeness score, a recommendation, 'kind to the user')
fail.
recordsNonTestCallerSearch keeps the value-card contract (non_test_callers
recorded) and accepts git grep, rg, grep or the Grep tool as the search.
Severity checks and the golden exclusion are unchanged. Stored evidence:
four known-good (three the old check rejected) pass; three known-bad (no
search, no caller count, prose only) fail.
prCoverageRequestViolations keeps the skill's stated older-open-PR budget
(five metadata pages, fifty file-list pages plus recent PR 7) and the coverage
outcome (PR 42's file list read from page 1 through page 2). It drops the PR 42
three/four-page and single-first-page ceilings, which were tuned to one
observed capture with a truncated first view. The judge criteria are
unchanged. Stored endpoint lists: three known-good (including a re-read first
page the old ceilings rejected) pass; four known-bad plus an over-budget
file-list run fail. incompleteFirstFileView is no longer used by the paid case.
The regression is a candidate silently dropped to fit the four-option cap.
The scope validator already requires an offered Include/Defer/Cut disposition
for all five candidates; the extra N-1 call floor failed agents that batch
compatible candidates, which the test header calls legitimate. FLOOR now comes
from CEO_SPLIT_CALL_FLOOR = 2. Stored shapes (ceo-split-floor.test.ts): five
sequential, 2-call and 3-call batches pass; a dropped candidate fails with
'missing target decisions'. The registration controls keep their verdicts;
the 3-call plan_ready scenario still fails, now on missing decisions.
Completion now needs at least one allowed Edit, the correction in place, at
least one verifier run whose final status is 0, and no edit after the final
verification. The ending-error case still requires a single verifier run
(status 69), because that fixture's error tells the actor to stop without
retrying. Ownership acquisition/release checks are unchanged. New free
protocol controls: a passing re-verification passes (the old grader failed
it); a failing final re-verification and a retry after the ending error both
fail, as they did before.
The false-completion oracle kept being patched to tell historical review-log
mentions from current claims. The E2E prompt now asks for a closing JSON line
{"outside_review_this_run": ...}. When present, the coverage claim is that
report ('completed' fails) plus any undenied 'both reviewers agree';
'disabled' also satisfies attribution. Runs without it keep the existing
prose oracle, so all stored captures and mutation rows keep their verdicts.
Dispatch, CLI execution, preflight and persisted-disabled checks are
unchanged. Stored outputs: three known-good pass (the old grader rejected
the table form); four known-bad fail.
The diagram grammar grew through four waves of capture patches to accept
correct-but-different diagrams. The E2E prompt now asks for one closing JSON
line {"tested": [...], "untested": [...]}; coverageAuditVerdict passes the
outcome when processPayment is only in tested and refundPayment only in
untested (last summary wins; quoted or example summaries ignored). The legacy
grammar stays as a fallback, so every stored capture keeps its verdict
(all existing rows unchanged). Read evidence and capture success are still
required. Stored summaries: three known-good (which the old grader rejected)
pass; seven known-bad fail.
auqMachineFormatProblems() fails a first-question run when the
Recommendation: line is missing or empty, or when the brief does not label
exactly one option (recommended); recommendation substance >= 4 is unchanged.
ELI10, Pros / cons, ✅/❌ and Net: are still scored and logged but no longer
fail a paid case. scoreAuqFormat and the consistency suite are unchanged.
Stored briefs: three known-good (including one without emoji or Net) pass;
four known-bad (no or empty Recommendation, zero or two (recommended)) fail.
docsCommandAllowed now unwraps, on read-only commands only (pwd, ls, cat,
stat, sha256sum, Git reads): a leading 'cd <fixture repo> &&', '2>/dev/null',
a trailing '|| true', '--no-pager', '-C <fixture repo>' and '-c' with
core.pager=cat / core.quotepath / color.* settings. Lifecycle and script calls
stay literal so their failures cannot be suppressed. Writes, other repos,
--git-dir/--work-tree, config that can execute code, real composition and
redirects stay rejected.
Stored commands (fixtures/docsync-observer-commands.json): 11 known-good read
forms, which the old observer rejected, now pass; all 17 known-bad forms still
fail under the new and old observers. docsync-command-grammar moves its three
harmless rows to an accept test and keeps unsafe variants of them as rejects.
Drops the exact 'Keep <x> in scope' / 'Defer <x> to TODOS' labels, the
'D<n> — Keep … in scope, or defer' title, the '<plan>.md lists' phrase and the
2-of-5 keyword quota. Keeps the evidence bindings (native answer, session,
timestamps, source read, mode context, later withdrawal) and now requires: one
keep option and one defer option, the answer selects keep, a word of the kept
label names a listed plan requirement, no expansion language, and an ELI10 +
Stakes rationale of at least 20 words (so a bare mode label still fails).
Stored captures: both retained known-good runs pass; every existing known-bad
mutation still fails; four rewordings the old grader rejected now pass; three
new known-bad cases fail. The Net/Note single-brief checks (t02 #3) are
unchanged: the 'second sentence after Net' row adds scope ('Also add shared
views'), so it stays a known-bad.
Keeps the engine-routing checks (Aside driven, retired binary unused) and adds
the outcome: design-audit.md exists, names a seeded defect, and index.html or
style.css differs from the initial fixture commit. The style(design) commit
prefix is not required. This case still cannot gate the design-review
stop-condition rewrite (it self-skips without Aside and caps fixes in its
prompt), per the Eng item.
The old check passed if the words 'in-branch', 'pre-existing', 'math' and
'string' appeared anywhere in the final message, in any pairing. The prompt
now asks for a closing JSON map; helpers/ship-triage-labels.ts reads it (or
unambiguous per-failure lines) and the test requires truncate/string ->
in-branch and divide/math -> pre-existing, plus a Bash call that ran the
suite. Stored outputs show swapped, words-only and one-label runs fail.
Drops the untyped recordE2E extras (typecheck baseline shrinks by one).
The workflow, first-run and review-gate prompts drop the 'IMPORTANT … simulate:
1. Detect … 2. Infer … 3. Note …' scripts and the answers they supplied
(platform, app, URL, merge method, 'Eng Review should be NOT RUN'). They keep
the facts the fixture cannot provide (no remote, no PR, gh/fly unavailable,
gstack-review-read output, report directory). Assertions now check the outcome:
the artifacts name Fly and test-app, and the readiness report records the
missing review. Canary, benchmark and setup-deploy prompts are unchanged.
- CLAUDE.md fixture: the shipped ## Skill routing section replaces 'ALWAYS
invoke it via the Skill tool as your FIRST action' and its hand-written rules.
- scrape-match-path: prompt is 'Run /scrape latest hacker news stories'; the
'You MUST follow … 1. Run skill list … Do NOT enter the prototype phase'
script is gone. Asserts the outcome: the bundled skill ran, no goto
prototyping, no new skill or skillify staging written. Listing is logged.
- scrape-prototype-path: drops the five-step script and the self-fulfilling
'Suggest /skillify'; keeps the user's JSON shape request.
- skillify-provenance-refusal: prompt no longer dictates the refusal.
- Narration grader moves to helpers/skill-body-narration.ts: line-start
'I '/'I'll '/'Let me ' plus mid-sentence 'let me <verb>', so 'let me know'
is legal. Stored known-good/known-bad bodies show it still rejects every
known-bad case the old /Let me /i ban caught.
Replaces the shouted 'ALWAYS invoke it … FIRST action' fixture and its
hand-written context-save/restore rules with the section gstack appends to
CLAUDE.md, which already routes save/resume to those skills.
readShippedSkillRouting() parses the '## Skill routing' section from the
routing-injection heredoc in bin/gstack-skill-start at test time. The journey
fixture now carries that instruction instead of 'ALWAYS invoke … FIRST action.
Do NOT answer directly', and still withholds the per-skill rule list so a
regressed description can fail. bin/gstack-skill-start and the helper join the
journey, context-skills and skillify touchfiles. A free check catches heredoc
drift. Runs on unchanged templates set the new baseline.
Drops the IMPORTANT block at skill-e2e-workflow.test.ts:76-83, including
'auto-approve everything' and the CHANGELOG/VERSION instructions; keeps only
the environment facts (non-interactive, no remote). Asserts the outcome: the
original CHANGELOG entries remain, VERSION still reads 1.1.1, no commit after
the fixture head touches VERSION, and README mentions Feature C.
Every ON trial must pass the fixture's verify (already a valid-measurement
requirement) and lowerIsBetter20Pct must hold; when the OFF median is 0, every
ON trial must also be 0. assessOverlayArms enforces both through a new
fixture.gate. Overlay contract bumps to v3 so v2 records are never
reinterpreted. overlay-measurement gains rows for correct+worse (fails),
20%+incorrect (fails) and the zero baseline (passes only at 0). The prose
comment follows the s01 M1 claude.md wording.
garrytan