mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-12 19:44:48 +01:00
ca395095a19cffa721db7ed966498cdffd29a456
389
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 garrytan
|
ca395095a1 |
test(W7h): delete the codex-offering prompt builder left unused by the retirement
Dead-code follow-up to the codex-offered-* retirement: buildCodexOfferingPrompt (test/helpers/codex-offering-fixture.ts) had no caller besides those cases. Deletes the helper, its free test (test/codex-offering-fixture.test.ts), the two captured public fixtures it replayed, its scripts/free-test-durations.json entry, and the helper/fixture paths still listed as touchfiles of six plan review E2Es in test/helpers/touchfiles-data.ts. Retirement card: - test: codex-offering-fixture.test.ts (pinned the builder's prompt text and replayed captured timeout/CDD transcripts against it). - detects: changes to a prompt that no paid case sends any more. - non_test_callers: none (grep -rn codex-offering test scripts is empty). - replacement: none needed; the behavior it supported is now the free probe check in gen-skill-docs.test.ts. - validation: bun test test/touchfiles.test.ts test/touchfiles-map-diff.test.ts test/paid-shards.test.ts test/e2e-tier-alignment.test.ts; bun run typecheck. |
||
|
garrytan
|
fc9049a8b5 |
test(W7h): retire the paid SKILL.md-summary cases in skill-e2e-plan
W7h retirement (t14 F6-F8): office-hours-spec-review, plan-ceo-review-benefits
and codex-offered-{office-hours,ceo-review,design-review,eng-review}. Each
asked a model to read a generated SKILL.md, write a summary, and grepped the
summary for facts the generated file states directly.
Registration cleanup in the same commit: E2E_TOUCHFILES, E2E_TIERS (all six
gate) and E2E_KINDS (rule) in test/helpers/touchfiles-data.ts; six
scripts/paid-test-durations.json case-shard entries; no
test/helpers/periodic-exclude-data.ts entries existed. The fast PR profile
(scripts/test-pr-profile.ts) drops plan-ceo-review-benefits and
office-hours-spec-review from PR_PROFILE_CASE_IDS and PR_PROFILE_FILES. Tests
that named the retired ids as live examples move to existing ones:
paid-free-boundary and paid-pr-profile (a CEO template change now plans
auq-format-gate plus the CEO judge; the case-shard fixture uses
plan-review-report), test-pr-profile, touchfiles (CEO selection count
21 -> 19), the touchfile-moved-code w5 golden (retired ids removed), the
behavior-lane case-shard census (78 -> 72) in cookie-validation-phases, and
the skill-e2e-plan timeout list and full-gate PR floor (78,425 -> 72,755 s)
in paid-retry-supervision. e2e-tier-alignment, eval-kinds,
eval-detach-timeout-floor and paid-shards stay green.
Coverage consequence: per the Eng Retirements item, plan-ceo-review-benefits
was the only gate-tier CEO E2E in the fast PR profile, so CEO invariants are
treated as periodic-only under the W7c gate (every CEO conversion in this
branch keeps one free check per invariant); the gate judge
'plan-ceo-review/SKILL.md modes' still runs for CEO prompt changes. An
office-hours prompt change now has no fast PR E2E and is reported as
deferred to broad coverage rather than run.
Retirement cards:
- office-hours-spec-review: detects the Spec Review Loop's stated dimensions,
dispatch tool, iteration cap and metrics. Replacement: gen-skill-docs.test.ts
'SPEC_REVIEW_LOOP resolver' (round-3 cap, prepare/check/finalize commands,
verdict and round-file wiring) and test/office-hours-review.test.ts (verdict
parsing). Control: changing the round limit in
scripts/resolvers/spec-review.ts turns gen-skill-docs red.
- plan-ceo-review-benefits: detects the CEO /office-hours prerequisite and
mid-session offer. Replacement: gen-skill-docs.test.ts 'plan-ceo-review
offers /office-hours when no design doc exists and detects a lost user
mid-session' (previous commit). Control: 4/4 red mutations there.
- codex-offered-* (x4): detects the `command -v codex` probe, a fallback and
optionality. Replacement: gen-skill-docs.test.ts 'Codex outside voice is
offered behind an executed availability probe' (probe + not_installed
fallback for all four skills) and skill-validation.test.ts 'plan reviews run
the Codex outside voice default-on'. Control: 4/4 red there.
- non_test_callers: none; history:
|
||
|
garrytan
|
51f1d947ee |
test(W7h): pin the CEO /office-hours prerequisite structure free
Replacement check for the plan-ceo-review-benefits paid retirement (t14 F7). The paid case had a model summarize plan-ceo-review/SKILL.md and grep the summary for "office hours" and "design doc". The rendered text is static, so this free test checks it directly: the Prerequisite Skill Offer triggers on the design-doc check's "No design doc found", offers 'A) Run /office-hours now' and 'B) Skip — proceed with standard review', does not re-offer later in the session, and Mid-session detection (0A) offers `/office-hours`. Per the Eng Retirements item, CEO invariants become periodic-only once this gate case retires, so they keep this structural free check. Control (scratch worktree, generated plan-ceo-review/SKILL.md): removing the /office-hours option, the no-design-doc trigger, the mid-session offer or the no-re-offer rule goes red; rewording the pitch stays green (5/5). Log: .context/mutation-ceo-benefits.md. |
||
|
garrytan
|
d3491c1734 |
test(W7h): pin the Codex availability probe free for the four offering skills
Replacement check for the codex-offered-* paid retirement (t14 F8): each of office-hours, plan-ceo-review, plan-design-review and plan-eng-review must render the executed `command -v codex` probe and its `not_installed` fallback mode in its generated SKILL.md + sections. These are bash lines the agent runs, so a deterministic grep replaces a paid model summary of them. skill-validation.test.ts already covers the default-on outside voice for eng/CEO/devex; this adds office-hours and plan-design-review and the fallback token. Control (scratch worktree, generated files): removing the probe from office-hours, plan-design-review or plan-eng-review, or the CEO not_installed branch, turns the new test red (4/4). Log: .context/mutation-codex-offering.md. |
||
|
garrytan
|
d636245c38 |
test(W7h): retire the bws session-awareness (ELI16) paid case
W7h retirement (t14 F5). Removes the case from test/skill-e2e-bws.test.ts and
its E2E_TOUCHFILES / E2E_TIERS (gate) / E2E_KINDS (rule) entries in
test/helpers/touchfiles-data.ts; scripts/paid-test-durations.json is
per-file and test/helpers/periodic-exclude-data.ts had no entry.
test/touchfiles.test.ts selection examples now name sibling cases with the
same touchfiles (skillmd-outside-git, skillmd-no-local-binary).
e2e-tier-alignment, eval-kinds and touchfiles stay green.
Retirement card:
- test: skill-e2e-bws.test.ts 'session-awareness' (gate, paid).
- detects: nothing gstack ships. The prompt coaches a mode that no longer
exists ("_SESSIONS=4, so ELI16 mode is active ... Re-ground them";
`git grep ELI16` outside test/ and CHANGELOG is empty) and says "DO NOT
actually call AskUserQuestion"; every asserted word (billing/acme,
payment/feature, stripe/checkout, recommend) comes from the prompt itself.
- non_test_callers: none.
- search_command: git grep -n "session-awareness\|ELI16" -- test scripts
- stronger_proof / replacement: the AUQ format's grounding line
'Project/branch/task: <1 short grounding sentence using _BRANCH>' is pinned
free by plan-review-calibration.test.ts (format tokens) and graded by gate
E2E auq-format-gate; session identity in the preamble stays pinned by
skill-validation.test.ts 'contains session awareness' and
gen-skill-docs.test.ts 'session awareness lives in gstack-skill-start'.
- history:
|
||
|
garrytan
|
8ac565d615 |
test(W7h): replace workflow-judge-cache source-grep counts with an invariant
W7h retirement (t09 F16, t13 F6) of the source-grep counts in 'workflow registration preserves model work and reserves only terminal-recording grace'. The stage-order check over runWorkflowJudge (the cheapest guard) and every behavioral callback test stay. Retirement card: - test: workflow-judge-cache.test.ts pins of skill-llm-eval.test.ts source: "execution: reused ? 'reused' : 'executed'", 'const workDeadline = started + JUDGE_MS;', 'const WORKFLOW_JUDGE_RECORD_MS = 5_000;', 'const WORKFLOW_JUDGE_TEST_MS = JUDGE_MS + 10_000;', "testName: 'review/SKILL.md workflow'", and the magic counts /}, WORKFLOW_JUDGE_TEST_MS\);/ x17 and /}, JUDGE_MS\);/ x7. - detects: any workflow eval added or removed, any constant reformatted. - non_test_callers: none. - stronger_proof / replacement: (a) a new invariant in the same test: every `await runWorkflowJudge(` registration closes with WORKFLOW_JUDGE_TEST_MS (the property the 17-count stood in for, now independent of how many evals exist); (b) the real-callback tests in this file exercise execute-once, reuse with provenance (execution reused/executed), timeout, supersede, expiry and the recording deadline. - history: |
||
|
garrytan
|
93d2d43c2c |
test(W7c): shrink gen-skill-docs to structural and meaning checks
W7c conversion of the sentence/caps pins in test/gen-skill-docs.test.ts (t01 F2-F9, F18, F19; t08 F12; t10 F4; t02 #5 duplicate STOP literal). t01 F1 ('Every commit must map to at least one bullet point') was already gone on the base. Pins (grep -cE 'toContain|toMatch|indexOf|includes') 1446 -> 1360; tests 457 -> 457. Untouched: the FRESH/STALE freshness check, the routing soft-policy negative, skill-start/end script pins, the SESSION_KIND spawned line, and the old-wording negative pins (delete-then-append, 0D/0F, gate "if applicable"). Machine markers stay byte-exact: NO UNRESOLVED DECISIONS, `**UNRESOLVED DECISIONS:**`, `Approval readiness: PASS`, `D<N> — <ROW-ID>: ...`, `## EXIT PLAN MODE GATE (BLOCKING)` anchors, `**Gate outcome: Blocked**`, helper commands/flags (gstack-office-hours-review prepare/check/finalize --design, --out-dir, --report, --unreviewed), helper stop reasons, analytics commands. Consent, stop, approval, decision-cycle and navigation lines (incl. the Eng W7c-gate lines at the old 2362-2370) are kept as short case-insensitive checks. Removed assertion groups (invariant -> remaining protection): - qa-only 'Never fix bugs' + 'NEVER fix anything' (t01 F8): report-only skill -> kept free: /never fix/i in Additional Rules + frontmatter no Edit/Glob/Grep. - qa 'WTF' token (t08 F12): fix-loop self-check -> kept free: /every 5 fixes/i. - TEST_COVERAGE_AUDIT sharedPhrases sentences and the three Test-step sentences (t01 F4): read source before tracing/diagramming in plan and ship -> kept free: order 'concrete source and test files' < Trace data flow < Diagram the execution < Quality scoring rubric in both, star/GAP markers, the plan/diff mode split, the Test review subheading structure test; gate judges 'plan-eng-review/SKILL.md sections', 'ship/SKILL.md workflow'. - 'REGRESSION RULE' / 'IRON RULE' labels (t01 F4): regression coverage is required -> kept free: REGRESSION RULE heading, /without regression coverage, that coverage is a critical requirement/i (plan), /regression test is written immediately/i (ship), plus the existing regression-contract tests. - '**STOP for each pending decision.**' (t01 F7, retired) -> kept free: 'Wait for its answer before applying that remedy' (control C1). - F19 rule sentences: step numbers dropped; kept free /prevent duplicate bumps, pushes and PRs, never verification/i, /native result never supplies outside coverage/i, /required-probe gate, never silently waive/i, /never offer a conversion/i, /actual counts, never estimates/i (deduped 2 -> 1). - SPEC_REVIEW_LOOP (t01 F3, about 74 sentence pins): - office-hours dispatch sentences (Before EACH dispatch, parent Read does not deliver, string unchanged) -> kept free: prepare --design/--out-dir, order mktemp < prepare < round-N.prompt.md < `dispatch`, /all preceding valid round files in order/i, /read the entire prepared prompt/i. - reviewer prompt sentences -> kept free: document/verdict path and schema wiring, mode sections ('The Assignment', 'What I noticed...'), dimension order, short checks for coaching protection, no invented customer answers, minor findings included, identical JSON, classify every prior finding, open questions vs committed behavior. Dropped: 'Could an engineer implement this...' ban (positive design-and-coaching check carries it), absence-is-not- confirmation, unsupported-claims and labeling sentences -> gate E2E office-hours-spec-review; prior-status validation in office-hours-review.test.ts. - convergence evidence sentences, 'Recording concerns does not mark them fixed', Disposition, observation/attempt counting, 'unavailable score is null', 'missing or invalid verdict is ... never PASS', 'Preserve the failed output' -> office-hours-review.test.ts (convergence, finalize, invalid artifact vs --unreviewed, null quality_score) + gate E2E office-hours-spec-review; kept free: helper stop-reason order and round 3, /before fixing any findings or dispatching again/i, /on a stop, do not fix again or re-dispatch/i, /do not edit that generated section/i, finalize before /existing user approval/i, --unreviewed wiring, /do not fabricate JSON/i, /not an approval gate/i. - CEO spec-review sentences (Read Agent's tool definition, omit field, consume completed review, Keep both consistent, duplicated 'both complete labeled texts'/'all five dimensions'/'Preserve the failure'/'citing the owning input'/'## Reviewer Concerns', 'Read these documents' ban, strategy-only and narrow-decision sentences) -> CEO is periodic-only, so kept free: one short check per invariant (wait tool, no-wait resume, no advance while waiting, three launches, stop after third, stop the loop, missing score, null grade, SCORE 1-10, not persisted, mkdir/append stop, save-failure stop, ITERATIONS, FOUND/FIXED/REMAINING order, both inputs in full, partial-input refusal, summary-is-not-the-plan, 1-10 score, PASS rule, cite input) plus the existing order/count/command checks. - Design approval reconciliation (t01 F2, about 79 pins): - Pass 5 sentences and 'AskUserQuestion once per issue. Do NOT batch. Recommend + WHY.' -> kept free: /stop\W+AskUserQuestion once per issue/i located after four short reuse/new-evidence/first-decision checks; batching -> auq-format-always-loaded.test.ts PER_SKILL_RULES ('One issue = one AskUserQuestion call'). - duplicate 'Never group distinct issues.' pins (2 -> 1) and the rest of the Design gate 0 sentences -> kept free: 0 < 1 order, short checks per approval line, refresh < Read-back gate < rerun order; gate judge 'plan-design-review/SKILL.md passes'. - CEO readiness/gate/0D sentences -> kept free: one short check per approval line, readiness order, `Approval readiness: PASS`, gate order Read-only < Verify readiness < 1. Read, route order, /stop for the actual answer/i in step 4, `D<N> — <ROW-ID>` header, row fields; admin-route sentences -> ceo-workflow-clarity.test.ts kept checks. Dropped: 'No report or completion log is needed' (order check carries it), 'Save or present the updated plan', 'checked row IDs...' (marker kept), strategy-only/narrow-decision sentences (default kept), cross-model/integration sentences reduced to external-only and native-fallback checks. - Eng readiness/procedure sentences -> kept free: short approval checks, gate numbering/order, AUQ call < /stop until the actual answer arrives/i < Record the answer, recovery order; 'separate it and rebuild this comparison' -> auq-format-always-loaded.test.ts PER_SKILL_RULES. - Exit gate 'contract violation' sentence (t01 F9, design/devex/codex) -> kept free: heading anchor + /do not call ExitPlanMode/i inside the gate. - Unresolved-decisions gate (t01 F5): '(MANDATORY', 'Never omit'/'never omitted', 'FINAL non-whitespace line is the unresolved-decisions', 'FAILS the gate', 'DROP the current skill' -> kept free: NO UNRESOLVED DECISIONS, /final\s+non-whitespace\s+line/i, `**UNRESOLVED DECISIONS:**`, /missing\s+status [^.]*fails/i in every gate skill and the resolver, /avoids double-counting/i, the old soft-wording negative pin. - delete-then-append 'never mid-file' / 'Do NOT replace the section in place' (t01 F6) -> kept free: 'delete-then-append flow' + /append the new report at EOF/i; old-wording negatives kept. - design-html '**Never include by default (AI slop blacklist):**' (t01 F18) -> kept free: /AI slop blacklist/i, catalog id comment, lib/design-catalog.ts. - CODEX_BOUNDARY 'Do NOT read or execute any' casing (t10 F4) -> kept free: /do not read or execute any/i in every codex-calling skill. - CEO 0D order anchor '**STOP for the actual answer...**' (one of the 8) -> kept free: /stop for the actual answer/i at the same position. Mutation pass (scratch worktree, generated SKILL.md/sections plus spec-review.ts and lib/office-hours-review.ts; the freshness test is excluded with -t because any mutation makes it STALE): 43/43 as expected. 25 kept-step, marker, safety, numeric and retirement-control mutations go red; 18 rewordings of dropped sentences and lowered-register STOP/NEVER/FAILS lines stay green. Log: .context/mutation-gen-skill-docs.md. Runtime (Bun 1.4.0, wall, 3 runs): before 4.42/4.49/4.70s (median 4.49s), after 4.42/4.53/4.63s (median 4.53s). |
||
|
garrytan
|
af697dc6ec |
test(W7h): retire the paid first-task-scaffold E2E duplicate
W7h retirement (t12 F12). Removes the case from test/helpers/touchfiles-data.ts
(E2E_TOUCHFILES, E2E_TIERS periodic, E2E_KINDS rule) and both
scripts/paid-test-durations.json entries; test/helpers/periodic-exclude-data.ts
had no entry. e2e-tier-alignment, eval-kinds, eval-detach-timeout-floor,
periodic-exclude-policy, touchfiles and paid-shards tests stay green.
Retirement card:
- test: test/skill-e2e-first-task-scaffold.test.ts ('first-task-scaffold',
periodic, ~$0.02): Haiku is told to run exactly one detector command and the
test greps the tool result for code_node / greenfield.
- detects: detector bucket regressions in bin/gstack-first-task-detect.
- non_test_callers: none.
- search_command: grep -rn "first-task-scaffold" test scripts .github
- stronger_proof / replacement: test/preamble-first-task-scaffold.test.ts
(free; executes the real detector on fixture repos for every bucket,
including code_node and greenfield, plus first-run gating and telemetry in
bin/gstack-skill-start). The model added nothing: the prompt dictated the
exact command.
- history:
|
||
|
garrytan
|
958f27a2b8 |
test(W7h): retire the literal 75 KB autoplan budget duplicate
W7h retirement (t04 F18). test/skill-preflight-budget.test.ts asserted the
same four-skill autoplan sum twice: once against
AUTOPLAN_PREFLIGHT_BUDGET_BYTES * 3 (25,600 x 3 = 76,800 = 75 KiB) and once
against a literal 75 * 1024, which would drift silently if the constant
changed.
Retirement card:
- test: skill-preflight-budget.test.ts 'autoplan total under 25 KB
instruction cap × 3 (75 KB instruction budget)'.
- detects: the four plan-* skills' brain preflight + refresh + write-back
instructions exceeding 75 KiB.
- non_test_callers: none.
- stronger_proof / replacement: 'autoplan: sum across 4 plan-* skills stays
under AUTOPLAN_PREFLIGHT_BUDGET_BYTES × 3 (instructions)' in the same file,
which reads the constant from scripts/brain-cache-spec.ts.
- history:
|
||
|
garrytan
|
6423617fe6 |
test(W7h): retire spec-template-sync as a duplicate of the freshness check
W7h retirement (t05 F18; Eng factual correction: the file takes 145 ms, its "2 minutes" was the 120 s timeout, so this is a duplicate-only retirement). Removes its scripts/free-test-durations.json entry. Retirement card: - test: test/spec-template-sync.test.ts (regenerate /spec into an out-dir and compare spec/SKILL.md + spec/sections/gate-and-file.md with the tracked bytes; check the generated header). - detects: /spec template edited without regeneration, or a hand-edited generated file. - non_test_callers: none. - search_command: grep -rn spec-template-sync package.json scripts test .github - stronger_proof / replacement: gen-skill-docs.test.ts 'generated files are fresh (match --dry-run)' (FRESH for every skill, no STALE anywhere; the dry-run covers carved sections such as spec/sections/gate-and-file.md); the generated-banner checks in the same file cover the header. - history: |
||
|
garrytan
|
d2f53fcb9c |
test(W7h): retire CEO TODO-menu prose pins from plan-review-calibration
W7h retirement (t04 F11; W7c list). The calibration corpus and
buildPlanReviewDecisionPrompt never read this CEO prose; the AUQ format
tokens (consumed by plan-tune and judges) and the calibration-harness
outcome tests are unchanged.
Retirement card:
- test: plan-review-calibration.test.ts, the six CEO TODO-menu pins ('Only
unanswered TODO proposals reach this menu', 'Do not ask again about an item
already deferred, skipped or kept', 'carry its actual answer and destination
forward', 'Resolve each remaining proposal through all four steps of 0D,
using the menu below', 'Never batch TODOs — one per question').
- detects: edits to plan-ceo-review TODO-menu prose.
- non_test_callers: none (test-only reads of the template).
- search_command: grep -rn "Never batch TODOs\|Only unanswered TODO" test/
- stronger_proof / replacement: skill-ceo-section-ordering.test.ts 'TODOS.md
updates' slice keeps /i meaning checks for unanswered-only, no re-asking
and one TODO per question (CEO invariants are periodic-only under the W7c
gate, so they keep a free check); 'route through 0D' stays covered by that
file's 0D route checks.
- history:
|
||
|
garrytan
|
561b82e861 |
test(W7c): shrink plan-review-cases to structural and meaning checks
W7c conversion (t03 F1-F4; t02 F4/F5 duplicate STOP/threshold pins). Pins
713 -> 208 lines by the brief's grep (toContain|toMatch|indexOf|includes);
prose assertions 592 -> 366 (toContain/toMatch calls plus expectAll regex
rules; expect() calls per run 3232 -> 2808). Tests 218 -> 215. t03 F1 (stale
~/.gstack/projects discovery path) was already fixed by W7a on the base; its
$GSTACK_STATE_ROOT check and the stale-path ban are kept.
Kept byte-exact or structural: marker-order arrays (### Prepare an unanswered
choice -> ### Send once and wait -> ### Record the answer, heading list
equality, Pending-record checkpoint, finish steps 1-6, entrypoint recovery
order, output placeholder order), the parsed write-policy table routes, the
generateAntiShortcutClause host matrix, every not.toContain(stale) ban,
numeric limits (8+ files / 2+ new classes, fewer than 8 / 2, under 5 words,
2-3 options, six-column / five-field / four-step), consumed tokens and field
names (`scope accepted as-is`, `scope reduced per recommendation`, FULL_REVIEW/
SCOPE_REDUCED mapping, **issues_found**, SOURCE=/OUTSIDE_STATUS=/STATUS=,
OUTCOME=, `BLOCKED`, `## GSTACK REVIEW REPORT`, `currentDecision`, D<N> and
ROW-ID formats, prescribed A-D menus), placeholders, commands/paths and all
picker behavior tests (unchanged). STOP/wait/approval/safety lines become
case-insensitive meaning checks located between their headings (e.g.
/stop until the actual answer arrives/i between Send once and Record the
answer; /do not replace the section in place/i plus /append the new report
at EOF/i). CEO invariants (periodic-only once plan-ceo-review-benefits
retires) each keep one free check. Eng/Design/DX invariants keep one free
check too; nothing below relies on a periodic-only E2E.
Removed assertion groups (invariant -> remaining protection):
- CEO route/narrow depth (:19) full sentences: admin answers approve nothing,
plan decisions run steps 1-4, settled choices go to 0E -> kept free:
Admin < Plan decision order, checkpoint order, five short /i checks, stale ban.
- Eng evidence paths/labels (:33) 'Prior Learnings configuration', path rule
sentences -> kept free: the three path field labels, /never invent paths/i,
`not available`, label tokens, stale ban.
- Eng first report save (:47) 9 requirement sentences -> 5 short /i
requirements with the same remove-one-must-throw self-control; parent-dir,
original-plan-copy and scope-record clauses dropped (initialization
invariant kept by target-header/preserve/no-findings/placement/terminal-
report checks; collision suffix kept in the write-policy test).
- Eng setup selectors (:79) and its duplicate in the big Eng ordering test:
selectors approve no remedy -> kept free: /these selections approve no
engineering remedy/i, B selectors, no pre-answer ledger, stale ban; the
'For the next choice' sentence -> kept free: transaction marker order.
- Eng performance (:94) exact bullet list -> kept free: four concern
keywords from the parsed bullets; Decision-procedure sentence -> kept free
in the four-section test.
- Eng complexity/MODE (:105) per-target counting, 'Count each once', 'Keep it
current', 'Continue to Section 1' sentences -> kept free: count-selected-
work, both thresholds, MODE tokens and mapping, scope-reduction meaning,
'approves no pending remedy', Continue-to-Section-1 in the C order array.
- Eng three-stage transaction (:125) sentence markers -> same 13-step order
array with short /i anchors and the remove-one-must-throw control.
- Eng Test Plan refresh, coverage trace (:150, :162) sentences -> kept free:
placeholder count/order, prepare step count, short /i checks; coverage audit
also graded by plan-eng-coverage-audit (gate).
- CEO fallback (:178) sentences -> kept free: host mode token matrix, status
field values, /final report is written later in Required Outputs/i, ban.
- Persistence per skill (:229) Eng duplicates ('If the required log is
forbidden', 'Neither supplies', FULL_REVIEW mapping, 'after successful
Read-back') -> kept free in the readiness, complexity and write-policy
tests; CEO 'retain its diagnostic' -> kept free: /do not claim that entry
was recorded/i and three other CEO log checks.
- Report resolver per host (:261) duplicate 'Do not choose another
destination', 'Review record and write policy', 'full review output',
explicit/named/active sentences -> kept free: target precedence order;
CEO writer's seven sentences -> kept free: six-anchor order ending at
'4. **Read-back gate:**' plus the blocked-return check.
- Generated carriers (:327) bold STOP pin -> /stop until the actual answer
arrives/i in the dispatch order array.
- Big Eng ordering test (:358) duplicates of the gate describe (one question
object, 8+/STOP, both-retain, setup, separate instrumentation, keep chosen
values, calibration<procedure<scope, Decision procedure < Architecture,
outside-voice consent lines) -> kept free once each (boundary list, queue
test, preparation order array now ending at ### 1. Architecture review);
'Use existing tests ... bounded probes', 'post-answer scope summary',
'Compare only the file/class arrangement', 'contracts and approved
security...', 'With no proposed cuts', 'Present numbered...' sentences ->
kept free: build-boundary order check, structure-question and
unapproved-fixes checks, C-step order array.
- Eng gate describe: exact-duplicate pins (same sentence 2-4x in one test or
repeated per section inside the 4-section loop) -> one copy kept; ledger
placeholder descriptions (Plan baseline/Runtime evidence/Actual answer/
Accepted scope text) -> kept free: parsed ledger field-name list, R1 header,
State values, A)/B) selectors, ledger position.
- Gate tests 'identifies commitments before comparing values', 'common new
defaults...', 'exact prior answers authorize...', 'finding evidence, stable
decision identity...' merged: every non-duplicate invariant (optional
depths, shared new value needs approval, unknown risks visible, drafts grant
nothing, reopened ID keeps D<N>, test scores not answer status) -> kept free
as one /i check in 'current contracts...' or 'all four section gates...';
their order checks -> kept free: prepare<identify<compare<save<send<record.
- Remove-old-occurrence, 'Otherwise leave the remedy pending', 'no question
or comparison grid is needed', 'never the saved-report gate', 'Compare the
label separately', 'When revising, replace the whole current payload',
'It may be the selected plan', 'Present each completely as not persisted',
'If none is permitted...', 'A forbidden write follows its row', 'Run
Learning hooks ...', 'Refresh affected tests...', 'If startup values...',
'instead of creating another approval row', 'Keep that remedy pending' ->
same invariant kept free by an adjacent check (State replace/once checks,
approves-no-implementation, factual-correction, recovery routing, payload
rebuild, write-policy routes, return-to-entrypoint count, late-change and
blocked checks, common-work check).
- Worked example anchor sentence -> anchored on the Compare section and
'Ask about jitter first:'; parsed R1/R2 table equality kept.
- Outside-voice queue: dead generic branch (all three skills `continue`
before it; never executed) removed -> none needed (unreachable). Eng
procedure re-checks inside the queue (5 sentences) -> kept free in the
gate describe. Shared CEO/Eng/DX scope-revision sentences hoisted to one
shared /i block per skill; CEO 'Keep candidate scope ...' etc. -> kept free
as short /i checks; prescribed A-D menus kept exact.
- Design register (:1771) sentences -> kept free: grounding-before-question
order and three /i checks; 'Surface real missing decisions' dropped
(positive restatement of the same rule).
Gate-tier coverage that also grades this prose: judges 'plan-eng-review/
SKILL.md sections', 'plan-design-review/SKILL.md passes', 'plan-ceo-review/
SKILL.md modes'; E2Es plan-eng-coverage-audit, plan-review-report,
plan-devex-review-plan-mode (gate). Cosmetic pins dropped without controls:
bold/caps STOP and NOT, issue-number comments unchanged.
Mutation pass (scratch worktree; eng/ceo/design templates and the
outside-voice, testing and review-dashboard resolvers): 33/33 as expected;
24 kept-step/marker/safety/numeric mutations go red, 6 rewordings of dropped
sentences and 3 lowered-register STOP/NOT lines stay green. Log:
.context/mutation-plan-review-cases.md.
Runtime (bun test, Bun 1.4.0, median of 3): 0.85 s -> 0.77 s.
|
||
|
garrytan
|
6a83ba86c3 |
test(W7c): shrink skill-ceo-section-ordering to structural and meaning checks
W7c conversion of test/skill-ceo-section-ordering.test.ts: t06 #1-#6, t05 F5 (cross-reference: the remaining bold-STOP pins), t13 F3 / W7.md CRITICAL RULE anchor. #1 (ASK_QUESTIONS_HEADING anchor) was already on the base branch; #5 now slices the blocked chat return to the next "\n## " heading, so removing or renaming `## Review Log` cannot widen or break the window. Pins (grep -cE 'toContain|toMatch|indexOf|includes'): 647 -> 388 lines; toContain( 535 -> 136; expect() calls per run 1657 -> 1399. The carve block, section-capture fixtures and validateCeoReviewCompletion tests are unchanged (t06 keep list) except two exact-duplicate 0D pins. CEO invariants count as periodic-only (plan-ceo-review-benefits retires), so every invariant below keeps a free structural or case-insensitive check. Gate-tier judge that also grades this prose: 'plan-ceo-review/SKILL.md modes'. New helpers: ordered() (exact labels/headings/placeholders/table cells as strings, meaning anchors as /i RegExp) and between() (fails loudly on a missing end anchor instead of silently running to EOF). Removed assertion groups (invariant -> remaining protection): - t06 #2 / t05 F5: '**STOP for the actual answer, even for a lone option.**' (x6 exact pins + 3 split/indexOf anchors) and '**STOP for the answer**' (x3): wait for the actual answer before recording -> kept free: one /stop for the actual answer/i per slice (template 0D in dispatch order between the ask-one-row check and the answer save; every generated host carrier between **4. Ask and **Post-answer checkpoint:**), /stop for the answer/i in 0E step 3 (template and generated), plus the Pre-question < 4. Ask < Post-answer checkpoint order and the 2-checkpoint count. Dispatch slices now end at **Post-answer checkpoint:**. - t06 #4 'Recommendation is ALWAYS present': recommendation label contract -> kept free: '(recommended)' and 'AUTO_DECIDE depends on it' in generated SKILL.md; exactly-one-(recommended) rule in 0D's currentDecision table. - t06 #6 and other bans of retired phrasings ('If the section has findings, you MUST call AskUserQuestion', 'Otherwise, use AskUserQuestion for each finding', 'After each section, pause and wait for feedback', 'Save pending rows before comparisons', 'using the Step 0E mode-handoff format', 'Save a chat-only plan', 'quality bonus, not a gate', 'If no file is in scope, skip this section', 'always write', 'For every question', 'for architecture choices', 'These approaches differ in coverage (...)', 'Finish with ExitPlanMode', 'Before summaries, review logs or next-step menus...', "put the complete comparison in the question's brief", 'Use this exact route:', 'citing its ID', /\blog (?:with|that ID)\b/): same invariants -> kept free: the positive checks (one **Decision gate.** per section wired to Analyze → Resolve → Apply; pending-rows-before-comparisons order; **Otherwise:** in the 0H order; report not persisted + Read-back gate; write-when-permitted incl. zero tasks; Outcome routing count 1; record-provenance-after-handoff order; literal dispatch-object comparison). 'Do NOT proceed ... 0C-bis' becomes not.toContain('0C-bis') on 0D. Short stale-token bans stay (<approved 0D approach>, reference | commitment | current value, ___ methods, retry once, telemetry has logged, NUMBER + LETTER / "3A", the Forbidden-storage regex, removed menu option, unbound placeholders). - t06 #3, magic counts: literal 11 sections / toHaveLength(7) closing steps -> kept free: section numbers 1..N derived from '## Review Sections (N sections' and checked against the ### Section headings; closing-step count derived from the 7 route labels. - Decision cycle / storage / comparison sentences (Start at step 1, run steps 2–4 only when needed, mode-change pointer, storage-policy save sentence, Copy the grid, Validate every field, Show unchanged/shared/pending repeated x4, other rows stay fixed or pending x5, never prewrite x3, Build one currentDecision x3, Ask one row per call x6, Save the answer reference x5, Keep independent changes separate x4): full decision protocol in order -> kept free: label order 1. Check < 2. Record < 3. Compare < grid < Pre-question < Read-back < 4. Ask < Post-answer (template + every host), one /i check each for pending-rows-before-comparisons, save-before-ask, amend only authorized work, failed save stops, decline-all stop, show unchanged/shared/pending, independent changes separate. Admin/plan routes also kept in ceo-workflow-clarity.test.ts. - Handoff/provenance sentences (After selection, send brief chat, Keep rows separate, logging-row repeats, Section 1 current-scope sentences): mode handoff applies before questions and logs provenance after -> kept free: /before tools or further questions/i, handoff < Other selections < provenance < selecting-a-mode order, the two exact output-format lines with <rows or none> / <Application and rationale>, auto_decided true/false fields, 'No new approach decision was needed' (quoted chat output). Section 1 dispositions -> ceo-workflow-clarity's Section 1 test. - Scope/mode-selection sentences (REDUCTION, HOLD and SELECTIVE's HOLD checks; For both expansion modes...; Skip/Defer destination repeats; present both inputs for final approval; 0E counts changed files; Count all deliverables; >15/greenfield repeats; SELECTIVE balances...): scope decisions per item with exact destinations -> kept free: prescribed menu label orders (A Add/B Defer/C Skip; A Defer/B Keep; TODO A/B/C; depth A/B), Defer -> TODOS.md + NOT in scope, Skip -> no TODO, /ask separately/i, /all four 0D steps for each unanswered addition or deferral/i, routing rows and >15 / 8-file / 2-class numbers (once each; ceo-workflow-clarity keeps the reuse-count checks), carve block's document-approval order. - TODO menu (per plan-review-calibration t04 F11 retirement, which names this file as the remaining free protection): kept free in the TODOS.md-updates slice: /only unanswered TODO proposals reach this menu/i, /do not ask again about an item already deferred, skipped or kept/i, /one per question/i. - Resolver prose (spec review, plan report, tasks emit, exit gate, codex plan review, brain cache refresh, approval readiness, design clause): each sentence pin -> one short /i check per invariant on every host (five dimensions, at most three launches, required save stops, destination-file Read/Write order, zero tasks, unknown files JSONL format, ExitPlanMode only after checks, forbidden history, outcome routing and skip routes, native fallback never outside coverage); exact strings kept only for machine/output tokens (NO UNRESOLVED DECISIONS, '- + N unresolved from prior reviews', STATUS/SOURCE/OUTSIDE_STATUS assignments, CROSS-MODEL, OUTSIDE COVERAGE, 'N findings; R resolved; U unresolved', quoted report phrases, Gate outcome: Blocked, Read-back gate). Dropped: 'If both counts are zero', 'For **Outside Review**, use ...', 'Enter after either an external reviewer...', 'identify the causal mechanism', 'A credible material risk...', "check the assembled set's capacity", 'including findings that needed only factual correction', 'show unsaved fields and errors', Strategy-only tasks sentence, verified-report-can-pass sentence -> neighbouring kept /i checks in the same slice plus the gate-tier 'plan-ceo-review/SKILL.md modes' judge. - Review-section prose (continuity requirement lists, test-review and outside rule duplicates, strategy-only diagram sentences, Review facts repeats, Stage 3 publishes it, Count a reopened choice once, Keep them before the terminal report repeats, EXIT gate verifies..., no report or log needed yet, then run telemetry as the last review operation): per-section gate and ledger continuity -> kept free: Analyze < Resolve < Apply, Resolve paths via check-scope < correct < record order, /resolve critical risks now/i, /review only; do not change code/i, ledger header + status tokens, one /i per approval/stop line; reopened-once lives in ceo-workflow-clarity; terminal order kept as cache refresh < {{BRAIN_CACHE_REFRESH}} < Telemetry < finished < Call ExitPlanMode < handoff. - Duplicate pins removed outright (same string asserted 2-6 times across tests, incl. carve-block 'Ask one row per call' and bold-STOP pins): each keeps its single remaining check named above. Cosmetic pins dropped without controls: bold/caps (**STOP**, ALWAYS, MUST), double compactProse wrappers. Mutation pass (scratch worktree; templates, generated SKILL.md and review-dashboard resolver): 39/39 as expected; 28 kept-step/marker/ placeholder/safety/consent/numeric mutations go red (incl. the three TODO-menu lines), 11 rewordings of dropped sentences, the CRITICAL RULE heading rename, re-adding the retired #6 sentence, and lowered-register STOP lines stay green. Log: .context/mutation-skill-ceo-section-ordering.md. Runtime (Bun 1.4.0, 3 runs): before 2.05/1.84/1.77 s (median 1.84), after 1.87/1.80/2.08 s (median 1.87); dominated by runGeneration and the capture subprocess fixtures. |
||
|
garrytan
|
d0fc479073 |
test(W7c): check the codex boundary on meaning and its path list
W7c conversion (t02 #14, W7a host-config.test.ts:565). The boundaryInstruction bytes sent to Codex are unchanged; only the test loosens. Removed assertion group (invariant -> remaining protection): - toContain('Do NOT read') (caps fragment): Codex must not read or execute Claude skill definitions -> kept free: /do not read or execute any files under/i plus each excluded path ('~/.claude/', '~/.agents/', '.claude/skills/', 'agents/'), which is the actual contract. Mutation pass (scratch worktree, hosts/codex.ts): 3/3 as expected; replacing the prohibition or dropping an excluded path goes red; lowering IMPORTANT/NOT stays green. Log: .context/mutation-host-config.md. |
||
|
garrytan
|
a5c5162d33 |
test(W7c): match redact scan-at-sink exact-bytes rule case-insensitively
W7c conversion (t02 #13). Credential-handling checks (mktemp, --from-file, exit codes 3/2/0, visibility lookups, no HIGH skip flag) are unchanged. Removed assertion group (invariant -> remaining protection): - /EXACT bytes/ (caps): scan the exact bytes that will be sent -> kept free: /exact bytes/i in the same test, next to the mktemp and --from-file checks. Mutation pass (scratch worktree, scripts/resolvers/redact-doc.ts): 3/3 as expected; scanning a summary instead of the exact bytes or dropping the Exit 3 (HIGH) branch goes red; lowering EXACT stays green. Log: .context/mutation-redact-doc-resolver.md. |
||
|
garrytan
|
d30812a109 |
test(W7c): match retro clock rule case-insensitively, drop proximity regex
W7c conversion (t02 #11, #12). Removed assertion groups (invariant -> remaining protection): - /NEVER from `date`/ (caps): today comes from the session reminder, not the system clock -> kept free: /session reminder/ and /never from `date`/i. - /(?:disclosure[\s\S]{0,200}narrative|narrative[\s\S]{0,200}disclosure)/ (200-char word-proximity window): offline/local-only runs disclose that the window is not freshness-verified -> kept free: the exact disclosure string 'offline run, window not freshness-verified' (it reaches the retro output), RETRO_REF disclosure and the GUARD_* script checks; gate judge 'retro/SKILL.md instructions'. Mutation pass (scratch worktree, retro/SKILL.md.tmpl): 4/4 as expected; allowing `date` or deleting the disclosure string goes red; lowering NEVER and rewording the narrative clause stays green. Log: .context/mutation-regression-1624-retro-stale-base.md. |
||
|
garrytan
|
27dac76763 |
test(W7c): match document-release spawned-dispatch contract on meaning
W7c conversion (t04 F3; W7a run-in-background-guidance.test.ts:472,474, named by the Eng W7c gate as anti-injection and VERSION invariants that stay as case-insensitive meaning checks). The run_in_background flag pins, structural scanner and inverted-phrasing ban are unchanged. Removed assertion groups (invariant -> remaining protection): - 'NEVER trigger it on their own' (caps literal): prompt/file/tool claims do not enable spawned mode (anti-injection) -> kept free: /claims never trigger it on their own/i in the generated skill and template. - 'The NEVER-do invariants below do' (short form of a wrapped sentence that names the caps headings s10 rewrites): spawned auto-pick still never rewrites CHANGELOG or changes VERSION -> kept free: /skip any recommendation that rewrites CHANGELOG or changes VERSION/i (whitespace-normalized) in both files; gate E2E document-release. - 'A spawned run must never change VERSION': same rule, now /i. Mutation pass (scratch worktree; document-release skill, template and release-body section, ship/sections/documentation.md): 7/7 as expected; deleting the anti-injection clause, the CHANGELOG/VERSION carve-out, the spawned VERSION line, the run_in_background flag or the dispatch heading goes red; lowering NEVER and rewording 'NEVER-do invariants' stays green. Log: .context/mutation-run-in-background-guidance.md. |
||
|
garrytan
|
5b2bb3a1a5 |
test(W7c): match plan-scope-selection scope-gate stop lines case-insensitively
W7c conversion, scope-gate caps pins only (t06 #14). The scope classifiers (nativeSeededPlanSelection, isScopeGateAutoSelectVisible, isScopeGateQuestionVisible), the consumed announcement and A/B/C menu bytes, and every non-caps assertion are unchanged. Removed assertion groups (invariant -> remaining protection): - 'Reply with A, B, or C. STOP and wait' order anchors (design, template and every host expansion): menu precedes bootstrap -> kept free: the consumed prefix 'Reply with A, B, or C' (pty/classify.ts, pty/judge.ts), now also required to exist after '## Scope gate' so a missing menu cannot pass the order check with indexOf -1. - 'call AskUserQuestion FIRST and wait.', 'does NOT count as the mode signal', 'If the user explicitly named a DIFFERENT target' (design and eng), 'STOP and wait for the answer — only after the user picks', 'STOP and wait for the answer' (toContain and indexOf before the first ```bash on every host): ask first, pasted text is not a mode signal, explicit targets win, stop until answered -> kept free: the same rules as case-insensitive meaning checks, including the per-host ordering (search /stop and wait for the answer/i before the first ```bash, with a presence check). Mutation pass (scratch worktree; plan-design-review and plan-eng-review templates): 12/12 as expected; gate-after-preamble, removed design menu prefix, removed stop lines (design, eng), removed mode-signal, ask-first and explicit-target rules go red; lowering STOP/FIRST/NOT/DIFFERENT stays green. The eng menu prefix is not pinned here (eng-scope-entry-ap.test.ts M1 covers it). Log: .context/mutation-plan-scope-selection.md. |
||
|
garrytan
|
7cb97e61b3 |
test(W7c): shrink autoplan-phase-order to structural and meaning checks
W7c conversion (t06 #13; W7c list item 'Eng runs LAST, always', 'ONLY/FINAL'). Pin lines 203 -> 166. autoplan has only periodic E2Es (autoplan-dual-voice, autoplan-journal-drift: E2E_TIERS periodic), so every invariant keeps one free check: sentence pins collapse to short /i meaning checks; commands, CLI signatures, placeholders, tool parameters, report fields, packet outputs (code-generated) and stale-string bans stay exact. Removed assertion groups (invariant -> remaining protection): - 'Eng runs LAST, always' (caps sentence): Eng is the terminal phase -> kept free: /CEO → Design.*→ DX.*→ Eng/s, heading order 1 < 2 < 2.5 < 3 and the 'CEO → Design → Eng → DX' ban. - Final-gate sentences ('...Use Decision Classification; ask once at Final Approval Gate, never mid-run', 'Queue clearly-wrong/challenged premises', 'The user decides there; never stop mid-pipeline'): premises queue for the single final gate -> kept free: /never auto-decide user challenges/i, /ask once at final approval gate, never mid-run/i, 'as User Challenges for Phase 4', /never stop mid-pipeline/i, the two old-gate bans. - Generated setup sentences ('Resolve this phase', 'Run all applicable skills and lazy sections fully', 'log successful ranges/total to EOF'): phases load at their own boundary -> kept free: /never prefetch future phases/i, /missing skill: report phase and setup repair/i, per-phase methodology CLI and Read-before-create order, no in-full-now Reads. - Intake load sentence ("Resolve this phase's source to absolute `<REVIEW_SKILL>`; load via its checkpoint") -> kept free: `<REVIEW_SKILL>` token and /never prefetch future phases/i; section load keeps `readRanges`/EOF/skip-listed checks. - Native dispatch sentences ("ONLY/FINAL tool call", 'Keep native Reads enabled', 'all criteria + plan', 'if its schema exposes it', 'Claude Code: end response immediately', 'Other hosts await that ID', "Then outside → this phase's review ONLY", 'No inline substitute; apply failure policy'): the native review blocks and completes before outside review -> kept free: native < outside order, run_in_background: false, isAsync: true, /final tool call/i, /end response immediately/i, /no further tool calls\/review until/i, `nativePromptPath` to EOF, INPUT-hash match, /retry invalid input once/i, /no inline substitute/i, 720000ms timeout, provenance placeholder, N/A and CONFIRMED status tokens, /primary cannot replace/i. - Sequential Execution sentences (Keep ONE phase active, compaction reload, 'reconcile saved artifacts and sent conversation messages separately', 'Publication is a separate operation', 'an earlier Read is not this close', "Then continue to the next phase's tool calls in the same turn", 'record verbatim into fenced history', 'retaining its original source SHA', 'compaction resumes the existing invocation', the full LOG/amend-input sentence): one phase at a time, publish before advancing -> kept free: /keep one phase active/i, /never draft future-phase reviews/i, compaction reload (/i), two ordered chains (load < snapshot < dispatch < consume < remaining sections; phase-close < parent message < driver advance < after Eng), skip/resume/step 6 (Publish) routes, open-gate, uptake, pending, never-skip and raw-transcript lines (/i), rerun fields `baselineEdits.record`/`sourceSha256`, `amend-input`. - Amendment-binding sentences ('load the shared close steps afresh, even if read earlier', 'Keep this checkpoint for this invocation; review exports do not replace it') -> kept free: checkpoint field per phase, `methodologyPath`, /afresh/i, /review exports do not replace it/i. - Shared-close sentences (~20: completion prerequisites, readRanges Read choreography, 'A Read of only the edited tail does not satisfy...', 'If a result is truncated...', 'If a Read fails...', 'regenerate the packet...', 'Retention checks prove bytes; counts, hashes...', 'Review history stays in Review record', 'Taste remains provisional', 'Apply the phase's failure policy...'): read the whole packet, verify, then publish -> kept free: the seven-step order, terminal-results/INPUT/pending/no-credit/User Challenges/None/fixed-checkpoint lines (/i), prepare-close CLI, `readRanges` with exact `offset` and `limit`, /through EOF/i, no advance without a result, step-3 return, verification compare/recheck/keep-open, stale ban, generated copy contains the template. - Publication sentences ('After sending the actual parent report', 'using actual findings and voice statuses', "The sent conversation message is step 6's output", 'This message is the next operation...', the duplicated driver pins) -> kept free: publish < report < return order, /send the filled report ... visible parent assistant text/i, /before any next-phase tool call/i, both-voices rule, same-turn/driver-alone/no-continue-wait/skip lines (/i), Bash-is-not-publication (/i), stale ban. - Report-format sentences ("the actual host's reviewer names", 'Include the DX metrics line only when ...', "Resolve `report.next` using the driver's applicable scope/skip rules") -> kept free: every report field line, `report.includeDxMetrics`, `report.next`, and the code-generated packet checks. - Intake/DX-scope/call-site sentences ('without losing requirements', 'Use returned paths/`scope`', 'false and neither semantic trigger applies', 'checks exact retention', the 'for both voices' tail) -> kept free: ACTIVE_PLAN field, save-to-ACTIVE_PLAN, conversation approval, backup-exactly, stop on helper error, `## Review record` (all /i where prose), init/scope/create/ prepare-close CLIs, `dxRequired`, 2+ threshold, enabling flags, no-negation rule, snapshot/verbatim dispatch/outside-inline lines, stale path bans. Mutation pass (scratch worktree; autoplan template, sections and generated phase-close): 23/23 as expected; 16 reorder/old-order/tool-param/placeholder/ numeric/consent/stop/safety mutations go red, 7 rewordings and lowered caps (ONLY/FINAL, Keep ONE phase) stay green. Log: .context/mutation-autoplan-phase-order.md. |
||
|
garrytan
|
227ec905a7 |
test(W7c): loosen auq-format-always-loaded per-skill rule pins
W7c conversion (t01 F10-F12; W7c list item auq-format-always-loaded:61-72).
The MANDATORY format list (machine-read Recommendation:/(recommended) and the
AUQ format labels, W7 keep list) is unchanged.
Removed assertion groups (invariant -> remaining protection):
- plan-ceo-review /Do NOT batch/i (t01 F10): one decision per question ->
kept free: /One decision unit = one AskUserQuestion call/i in the same rule
list.
- plan-eng-review full-sentence regexes 'Send `AskUserQuestion({ questions:
[currentDecision] })` only after the pending-record checkpoint passes. Send
one question object for one choice; other IDs wait' and 'If you discover
another independent choice, separate it and rebuild this comparison before
saving or sending the question' (t01 F11): one question object per
independent choice survives the carve -> kept free: /one\s+question object
for one choice/i and /independently selectable changes separate IDs/i; the
checkpoint-before-send order stays pinned in eng-review-routing.test.ts.
- codex /Synthesis recommendation \(REQUIRED\)/i (t01 F12, caps marker):
codex keeps its synthesis rule always loaded -> kept free: /Synthesis
recommendation/i and the machine format /Recommendation\s*:\s*<action>\s*
because/i.
Mutation pass (scratch worktree, generated SKILL.md and sections): 8/8 as
expected; removing Recommendation: <action> because, the synthesis rule, or
the CEO/eng/design decision-unit rules goes red; dropping (REQUIRED) and
rewording the two dropped eng sentences stays green. Log:
.context/mutation-auq-format-always-loaded.md.
|
||
|
garrytan
|
7dac647939 |
test(W7c): drop coverage-audit checklist tool-call prose pins
W7c conversion (t05 F11; unblocks s06 L9). Pin lines 17 -> 14. Removed assertion group (invariant -> remaining protection): - review/specialists/testing.md prose pins 'If the caller explicitly asks for an ASCII coverage diagram', 'dedicated tool call' and /Read diffs, package files, configs, or other context in separate tool calls\./ (tool-use choreography): the agent reads the named source and test files before drawing the diagram -> kept free: the same test still extracts the checklist's example diagram and requires coverageAuditVerdict to accept it (sourceRead/testsRead/diagram/passed), plus the kept [OK]/[GAP] row and legend pins; gate-tier E2Es review-coverage-audit, plan-eng-coverage-audit and ship-coverage-audit (E2E_TIERS: gate) grade the reads from transcript evidence with the same oracle. Mutation pass (scratch worktree, review/specialists/testing.md): 6/6 as expected; reordered diagram rows, deleted legend, deleted [OK] marker and a removed text fence go red; two rewordings of the dropped tool-call sentences stay green. Log: .context/mutation-coverage-audit.md. |
||
|
garrytan
|
5aad90bc3f |
test(W7c): shrink eng-scope-entry-ap to structural and meaning checks
W7c conversion (t05 F1-F4; W7a s04 F15/F18 pins). Pin lines 97 -> 69
(toContain|toMatch|indexOf|includes|ordered). The AskUserQuestion-format
byte-equivalence test (t05 Kept) is unchanged. eng scope-gate E2Es are
periodic (plan-eng-review-plan-mode), so every scope-gate invariant keeps a
free check; duplicates of eng-review-routing cite that file.
Removed assertion groups (invariant -> remaining protection):
- 'Reply with A, B, or C. STOP and wait' order anchors (t05 F1): menu before
bootstrap -> kept free: the consumed prefix 'Reply with A, B, or C'
(pty/classify.ts, pty/judge.ts) before {{PREAMBLE}}, {{GBRAIN_CONTEXT_LOAD}}
and the first ```bash; new presence checks for the announcement and menu
after '## Scope gate' (an indexOf of -1 used to pass the order check).
- Preamble 'starting from the Scope gate...' exact pin: kept as /i meaning.
- Scope entry sentences ('Before discovery tools or preamble, check provided
messages...', 'Do not probe for session state', 'When no exception above
applied:', 'Announce an auto-selected plan in one line...', 'A fresh
announcement made before skill loading...', 'Step 0 below still verifies or
sends the public auto-selection line...') (t05 F4): target bound before
bootstrap and announced -> kept free: announcement/menu-before-bootstrap
order checks above, /first tool call = AskUserQuestion/i, /clarify
ambiguous, conflicting, quoted or stale targets/i; periodic E2E
plan-eng-review-plan-mode.
- Startup step sentences ('3. Check web-research readiness at ...', '4. Run
**Design Doc Check**, then ...', 'Keep the reviewed target fixed', 'Continue
at ...'): startup order -> kept free: ordered startup anchors (now including
Prerequisite Skill Offer), '**Engineering review → Step 0**' pointer, the
stale-path ban; 'Keep the reviewed target fixed' stays pinned (/i) in
eng-review-routing.test.ts.
- Entry anchors 'Before Step 0, require resolved scope' and '**Complexity
gate:** while a Scope Challenge complexity question' -> shortened anchors
'require resolved scope' and '**Complexity gate:**', same order check.
- Plan-selection exception sentences and 'If the tool is unavailable,
disallowed (`--disallowedTools`) ... send the menu as plain prose and STOP'
(t05 F4): pasted text is not a mode signal; ambiguity and named targets ask;
unavailable tool stops -> kept as /i meaning checks for each (safety and
consent lines); A/B/C menu body kept exact (consumed by the PTY classifier).
- 'Reply with A, B, or C. STOP and wait for the answer.' exact: kept as the
exact prefix plus /reply with A, B, or C\. stop and wait for the answer/i.
- Judge-bundle sentences ('format/transport rules apply throughout', 'Follow
the blocks below in order after startup', 'It may be the selected plan or a
separate file', 'Save permitted auxiliary artifacts under the write policy'):
ordering and write permission -> kept free: stage order, write-policy table
rows exact, /permission for one path authorizes no other/i, /defer ...
telemetry and plan status footer to finish/i, forbidden-log Blocked outcome
route (/i), section Read paths, log exit-code tails.
- Complexity-path duplicates of eng-review-routing (t05 F2: 'Complete these
checks before...', 'After verification, apply only accepted scope changes',
'Run C whether B...', 'Always ask the structure question...', post-answer
summary/Read-back sentences, 'A failed save or Read blocks advancement',
'Findings and scope answers approve no remedies', 'Continue to Section 1
only when...', 'Send one question object for one choice', 'Compare every
native field...', 'Repair any difference and repeat the complete Read
before asking', 'Read the selected saved label...', 'Check the save result,
then Read the entire resolution block', '**STOP until the actual answer
arrives.**', the Paused question / Blocked outcome lines): the same
invariants -> kept free in eng-review-routing.test.ts (order of
Send/AskUserQuestion/Record, /stop until the actual answer arrives/i, failed
save blocks, approve-no-remedies, Paused/Blocked recovery checks) and
auq-format-always-loaded.test.ts (one question object per choice); here:
A/B/C stage order and the 8-file / 2-class thresholds (/i, no caps STOP
literal), Recovery routing pointer for unreadable records. The Read-back
choreography is the dated text s04 F18 flags; judge
'plan-eng-review/SKILL.md sections' (gate) grades the procedure.
- Calibration note exact sentences and the one-line formatting pin (t05 F3,
W7a s04 F15): no shipped source enables BRAIN_CALIBRATION_WRITEBACK ->
kept free: {{BRAIN_WRITE_BACK}} once after {{LEARNINGS_LOG}}, /never set it
yourself/i, and a new code check that no file in bin/, lib/ or scripts/
assigns or exports BRAIN_CALIBRATION_WRITEBACK (only the gbrain resolver
names it).
Mutation pass (scratch worktree; plan-eng-review templates, generated
plan-eng-review skill, bin/gstack-config): 21/21 as expected; 14
reorder/marker/table/stop/safety/consent/code/numeric mutations go red, 7
rewordings and lowered-register lines stay green. Log:
.context/mutation-eng-scope-entry-ap.md.
|
||
|
garrytan
|
64dcd18df4 |
test(W7c): shrink eng-review-routing to structural and meaning checks
W7c conversion (t02 #4-#10, W7a s04 F15 pin). Pin lines 88 -> 74 (toContain|toMatch|indexOf|includes|ordered); ~50
full-sentence pins become order checks, field/marker checks and short
case-insensitive meaning checks. Remaining gate-tier judge for the plan-eng
prose: 'plan-eng-review/SKILL.md sections' (judge input is ENG_REVIEW_EXCERPT,
which the first test proves contains review-sections and Recovery routing).
Removed assertion groups (invariant -> remaining protection):
- Preparation sentences ('Follow the blocks below in order after startup',
'Confidence Calibration and Decision procedure are reference rules'): blocks
run in order -> kept free: heading order check now starts at
'## Review preparation'; judge 'plan-eng-review/SKILL.md sections'.
- Priority/compression sentences ('Shorten only optional commentary', 'never
Scope Challenge, Sections 1-4...', 'Never condense, abbreviate or skip a
section'): compression cannot drop required stages -> kept free:
/complete every required stage, decision gate and output/i and the
'Everything else' stale ban; judge.
- Exact seven-label toEqual on Scope Challenge assessments (t02 #8): all
seven checks present -> kept free: arrayContaining on the same labels plus
the A/B/C heading toEqual (excerpt anchors).
- 'Complete these checks before the complexity decision in B': A runs before
B -> kept free: heading order A < B < C.
- Below-threshold / findings sentences (numbered step full sentences): C
always runs, findings approve nothing -> kept free: step-number order,
/fewer than 8 files AND fewer than 2 new classes/i, /run C whether B was
completed or skipped/i, /approve no remedies/i, '"No issues found"'.
- '... STOP before Section 1' exact caps (t02 #4): high complexity halts ->
kept free: /at 8\+ files or 2\+ new classes\/services, stop before
section 1/i.
- High-complexity sentences ('With no proposed cuts...', 'Both retain the same
approved feature list...', 'Initial scope selectors need no grid...'): cuts
and structure asked separately -> kept free: step order with short anchors,
'Pending remedies not decided here: <ids>' field; judge.
- Smaller-arrangement sentences ('offer confirmation of the original
arrangement or a pause', 'then return to this structure selector'): a pause
authorizes no cut -> kept free: order + /do not continue to C until it is
settled/i + /investigate only the agreed question/i.
- Selector summary sentences ('post-answer scope summary, not a remedy's pending
ledger record', 'Do not invent a pre-answer record afterward', 'Scope
Challenge B also uses its own selectors'): answers saved before applying ->
kept free: field template order, /a failed save or read blocks
advancement/i, '**not persisted**', /these selections approve no
engineering remedy/i.
- Decision-procedure Read-choreography anchors ('use Read to fetch the entire
saved record', 'Read the entire resolution block, including State') and the
bold '**STOP until the actual answer arrives.**' anchor (t02 #5/#6, s04 F18):
ask once, wait, record before the next choice -> kept free: Compare <
Pending-record checkpoint < Send once < AskUserQuestion call < Record < next
choice, /stop until the actual answer arrives/i located between Send and
Record, /do not apply a remedy ... while the choice awaits an answer/i,
/approves no implementation/i; AUQ completeness and kind-note strings kept
exact (plan-tune reads them).
- 'An Investigate/Defer option must bound the investigation', 'Apply the
preamble's Completeness scores or kind-note accordingly': judge only (gate).
- Research/outside-voice sentences ('If Aside is unavailable, use host
WebSearch', 'Record the actual coverage...'): unavailable research is
recorded, not hidden -> kept free: exact 'Search unavailable' output string,
'Outside voice: recorded provider, completed / unavailable / disabled /
skipped (reason)' field, /only completed reviews enter cross-model tension/i,
TODO < Approval readiness < Required outputs order.
- Recovery sentences: kept as /i meaning checks (ExitPlanMode/telemetry wait,
do not resend, stop before dependent output, no chat-only success, skip
forbidden writes); 'If no recovery is specified or it fails' full sentence
-> kept free: 'follow **Blocked outcome**' routing; OUTCOME=error kept.
- Late-change sentences ('Refresh affected tests, tasks, dependencies and
parallelization', 'Unchanged saved outputs may reuse their successful Review
Log'): rerun affected outputs before navigation -> kept free: order of
return/Decision procedure/Approval readiness/Required outputs/navigation and
the stale-evidence Blocked outcome route; judge.
- Final sequence anchor 'Run Learning hooks, including gated Brain Calibration
Write-Back; then return to the entrypoint's Section self-check' -> kept
free: 'Section self-check' as the last ordered anchor.
- Coverage-audit sentences ('For each existing or proposed component in the
selected target', 'the selected target. For each existing or proposed
feature', 'dedicated tool call before drawing the diagram' (s06 L9
choreography)): plan diagrams cover proposed paths, ship covers changed code
-> kept free: /existing or proposed (component|function\/method)/,
/future paths remain proposals/i, the four 'Every ...' obligations,
/no skipping regression coverage/i, ship has no 'existing or proposed';
gate E2E plan-eng-coverage-audit.
- Brain write-back ordered prose list ('reserved default-off gate', 'this
runtime does not set it', 'not an instruction to write now', t02 #9 / W7a
s04 F15): no gbrain write without the default-off gate and trust policy ->
kept free: 'Skip unless `BRAIN_CALIBRATION_WRITEBACK` is set' precedes both
write tools, eng prefix says /skip this section/i and /do not enable it or
infer permission/i, trust-policy key, /if unknown, skip/i, no env
interpolation, other skills carry no eng-only skip prefix.
Mutation pass (scratch worktree; plan-eng-review templates and
scripts/resolvers/gbrain.ts): 21/21 as expected; 14 reorder/marker/stop/
approval/consent/numeric mutations go red, 7 rewordings of dropped sentences
and lowered-register STOP lines stay green. Log:
.context/mutation-eng-review-routing.md.
|
||
|
garrytan
|
03a4521dda |
test(W7c): shrink ceo-workflow-clarity to structural and meaning checks
W7c conversion (t05 F5-F8). Pin lines 48 -> 29 (toContain|toMatch|indexOf|includes|ordered); sentence pins
collapse to one case-insensitive meaning check per invariant. Per the Eng
"Retirements" item, CEO invariants count as periodic-only once
plan-ceo-review-benefits retires, so every invariant below keeps a free check.
Removed assertion groups (invariant -> remaining protection):
- Required/Pending/Settled status sentences (4 pins): pending work stays
unapproved; only exact authority settles a choice -> kept free: label order
plus /Pending: unapproved/i and /Settled: ... a recommendation does not/i.
- Admin-route sentences (Skip steps 1-4, 0E/0H ownership, admin answer that
requests a plan change, 0G proposals use this route, mode-change pointer):
admin menus approve nothing and return locally -> kept free: Admin < Plan
decision < steps 1-4 < checkpoints order, /this approves no plan changes/i,
/0D returns to its caller, not to mode selection/i.
- not.toContain('0D never restarts mode selection') and
not.toContain('using the Step 0E mode-handoff format') (t05 F8, bans of
retired phrasings): same invariants -> kept free: the positive route checks.
- '**STOP for the actual answer, even for a lone option.**' exact bold pin:
wait for the answer before recording -> kept free: /stop for the actual
answer/i, located between step 4 and the post-answer checkpoint.
- Mode-change sentences (repeat handoff record, complete newly applicable
Step 0 work, resume the paused step): a mode change waits for an answer ->
kept free: /pause and ask ... keep the mode until answered/i and
/selecting a mode does not approve changes/i; routing table rows kept exact.
- Scope-count sentences: reuse counts against limits, mode recommendation
excludes unchanged reuse, neither approves -> kept free: three short
case-insensitive meaning checks plus the >15 / 8-file / 2-class numbers and
the route order.
- Completion-fraction sentences (select answered questions, exclude unscored,
Y/X definitions): fraction definition -> kept free: 10/7/3 scale, reopened
choice counted once, 'Report X/Y, or `N/A` when Y is zero' (output field).
- Section-gate sentences ('0D's Plan decision route through its post-answer
save', 'Resolve critical risks now') and the literal Decision gate count 11
(t05 F7): one gate per section -> kept free: path order, /no path selects
the mode again/i, gate count derived from '### Section N:' headings.
- Section 1 opening sentences (retain mode/rationale, governing row ID): Section
1 publishes current dispositions, not a mode handoff -> kept free:
**Current scope** present, the four disposition names, /not another mode
handoff; do not ask or log the mode again/i.
Gate-tier judge that also grades this prose: 'plan-ceo-review/SKILL.md modes'.
Mutation pass (scratch worktree, generated plan-ceo-review skill): 16/16 as
expected; 11 kept-step/marker/safety/numeric mutations go red, 5 rewordings of
dropped sentences (and a lowered-register STOP line) stay green. Log:
.context/mutation-ceo-workflow-clarity.md.
|
||
|
garrytan
|
23a0be82d7 | chore: regenerate SKILL.md, goldens and context-budget fixture for the Opus 5.5 prompt cleanup | ||
|
garrytan
|
841a766eec | Merge remote-tracking branch 'origin/garrytan/opus55-wip-D' into garrytan/opus-5-5-prompt-audit | ||
|
garrytan
|
737bf478d9 | test: dedicated-tools overlay case pinned to claude-opus-5-5 (E2) | ||
|
garrytan
|
522e300d4e | Merge remote-tracking branch 'origin/garrytan/opus55-wip-B' into garrytan/opus-5-5-prompt-audit | ||
|
garrytan
|
806dda8fa3 |
Merge remote-tracking branch 'origin/garrytan/opus55-wip-C' into garrytan/opus-5-5-prompt-audit
# Conflicts: # test/skill-e2e-plan-ceo-plan-mode.test.ts |
||
|
garrytan
|
0355b91c89 | Merge remote-tracking branch 'origin/garrytan/opus55-wip-E' into garrytan/opus-5-5-prompt-audit | ||
|
garrytan
|
e28ef17040 |
test: plan-design SDK fixture pins its harness prompt once, by ordered properties (W7e plan-design-sdk-fixture.test.ts:91-120; t01 F13)
The prompt was pinned three ways (sha256, ~20 sentences, an order check). Kept: the skill and lazy-section paths, the plan path, the ordered read -> review -> edit -> Write -> read-back -> summary steps, and the no-browsing rule (case-insensitive). Dropped the hash and sentence pins. Negative control: removing the read-back step turns it red. |
||
|
garrytan
|
c2d3182f24 |
test: shared-libs revalidation prompt keeps its machine-shaped checks, not ~69 phrase pins (W7e shared-libs-revalidation-prompt; t11 #3, #8; t05 F17)
reviewRevalidationPrompt and reviewPrompt are test-owned harness prompts (non-test callers: 0); the paid shared-libs E2Es measure whether they work. Kept: the three exact receipt commands, base-prompt prefix and resumed suffix identity, branch selection (actor / default / resumed coverage), the state path and turn-budget interpolations, read-record-before-diff and first-response-before-start ordering, no embedded token, no supplied Skip answer, reusable:true before reusable:false, the captured R59 stage boundaries in the exclusion list, asset roots and interfaces, and the stale step-6 reference ban. The review template's Step 4.7/4.8 completion rule is matched by step number on one line, not by sentence. Negative control: reordering the diff-read instruction turns the test red. |
||
|
garrytan
|
a31c0a7ece |
test: docsync lifecycle interface drops pins on its own helper prompt (W7e docsync-lifecycle-interface.test.ts; t02 #17)
docsNativeInterface is the test helper's own prompt, so its sentence pins only checked that the helper says what it says. Kept: owned skill path, the two extracted lifecycle commands (exact), the scripts-list interpolation, the repo path and Git read examples, and every docsCommandAllowed / docsToolFailures behavior check that enforces the interface. |
||
|
garrytan
|
744e5a2392 |
test: office-hours review-preservation judge test keeps its schema and untrusted-data checks, not its wording (W7e office-hours-completion.test.ts:416-429; t09 F10)
Drops 13 phrase pins on the judge prompt. Kept: both inputs carried as JSON, the response schema (wire format), and the separate untrusted-DATA test. The validator behavior tests already use stub judges. Negative control: changing the schema string in the prompt builder turns the test red. |
||
|
garrytan
|
8962715176 |
test: section-capture native tools checks harness prompt properties, not a copied golden (W7e section-capture-native-tools.test.ts:467; t04 F4)
Asserts the absolute skill path is the only SKILL.md named, the scenario, AskUserQuestion unavailable, the section Read rule (case-insensitive, also at :308), no git/gh/mutation, and the REPORT.md path. argv/tool-list checks are unchanged. Negative control: dropping the no-mutation rule turns it red. |
||
|
garrytan
|
903c46f3d4 |
test: session-runner-tools checks harness prompt properties, not a byte-for-byte copy (W7e session-runner-tools.test.ts:544; t09 F7, F8)
The ship capture prompt test asserts its purpose (no CEO-only writing guidance) plus the functional rules: one allowed skill path, AskUserQuestion unavailable, the section Read rule, no git/gh/mutation, and the REPORT.md path. The two 'MUST actually Read' pins match the Read rule case-insensitively. Negative control: dropping the no-mutation rule from the harness prompt turns the test red. |
||
|
garrytan
|
d1e3eadff8 |
test: plan-eng artifact case binds to the hermetic state root and requires the artifact; report case drops its shouted requirement (W7g plan-artifact fixture binding; W7g skill-e2e-plan.test.ts:479,720)
- plan-eng-review-artifact: actor, observer and cleanup use one isolated GSTACK_HOME (the hermetic run's, or a seeded temporary one when hermetic mode is off) and the slug gstack-slug resolves for the fixture, instead of os.homedir()/.gstack/projects/test-project. A guard refuses a state root under the real home, and cleanup removes only the fixture's project directory. The 'IMPORTANT … you MUST write the test-plan artifact' line is replaced by the environment fact (state root and slug). The soft warning becomes an assertion: exactly one new *eng-review-test-plan-*.md about the reviewed dashboard change. - plan-review-report: removes 'CRITICAL REQUIREMENT … MOST IMPORTANT deliverable' and the restated report contract; keeps 'plan.md is the plan file'. The existing assertions (report present, last, rows, unresolved status) now measure the skill's own report prose. |
||
|
garrytan
|
f43fe0de13 |
test: eval-kinds drops the exact 22 behavior / 25 judge census (W7d exact count in eval-kinds; t12 F6)
The kind contract is already asserted: every live case has one kind, kinds are rule/behavior/judge, every LLM judge is judge-kind, every behavior case has a literal registration and a BEHAVIOR_WHY, and the named contract-shaped cases stay rule. The census count only broke on adding or retiring a case. |
||
|
garrytan
|
0725d99d99 |
test: browse reference judge gates on its stated floors; the baseline comparison is recorded, not a ratchet (W7d browse judge baseline ratchet; t08 F7)
expect(regressions).toEqual([]) compared a noisy three-sample mean against eval-baselines.json (clarity 4), silently raising the stated clarity floor of 3. The comparison stays in judge_reasoning; pass and assertions use BROWSE_JUDGE_FLOORS (clarity 3, completeness 4, actionability 4). Judge prompt and schema are unchanged. Stored panel means: three known-good (two the ratchet rejected) pass; three below-floor panels fail. |
||
|
garrytan
|
99720e744e | test: CEO plan-mode failure message names the current rule, not the '## CRITICAL RULE' heading (t13 F20) | ||
|
garrytan
|
833a35de8a |
test: plan-design judge selection check anchors on the question-procedure heading (W7a judge-anchor move; touchfiles.test.ts:219)
Same structural anchor as the other judge-anchor moves; selects the same text on the current templates and after the '## How to ask questions' rename. |
||
|
garrytan
|
ea116a16d1 |
test: plan-mode smokes check the targeted plan, not the scope-gate announcement, and reject plan-file ## Decisions (W7d exact scope-gate line; W7g plan-mode ## Decisions)
- Seeded eng/design plan-mode cases keep 'no scope menu' and replace the exact 'Scope gate: plan mode — auto-selected B' requirement with seededPlanTargeted(): the announcement or native seeded-plan selection, or seed-only plan names appearing in the review output after the slash command (trackTokens). The announcement is logged. - The AskUserQuestion format forbids writing a decision to the plan file as a substitute for asking. assertNoPlanFileDecisions() fails any plan_ready run whose plan carries ## Decisions (eng, design, devex, office-hours auto-mode). The AUQ-blocked cases (eng seeded, devex, office-hours) now require outcome 'asked' (AskUserQuestion variant or prose fallback). - plan-mode-evidence registration controls keep their verdicts; the missing-decisions control now fails on 'without a question'. - plan-mode-targeting.test.ts stores known-good (announcement; paraphrase plus seed content) and known-bad (neither; an untracked token) observations and plan files with and without ## Decisions. |
||
|
garrytan
|
0420a06e91 |
test: plan-format accepts any statement that kind-differentiated options are not comparable on coverage (W7d 'options differ in kind' phrase)
The outcome, no fabricated Completeness: N/10, stays asserted. The required note now matches any brief wording of the same fact (KIND_NOTE_RE in helpers/plan-format-kind-note.ts). The format-compliance suite that pins the declared kind-note string is unchanged. Stored lines: four known-good pass, three known-bad (a completeness score, a recommendation, 'kind to the user') fail. |
||
|
garrytan
|
93ef6f69be |
test: test-value review accepts any caller search, not only the literal 'git grep' (W7d literal git grep in test-value)
recordsNonTestCallerSearch keeps the value-card contract (non_test_callers recorded) and accepts git grep, rg, grep or the Grep tool as the search. Severity checks and the golden exclusion are unchanged. Stored evidence: four known-good (three the old check rejected) pass; three known-bad (no search, no caller count, prose only) fail. |
||
|
garrytan
|
e0f30196a7 |
test: shared-libs PR coverage checks the stated page budget, not per-PR ceilings from one run (W7d shared-libs per-endpoint page caps)
prCoverageRequestViolations keeps the skill's stated older-open-PR budget (five metadata pages, fifty file-list pages plus recent PR 7) and the coverage outcome (PR 42's file list read from page 1 through page 2). It drops the PR 42 three/four-page and single-first-page ceilings, which were tuned to one observed capture with a truncated first view. The judge criteria are unchanged. Stored endpoint lists: three known-good (including a re-read first page the old ceilings rejected) pass; four known-bad plus an over-budget file-list run fail. incompleteFirstFileView is no longer used by the paid case. |
||
|
garrytan
|
fae9a05c12 |
test: CEO split-overflow call floor is two, the minimum for five options at four per call (W7d split-overflow 4-call floor)
The regression is a candidate silently dropped to fit the four-option cap. The scope validator already requires an offered Include/Defer/Cut disposition for all five candidates; the extra N-1 call floor failed agents that batch compatible candidates, which the test header calls legitimate. FLOOR now comes from CEO_SPLIT_CALL_FLOOR = 2. Stored shapes (ceo-split-floor.test.ts): five sequential, 2-call and 3-call batches pass; a dropped candidate fails with 'missing target decisions'. The registration controls keep their verdicts; the 3-call plan_ready scenario still fails, now on missing decisions. |
||
|
garrytan
|
e30fcf732a |
test: investigate boundary grader checks the verified fix, not one Edit and one verify (W7d investigate exactly-one ritual)
Completion now needs at least one allowed Edit, the correction in place, at least one verifier run whose final status is 0, and no edit after the final verification. The ending-error case still requires a single verifier run (status 69), because that fixture's error tells the actor to stop without retrying. Ownership acquisition/release checks are unchanged. New free protocol controls: a passing re-verification passes (the old grader failed it); a failing final re-verification and a retry after the ending error both fail, as they did before. |
||
|
garrytan
|
5701c3d903 |
test: disabled outside-plan grader reads the run's own coverage self-report (W7d disabled-plan-review-evidence)
The false-completion oracle kept being patched to tell historical review-log
mentions from current claims. The E2E prompt now asks for a closing JSON line
{"outside_review_this_run": ...}. When present, the coverage claim is that
report ('completed' fails) plus any undenied 'both reviewers agree';
'disabled' also satisfies attribution. Runs without it keep the existing
prose oracle, so all stored captures and mutation rows keep their verdicts.
Dispatch, CLI execution, preflight and persisted-disabled checks are
unchanged. Stored outputs: three known-good pass (the old grader rejected
the table form); four known-bad fail.
|
||
|
garrytan
|
a6b0240971 |
test: coverage audit grades a closing tested/untested summary before the diagram grammar (W7d coverage-audit-evidence)
The diagram grammar grew through four waves of capture patches to accept
correct-but-different diagrams. The E2E prompt now asks for one closing JSON
line {"tested": [...], "untested": [...]}; coverageAuditVerdict passes the
outcome when processPayment is only in tested and refundPayment only in
untested (last summary wins; quoted or example summaries ignored). The legacy
grammar stays as a fallback, so every stored capture keeps its verdict
(all existing rows unchanged). Read evidence and capture success are still
required. Stored summaries: three known-good (which the old grader rejected)
pass; seven known-bad fail.
|
||
|
garrytan
|
dbff2527b0 |
test: AUQ matrix fails only on the format fields software reads (W7d AUQ matrix emoji check)
auqMachineFormatProblems() fails a first-question run when the Recommendation: line is missing or empty, or when the brief does not label exactly one option (recommended); recommendation substance >= 4 is unchanged. ELI10, Pros / cons, ✅/❌ and Net: are still scored and logged but no longer fail a paid case. scoreAuqFormat and the consistency suite are unchanged. Stored briefs: three known-good (including one without emoji or Net) pass; four known-bad (no or empty Recommendation, zero or two (recommended)) fail. |