mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-12 19:44:48 +01:00
a686a40ba0c2f55015ea0a83134bc17da067a667
434
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 garrytan
|
a686a40ba0 | chore: regenerate qa-only | ||
|
garrytan
|
93b6e11fe9 | qa-only: one rule for mixed-run clocks; annotations and materialize are per probe directory | ||
|
garrytan
|
8f34e6eb99 | test: skillify E2E sweeps the state root (GSTACK_HOME) where the global tier actually writes | ||
|
garrytan
|
f4cfae6661 | test: calibration-gate scan reads tracked files with POSIX paths (windows-free-tests) | ||
|
garrytan
|
6c2e9a70a0 | chore: regenerate plan-tune, office-hours, retro | ||
|
garrytan
|
eb768a55dd | plan-tune, office-hours, retro: read gate and profile files from the state root (E1 follow-up) | ||
|
garrytan
|
4d371a98ad | chore: regenerate SKILL.md and context budget after manifest state-root globs | ||
|
garrytan
|
44e9c1ed3a |
fix(state-root): gbrain context_queries globs use {gstack_state_root}/
bin/gstack-brain-context-load.ts resolves the {gstack_state_root}/ prefix
through gstack-paths (configuredStateGlob); the legacy ~ form only matched the
default root. The nine projects/analytics manifest globs in design-consultation,
design-shotgun, investigate, office-hours, plan-ceo-review and retro now use the
prefix, as does office-hours' builder-profile developer-profile.json glob
(plan-tune writes it under $GSTACK_STATE_ROOT).
Allowlist: drops the nine MANIFEST_GLOB rows and the now-unused MANIFEST_GLOB
and PENDING reasons; the header lists the remaining row kinds.
Tests: new brain-context-load case loads the shipped investigate and
office-hours queries against a configured root and proves a file only under
~/.gstack is not read.
|
||
|
garrytan
|
3215981a2e | chore: regenerate SKILL.md, ship goldens and context budget after telemetry prose state-root change | ||
|
garrytan
|
d7e9f0b501 |
fix(state-root): completion-status telemetry prose names $GSTACK_STATE_ROOT/analytics
gstack-skill-end writes analytics under the resolved state root ($_GH), so the PLAN MODE EXCEPTION note now names $GSTACK_STATE_ROOT/analytics/ instead of the ~/.gstack default. Allowlist: drops the generate-completion-status.ts PENDING row. Pin (W7a): gen-skill-docs 'render documents the analytics sink' now checks that the render names the state-root analytics dir. |
||
|
garrytan
|
fc0aacd080 | chore: regenerate SKILL.md and context budget after taste-profile state-root glob | ||
|
garrytan
|
10d8c1c027 |
fix(state-root): taste-profile legacy fallback globs $GSTACK_STATE_ROOT designs
The legacy approved.json fallback in {{TASTE_PROFILE}} (design-consultation,
design-shotgun) named the literal ~/.gstack/projects path while the approval
writers use $GSTACK_STATE_ROOT/projects/$SLUG/designs.
Allowlist: drops the design.ts PENDING row.
Pin (W7a): design-consultation-contract's exact glob string becomes a
meaning-level check (legacy fallback names the state-root designs dir and
approved.json, and no ~/.gstack literal).
|
||
|
garrytan
|
b7a1d77050 | chore: regenerate SKILL.md, ship goldens and context budget after history/ledger state-root paths | ||
|
garrytan
|
1fdda72ab9 |
fix(state-root): history, report and ledger paths name $GSTACK_STATE_ROOT
Agent-facing read/write paths that named the literal ~/.gstack default now
name the state root their writers use:
- canary-history.jsonl and health-history.jsonl (the blocks above already
resolve the root and print PROJECT_DIR)
- ios-design-review report: adds {{SLUG_SETUP}} so the report path is the
PROJECT_DIR it prints
- plan-tune dream-cycle gate's distillation-proposals.json
- retro's skill-usage and eureka reads: read the path gstack-retro-metrics
prints (it already checks $_gstack_sr_root/analytics)
- ship's apple-effects.log durable-effect ledger and the gstack-evidence
ledger description (bin/gstack-evidence writes under resolveStateRoot())
Allowlist: drops the nine matching PENDING rows from STATE_ROOT_ALLOWLIST.
|
||
|
garrytan
|
3c3a013a3b | chore: regenerate SKILL.md and context budget after design-doc discovery state-root change | ||
|
garrytan
|
569cdae04f |
fix(state-root): design-doc discovery reads $GSTACK_STATE_ROOT/projects
DESIGN_DOC_DISCOVERY and autoplan's Step 2 design-doc lookup read the literal ~/.gstack/projects path, so under a non-default state root they missed the docs office-hours writes. The block now globs "$GSTACK_STATE_ROOT"/projects, and each caller's bash block (autoplan, plan-ceo/devex/eng-review, the spec-review prerequisite recheck) resolves the root with the guarded gstack-paths eval first. Allowlist: drops the autoplan ls row, both _LOCALDOC rows and the resolver source-comment row from STATE_ROOT_ALLOWLIST. Tests: the eng Design Doc Check fixture installs gstack-paths and seeds docs under a non-default root; the CEO discovery fixture passes an explicit GSTACK_STATE_ROOT. |
||
|
garrytan
|
5d44196e80 | chore: constants comment names the /ship documentation dispatch, not a stale step number | ||
|
garrytan
|
39e670035a | test: integration repairs after F2 (calibration test renders the AUQ format; census counts and release floors without the reverted overlay case) | ||
|
garrytan
|
7ec098f9a1 | Merge remote-tracking branch 'origin/garrytan/opus55-wip-F2' into garrytan/opus-5-5-prompt-audit | ||
|
garrytan
|
727481797e |
test(E1): pin CLAUDE.md facts, /ship step references and state-root literals
New free test test/instruction-facts.test.ts (CEO/DX/Eng E1 items; DX-13 one
named test() per check, each failure prints both values, file:line and the
fix):
- 'CLAUDE.md names every TREE_MUTATING file and does not call the set
empty': imports TREE_MUTATING from scripts/test-free-shards.ts (today
test/bootstrap-retention.test.ts).
- 'CLAUDE.md states the current catalog ceiling from
test/catalog-budget.test.ts': the "`CATALOG_BUDGET_TOKEN_EQUIVALENTS`
(N today" value must equal the constant (1,194).
- 'CLAUDE.md points at package.json for detach timeouts instead of stating
numbers': no eval:bg* --timeout value appears in CLAUDE.md (plain or with
commas), and the detach section names package.json's eval:bg:gate /
eval:bg:periodic and states no timeout number.
- 'every explicit /ship Step reference names a defined /ship step with the
expected title': matches only `/ship Step N` and `/ship's Step N`
(backticked or not) across every .tmpl and scripts/resolvers/; each must
be a table row whose step is a heading in ship/SKILL.md.tmpl or
ship/sections/*.md.tmpl whose title contains the row's subject, and the
reference's line or enclosing heading must mention that subject; stale rows
fail. Today: document-release release-body Step 7 -> /ship Step 14
(TODOS.md), plus the incident-history comment in
scripts/resolvers/constants.ts ("/ship Step 18"), kept as a historical row
with a reason (handoff: that comment's step is stale; the dispatch is Step
14.5).
- 'negative control: moving a reference to another existing /ship step
fails': rewrites the TODOS reference to Step 12 in memory and asserts the
checker reports it.
- 'every ~/.gstack/projects/ and ~/.gstack/analytics/ hit is allowlisted with
a reason' and 'every state-root allowlist row still matches a hit': all 37
current hits in .tmpl and scripts/resolvers/ are covered by 36 rows, each with one of four
reasons (user-facing description; gbrain context_queries manifest glob read
literally by bin/gstack-brain-context-load.ts; source comment; pending
conversion tracked in TODOS.md). The readiness-gate EVAL_DIR rows reuse the
reason from test/state-root-ratchet.allowlist.json. New hits fail, and
converting a literal fails until its row is deleted.
Mutation pass (scratch worktree): 15/15 as expected; drifting the catalog
number in CLAUDE.md or the constant, dropping or emptying TREE_MUTATING text,
adding a TREE_MUTATING key, copying a detach timeout into CLAUDE.md,
moving/undefining/retitling a /ship reference, a new /ship reference, a new
state-root literal and a converted allowlisted literal all go red; unrelated
rewording stays green. Log: .context/mutation-instruction-facts.md.
|
||
|
garrytan
|
4819d8e2b8 |
test: update the periodic release floor for the opus-5-5 overlay case
paid-retry-supervision.test.ts 'detached PR fallback and release commands cover their actual default worker budgets' was red on the integrated base |
||
|
garrytan
|
b27c77c8bc |
test(W7h): delete incompleteFirstFileView, which no paid case uses
W7h retirement (task scope: "incompleteFirstFileView + its free test in test/shared-libs-fixture.test.ts if no paid case uses it"). `grep -rn incompleteFirstFileView test scripts` showed only the export in test/helpers/shared-libs-eval-fixture.ts and its own free describe block; no paid case or grader calls it, so the helper and the two tests that pinned it are dead code. Retirement card: - test: shared-libs-fixture.test.ts 'incompleteFirstFileView (census 36641820398 shared-libs-pr-coverage)' (2 tests). - detects: regressions in a page-1 recovery classifier nothing consumes. - non_test_callers: none. - replacement: none needed; the shared-libs page budgets that a recovery read is charged to stay enforced by the kept budget function and its tests. - validation: bun test test/shared-libs-fixture.test.ts (241 pass); bun run typecheck. |
||
|
garrytan
|
1f3e192098 |
test(W7h): drop the no-op review-report check from setup-gbrain-remote
W7h retirement (t12 F9) of one assertion in the paid setup-gbrain-remote
case. No registration changes (the case stays).
Retirement card:
- assertion: `expect(/## GSTACK REVIEW REPORT|critical_gaps/i
.test(result.output)).toBe(false)` with its comment.
- detects: /setup-gbrain writing a plan-review report or critical_gaps
findings, which it never produces; the removed comment itself said
"Setup-gbrain doesn't have a review report contract". Copied from the
plan-review tests (
|
||
|
garrytan
|
56e4fa201a |
test(W7h): delete validateOfficeHoursSpecSummary left unused by the retirement
Dead-code follow-up to the office-hours-spec-review retirement:
validateOfficeHoursSpecSummary in test/helpers/office-hours-completion.ts graded
only that paid case's model-written summary. Deletes it and its free
describe block in test/office-hours-completion.test.ts ('office-hours
spec-review summary completion'); every other validator and test in both
files is unchanged.
Retirement card:
- detects: a summary missing five dimensions, the Agent dispatch, the 3
iteration cap or the four metrics.
- non_test_callers: none.
- replacement: the facts it graded are pinned free on the generated text by
gen-skill-docs.test.ts 'SPEC_REVIEW_LOOP resolver' (round-3 cap and loop
wiring); see the retirement commit's control.
- validation: bun test test/office-hours-completion.test.ts.
|
||
|
garrytan
|
ca395095a1 |
test(W7h): delete the codex-offering prompt builder left unused by the retirement
Dead-code follow-up to the codex-offered-* retirement: buildCodexOfferingPrompt (test/helpers/codex-offering-fixture.ts) had no caller besides those cases. Deletes the helper, its free test (test/codex-offering-fixture.test.ts), the two captured public fixtures it replayed, its scripts/free-test-durations.json entry, and the helper/fixture paths still listed as touchfiles of six plan review E2Es in test/helpers/touchfiles-data.ts. Retirement card: - test: codex-offering-fixture.test.ts (pinned the builder's prompt text and replayed captured timeout/CDD transcripts against it). - detects: changes to a prompt that no paid case sends any more. - non_test_callers: none (grep -rn codex-offering test scripts is empty). - replacement: none needed; the behavior it supported is now the free probe check in gen-skill-docs.test.ts. - validation: bun test test/touchfiles.test.ts test/touchfiles-map-diff.test.ts test/paid-shards.test.ts test/e2e-tier-alignment.test.ts; bun run typecheck. |
||
|
garrytan
|
fc9049a8b5 |
test(W7h): retire the paid SKILL.md-summary cases in skill-e2e-plan
W7h retirement (t14 F6-F8): office-hours-spec-review, plan-ceo-review-benefits
and codex-offered-{office-hours,ceo-review,design-review,eng-review}. Each
asked a model to read a generated SKILL.md, write a summary, and grepped the
summary for facts the generated file states directly.
Registration cleanup in the same commit: E2E_TOUCHFILES, E2E_TIERS (all six
gate) and E2E_KINDS (rule) in test/helpers/touchfiles-data.ts; six
scripts/paid-test-durations.json case-shard entries; no
test/helpers/periodic-exclude-data.ts entries existed. The fast PR profile
(scripts/test-pr-profile.ts) drops plan-ceo-review-benefits and
office-hours-spec-review from PR_PROFILE_CASE_IDS and PR_PROFILE_FILES. Tests
that named the retired ids as live examples move to existing ones:
paid-free-boundary and paid-pr-profile (a CEO template change now plans
auq-format-gate plus the CEO judge; the case-shard fixture uses
plan-review-report), test-pr-profile, touchfiles (CEO selection count
21 -> 19), the touchfile-moved-code w5 golden (retired ids removed), the
behavior-lane case-shard census (78 -> 72) in cookie-validation-phases, and
the skill-e2e-plan timeout list and full-gate PR floor (78,425 -> 72,755 s)
in paid-retry-supervision. e2e-tier-alignment, eval-kinds,
eval-detach-timeout-floor and paid-shards stay green.
Coverage consequence: per the Eng Retirements item, plan-ceo-review-benefits
was the only gate-tier CEO E2E in the fast PR profile, so CEO invariants are
treated as periodic-only under the W7c gate (every CEO conversion in this
branch keeps one free check per invariant); the gate judge
'plan-ceo-review/SKILL.md modes' still runs for CEO prompt changes. An
office-hours prompt change now has no fast PR E2E and is reported as
deferred to broad coverage rather than run.
Retirement cards:
- office-hours-spec-review: detects the Spec Review Loop's stated dimensions,
dispatch tool, iteration cap and metrics. Replacement: gen-skill-docs.test.ts
'SPEC_REVIEW_LOOP resolver' (round-3 cap, prepare/check/finalize commands,
verdict and round-file wiring) and test/office-hours-review.test.ts (verdict
parsing). Control: changing the round limit in
scripts/resolvers/spec-review.ts turns gen-skill-docs red.
- plan-ceo-review-benefits: detects the CEO /office-hours prerequisite and
mid-session offer. Replacement: gen-skill-docs.test.ts 'plan-ceo-review
offers /office-hours when no design doc exists and detects a lost user
mid-session' (previous commit). Control: 4/4 red mutations there.
- codex-offered-* (x4): detects the `command -v codex` probe, a fallback and
optionality. Replacement: gen-skill-docs.test.ts 'Codex outside voice is
offered behind an executed availability probe' (probe + not_installed
fallback for all four skills) and skill-validation.test.ts 'plan reviews run
the Codex outside voice default-on'. Control: 4/4 red there.
- non_test_callers: none; history:
|
||
|
garrytan
|
51f1d947ee |
test(W7h): pin the CEO /office-hours prerequisite structure free
Replacement check for the plan-ceo-review-benefits paid retirement (t14 F7). The paid case had a model summarize plan-ceo-review/SKILL.md and grep the summary for "office hours" and "design doc". The rendered text is static, so this free test checks it directly: the Prerequisite Skill Offer triggers on the design-doc check's "No design doc found", offers 'A) Run /office-hours now' and 'B) Skip — proceed with standard review', does not re-offer later in the session, and Mid-session detection (0A) offers `/office-hours`. Per the Eng Retirements item, CEO invariants become periodic-only once this gate case retires, so they keep this structural free check. Control (scratch worktree, generated plan-ceo-review/SKILL.md): removing the /office-hours option, the no-design-doc trigger, the mid-session offer or the no-re-offer rule goes red; rewording the pitch stays green (5/5). Log: .context/mutation-ceo-benefits.md. |
||
|
garrytan
|
6d7935eae8 | Merge remote-tracking branch 'origin/garrytan/opus55-wip-F1' into garrytan/opus-5-5-prompt-audit | ||
|
garrytan
|
f8043bd8fd |
test(W7c): qa-exploratory-callers keeps phase order and authority, drops duplicated prose
W7c (t04 F8, F16). One test file; only the generated-parent-path and
runner-prompt pins change. The validator-backed caller tests (observer,
checkpoint causality, read-only diff grammar, write watcher, deadline
authority) are unchanged.
Removed assertion groups (invariant -> remaining protection):
- Parent-path loop and preparation pins duplicated from qa-probe-gates
(Reads in order, "Reuse Step 4's surfaces ...", "Finish missing methods
before charters", isolation preflight, shared preparation order, the six
plan-execution sentences, the three shared loop contracts) -> owned by
qa-probe-gates (free, every host, with semantic mutation controls).
- Parent prose ("Templates cannot replace them", "Only the parent runs
report-only discovery", "Read agent/user updates and await results ...",
report-clean and not-run list duplicates of qa-caller-authority) -> caller
observer attribution tests in this file (free) and gate E2Es
review-exploratory-small-cli, ship-exploratory-small-cli (gate); report
gates stay in qa-caller-authority (free, every host).
- Shared-loop duplicates (preserve key/value, Q supplies observed, write the
report, return to step 2, pass requires) -> qa-probe-gates /
qa-caller-authority; kept the observationCommand/observed field definitions
and identity-hash rule here.
- Runner harness prompt sentences (t04 F16: receipt-interface prose,
appendSystemPrompt rules, scheduling prompt) -> reduced to one anchor per
rule; `at most ${maxTurns} assistant turns`, maxTurns/timeout/reserve
values, `(probe-...)`, `reports/review.md`, and every `not.toMatch`
scenario-answer ban stay. validateCallerEvidence remains the behavioral
oracle.
Kept as case-insensitive meaning checks: phase order 1-4 per caller,
review readiness follows report-only rules and never installs/imports
cookies/bootstraps, smoke stop at 5 minutes or 12 probes, G enforces the
deadline, never reset D/bypass G, plan checks remain required past the
smoke budget, /review incomplete and /ship blocked unless the user accepts
the named risk, repeat step 3 for affected checks, harness keeps parent
decision gates, turn limit never authorizes skipping work, never group
probes, Hard deadline UTC rule, HANDOFF.md read before completed records,
no checkpoint for deferred ideas.
Cosmetic pins dropped: AND / Never capitals.
Mutation log: .context/mutation-qa-exploratory-callers.md (15 mutations, 0
unexpected: 2 reorders, 2 markers, 6 safety red; 5 rewordings green).
|
||
|
garrytan
|
3a4c34f184 |
test(W7c): ship-workflow-clarity becomes a structural check of ship-only invariants
W7c (t07 F1, F2, F15). One test file: 59 tests / ~438 pins -> 33 tests.
Round-tagged titles (r6/r12/r22) are gone. Every invariant this file shared
with the other converted ship files is now checked once, in its owner:
ship-control-flow (work list, repair ranges, docs/test/push recovery,
adversarial finish order, Step 11.5 binding), ship-plan-completion-
invariants (evidence ledger, push/commit, waivers), ship-reentry-gates (stop
lines, docs reentry, NEW_TITLE), workflow-excerpt (review cycle order,
coverage/probe gates), qa-probe-gates / qa-caller-authority (QA loop).
Removed assertion groups (invariant -> remaining protection):
- Duplicated literals (45 repeated 2-7x, e.g. "Keep the same attempt counts
throughout the invocation" x7, "Docs, TODO edits ... make evidence STALE"
x5, the CYCLES >= 3 / fixes-below-cap / no-edits sentences, push-failure
and native-recovery sentences) -> one free check each in the owners above.
- Smoke/plan-check loop sentences and "Required even for small diffs ..."
-> qa-probe-gates (free, every host); gate E2E ship-exploratory-small-cli
(gate). Kept here: smoke guard started once before step 3, every host.
- Invocation-record field lists (receipt/handle/attempt field prose, "reuse
this invocation's saved level", FRESH/ALREADY_BUMPED level derivation,
receipt definition, start-token definition) -> kept `BUMP_LEVEL`, record
outside the product tree, approval scope, never borrow/replace a token,
REVIEW_START/PASS_START/DESIGN_START, `gstack-review-log --start`,
`gstack-wtree`; version derivation also in test/ship-version-sync.test.ts
(free); judge `ship/SKILL.md workflow` (gate lane).
- Reuse/judge-evidence prose ("original command, result/counts ...",
"Compare hashes or complete bytes", "consumed files, fixtures ...",
"complete expanded request, rubric ...") -> kept never resample, changed/
unknown dependencies rerun, mandatory reviews still run, freeze inputs;
judge `ship/SKILL.md workflow` (gate lane).
- Build/route prose ("If none exists, record not applicable ...", "declared
generation/build commands ...", "Run them and save results", "Explain why
other changes cannot affect it", "Inspect writer handles, including the
docs child", "Keep Step 8's original child report and counts", "Recheck
affected plan items ...", "append current evidence ...") -> kept stage
order, route-label order, prompts/templates count as behavior, no-change ->
stage 3, writer termination before another writer, stop until confirmed,
build-failed label, no substitute command; gate E2Es ship-docsync-*
(gate).
- Docs-collector prose ("Terminal completion or confirmed termination is
sufficient", "Require every field/type ...", "Compare saved base and input
hashes ...", "Otherwise save post-child hashes ...", "A stale snapshot is
neither ...", "Reconcile those before proceeding") -> kept attempt counting
incl. failed launches, request alone insufficient, exited child blocked,
HEAD/index unchanged + `files_updated`, protected-file exclusions,
`documentation_section`, failed/blocked -> recovery, actual user exception,
blocked status retained, same approved scope and exact content, retry
restarts stage 1; gate E2Es ship-docsync-missing-marker,
ship-docsync-launch-failure, ship-docsync-late-result (gate).
- Release/commit prose ("Record the release decision after a version was
actually written", "Skip unchanged versions and manifest-only repairs",
"Best-effort, non-interactive, non-blocking", "Group VERSION + CHANGELOG
+ TODOS.md ...", "Only the final VERSION/CHANGELOG commit gets the release
version and co-author trailer", "use the first changed component ...
(major/minor/patch/micro") -> kept ALREADY_BUMPED, the stale "skip if
ALREADY_BUMPED" ban, `baseVersion`/`currentVersion`, absent fourth
component is zero, no automatic rebump, digest tokens and pre-push digest
check, no Git tag, commit-before-push order; gate E2E ship-local-workflow
(gate).
- Coverage/design prose ("Count each generation-authorized attempt ...",
"including the initial audit, failures and zero-test results", "Two passes
already used ...", "read-only reassessment uses no pass", design-lite
ownership, same path/line merge, specialist stats, distinct defects) ->
kept the 2-pass and 30/5/2-minute limits, re-entry never resets, missing
permission is not approval, audit-only header/override order/no commit/
return decisions, child JSON, one item with both sources and stricter ASK,
neither pass substitutes; gate E2Es ship-coverage-audit,
ship-coverage-value (gate). Design-review-lite E2E is periodic, so the
merge rule keeps its free meaning check here.
- Spec/title/comment prose ("existing open PR/MR", "one match supplies the
existing title/identity", "Save the result for Step 19's recheck", "In a
new shell, restore ...", "Partial, deferred, failed, dropped or unverified
scope uses ...", "newest", "Try to auto-resolve if they are simple",
"distinguish an app from a library/CLI", "If the ask is App Store/
TestFlight distribution", "a new package.json or Cargo.toml alone ...",
"After edits, the full re-review repeats ...") -> judge `ship/SKILL.md
workflow` (gate lane); ship-apple-gate (free) for the Apple route.
Kept as case-insensitive meaning checks (stop/consent/approval/safety,
always kept): never invent credentials, ask for unknown targets first,
include the workflow in tests/review, frontmatter read as data never shell
source, exact spec_branch match, positive-integer issue number, Closes #N
only for fully completed scope, lookup failure/ambiguous match -> stop never
"no PR", never publish an unprefixed title, ambiguous conflicts -> stop,
AskUserQuestion and wait, unclear Apple target -> AskUserQuestion and wait,
confirmed app -> stop and Read, stop for repair (recommended) or accept each
named probe risk, skipping a fix is not risk acceptance, accepted risk is not
passing verification, queue without editing in Steps 10/11, never credit
outside coverage, do not report a clean review, unavailable outside
challenge never resets the three-cycle limit, records mismatch blocks
release, missing snapshots block regardless of HEAD, uncertain edits cannot
be exempted, scripts/dependencies/runtime config need live tests, named
testing gap A/B/C (recommended), reuse prior B only for same scope, declared
but unavailable suite blocks, explicit untested-scope approval, report the
gap never FRESH, no documentation writer after push, do not run create
commands for an existing PR, Greptile status truthfulness.
Kept unchanged: all stale-string bans, machine tokens and JSON contracts
(`gh pr list ... --json number,title,url`, glab list, `v$NEW_VERSION `,
`${NEW_TITLE:?...}` guard, exit 3, `--allow-paths ...`, `| FRESH (exit 0) |`,
`--label <lane> --expect-cmd`, `gstack-review-read`, `git diff <reviewed-tree>
<current-tree>`, `"status":"complete|no_pr|unavailable"`, specialist and
coverage JSON, `status:"unavailable"`/`completed:false`/`converged:false`,
`Pre-Landing Review: INCOMPLETE`, `ALREADY_PUSHED`), the threshold/branch/
installed-asset reference test, the section-index row check.
Cosmetic pins dropped: STOP / ONE / ONLY capitals and bold (t07 F2).
Mutation log: .context/mutation-ship-workflow-clarity.md (23 mutations, 0
unexpected: 4 reorders, 4 markers, 9 safety red; 6 rewordings green, all of
which turned the old test red).
|
||
|
garrytan
|
d3491c1734 |
test(W7h): pin the Codex availability probe free for the four offering skills
Replacement check for the codex-offered-* paid retirement (t14 F8): each of office-hours, plan-ceo-review, plan-design-review and plan-eng-review must render the executed `command -v codex` probe and its `not_installed` fallback mode in its generated SKILL.md + sections. These are bash lines the agent runs, so a deterministic grep replaces a paid model summary of them. skill-validation.test.ts already covers the default-on outside voice for eng/CEO/devex; this adds office-hours and plan-design-review and the fallback token. Control (scratch worktree, generated files): removing the probe from office-hours, plan-design-review or plan-eng-review, or the CEO not_installed branch, turns the new test red (4/4). Log: .context/mutation-codex-offering.md. |
||
|
garrytan
|
d636245c38 |
test(W7h): retire the bws session-awareness (ELI16) paid case
W7h retirement (t14 F5). Removes the case from test/skill-e2e-bws.test.ts and
its E2E_TOUCHFILES / E2E_TIERS (gate) / E2E_KINDS (rule) entries in
test/helpers/touchfiles-data.ts; scripts/paid-test-durations.json is
per-file and test/helpers/periodic-exclude-data.ts had no entry.
test/touchfiles.test.ts selection examples now name sibling cases with the
same touchfiles (skillmd-outside-git, skillmd-no-local-binary).
e2e-tier-alignment, eval-kinds and touchfiles stay green.
Retirement card:
- test: skill-e2e-bws.test.ts 'session-awareness' (gate, paid).
- detects: nothing gstack ships. The prompt coaches a mode that no longer
exists ("_SESSIONS=4, so ELI16 mode is active ... Re-ground them";
`git grep ELI16` outside test/ and CHANGELOG is empty) and says "DO NOT
actually call AskUserQuestion"; every asserted word (billing/acme,
payment/feature, stripe/checkout, recommend) comes from the prompt itself.
- non_test_callers: none.
- search_command: git grep -n "session-awareness\|ELI16" -- test scripts
- stronger_proof / replacement: the AUQ format's grounding line
'Project/branch/task: <1 short grounding sentence using _BRANCH>' is pinned
free by plan-review-calibration.test.ts (format tokens) and graded by gate
E2E auq-format-gate; session identity in the preamble stays pinned by
skill-validation.test.ts 'contains session awareness' and
gen-skill-docs.test.ts 'session awareness lives in gstack-skill-start'.
- history:
|
||
|
garrytan
|
8ac565d615 |
test(W7h): replace workflow-judge-cache source-grep counts with an invariant
W7h retirement (t09 F16, t13 F6) of the source-grep counts in 'workflow registration preserves model work and reserves only terminal-recording grace'. The stage-order check over runWorkflowJudge (the cheapest guard) and every behavioral callback test stay. Retirement card: - test: workflow-judge-cache.test.ts pins of skill-llm-eval.test.ts source: "execution: reused ? 'reused' : 'executed'", 'const workDeadline = started + JUDGE_MS;', 'const WORKFLOW_JUDGE_RECORD_MS = 5_000;', 'const WORKFLOW_JUDGE_TEST_MS = JUDGE_MS + 10_000;', "testName: 'review/SKILL.md workflow'", and the magic counts /}, WORKFLOW_JUDGE_TEST_MS\);/ x17 and /}, JUDGE_MS\);/ x7. - detects: any workflow eval added or removed, any constant reformatted. - non_test_callers: none. - stronger_proof / replacement: (a) a new invariant in the same test: every `await runWorkflowJudge(` registration closes with WORKFLOW_JUDGE_TEST_MS (the property the 17-count stood in for, now independent of how many evals exist); (b) the real-callback tests in this file exercise execute-once, reuse with provenance (execution reused/executed), timeout, supersede, expiry and the recording deadline. - history: |
||
|
garrytan
|
e149521414 |
test(W7c): ship-control-flow keeps step routing, tokens and gates, drops rule prose
W7c (t09 F1, F2, F3, F4). One test file.
Removed assertion groups (invariant -> remaining protection):
- Work-list prose ("Expand a repair into individual steps ...", "This
replaces the current item ...", "Add its destination only if ...", "The
saved list takes precedence ...", "For another repair, repeat rule 2 ...",
"The unchanged release steps follow", "Follow the saved work list",
"Keep CYCLES and scoped approvals across this repeat", "Keep the findings
and their approvals", "any further repair inserts its checks ...") ->
repairs insert ordered steps ahead of pending work: kept free checks on the
example sequences (`5 → … → 11.5`, `9 → 10 → 11` before 11.5, `5–11.5 →
12–14 → 16`, `14.5 → 15 → 16`, `9 → 10 → 11 → 11.5`), insert-before-
pending routing lines, no unlisted steps, same attempt counts, no reset of
the one-repair limit; judge `ship/SKILL.md workflow` (gate lane); gate
E2Es ship-docsync-recovery, ship-docsync-stale-after (gate).
- Queue-qualification sentence pins duplicated in ship-version-sync (t09 F4:
"require successful utility output ...", "`offline:false` qualifies ...",
"Offline output without that fallback ...") -> kept in
test/ship-version-sync.test.ts next to its script-behavior tests (free);
here the Qualify < Usable < No-usable order, `NEW_VERSION=CANDIDATE_VERSION`,
`>= NEW_VERSION`, `BUMP_LEVEL`, `currentVersion` and the approval line stay.
- Distribution prose ("List candidate distribution paths", untracked files,
manifest sentences) -> kept the New-artifact branch label, AskUserQuestion
and the no-release-during-/ship safety line; gate E2E ship-local-workflow
(gate).
- Dispatch/dashboard/coverage prose ("Omitting the flag runs the subagent in
the background", "keeping a fresh context", "Use one row for each entry in
step 1", "including its undetermined-percentage and test-only rules") ->
kept the `run_in_background: false` contract at all three sites, the
dashboard header/verdict tokens, no inline Skill fallback, coverage gate
not bypassed; gate E2E ship-coverage-audit (gate).
- Recovery/docs/test sentences ("If an attempt remains and either ...", "a
concrete launch/input/permission correction ...", "restart Step 16 stage 1
to regenerate ...", "This recovery also applies ... stage 5", "Content-
preserving commits keep valid evidence", "Retain queued fixes and restore
coverage", "If this pass made edits, resume at the next decision ...",
"Undispatched host-unsupported/gated specialists do not block", "count it
in the invocation record before launch", "Keep Step 9.4's incomplete
flags", "saved handle, original token and source", "unknown or missing
status is unavailable", "Finish the saved replies ...", "fetch and inspect
the remote, then merge ...") -> judge `ship/SKILL.md workflow` (gate
lane); gate E2Es ship-docsync-* (gate) and ship-skipped-queued-finding
(gate); the adjacent stop/approval lines stay free.
Kept as case-insensitive meaning checks (t09 F1/F3; stop/consent/approval
and safety lines): STOP and AskUserQuestion gates apply during repairs, a
repair never resets or expands approvals, children return evidence not
permission, prior-invocation audits never qualify, one repair/re-audit and
no third attempt (initial-plus-one, once), Otherwise -> stop unless the user
accepts the named docs risk, never silently refresh hashes, never run a third
audit, dispatched-reviewer-missing -> stop, CYCLES >= 3 -> stop with
`converged:false`, no fourth cycle, missing coverage cannot be replaced,
failed probes never clean, required-probe gate before Step 10, cannot waive
reviewer/test/security gates, native recovery stops the task first and
remains blocked, outside output cannot replace native, never attach new
tokens to old work, failed build stops shipping, never edit the child's
counts, test failure stops publication, push failure stops with no Step 19,
never rewrite history, never bypass failed guards, both redaction scans
before publishing, no release publish during /ship, only approval changes the
version, declining bootstrap is not test approval.
Kept unchanged: the executed plan-tune nudge shell test (3 roots), section/
step ordering checks, Step 11.5 placement and `review_binding.*` fields,
branch-label order in the adversarial finish (every host), all stale-
wording bans, `(recommended)` marker on Add tests.
Cosmetic pins dropped: STOP / ONE capitals (t09 F1, F3).
Mutation log: .context/mutation-ship-control-flow.md (21 mutations, 0
unexpected: 3 reorders, 3 markers, 9 safety red; 6 rewordings green).
|
||
|
garrytan
|
9c5aedea12 |
test(W7c): review-workflow-clarity keeps order, tokens and gates, drops judged prose
W7c (t06 #12). One test file. Each removed pin was the text fix for a past `review/SKILL.md workflow` judge finding; that judge (gate lane) still grades the prose, so the free test keeps only structure, tokens and gates. Removed assertion groups (invariant -> remaining protection): - Plan-audit sentences ("retain the exact command ...", "A mixed item contributes to both lists", "Keep external-state ...", "If only behavioral checks remain ...", "File-existence checks ... are static audit checks") -> audited deliverables vs deferred behavioral checks: kept free checks on the separation, never DONE from a diff, zero-deliverable no-waive, UNVERIFIABLE -> Step 4.7, read-only audit (no apps/APIs/state) and inspect -before-invoke order; gate E2E review-exploratory-small-cli (gate). - QA loop/report duplicates of qa-probe-gates / qa-caller-authority (probe loop sentences, shared contracts, report layout prose, "Only the parent runs report-only discovery", "Read agent/user updates ...", "Prepare one provisional QA section", smoke composition) -> owned there (free, every host); here only `5 minutes/12 probes`, headings, link-every-checkpoint, no second report, never overwrite another run, freshness vs current inputs. - Persistence/finalization prose (~45: "final zero-edit pass, reconcile", "by structural identity", "without requiring deleted pre-extraction blocks", "raw bytes equal the bound snapshot blobs", "Keep both through re-reviews", "Earlier core tokens remain unused", "Re-read its final- snapshot supporting source ...", "Fixed advice needs no skip coverage", "The required in-host adversarial result controls native completion", "Optional outside attempts keep their own incomplete records ...", etc.) -> kept free token list (REVIEW_START/PASS_START, COMPLETED, CONVERGED, CYCLES=0 once, `review_binding`, `snapshot_covered_paths`, sharedLibsFingerprint, `completed:false`, evidence_paths/helper_target), cycle limit (at most 3, Below 3 repeat, At 3 persist `converged:false`), section order, and the integrity gates below; judge `review/SKILL.md workflow` (gate lane); gate E2Es review-army-delivery-audit, review-army-json-findings, review-army-quality-score (gate). - Confidence-calibration prose (data loss/outage examples, missing field / nullable / race quoting, metaclass/ORM sentence, missing-literal sentence, learning-log sentence) -> kept the score table as a routing table (each band's action), `[CRITICAL]` / `[CRITICAL|INFORMATIONAL]`, no P0-2, `file:line`, force 4-5, never invent 7+, unverified-without-quote, gate before table; judge `review/SKILL.md workflow` (gate lane). - Shared-code reuse safeguard enumeration (assume-unchanged/skip-worktree, sparse index, symlinks, submodules, Git filters, encodings, fsmonitor, textconv, provenance, binding wording) -> enforced in code and covered by free tests test/shared-libs-snapshot-check.test.ts, test/shared-libs-evidence.test.ts, test/shared-libs-checker-interface- evidence.test.ts; kept the checker command/heredoc/JSON tokens and the fail-closed lines. - Misc definitions ("A probe is affected when ...", "An invocation is this /review run ...", "Step 1.5's provisional notes ...", "Otherwise record each considered specialist", "Core findings use the confidence gates below", "highest confidence", "distinct specialists", "all source names", "active or unknown reader/writer", Greptile "Before edits, inspect ...") -> judge `review/SKILL.md workflow` (gate lane). Kept as case-insensitive meaning checks (stop/consent/approval/safety): no edits before Step 5, readers returned or confirmed stopped, no edits in Step 4.8, stop without edits on unconfirmed readers (review and ship), stop at 3 cycles with no clean summary or fourth pass, failed/blocked probes mean incomplete, ship named-risk cannot complete review, native vs outside cannot substitute, structured-review gate, never carry coverage forward, never invent a binding, skipped advisory cannot suppress a defect, user-skip-only suppression, Greptile reply != code approval and wait for approval before editing, HIGH-impact gate options (A ends, read-only audit, no shipping authorization), never downgrade severity, never trust a supplied hash, missing metadata cannot dedupe, test creation needs approval, report-only review never runs setup/installs/cookie import. Kept unchanged: all marker-order arrays, manifest order, Pre-Landing summary after the logger, finish-token count and negatives, stale-wording bans (incl. `git diff origin/main`), ship/review routing negatives. Cosmetic pins dropped: STOP / NON / COUNT capitals; the test titled "resolves the judged ... ambiguities" is renamed to say what it keeps. Mutation log: .context/mutation-review-workflow-clarity.md (23 mutations, 0 unexpected: 3 reorders, 4 markers, 11 safety red; 5 rewordings green). |
||
|
garrytan
|
93d2d43c2c |
test(W7c): shrink gen-skill-docs to structural and meaning checks
W7c conversion of the sentence/caps pins in test/gen-skill-docs.test.ts (t01 F2-F9, F18, F19; t08 F12; t10 F4; t02 #5 duplicate STOP literal). t01 F1 ('Every commit must map to at least one bullet point') was already gone on the base. Pins (grep -cE 'toContain|toMatch|indexOf|includes') 1446 -> 1360; tests 457 -> 457. Untouched: the FRESH/STALE freshness check, the routing soft-policy negative, skill-start/end script pins, the SESSION_KIND spawned line, and the old-wording negative pins (delete-then-append, 0D/0F, gate "if applicable"). Machine markers stay byte-exact: NO UNRESOLVED DECISIONS, `**UNRESOLVED DECISIONS:**`, `Approval readiness: PASS`, `D<N> — <ROW-ID>: ...`, `## EXIT PLAN MODE GATE (BLOCKING)` anchors, `**Gate outcome: Blocked**`, helper commands/flags (gstack-office-hours-review prepare/check/finalize --design, --out-dir, --report, --unreviewed), helper stop reasons, analytics commands. Consent, stop, approval, decision-cycle and navigation lines (incl. the Eng W7c-gate lines at the old 2362-2370) are kept as short case-insensitive checks. Removed assertion groups (invariant -> remaining protection): - qa-only 'Never fix bugs' + 'NEVER fix anything' (t01 F8): report-only skill -> kept free: /never fix/i in Additional Rules + frontmatter no Edit/Glob/Grep. - qa 'WTF' token (t08 F12): fix-loop self-check -> kept free: /every 5 fixes/i. - TEST_COVERAGE_AUDIT sharedPhrases sentences and the three Test-step sentences (t01 F4): read source before tracing/diagramming in plan and ship -> kept free: order 'concrete source and test files' < Trace data flow < Diagram the execution < Quality scoring rubric in both, star/GAP markers, the plan/diff mode split, the Test review subheading structure test; gate judges 'plan-eng-review/SKILL.md sections', 'ship/SKILL.md workflow'. - 'REGRESSION RULE' / 'IRON RULE' labels (t01 F4): regression coverage is required -> kept free: REGRESSION RULE heading, /without regression coverage, that coverage is a critical requirement/i (plan), /regression test is written immediately/i (ship), plus the existing regression-contract tests. - '**STOP for each pending decision.**' (t01 F7, retired) -> kept free: 'Wait for its answer before applying that remedy' (control C1). - F19 rule sentences: step numbers dropped; kept free /prevent duplicate bumps, pushes and PRs, never verification/i, /native result never supplies outside coverage/i, /required-probe gate, never silently waive/i, /never offer a conversion/i, /actual counts, never estimates/i (deduped 2 -> 1). - SPEC_REVIEW_LOOP (t01 F3, about 74 sentence pins): - office-hours dispatch sentences (Before EACH dispatch, parent Read does not deliver, string unchanged) -> kept free: prepare --design/--out-dir, order mktemp < prepare < round-N.prompt.md < `dispatch`, /all preceding valid round files in order/i, /read the entire prepared prompt/i. - reviewer prompt sentences -> kept free: document/verdict path and schema wiring, mode sections ('The Assignment', 'What I noticed...'), dimension order, short checks for coaching protection, no invented customer answers, minor findings included, identical JSON, classify every prior finding, open questions vs committed behavior. Dropped: 'Could an engineer implement this...' ban (positive design-and-coaching check carries it), absence-is-not- confirmation, unsupported-claims and labeling sentences -> gate E2E office-hours-spec-review; prior-status validation in office-hours-review.test.ts. - convergence evidence sentences, 'Recording concerns does not mark them fixed', Disposition, observation/attempt counting, 'unavailable score is null', 'missing or invalid verdict is ... never PASS', 'Preserve the failed output' -> office-hours-review.test.ts (convergence, finalize, invalid artifact vs --unreviewed, null quality_score) + gate E2E office-hours-spec-review; kept free: helper stop-reason order and round 3, /before fixing any findings or dispatching again/i, /on a stop, do not fix again or re-dispatch/i, /do not edit that generated section/i, finalize before /existing user approval/i, --unreviewed wiring, /do not fabricate JSON/i, /not an approval gate/i. - CEO spec-review sentences (Read Agent's tool definition, omit field, consume completed review, Keep both consistent, duplicated 'both complete labeled texts'/'all five dimensions'/'Preserve the failure'/'citing the owning input'/'## Reviewer Concerns', 'Read these documents' ban, strategy-only and narrow-decision sentences) -> CEO is periodic-only, so kept free: one short check per invariant (wait tool, no-wait resume, no advance while waiting, three launches, stop after third, stop the loop, missing score, null grade, SCORE 1-10, not persisted, mkdir/append stop, save-failure stop, ITERATIONS, FOUND/FIXED/REMAINING order, both inputs in full, partial-input refusal, summary-is-not-the-plan, 1-10 score, PASS rule, cite input) plus the existing order/count/command checks. - Design approval reconciliation (t01 F2, about 79 pins): - Pass 5 sentences and 'AskUserQuestion once per issue. Do NOT batch. Recommend + WHY.' -> kept free: /stop\W+AskUserQuestion once per issue/i located after four short reuse/new-evidence/first-decision checks; batching -> auq-format-always-loaded.test.ts PER_SKILL_RULES ('One issue = one AskUserQuestion call'). - duplicate 'Never group distinct issues.' pins (2 -> 1) and the rest of the Design gate 0 sentences -> kept free: 0 < 1 order, short checks per approval line, refresh < Read-back gate < rerun order; gate judge 'plan-design-review/SKILL.md passes'. - CEO readiness/gate/0D sentences -> kept free: one short check per approval line, readiness order, `Approval readiness: PASS`, gate order Read-only < Verify readiness < 1. Read, route order, /stop for the actual answer/i in step 4, `D<N> — <ROW-ID>` header, row fields; admin-route sentences -> ceo-workflow-clarity.test.ts kept checks. Dropped: 'No report or completion log is needed' (order check carries it), 'Save or present the updated plan', 'checked row IDs...' (marker kept), strategy-only/narrow-decision sentences (default kept), cross-model/integration sentences reduced to external-only and native-fallback checks. - Eng readiness/procedure sentences -> kept free: short approval checks, gate numbering/order, AUQ call < /stop until the actual answer arrives/i < Record the answer, recovery order; 'separate it and rebuild this comparison' -> auq-format-always-loaded.test.ts PER_SKILL_RULES. - Exit gate 'contract violation' sentence (t01 F9, design/devex/codex) -> kept free: heading anchor + /do not call ExitPlanMode/i inside the gate. - Unresolved-decisions gate (t01 F5): '(MANDATORY', 'Never omit'/'never omitted', 'FINAL non-whitespace line is the unresolved-decisions', 'FAILS the gate', 'DROP the current skill' -> kept free: NO UNRESOLVED DECISIONS, /final\s+non-whitespace\s+line/i, `**UNRESOLVED DECISIONS:**`, /missing\s+status [^.]*fails/i in every gate skill and the resolver, /avoids double-counting/i, the old soft-wording negative pin. - delete-then-append 'never mid-file' / 'Do NOT replace the section in place' (t01 F6) -> kept free: 'delete-then-append flow' + /append the new report at EOF/i; old-wording negatives kept. - design-html '**Never include by default (AI slop blacklist):**' (t01 F18) -> kept free: /AI slop blacklist/i, catalog id comment, lib/design-catalog.ts. - CODEX_BOUNDARY 'Do NOT read or execute any' casing (t10 F4) -> kept free: /do not read or execute any/i in every codex-calling skill. - CEO 0D order anchor '**STOP for the actual answer...**' (one of the 8) -> kept free: /stop for the actual answer/i at the same position. Mutation pass (scratch worktree, generated SKILL.md/sections plus spec-review.ts and lib/office-hours-review.ts; the freshness test is excluded with -t because any mutation makes it STALE): 43/43 as expected. 25 kept-step, marker, safety, numeric and retirement-control mutations go red; 18 rewordings of dropped sentences and lowered-register STOP/NEVER/FAILS lines stay green. Log: .context/mutation-gen-skill-docs.md. Runtime (Bun 1.4.0, wall, 3 runs): before 4.42/4.49/4.70s (median 4.49s), after 4.42/4.53/4.63s (median 4.53s). |
||
|
garrytan
|
af697dc6ec |
test(W7h): retire the paid first-task-scaffold E2E duplicate
W7h retirement (t12 F12). Removes the case from test/helpers/touchfiles-data.ts
(E2E_TOUCHFILES, E2E_TIERS periodic, E2E_KINDS rule) and both
scripts/paid-test-durations.json entries; test/helpers/periodic-exclude-data.ts
had no entry. e2e-tier-alignment, eval-kinds, eval-detach-timeout-floor,
periodic-exclude-policy, touchfiles and paid-shards tests stay green.
Retirement card:
- test: test/skill-e2e-first-task-scaffold.test.ts ('first-task-scaffold',
periodic, ~$0.02): Haiku is told to run exactly one detector command and the
test greps the tool result for code_node / greenfield.
- detects: detector bucket regressions in bin/gstack-first-task-detect.
- non_test_callers: none.
- search_command: grep -rn "first-task-scaffold" test scripts .github
- stronger_proof / replacement: test/preamble-first-task-scaffold.test.ts
(free; executes the real detector on fixture repos for every bucket,
including code_node and greenfield, plus first-run gating and telemetry in
bin/gstack-skill-start). The model added nothing: the prompt dictated the
exact command.
- history:
|
||
|
garrytan
|
851328b35f |
test(W7c): workflow-excerpt keeps heading order and tokens, drops judged prose
W7c (t04 F5, F6, F7, F15; F1/F2 already landed with W7a). One test file.
This file guards judge-input fidelity; the judges that read these excerpts
grade the prose itself.
Removed assertion groups (invariant -> remaining protection):
- Ship rule sentences ("Use the documented selector and pre-merge command.",
"During pre-flight, read the existing review log", "Tests must pass or
retain approval ...", "For another repair, repeat rule 2 ...", "The
unchanged release steps follow", "Use this example only after confirming
...", "Retain queued fixes and restore coverage", "Each commit must work
independently", "Keep actual outcomes and incomplete flags", the long
"Continue to Step 9.2 with the core/design-lite findings ..." form, the
Y'd/D'd evidence sentence) -> judge `ship/SKILL.md workflow` (gate lane),
which reads this exact excerpt; kept free checks on step routing (`Step 4`,
`Step 8.1`, 9.2 -> 9.2.1 QA -> 9.3, `Insert Step 5, affected Steps 6–8 and
all of Step 9 before the pending Step 10`, `5 → 6 → … → 11.5`,
`continue to Step 16`), tokens (`BASE_VERSION`, `VERIFY_RESULT stays
fail`, `"partial":N,"not_done":N`, "intentionally dropped"), and the review
cycle order (9.1 < 9.2 < 9.2.1 < 9.3 < 9.4) as a structural check.
- Outside-review routing sentences ("Other preflight mode, including harness
mismatch", "Outside execution or output validation fails", "Retain its
output and diagnosis ...", "No prompt, outside process or native
replacement", "Other preflight failures retain their printed diagnosis",
the full fallback-list regex) -> disabled means no outside call: kept free
order of disabled/prompt/ready/fallback blocks, `outside_status` tokens,
`_DISABLED_REVIEW_MODE`, disabled recheck before dispatch, "disabled means
no replacement", fallback-not-completion; judges
`document-release/SKILL.md workflow`, `plan-eng-review/SKILL.md sections`
(gate lane).
- Eng/design prose ("quote the motivating plan requirement", "Use a scoped
Edit to save this record ... Leave other choices unchanged" full form,
"edit the plan and re-rate") -> judge `plan-eng-review/SKILL.md sections`
and `plan-design-review/SKILL.md passes` (gate lane); approval meaning
checks kept.
- Retro prose ("does not require saved history", "PRs referenced", "not
newly added test cases", "is the live **team** streak", "draft the
tweetable summary using the format in Step 14, then save") -> judge
`retro/SKILL.md instructions` (gate lane); kept headings, `prs_merged:
null`, `streak_days`, the one-second compare window, Capture Learnings
after Step 14, no `$(date`, two `today="<today>"`.
Loosened to case-insensitive meaning checks (stop/consent/approval and
safety lines, always kept): STOP for each pending decision (now a regex
anchor between Step 5 and the Test Plan Artifact), STOP until the actual
answer arrives, no remedy/next section/ExitPlanMode while a choice awaits an
answer, evidence-not-approval, reopened choices need answers, wait for
approval (design), AskUserQuestion once (t04 F6), per-finding approvals,
dispatched-reviewer-missing -> stop, CYCLES >= 3 -> stop with
`converged:false`, no fourth cycle, required-probe gate before Step 10,
failed probes block continuation, coverage-gate not bypassed, failed/unrun
probes not passed, three snapshots match, document-release VERSION-absent
rule, never create an empty commit, `read any other SKILL.md` (t04 F15),
AskUserQuestion-and-wait in the capture prompt.
Kept unchanged: heading/step order checks, `## Review Sections` single-copy
check, fail-closed marker tests, redact-before-edit ordering, approval gates
outside subagent prompts, plan-tune auto-decide output lines, CEO capture
harness contract, deploy formula execution, all stale-string bans.
Cosmetic pins dropped: STOP / ONCE / NOT / OTHER capitals.
Mutation log: .context/mutation-workflow-excerpt.md (21 mutations, 0
unexpected: 3 reorders, 3 markers, 8 safety red; 7 rewordings green).
|
||
|
garrytan
|
958f27a2b8 |
test(W7h): retire the literal 75 KB autoplan budget duplicate
W7h retirement (t04 F18). test/skill-preflight-budget.test.ts asserted the
same four-skill autoplan sum twice: once against
AUTOPLAN_PREFLIGHT_BUDGET_BYTES * 3 (25,600 x 3 = 76,800 = 75 KiB) and once
against a literal 75 * 1024, which would drift silently if the constant
changed.
Retirement card:
- test: skill-preflight-budget.test.ts 'autoplan total under 25 KB
instruction cap × 3 (75 KB instruction budget)'.
- detects: the four plan-* skills' brain preflight + refresh + write-back
instructions exceeding 75 KiB.
- non_test_callers: none.
- stronger_proof / replacement: 'autoplan: sum across 4 plan-* skills stays
under AUTOPLAN_PREFLIGHT_BUDGET_BYTES × 3 (instructions)' in the same file,
which reads the constant from scripts/brain-cache-spec.ts.
- history:
|
||
|
garrytan
|
6423617fe6 |
test(W7h): retire spec-template-sync as a duplicate of the freshness check
W7h retirement (t05 F18; Eng factual correction: the file takes 145 ms, its "2 minutes" was the 120 s timeout, so this is a duplicate-only retirement). Removes its scripts/free-test-durations.json entry. Retirement card: - test: test/spec-template-sync.test.ts (regenerate /spec into an out-dir and compare spec/SKILL.md + spec/sections/gate-and-file.md with the tracked bytes; check the generated header). - detects: /spec template edited without regeneration, or a hand-edited generated file. - non_test_callers: none. - search_command: grep -rn spec-template-sync package.json scripts test .github - stronger_proof / replacement: gen-skill-docs.test.ts 'generated files are fresh (match --dry-run)' (FRESH for every skill, no STALE anywhere; the dry-run covers carved sections such as spec/sections/gate-and-file.md); the generated-banner checks in the same file cover the header. - history: |
||
|
garrytan
|
d2f53fcb9c |
test(W7h): retire CEO TODO-menu prose pins from plan-review-calibration
W7h retirement (t04 F11; W7c list). The calibration corpus and
buildPlanReviewDecisionPrompt never read this CEO prose; the AUQ format
tokens (consumed by plan-tune and judges) and the calibration-harness
outcome tests are unchanged.
Retirement card:
- test: plan-review-calibration.test.ts, the six CEO TODO-menu pins ('Only
unanswered TODO proposals reach this menu', 'Do not ask again about an item
already deferred, skipped or kept', 'carry its actual answer and destination
forward', 'Resolve each remaining proposal through all four steps of 0D,
using the menu below', 'Never batch TODOs — one per question').
- detects: edits to plan-ceo-review TODO-menu prose.
- non_test_callers: none (test-only reads of the template).
- search_command: grep -rn "Never batch TODOs\|Only unanswered TODO" test/
- stronger_proof / replacement: skill-ceo-section-ordering.test.ts 'TODOS.md
updates' slice keeps /i meaning checks for unanswered-only, no re-asking
and one TODO per question (CEO invariants are periodic-only under the W7c
gate, so they keep a free check); 'route through 0D' stays covered by that
file's 0D route checks.
- history:
|
||
|
garrytan
|
08f53f3bc3 |
test(W7c): qa-caller-authority keeps authority and safety lines, drops report prose
W7c (t10 F12, F13). One test file; still renders every host x caller.
Removed assertion groups (invariant -> remaining protection):
- `STOP and wait` / `STOP for the answer` / `AskUserQuestion and WAIT`
capitals (t10 F12) -> approval before `./setup` or install: now an order
check (explicit approval < "only after approval" < `cd <SKILL_DIR> &&
./setup`; OK to proceed? < setup; AskUserQuestion-and-wait < install only
the actual choice), plus case-insensitive "stop for the answer". Stronger:
the mutation that moves setup ahead of the approval ask now goes red (the
old test stayed green). New negative control removes the approval ask.
- Report-layout prose for /review and /ship (keep metadata/outcome tables,
demote headings one/two levels, fields as subsections, keep scores
separate, save baseline normally) -> one final linked report section:
kept free checks on the `## Exploratory QA and Verification Results` /
`### Browser results` / `## Exploratory QA` headings, link every checkpoint
and no second report; gate E2Es review-exploratory-small-cli,
ship-exploratory-small-cli (gate).
- Smoke-pacing prose ("one success and the riskiest changed failure/edge",
"Required even for small diffs ...", "Required: plan commands/assertions,
listed separately", "Other ideas are optional, untested") and duplicate
report-status pins -> kept the numeric `5 minutes/12 probes` bound and the
report-clean-only-when-all-required-pass gate; gate E2Es
ship-exploratory-small-cli, ship-exploratory-plan-checks (gate).
- Dedup prose ("Combine saved `findings` ...", "If both history ... classify
normally", "Report the suppressed count once if nonzero", "If N > 0, print
once:", "Otherwise skip the summary") -> only user `skipped` findings are
suppressed: kept free order check (read decisions before matching) and the
`skipped` / never `fixed` / `auto-fixed` lines; gate E2E
ship-skipped-queued-finding (gate).
- Preparation sentence pins duplicated from qa-probe-gates (Reads in order,
no repeated Reads, "in full and select the surfaces", "Write a **charter**",
"charter** per behavior", "announce finite command timeouts") -> owned by
qa-probe-gates (free, every host); this file keeps scope < methods <
charter structural order for every caller.
- Learnings prose ("Otherwise skip this optional lookup", "Reading old notes
never requires writing new ones") -> report-only lookup never writes: kept
free checks (directory already known, caller permits, do not run the
search helper, no bash/gstack-config/AskUserQuestion, injection throw); the
query interpolation stays checked via `"webhook retries"`.
- Misc prose ("Re-run affected contracts and adjacent happy paths ...", "Run
only for selected browser surfaces", "Confirm each issue by retrying once",
"parent applies queued fixes", "No test framework detected", "Report QA
setup blockers", "required QA") -> judge `qa/SKILL.md workflow` and
`review/SKILL.md workflow` (gate lane); the adjacent kept gates remain free.
Kept as case-insensitive meaning checks (authority, consent, safety): probe-
local blocker rule, report-only never bootstraps/invokes skills, unknown
caller -> report-only, declined -> blocked, credentials/passwords/other tabs,
local-target ownership, no unit-test/curl substitute, functional-only reads
nothing browser-side, never-pass list, named-risk acceptance, noninteractive
-> blocked, test_stub ASK approval, ship waiver cannot complete review,
caller-required rechecks, blocked rechecks never verify repairs, no edits
during Step 4.8, no inner repair loop, REVIEW_START/token-once, native pass
required for 5.8, coverage-cap stop lines, report-only never commits or
dispatches, CI/test-weakening ban, never delete a valid red regression.
Kept unchanged: machine tokens (`bun G start D`, `$B snapshot -i`,
`DIFF_START`, `CONSOLE_ERRORS=`, `$B handoff/resume`), resource paths, the
`If A and allowance remains:` count, `Minimum = 60%, Target = 80%`, all
`.not.toContain` stale-workflow bans, existing negative controls.
Cosmetic pins dropped: STOP / WAIT / LOCAL / Never capitals.
Mutation log: .context/mutation-qa-caller-authority.md (23 mutations, 0
unexpected: 3 reorders, 3 markers, 11 safety red; 6 rewordings green).
|
||
|
garrytan
|
561b82e861 |
test(W7c): shrink plan-review-cases to structural and meaning checks
W7c conversion (t03 F1-F4; t02 F4/F5 duplicate STOP/threshold pins). Pins
713 -> 208 lines by the brief's grep (toContain|toMatch|indexOf|includes);
prose assertions 592 -> 366 (toContain/toMatch calls plus expectAll regex
rules; expect() calls per run 3232 -> 2808). Tests 218 -> 215. t03 F1 (stale
~/.gstack/projects discovery path) was already fixed by W7a on the base; its
$GSTACK_STATE_ROOT check and the stale-path ban are kept.
Kept byte-exact or structural: marker-order arrays (### Prepare an unanswered
choice -> ### Send once and wait -> ### Record the answer, heading list
equality, Pending-record checkpoint, finish steps 1-6, entrypoint recovery
order, output placeholder order), the parsed write-policy table routes, the
generateAntiShortcutClause host matrix, every not.toContain(stale) ban,
numeric limits (8+ files / 2+ new classes, fewer than 8 / 2, under 5 words,
2-3 options, six-column / five-field / four-step), consumed tokens and field
names (`scope accepted as-is`, `scope reduced per recommendation`, FULL_REVIEW/
SCOPE_REDUCED mapping, **issues_found**, SOURCE=/OUTSIDE_STATUS=/STATUS=,
OUTCOME=, `BLOCKED`, `## GSTACK REVIEW REPORT`, `currentDecision`, D<N> and
ROW-ID formats, prescribed A-D menus), placeholders, commands/paths and all
picker behavior tests (unchanged). STOP/wait/approval/safety lines become
case-insensitive meaning checks located between their headings (e.g.
/stop until the actual answer arrives/i between Send once and Record the
answer; /do not replace the section in place/i plus /append the new report
at EOF/i). CEO invariants (periodic-only once plan-ceo-review-benefits
retires) each keep one free check. Eng/Design/DX invariants keep one free
check too; nothing below relies on a periodic-only E2E.
Removed assertion groups (invariant -> remaining protection):
- CEO route/narrow depth (:19) full sentences: admin answers approve nothing,
plan decisions run steps 1-4, settled choices go to 0E -> kept free:
Admin < Plan decision order, checkpoint order, five short /i checks, stale ban.
- Eng evidence paths/labels (:33) 'Prior Learnings configuration', path rule
sentences -> kept free: the three path field labels, /never invent paths/i,
`not available`, label tokens, stale ban.
- Eng first report save (:47) 9 requirement sentences -> 5 short /i
requirements with the same remove-one-must-throw self-control; parent-dir,
original-plan-copy and scope-record clauses dropped (initialization
invariant kept by target-header/preserve/no-findings/placement/terminal-
report checks; collision suffix kept in the write-policy test).
- Eng setup selectors (:79) and its duplicate in the big Eng ordering test:
selectors approve no remedy -> kept free: /these selections approve no
engineering remedy/i, B selectors, no pre-answer ledger, stale ban; the
'For the next choice' sentence -> kept free: transaction marker order.
- Eng performance (:94) exact bullet list -> kept free: four concern
keywords from the parsed bullets; Decision-procedure sentence -> kept free
in the four-section test.
- Eng complexity/MODE (:105) per-target counting, 'Count each once', 'Keep it
current', 'Continue to Section 1' sentences -> kept free: count-selected-
work, both thresholds, MODE tokens and mapping, scope-reduction meaning,
'approves no pending remedy', Continue-to-Section-1 in the C order array.
- Eng three-stage transaction (:125) sentence markers -> same 13-step order
array with short /i anchors and the remove-one-must-throw control.
- Eng Test Plan refresh, coverage trace (:150, :162) sentences -> kept free:
placeholder count/order, prepare step count, short /i checks; coverage audit
also graded by plan-eng-coverage-audit (gate).
- CEO fallback (:178) sentences -> kept free: host mode token matrix, status
field values, /final report is written later in Required Outputs/i, ban.
- Persistence per skill (:229) Eng duplicates ('If the required log is
forbidden', 'Neither supplies', FULL_REVIEW mapping, 'after successful
Read-back') -> kept free in the readiness, complexity and write-policy
tests; CEO 'retain its diagnostic' -> kept free: /do not claim that entry
was recorded/i and three other CEO log checks.
- Report resolver per host (:261) duplicate 'Do not choose another
destination', 'Review record and write policy', 'full review output',
explicit/named/active sentences -> kept free: target precedence order;
CEO writer's seven sentences -> kept free: six-anchor order ending at
'4. **Read-back gate:**' plus the blocked-return check.
- Generated carriers (:327) bold STOP pin -> /stop until the actual answer
arrives/i in the dispatch order array.
- Big Eng ordering test (:358) duplicates of the gate describe (one question
object, 8+/STOP, both-retain, setup, separate instrumentation, keep chosen
values, calibration<procedure<scope, Decision procedure < Architecture,
outside-voice consent lines) -> kept free once each (boundary list, queue
test, preparation order array now ending at ### 1. Architecture review);
'Use existing tests ... bounded probes', 'post-answer scope summary',
'Compare only the file/class arrangement', 'contracts and approved
security...', 'With no proposed cuts', 'Present numbered...' sentences ->
kept free: build-boundary order check, structure-question and
unapproved-fixes checks, C-step order array.
- Eng gate describe: exact-duplicate pins (same sentence 2-4x in one test or
repeated per section inside the 4-section loop) -> one copy kept; ledger
placeholder descriptions (Plan baseline/Runtime evidence/Actual answer/
Accepted scope text) -> kept free: parsed ledger field-name list, R1 header,
State values, A)/B) selectors, ledger position.
- Gate tests 'identifies commitments before comparing values', 'common new
defaults...', 'exact prior answers authorize...', 'finding evidence, stable
decision identity...' merged: every non-duplicate invariant (optional
depths, shared new value needs approval, unknown risks visible, drafts grant
nothing, reopened ID keeps D<N>, test scores not answer status) -> kept free
as one /i check in 'current contracts...' or 'all four section gates...';
their order checks -> kept free: prepare<identify<compare<save<send<record.
- Remove-old-occurrence, 'Otherwise leave the remedy pending', 'no question
or comparison grid is needed', 'never the saved-report gate', 'Compare the
label separately', 'When revising, replace the whole current payload',
'It may be the selected plan', 'Present each completely as not persisted',
'If none is permitted...', 'A forbidden write follows its row', 'Run
Learning hooks ...', 'Refresh affected tests...', 'If startup values...',
'instead of creating another approval row', 'Keep that remedy pending' ->
same invariant kept free by an adjacent check (State replace/once checks,
approves-no-implementation, factual-correction, recovery routing, payload
rebuild, write-policy routes, return-to-entrypoint count, late-change and
blocked checks, common-work check).
- Worked example anchor sentence -> anchored on the Compare section and
'Ask about jitter first:'; parsed R1/R2 table equality kept.
- Outside-voice queue: dead generic branch (all three skills `continue`
before it; never executed) removed -> none needed (unreachable). Eng
procedure re-checks inside the queue (5 sentences) -> kept free in the
gate describe. Shared CEO/Eng/DX scope-revision sentences hoisted to one
shared /i block per skill; CEO 'Keep candidate scope ...' etc. -> kept free
as short /i checks; prescribed A-D menus kept exact.
- Design register (:1771) sentences -> kept free: grounding-before-question
order and three /i checks; 'Surface real missing decisions' dropped
(positive restatement of the same rule).
Gate-tier coverage that also grades this prose: judges 'plan-eng-review/
SKILL.md sections', 'plan-design-review/SKILL.md passes', 'plan-ceo-review/
SKILL.md modes'; E2Es plan-eng-coverage-audit, plan-review-report,
plan-devex-review-plan-mode (gate). Cosmetic pins dropped without controls:
bold/caps STOP and NOT, issue-number comments unchanged.
Mutation pass (scratch worktree; eng/ceo/design templates and the
outside-voice, testing and review-dashboard resolvers): 33/33 as expected;
24 kept-step/marker/safety/numeric mutations go red, 6 rewordings of dropped
sentences and 3 lowered-register STOP/NOT lines stay green. Log:
.context/mutation-plan-review-cases.md.
Runtime (bun test, Bun 1.4.0, median of 3): 0.85 s -> 0.77 s.
|
||
|
garrytan
|
6a83ba86c3 |
test(W7c): shrink skill-ceo-section-ordering to structural and meaning checks
W7c conversion of test/skill-ceo-section-ordering.test.ts: t06 #1-#6, t05 F5 (cross-reference: the remaining bold-STOP pins), t13 F3 / W7.md CRITICAL RULE anchor. #1 (ASK_QUESTIONS_HEADING anchor) was already on the base branch; #5 now slices the blocked chat return to the next "\n## " heading, so removing or renaming `## Review Log` cannot widen or break the window. Pins (grep -cE 'toContain|toMatch|indexOf|includes'): 647 -> 388 lines; toContain( 535 -> 136; expect() calls per run 1657 -> 1399. The carve block, section-capture fixtures and validateCeoReviewCompletion tests are unchanged (t06 keep list) except two exact-duplicate 0D pins. CEO invariants count as periodic-only (plan-ceo-review-benefits retires), so every invariant below keeps a free structural or case-insensitive check. Gate-tier judge that also grades this prose: 'plan-ceo-review/SKILL.md modes'. New helpers: ordered() (exact labels/headings/placeholders/table cells as strings, meaning anchors as /i RegExp) and between() (fails loudly on a missing end anchor instead of silently running to EOF). Removed assertion groups (invariant -> remaining protection): - t06 #2 / t05 F5: '**STOP for the actual answer, even for a lone option.**' (x6 exact pins + 3 split/indexOf anchors) and '**STOP for the answer**' (x3): wait for the actual answer before recording -> kept free: one /stop for the actual answer/i per slice (template 0D in dispatch order between the ask-one-row check and the answer save; every generated host carrier between **4. Ask and **Post-answer checkpoint:**), /stop for the answer/i in 0E step 3 (template and generated), plus the Pre-question < 4. Ask < Post-answer checkpoint order and the 2-checkpoint count. Dispatch slices now end at **Post-answer checkpoint:**. - t06 #4 'Recommendation is ALWAYS present': recommendation label contract -> kept free: '(recommended)' and 'AUTO_DECIDE depends on it' in generated SKILL.md; exactly-one-(recommended) rule in 0D's currentDecision table. - t06 #6 and other bans of retired phrasings ('If the section has findings, you MUST call AskUserQuestion', 'Otherwise, use AskUserQuestion for each finding', 'After each section, pause and wait for feedback', 'Save pending rows before comparisons', 'using the Step 0E mode-handoff format', 'Save a chat-only plan', 'quality bonus, not a gate', 'If no file is in scope, skip this section', 'always write', 'For every question', 'for architecture choices', 'These approaches differ in coverage (...)', 'Finish with ExitPlanMode', 'Before summaries, review logs or next-step menus...', "put the complete comparison in the question's brief", 'Use this exact route:', 'citing its ID', /\blog (?:with|that ID)\b/): same invariants -> kept free: the positive checks (one **Decision gate.** per section wired to Analyze → Resolve → Apply; pending-rows-before-comparisons order; **Otherwise:** in the 0H order; report not persisted + Read-back gate; write-when-permitted incl. zero tasks; Outcome routing count 1; record-provenance-after-handoff order; literal dispatch-object comparison). 'Do NOT proceed ... 0C-bis' becomes not.toContain('0C-bis') on 0D. Short stale-token bans stay (<approved 0D approach>, reference | commitment | current value, ___ methods, retry once, telemetry has logged, NUMBER + LETTER / "3A", the Forbidden-storage regex, removed menu option, unbound placeholders). - t06 #3, magic counts: literal 11 sections / toHaveLength(7) closing steps -> kept free: section numbers 1..N derived from '## Review Sections (N sections' and checked against the ### Section headings; closing-step count derived from the 7 route labels. - Decision cycle / storage / comparison sentences (Start at step 1, run steps 2–4 only when needed, mode-change pointer, storage-policy save sentence, Copy the grid, Validate every field, Show unchanged/shared/pending repeated x4, other rows stay fixed or pending x5, never prewrite x3, Build one currentDecision x3, Ask one row per call x6, Save the answer reference x5, Keep independent changes separate x4): full decision protocol in order -> kept free: label order 1. Check < 2. Record < 3. Compare < grid < Pre-question < Read-back < 4. Ask < Post-answer (template + every host), one /i check each for pending-rows-before-comparisons, save-before-ask, amend only authorized work, failed save stops, decline-all stop, show unchanged/shared/pending, independent changes separate. Admin/plan routes also kept in ceo-workflow-clarity.test.ts. - Handoff/provenance sentences (After selection, send brief chat, Keep rows separate, logging-row repeats, Section 1 current-scope sentences): mode handoff applies before questions and logs provenance after -> kept free: /before tools or further questions/i, handoff < Other selections < provenance < selecting-a-mode order, the two exact output-format lines with <rows or none> / <Application and rationale>, auto_decided true/false fields, 'No new approach decision was needed' (quoted chat output). Section 1 dispositions -> ceo-workflow-clarity's Section 1 test. - Scope/mode-selection sentences (REDUCTION, HOLD and SELECTIVE's HOLD checks; For both expansion modes...; Skip/Defer destination repeats; present both inputs for final approval; 0E counts changed files; Count all deliverables; >15/greenfield repeats; SELECTIVE balances...): scope decisions per item with exact destinations -> kept free: prescribed menu label orders (A Add/B Defer/C Skip; A Defer/B Keep; TODO A/B/C; depth A/B), Defer -> TODOS.md + NOT in scope, Skip -> no TODO, /ask separately/i, /all four 0D steps for each unanswered addition or deferral/i, routing rows and >15 / 8-file / 2-class numbers (once each; ceo-workflow-clarity keeps the reuse-count checks), carve block's document-approval order. - TODO menu (per plan-review-calibration t04 F11 retirement, which names this file as the remaining free protection): kept free in the TODOS.md-updates slice: /only unanswered TODO proposals reach this menu/i, /do not ask again about an item already deferred, skipped or kept/i, /one per question/i. - Resolver prose (spec review, plan report, tasks emit, exit gate, codex plan review, brain cache refresh, approval readiness, design clause): each sentence pin -> one short /i check per invariant on every host (five dimensions, at most three launches, required save stops, destination-file Read/Write order, zero tasks, unknown files JSONL format, ExitPlanMode only after checks, forbidden history, outcome routing and skip routes, native fallback never outside coverage); exact strings kept only for machine/output tokens (NO UNRESOLVED DECISIONS, '- + N unresolved from prior reviews', STATUS/SOURCE/OUTSIDE_STATUS assignments, CROSS-MODEL, OUTSIDE COVERAGE, 'N findings; R resolved; U unresolved', quoted report phrases, Gate outcome: Blocked, Read-back gate). Dropped: 'If both counts are zero', 'For **Outside Review**, use ...', 'Enter after either an external reviewer...', 'identify the causal mechanism', 'A credible material risk...', "check the assembled set's capacity", 'including findings that needed only factual correction', 'show unsaved fields and errors', Strategy-only tasks sentence, verified-report-can-pass sentence -> neighbouring kept /i checks in the same slice plus the gate-tier 'plan-ceo-review/SKILL.md modes' judge. - Review-section prose (continuity requirement lists, test-review and outside rule duplicates, strategy-only diagram sentences, Review facts repeats, Stage 3 publishes it, Count a reopened choice once, Keep them before the terminal report repeats, EXIT gate verifies..., no report or log needed yet, then run telemetry as the last review operation): per-section gate and ledger continuity -> kept free: Analyze < Resolve < Apply, Resolve paths via check-scope < correct < record order, /resolve critical risks now/i, /review only; do not change code/i, ledger header + status tokens, one /i per approval/stop line; reopened-once lives in ceo-workflow-clarity; terminal order kept as cache refresh < {{BRAIN_CACHE_REFRESH}} < Telemetry < finished < Call ExitPlanMode < handoff. - Duplicate pins removed outright (same string asserted 2-6 times across tests, incl. carve-block 'Ask one row per call' and bold-STOP pins): each keeps its single remaining check named above. Cosmetic pins dropped without controls: bold/caps (**STOP**, ALWAYS, MUST), double compactProse wrappers. Mutation pass (scratch worktree; templates, generated SKILL.md and review-dashboard resolver): 39/39 as expected; 28 kept-step/marker/ placeholder/safety/consent/numeric mutations go red (incl. the three TODO-menu lines), 11 rewordings of dropped sentences, the CRITICAL RULE heading rename, re-adding the retired #6 sentence, and lowered-register STOP lines stay green. Log: .context/mutation-skill-ceo-section-ordering.md. Runtime (Bun 1.4.0, 3 runs): before 2.05/1.84/1.77 s (median 1.84), after 1.87/1.80/2.08 s (median 1.87); dominated by runGeneration and the capture subprocess fixtures. |
||
|
garrytan
|
d0fc479073 |
test(W7c): check the codex boundary on meaning and its path list
W7c conversion (t02 #14, W7a host-config.test.ts:565). The boundaryInstruction bytes sent to Codex are unchanged; only the test loosens. Removed assertion group (invariant -> remaining protection): - toContain('Do NOT read') (caps fragment): Codex must not read or execute Claude skill definitions -> kept free: /do not read or execute any files under/i plus each excluded path ('~/.claude/', '~/.agents/', '.claude/skills/', 'agents/'), which is the actual contract. Mutation pass (scratch worktree, hosts/codex.ts): 3/3 as expected; replacing the prohibition or dropping an excluded path goes red; lowering IMPORTANT/NOT stays green. Log: .context/mutation-host-config.md. |
||
|
garrytan
|
a5c5162d33 |
test(W7c): match redact scan-at-sink exact-bytes rule case-insensitively
W7c conversion (t02 #13). Credential-handling checks (mktemp, --from-file, exit codes 3/2/0, visibility lookups, no HIGH skip flag) are unchanged. Removed assertion group (invariant -> remaining protection): - /EXACT bytes/ (caps): scan the exact bytes that will be sent -> kept free: /exact bytes/i in the same test, next to the mktemp and --from-file checks. Mutation pass (scratch worktree, scripts/resolvers/redact-doc.ts): 3/3 as expected; scanning a summary instead of the exact bytes or dropping the Exit 3 (HIGH) branch goes red; lowering EXACT stays green. Log: .context/mutation-redact-doc-resolver.md. |
||
|
garrytan
|
d30812a109 |
test(W7c): match retro clock rule case-insensitively, drop proximity regex
W7c conversion (t02 #11, #12). Removed assertion groups (invariant -> remaining protection): - /NEVER from `date`/ (caps): today comes from the session reminder, not the system clock -> kept free: /session reminder/ and /never from `date`/i. - /(?:disclosure[\s\S]{0,200}narrative|narrative[\s\S]{0,200}disclosure)/ (200-char word-proximity window): offline/local-only runs disclose that the window is not freshness-verified -> kept free: the exact disclosure string 'offline run, window not freshness-verified' (it reaches the retro output), RETRO_REF disclosure and the GUARD_* script checks; gate judge 'retro/SKILL.md instructions'. Mutation pass (scratch worktree, retro/SKILL.md.tmpl): 4/4 as expected; allowing `date` or deleting the disclosure string goes red; lowering NEVER and rewording the narrative clause stays green. Log: .context/mutation-regression-1624-retro-stale-base.md. |
||
|
garrytan
|
27dac76763 |
test(W7c): match document-release spawned-dispatch contract on meaning
W7c conversion (t04 F3; W7a run-in-background-guidance.test.ts:472,474, named by the Eng W7c gate as anti-injection and VERSION invariants that stay as case-insensitive meaning checks). The run_in_background flag pins, structural scanner and inverted-phrasing ban are unchanged. Removed assertion groups (invariant -> remaining protection): - 'NEVER trigger it on their own' (caps literal): prompt/file/tool claims do not enable spawned mode (anti-injection) -> kept free: /claims never trigger it on their own/i in the generated skill and template. - 'The NEVER-do invariants below do' (short form of a wrapped sentence that names the caps headings s10 rewrites): spawned auto-pick still never rewrites CHANGELOG or changes VERSION -> kept free: /skip any recommendation that rewrites CHANGELOG or changes VERSION/i (whitespace-normalized) in both files; gate E2E document-release. - 'A spawned run must never change VERSION': same rule, now /i. Mutation pass (scratch worktree; document-release skill, template and release-body section, ship/sections/documentation.md): 7/7 as expected; deleting the anti-injection clause, the CHANGELOG/VERSION carve-out, the spawned VERSION line, the run_in_background flag or the dispatch heading goes red; lowering NEVER and rewording 'NEVER-do invariants' stays green. Log: .context/mutation-run-in-background-guidance.md. |
||
|
garrytan
|
5b2bb3a1a5 |
test(W7c): match plan-scope-selection scope-gate stop lines case-insensitively
W7c conversion, scope-gate caps pins only (t06 #14). The scope classifiers (nativeSeededPlanSelection, isScopeGateAutoSelectVisible, isScopeGateQuestionVisible), the consumed announcement and A/B/C menu bytes, and every non-caps assertion are unchanged. Removed assertion groups (invariant -> remaining protection): - 'Reply with A, B, or C. STOP and wait' order anchors (design, template and every host expansion): menu precedes bootstrap -> kept free: the consumed prefix 'Reply with A, B, or C' (pty/classify.ts, pty/judge.ts), now also required to exist after '## Scope gate' so a missing menu cannot pass the order check with indexOf -1. - 'call AskUserQuestion FIRST and wait.', 'does NOT count as the mode signal', 'If the user explicitly named a DIFFERENT target' (design and eng), 'STOP and wait for the answer — only after the user picks', 'STOP and wait for the answer' (toContain and indexOf before the first ```bash on every host): ask first, pasted text is not a mode signal, explicit targets win, stop until answered -> kept free: the same rules as case-insensitive meaning checks, including the per-host ordering (search /stop and wait for the answer/i before the first ```bash, with a presence check). Mutation pass (scratch worktree; plan-design-review and plan-eng-review templates): 12/12 as expected; gate-after-preamble, removed design menu prefix, removed stop lines (design, eng), removed mode-signal, ask-first and explicit-target rules go red; lowering STOP/FIRST/NOT/DIFFERENT stays green. The eng menu prefix is not pinned here (eng-scope-entry-ap.test.ts M1 covers it). Log: .context/mutation-plan-scope-selection.md. |
||
|
garrytan
|
3ab469198e |
test(W7c): qa-functional-prompt keeps the verifier contract, drops driver prose
W7c (t07 F6, F7, F8). One test file. The eval prompt itself is unchanged.
Removed assertion groups (invariant -> remaining protection):
- ~40 driver-prompt sentence pins in the disclosure, artifact and fix-budget
tests ("All four reads are required before probing ...", "command is the
exact full outer capture invocation, including ...", "different later
command", "not the required same-command replay", "Only the helper writes
observed fields", "Link the evidence and checkpoint files ...", "Evidence
rows contain ONLY complete JSON ...", "aim under 400 words", "one causal
sentence per checkpoint hypothesis ...", "a reproduced in-tier defect
requires ...", duplicates of "retain its headings ..." etc.) -> the run
produces verifiable evidence and a linked report: free validator tests in
this file (qaFunctionalVerdict replay, validateQACheckpoints on CI, R29 and
R85 captures) and gate E2Es qa-functional-cli-report,
qa-functional-webhook-report, qa-functional-cli-fix,
qa-functional-webhook-fix (gate). Kept the machine-read contract: read
list, `"command":"<exact full outer capture invocation>"`,
`annotations.learning`, the three copied field names, the >20-character
limit, evidence.json before report.md.
- `maxTurns: 40` / `completionReserveMs: timeout / 4` exact source pins ->
replaced by a `maxTurns >= 40` floor and presence of `completionReserveMs`.
- Fix-stage provenance sentences ("most recent completed native probe",
"Tests, source edits and clock reads ...", "write no checkpoint when there
is no next probe") -> test output never becomes `observed`: kept one free
meaning check plus the R29 validator test (free).
- Duplicate loop-order checks on qa-only/sections/exploratory.md and grep of
`scripts/resolvers/qa.ts` source (t07 F6, F8) -> loop ordering has one
owner, qa-probe-gates (free, every host). Here the source grep becomes a
check on `generateQAExploratory` output (one probe per native command,
never batch probes), and the caller test keeps resource paths, field names
and structural stage anchors.
Loosened to case-insensitive meaning checks (kept): never synthesize JSON,
never shorten native JSON, existing tests read-only, freeze tests after red,
repair scope `src/<family>.ts`, green suite does not replace native probes,
report incomplete honestly, report-only proposes test stubs, completion
reserve does not end coverage, all eight scenarios required, never backfill
notes, owned probe directory, only complete emitted JSON, wrapper/entrypoint
boundary lines.
Kept unchanged: paid-case selection, CI/R29/R85 validator replays, actor
boundary and command allowlist, fix-stage order, verification probe list,
scenario lists.
Cosmetic pins dropped: ONLY / NOT / Never capitals.
Mutation log: .context/mutation-qa-functional-prompt.md (19 mutations, 0
unexpected: 3 reorders, 4 markers, 7 safety red; 5 rewordings green).
|