mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-13 03:48:06 +01:00
* chore(scripts): retire hollow and legacy paid package scripts behind one-release stubs (W8a, DX-4, CEO-31) test:codex*, test:evals*, test:e2e*, test:gate, test:periodic, eval:bg, eval:bg:all, eval:flake-rank, eval:watch and test:audit now print their replacement and exit 1 via scripts/retired-command.ts. CONTRIBUTING.md carries the same table; test/retired-commands.test.ts pins both. Eval CLI empty-state hints point at eval:bg:pr. * chore: delete dead update-readme-throughput, eval-watch, orphan fixtures and QA_FUNCTIONAL_INPUTS (W8b, CEO-24) Retired with named behavior: - update-readme-throughput.ts + readme-throughput test: replaced a README anchor that no longer exists from a JSON file that was never produced; the script was a no-op. Guarantee withdrawn (nothing to protect). - eval-watch.ts + its dashboard tests: rendered ~/.gstack-dev/e2e-live.json and _partial-e2e.json, written only by unsharded runs, so it showed nothing for every supported (sharded) entry point. Guarantee withdrawn; live progress is the gstack-detach log or gh run watch. Manual-review rendering stays covered by eval-list and eval-summary tests. - Orphan fixtures (autoplan-pending-artifact-ae.json, autoplan-existing-app/, autoplan-sign-in-ui/, parity baselines v1.47.0.0/v1.57.7.0/v1.69.1.0): no test, touchfile or script reads them; superseded baselines stay in git history. - QA_FUNCTIONAL_INPUTS: exported, never imported. * refactor(test-free): delete dead WORKER_HOSTILE list and buildShardArgs parallel mode Neither had a production caller: full-suite mode runs shard processes, so the worker quarantine was inert and --parallel was never passed (W8b). * docs(test-value): narrow prompt-byte contract to machine-read tokens (D2) Define prompt-byte contract as machine-read tokens (markers, field names, command lines, enum values, file paths, security-boundary strings) and add a sentence-pin glossary entry: convert to structural checks or delete when a behavioral eval covers the behavior. * test(ratchet): count free tests that slice a paid test's source A free test that binds readFileSync(<paid test file>) and cuts it with .slice() tests the paid harness's code out of context even when it imports a lib/ symbol. Record the 15 existing files with reasons; new ones fail with fix text. Self-test covers positive and negative shapes. * test(browse): un-quarantine compare-board; all 16 board tests pass headless (W8d, CEO-24) The GSTACK_COMPARE_BOARD_TESTS opt-in hid a headless mismatch that no longer reproduces: all 16 tests pass on Bun 1.4.0 with the bundled headless Chromium, solo, repeated, and beside other in-process browser files. Protected behavior (the comparison-board DOM contract the plan-design-review/office-hours/design-consultation feedback loop polls) is checked on every free run again. * test: replace silent existsSync early returns with explicit skips or assertions (W8d) - browse/test/build.test.ts: server-node.mjs checks skip only when the bundle is absent and GSTACK_EXPECT_BINARIES is unset; the free CI lane and test:ubicloud build it and set the flag, so a dropped build step fails with the build command instead of passing with no assertions. - gstack-update-check: VERSION is tracked; the guard could only hide a bug. - gen-skill-docs host leakage/frontmatter checks assert the host tree exists (the module-level render always produces it). - cso-docker staged-runtime test uses test.skipIf so reports show it as skipped, not passed. * test(ship): convert ship-workflow-clarity sentence pins to structural checks Add test/helpers/prompt-structure.ts (between, expectTokens, expectAbsent, expectOrdered, expectMentions) with fix text on failure. ship-workflow-clarity now pins headings, step order and machine-read tokens (commands, fields, enum and status strings); safety rules are meaning-level keyword checks. Wording is judged by the 'ship/SKILL.md workflow' judge in skill-llm-eval. Sentence pins: 90 -> 0. * ci: add platform-qualification lane for opt-in ML, gitleaks and Swift tests (W8d, CEO-24) security-bench + security-live-playwright L4 (SECURITY_BENCH), the real gitleaks 8.30.1 memory-ingest cases (GSTACK_TEST_GITLEAKS) and the DebugBridge Swift build invariants (GSTACK_TEST_SWIFT) self-skip in every lane today. No scheduled lane has their prerequisites, so per CEO-24 they move to an owned dispatch + quarterly workflow instead of being deleted. Each job was run locally (ML 10 pass, gitleaks 130 pass; Swift needs macOS). test/platform-qualification-workflow.test.ts pins gate variable, job and file together. * fix(scripts): capture-baseline and preflight-agent-sdk no longer act on --help or typos (W8e) capture-baseline parses its flags strictly: --help prints usage, an unknown or valueless flag exits 2, and only a valid invocation writes a fixture (--help used to write test/fixtures/parity-baseline-current.json). preflight-agent-sdk gains --help, rejects unknown flags, and runs the live SDK query (API spend) only with --live. * test(evals): register the three keyless periodic paid files with case ids (W1b) skill-e2e-auq-consistency, skill-e2e-auq-verbose-vs-carved-ab and codex-e2e-recommendation-substance now carry touchfiles, a periodic tier and kind rule (one trial, unchanged), so editing one selects its own case instead of restoring the full PR gate. The verbose A/B case also registers the vendored pre-cut fixture it reads, and review-army-migration-safety registers the SQL fixture it reads. * feat(evals): narrow PR fallback for free-only inputs (W1a, CEO-13/20, ENG-13) The PR profile now exempts the duration seeds, every free-lane workflow (explicit list; inventory test fails on an unclassified workflow, .yaml and workflow_call included), scripts/ubicloud/** and fixtures declared in the explicit FREE_FIXTURES map. Each remaining unknown file carries a label and fix (needs touchfile entry / add to FREE_ONLY_PR_FILES / real unknown dependency). evals.yml itself still restores the full gate. * ci: delete ci-image.yml; the eval workflows already build and push the same content-hash tag (W8f) evals.yml, evals-periodic.yml and evals-marathon.yml each compute the identical hashFiles() tag and build/push it (with :buildcache and :latest) when missing, so ci-image's push trigger and weekly cron were no-ops (30-day median 0.5 min, 36/36 success). ci-image-tag-binding now pins the three eval workflows. * ci(evals): key periodic/marathon concurrency on ref+event; branch periodic dispatch skips the gate census (W2a, ENG-9, CEO-04) * refactor(paid): drop ~160 unused imports; leaf paid-types/paid-select modules break the paid-* import cycle (W8c) * ci(quality-gate): per-PR concurrency, free runner, drop unscoped bun audit (W2a, W8f) - Concurrency keys on pull_request.number || run_id, so back-to-back main pushes no longer cancel each other's credential scan. - ubuntu-24.04 instead of a billed 8-vCPU Ubicloud runner for a ~1 min job. - bun audit --audit-level=critical ran on every PR regardless of diff, so a newly published advisory turned unrelated PRs red, with its own ignore policy. dependency-review (lockfile PRs, fail >= high) and the weekly OSV sweep remain the dependency gates. * chore(test-free): refresh the free duration seed from the audit's Ubicloud recording Adds the 19 files merged after the last refresh, including the 115 s claude-overlay-setup that packed as ~1 s and doubled a shard (W6a). The seed is re-recorded on the final tree at the end of this wave. * ci: move tiny ubicloud-standard-2 jobs to free ubuntu-24.04 runners (W8f) actionlint, skill-docs, dependency-review, version-gate and pr-title-sync are sub-minute jobs; GitHub-hosted runners are free for this public repo. * feat(test-free): pessimistic unseeded packing, seed growth ratchet and CI health summary - Unseeded files pack at the seed's p99 instead of p75 (W6a). - --ci-plan warns in the job summary past 5 unseeded files; --ci-verify lists shards that outran their prediction by 1.5x (and 30 s) plus every flaky pass absorbed by the serial retry (W6a, W7a job summary). - test/free-seed-ratchet.test.ts fails when a seeded file over 60 s is not on scripts/free-test-seed-allowlist.json, when an entry is stale, and when the allowlist grows versus its merge-base copy unless the new entry is an annotated exception with a todo (CEO-10, DX-7). free-plan runs it with full history so the shrink check is real in the required lane. - Every message names the fix command and docs anchor (DX-6). * feat(evals): state PR coverage mode, counts, reuse and fallback files in the summary and comment (CEO-13) * test: make the tracked-tree skill-doc freshness check local-only; tidy skill-docs header (W8f) Every free CI shard regenerates the tracked SKILL.md files before running tests, so the --dry-run freshness assertion compared the regenerated tree with itself and could not fail there. It now skips under GitHub Actions; skill-docs.yml's git diff --exit-code is CI's freshness gate. Removes a stale concurrency comment in skill-docs.yml. * ci(free-tests): install zsh so the bash+zsh portability arms run (W6f) ~42 zsh-arm tests (aside-probe-shell, qa-browser-preservation, generated-docs-fences) skipped in CI because zsh is not in the runner image. All 141 tests in those files pass with zsh present. * feat(evals): executors write negative receipts at the execution boundary (CEO-25, ENG-6) A completed single-trial rule/judge shard that did not pass now writes its negative receipt beside the pass receipts in the slice artifact, so a cancelled run's FAIL still blocks an older PASS when its report never runs. Receipts merge negatives first and order by (run attempt, completedAt); a same-second FAIL/PASS tie resolves to FAIL. The runner call site is a one-line hunk in scripts/test-paid-shards.ts. * docs(cso): state that contained runtime/scanner execution is unavailable today The scanner catalog is empty while its build inputs await review and the runtime catalog holds no qualified profile until the private evaluator exists, so /cso runs static assessment only. AGENTS.md, README and docs/cso-release-qualification.md now say so; docs/cso-scanners.md already did. * test(helpers): expectMentions splits on sentence ends only Clauses joined by ';' or ':' are one rule; splitting there made a reworded rule fail for punctuation. * test(plan-ceo): convert skill-ceo-section-ordering sentence pins Safety and authority rules (never/do not/stop/only/without/wait/approval) are meaning-level expectMentions checks; other sentence pins are dropped. Headings, labels, option rows, placeholders and machine-read lines (auto-decide annotation, Mode:/approved decisions format, STATUS/SOURCE fields, CROSS-MODEL marker, gstack-ceo-mode-handoff) stay as tokens; ordering checks unchanged. Behavior is covered by the periodic PTY case in skill-e2e-plan-ceo-review-section-loading.test.ts. Sentence pins: 210 -> 23. * test(helpers): expectMentions treats wrapped lines as one sentence Templates hard-wrap prose; only sentence ends and blank lines separate rules. * test(gen-skill-docs): convert sentence pins to meaning checks or drop them Safety/authority sentences become expectMentions keyword checks; other English sentences are dropped (wording is judged by the per-skill workflow judges in skill-llm-eval and the E2E cases in touchfiles-data). Commands, config keys, markers (PLAN MODE EXCEPTION, NO UNRESOLVED DECISIONS, Gate outcome), field names and alternation-based meaning regexes stay. Removed the now-empty 'contains opt-in AskUserQuestion text' test (only pinned a sentence; the resolver's AskUserQuestion wiring is covered by the codex second-opinion tests in the same describe). Lane E's existsSync hunks are untouched. Sentence pins: 216 -> 65 (remaining lines carry machine tokens). * ci(windows): shard the Windows lane via --ci-plan/--ci-run/--ci-verify; move native campaigns to native-qualification.yml - windows-free-tests.yml: a Linux plan job curates and packs the Windows-safe files into 6 shards, one strict serial shard per windows-latest job, and a Linux aggregate verifies exact coverage and every result (8.1 min on one runner today). --windows-only now combines with the CI modes; the plan binds the curated file set (W6e). - Every Windows plan and --list run prints the curated count with its top exclusion reasons in the job summary; a free ratchet fails when the curated subset drops below its floor, with the fix (W6e). - The four Dia/cookie diagnostic dispatch campaigns move to the dispatch-only native-qualification.yml with one exclusive `mode` choice input, so their red dispatch runs no longer read as a red PR lane (W8f). Their workflow pins move to test/windows-native-workflows.test.ts; the boolean-exclusivity guard is replaced by the choice input. * feat(paid): per-slice CI job ceilings and an in-process slice deadline with not_run/hung outcomes and checkpoints (W2c, ENG-2) * fix(pass-rates): segment trial history strictly by EVAL_POLICY version Reader-compatibility floor for EVAL_POLICY v2 (ENG-14/CEO-35/DX-12). Only trials under the reader's own policy version feed labels and alarms; older policy trials stay visible but are never scored or used as a drift baseline; trials from a newer policy are ignored with a printed count and fix text. Reverting the later v2 writer commits leaves a reader that cannot pool v2 records into v1 series (the pre-PR reader at2db0b3adoes: it reports a mixed fixture as FAILING with drift alarms). * docs: one contributor command table; retired commands, seed sources and current workflows (W8g, DX-5, DX-13, CEO-12) - CONTRIBUTING.md gains 'Which command do I run?' (task -> command -> cost/time -> needs, including branch validation via gh workflow run); CLAUDE.md, AGENTS.md and TESTING_INTERNALS.md link to it instead of repeating command lists. Tier tables drop retired commands and stale per-run dollar estimates. - Retired names (test:evals, test:e2e, eval:bg, eval:flake-rank, eval:watch, ...) are gone from CLAUDE.md, CONTRIBUTING.md, TESTING_INTERNALS.md, ARCHITECTURE.md and USING_GBRAIN_WITH_GSTACK.md. - The flake-ledger claim now says what happens: CI uploads flake-ledger-* artifacts; eval:pass-rates reads only the local ledger. - TESTING_INTERNALS states that only test:ubicloud --record-durations produces an acceptable duration seed (DX-13). - CI section: quality gate no longer runs bun audit; dependency-review's real trigger; the platform-qualification lane. - docs/test-audit-2026-09.md gets a corrections section for its stale claims. - TODOS.md: audit follow-ups (post-merge metrics, stub removal, name reclaim, diff-aware planner); eval-watch shard-awareness and the image freshness tripwire closed as won't-do. * feat(evals): recover receipts of cancelled PR runs (W1c, CEO-14/34, ENG-4/5/7) A new recover-receipts job runs base-ref code with actions: read, restores the PR's store, merges the receipts of completed cancelled evals.yml runs for this PR (provenance-checked, at most 5, oldest first, negatives first), reconstructs completed panel negatives with panelVerdict() (partial panels get no credit) and keeps a recovery.json checkpoint that never advances past an unresolved run. plan-slices keeps contents: read, takes the store as an artifact, turns reuse off when recovery degraded or a run is unresolved, and carries the whole store to slices-report so nothing is lost. Any gh problem degrades to no recovered receipts with a named reason. * feat(evals): classify this wave's free-only tools and workflows for PR selection platform-qualification.yml, retired-command.ts, test-health-report.ts, bump-harness-version.ts and eval-bg.ts never run inside a paid case. * feat(evals): debounce PR push bursts before planning paid slices (W1d, CEO-15, ENG-11, DX-7) A tiny ubuntu-24.04 job waits 90 s only on synchronize when another run for the PR started in the last 15 minutes; cancel-in-progress supersedes, and a run that outlives the wait reports "superseded by <sha>" and plans nothing. Labels: evals-no-debounce skips the wait, evals-fresh disables receipt reuse. * perf(test): move the real-process browser settlement cases out of test-free-shards.test.ts (W6b) The 19 owned-browser settlement modes were 85 s of the file's 95 s and could not be packed apart from the fast runner unit cases. They move to a shared fixture with two files (lifecycle 47 s, ownership 39 s); the unit file drops to 10 s. Same modes, same assertions. * chore(evals): delete the finished validation_phase dispatch and its inline planner (W8f) Retirement (CEO-24): the protected behavior was a one-off cookie-repair validation subset (eight dependent cases) and a quality/behavior split of the census; that validation is complete, so the guarantee is withdrawn. Branch validation dispatches evals.yml with evals_all. * perf(test): inject the office-hours record grace in non-cooperative deadline tests * perf(test): shrink aside-render process slack via GSTACK_RENDER_SLACK_MS in timeout tests * perf(test): make the memorable lock give-up configurable for lock-contention tests * perf(test): check install-timeout knob normalization through its own setup lines * perf(test): inject a 2 s storage-reset deadline in the late isolated dispatch test * perf(test): shorten the slow gbrain fake to 1 s, still past the 300 ms probe timeout * ci(free-tests): run only cso-docker-integration in the free CSO Docker job (F16) cso-node-lifecycle-integration and cso-stack-cold-integration need GSTACK_CSO_TEST_STACK images and always skipped there; they stay in the CSO runtime workflows through test:cso:docker. * test(gen-skill-docs,plan-ceo): split mixed sentence pins into tokens and rules Sentences that wrap a backticked or bold token keep the token (expectTokens) and, when they state a safety rule, a keyword check; short safety sentences (25-39 chars) get the same treatment. * test(plan-ceo): convert ceo-workflow-clarity.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: periodic PTY case skill-e2e-plan-ceo-review-section-loading and the plan-ceo workflow judges. Sentence pins: 13 -> 1. * test(plan-eng): convert eng-review-routing.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: plan-eng E2E cases (skill-e2e-plan-eng-*) and the plan-eng workflow judge. Sentence pins: 28 -> 6. * test(plan-eng): convert eng-scope-entry-ap.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: plan-eng E2E cases (skill-e2e-plan-eng-*) and the plan-eng workflow judge. Sentence pins: 16 -> 5. * test(plan): convert plan-scope-selection.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: plan-review E2E cases and workflow judges in skill-llm-eval. Sentence pins: 19 -> 2. * test(qa): convert qa-lazy-sections.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 71 -> 11. * test(qa): convert qa-browser-preservation.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 25 -> 11. * test(qa): convert qa-probe-gates.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 20 -> 5. * test(qa): convert qa-checkpoint-evidence.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 7 -> 5. * test(qa): convert qa-evidence.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 4 -> 3. * test(design): convert design-consultation-contract.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: design-consultation E2E/judge cases in touchfiles-data. Sentence pins: 44 -> 11. * test(outside-voice): convert run-in-background-guidance.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: outside-voice E2E cases and plan-review workflow judges. Sentence pins: 35 -> 5. * test(spec): convert spec-template-invariants.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: spec E2E/judge cases in touchfiles-data. Sentence pins: 36 -> 21. * perf(evals): 420 s census slice budget; case-shard skill-e2e-qa-workflow; seed its case walls (W5c) * feat(evals): evals.yml dispatch contract for eval:bg (CEO-17/29, DX-1, ENG-8/12) workflow_dispatch gains base_ref, base_sha, expected_sha, nonce and pr_receipts. The nonce appears in run-name so the dispatcher resolves its own run; evals_all=false runs the PR profile against base_sha/base_ref; a checkout that differs from expected_sha fails before any paid work with a named fix; pr_receipts reads that PR's receipts read-only through the recover-receipts artifact path (never the cache) and never saves. * ci(windows): pack the Windows lane with Windows-measured durations (W6e) The first sharded run (37221451711) finished in 6.0 min: the Linux seed has browse/test/cookie-import-native-job.test.ts at ~0 s (its cases skip on Linux) but it takes 122 s on Windows, so one shard ran 242 s against an 84 s prediction. The plan now reads scripts/free-test-durations-windows.json; a record_durations dispatch times every Windows-safe file alone and uploads it. * test(free): settlement case files import the runner rule they exercise Keeps the split files out of the test-of-test ratchet: the retry-eligibility rule is a direct runner import passed to the shared fixture. * fix(marathon): no hollow plan-decision-classification shard; office-hours marathon scenario scoped to the pilot (W2f) * test(free): satisfy the typecheck:test ratchet in the seed ratchet test * refactor(ci-history): one shared GitHub history reader with the CI flake-ledger reader Extract pass-rates' gh fetch layer into scripts/lib/ci-history.ts (CEO-09) and add the W7a reader: paged run listings that fail whole on any page error, per-run artifact downloads (cached, size-capped, data only), flake-ledger parsing, the pre-registered flaky-file rule (> 5% of main free-tests runs over a window of at least 20 runs, widened to the newest 20 runs when the window is thinner), a job-summary table, and named unavailable reasons with next steps. * feat(test-health): test:health report of the audit success metrics scripts/test-health-report.ts (CEO-09/06/22/32, DX-8/9/12/13) prints every success metric with its counting rule from CI history: PR full-fallback rate, reused PR trial records, PR-lane spend per merged PR and per validated revision, cancelled slice-minutes per week, push-to-verdict p50/p95, census expected failed trials per run with raw failure rate per kind/class and P(lane red) under the recorded panels vs single-trial aggregation, INFRA re-dispatches, quarantine evidence (insufficient evidence distinct from healthy), free flaky-pass rate, free/Windows wall, free serial time and the net-complexity line. An unreadable input prints 'unavailable: <reason>; next: <step>', never 0. --enforce fails on a flaky file (W7a) or > 5 unseeded free files (CEO-10); unavailable checks never fail and keep the issue open. * test(review): convert review-workflow-clarity.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval. Sentence pins: 85 -> 17. * test(review): convert review-entry-and-design-clarity-au.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval. Sentence pins: 15 -> 4. * test(review): convert review-enum-lifecycle.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval. Sentence pins: 8 -> 0. * test(ship): convert ship-control-flow.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 74 -> 9. * test(ship): convert ship-skip-requeue.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 28 -> 3. * test(ship): convert ship-plan-completion-invariants.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 28 -> 12. * test(ship): convert ship-reentry-gates.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 16 -> 4. * test(ship): convert ship-document-release-dispatch.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 11 -> 4. * test(ship): convert ship-apple-gate.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 7 -> 0. * test(ship): convert ship-skip-actor.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 6 -> 1. * test(ship): convert ship-template-redaction.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 5 -> 4. * test(ship): convert ship-publication-gates.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 5 -> 2. * test(ship): convert ship-review-loop.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 3 -> 1. * test(ship): convert ship-version-sync.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 5 -> 2. * test(ship): convert ship-pr-liveness-policy.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 4 -> 0. * test(plan-review): convert plan-review-cases.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: plan-review E2E cases and workflow judges in skill-llm-eval. Sentence pins: 58 -> 15. * test(autoplan): convert autoplan-phase-order.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge. Sentence pins: 42 -> 11. * test(autoplan): convert autoplan-phase-handoff.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge. Sentence pins: 20 -> 4. * test(autoplan): convert autoplan-snapshot.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge. Sentence pins: 21 -> 5. * test(outside-voice): convert outside-voice-preflight.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: outside-voice E2E cases and plan-review workflow judges. Sentence pins: 26 -> 9. * test(ask-format): convert resolver-ask-user-format.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: AUQ E2E cases (skill-e2e-auq-*); prompt-injection rule kept as a meaning check. Sentence pins: 32 -> 15. * test(skills): convert binding-template-drift.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 19 -> 13. * test(skills): convert carve-plan-fixture.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 20 -> 6. * test(land-and-deploy): convert land-and-deploy-postfail.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge. Sentence pins: 21 -> 14. * test(land-and-deploy): convert land-and-deploy-ci-gate.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge. Sentence pins: 5 -> 4. * test(land-and-deploy): convert land-and-deploy-flow.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge. Sentence pins: 5 -> 3. * test(office-hours): convert office-hours-completion.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: office-hours E2E cases (skill-e2e-office-hours*). Sentence pins: 16 -> 1. * test(plan-ceo): restore the stale closing-route guard The negative check for 'Closing hooks' / 'return to section' guards a retired route token, not a sentence; the first conversion pass dropped it. * ci(test-health): weekly test-health workflow with an honest tracking issue New test-health.yml (CEO-18/DX-9): Monday 10:00 UTC after the periodic census plus dispatch, free ubuntu-24.04 runner, permissions contents:read + actions:read + issues:write only. Runs bun run test:health --since-days 7 --enforce; an enforced failure upserts one tracking issue, a run with unavailable inputs keeps it open and lists them, and it closes only when every enforced check has evidence and passes. Only main runs touch the issue. * feat(evals): an edited paid test file runs its own gate cases in the PR lane (DX-11) A gate case outside the audited fast profile is selected when its single owning paid test file changed and the runner can select it by name (its Bun test is named by the case id). The manifest records directCases and validation accepts only recorded, name-addressable ones; the runner and planner read the file->cases map from test-pr-profile. Cases the runner cannot address stay deferred with the fix. An inventory test covers every owned gate case. Touches one hunk each in test-paid-shards.ts and lib/paid-plan.ts, and pr-shared-input-selection.test.ts's ship-skip pin. * ci(osv): tracking issue for scheduled reds and a 14-day suppression expiry test F11/W8f: a failed main OSV scan now upserts one tracking issue and a clean scan closes it (scheduled reds notified no one during the August backlog). A free test fails 14 days before any .osv-scanner.toml ignoreUntil (all four expire 2026-11-30), with the fix: upgrade and delete, or re-justify and move the date. * test(ship,review): keep guards against retired fail-open instructions 'skip silently' on reviewer failure and 'do not block /ship on subagent failure' were harmful instructions, not prose style; keep them absent with the shortest distinctive phrase. * feat(eval-select): default to the pr profile DX-10: eval:select previews the PR lane's selection by default, matching test:pr and eval:bg:pr; --profile full (or EVALS_PROFILE=full) keeps the plain touchfile selection. * test(evals): type-clean the recovery gh fake (typecheck:test ratchet) * chore(windows): record the Windows duration seed (run 37221963236) 573 Windows-safe files timed alone on windows-latest; cookie-import-native-job is 102 s there. Six shards now predict 154 s each. * feat(evals): eval:bg dispatches the pushed ref to CI or runs locally under a 1.5x-planned cap (W5d, CEO-17/29, DX-1/2/3/7, ENG-12/19) * feat(runners): --help on both shard runners via the shared flag loop; paid runner header describes today's planner/executor/report (W8e) * test(windows): drop the stale Playwright exclusion and force-include probe-proven files (W6e) A temporary windows-latest probe (run 37221995077) ran each of the 97 files excluded for 'launches Playwright browser' or 'raw /tmp/ path' alone: all 17 Playwright files pass (Chromium is installed and Bun drives it there), and 41 of the /tmp files pass with executed cases. The Playwright pattern is removed and the 41 files are force-included with the probe as their reason; the 36 real failures stay excluded. The curated subset grows from 573 to 631 files and the ratchet floor follows. Windows curation moves to scripts/lib/windows-curation.ts (the runner drops to 2000 lines). * test(free): one install fixture cloned from a read-only seed (W6c, ENG-1) Merge test/helpers/setup-codex-scope-fixture.ts into install-fixture.ts. The checkout is built once per test file into a read-only seed and cloned per fixture with copy-on-write where the filesystem has it (full copy otherwise), restoring each file's own mode; clones never hard-link the seed because put() and tests write in place. Callers register afterAll(cleanupSeed); Bun's runner fires no exit event, so seeds of killed runs are swept by pid. test/install-fixture.test.ts is the ENG-1 tripwire: put/setVersion/chmod in one clone leave the seed and a sibling unchanged, and clones share no inodes with the read-only seed. * test(free): split claude-overlay-setup by describe (W6b) The 115 s serial file becomes default/lifecycle/installs files (5/4/3 tests, unchanged bodies) sharing test/helpers/claude-overlay-fixture.ts, so the shard packer can balance them. * test(cso): convert cso-contracts.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval). Sentence pins: 16 -> 8. * test(cso): convert cso-preserved.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval). Sentence pins: 10 -> 3. * test(cso): convert cso-eval.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval). Sentence pins: 7 -> 4. * test(browser): convert aside-driver.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: Aside/browse E2E cases; the installer ban stays a meaning check. Sentence pins: 27 -> 9. * test(browser): convert third-party-actions.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: Aside/browse E2E cases; the installer ban stays a meaning check. Sentence pins: 17 -> 2. * test(skills): convert skill-validation.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 19 -> 6. * test(skills): convert workflow-excerpt.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 37 -> 10. * test(skills): convert question-tuning-registry-path.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 4 -> 0. * test(skills): convert host-conformance.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 4 -> 4. * test(memory): convert memory-ingest-landing.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: memory-ingest free tests that run the binary. Sentence pins: 11 -> 9. * test(windows): exclude the skip-only claude-overlay setup files from the Windows subset Their real-setup cases are describe.skipIf(win32); after the split their bin/ spawns live in the shared helper, so the source scan stopped seeing them. * test(skills): convert agents-digest.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 1 -> 0. * test(skills): convert brain-preflight.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 2 -> 1. * test(skills): convert codex-skill-isolation.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 1 -> 0. * test(skills): convert office-hours-review.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 2 -> 1. * test(skills): convert onboarding-moved-literals.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 2 -> 2. * test(skills): convert open-gstack-browser-liveness.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 3 -> 2. * test(skills): convert plan-floor-review.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 3 -> 2. * test(skills): convert plan-review-report-recording.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 4 -> 2. * test(skills): convert regression-1539-review-self-verify.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 4 -> 0. * test(skills): convert skill-fixture.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 6 -> 3. * test(skills): convert template-context-parity.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 1 -> 0. * test: restore pins on renderer output and paid-fixture inputs cso-contracts (the report renderer's required empty phrase), memory-ingest (binary output), carve-plan-fixture, ship-skip-actor and cso-eval (inputs the paid harness builds) assert product or fixture output, not skill prose, so the value-bar narrowing does not apply to them. * feat(eval-policy): v2 series identity, HARNESS_VERSION guard and identity backtest W4a (CEO-27/ENG-3/DX-6), not yet wired to the writer: caseSeriesIdentitiesV2 keys a case by the bytes it owns (paid test file, fixtures, its skills' prompt files including the generated SKILL.md) plus HARNESS_VERSION, and records the full consumed-input fingerprint as provenance. scripts/harness-version.json pins the blob of every shared-harness file (GLOBAL_TOUCHFILES plus the session runners, PTY harness, e2e gate and LLM judge), one line per file; test/harness-version.test.ts fails on drift with both fixes, and scripts/bump-harness-version.ts records the decision: --bump (new series for every case) or --non-behavioral (compatibility entry, no reset). eval-trial-series --backtest replays identities over weekly main revisions and recorded trials (ENG-3/DX-12). * feat(pass-rates): weekly history is main's scheduled runs plus main dispatches W4b: pass-rates reads evals-periodic history from main only, keeping scheduled runs and main dispatches and never a branch dispatch, so back-to-back branch validation runs neither feed pass rates nor count as weeks for quarantine expiry or drift. --branch <name> still reads one branch's runs for inspection. * docs(test-audit): EVAL_POLICY v2 decision memo with the identity backtest D1 memo (CEO-33, ENG-3, DX-12, CEO-27): P(green) arithmetic, false-blocking targets, the v2 identity rule and harness guard, weekly history, W4c evidence (the named 'stochastic' cases pass 26/27 to 34/34, so they stay rule), the backtest over 29 weekly main revisions and 14,898 recorded trials, what it means for quarantine reachability, alternatives (gating vs trended, adaptive panels not adopted, buy vs build), and the rollback proof on the pre-PR reader. * feat(eval-policy): EVAL_POLICY v2 writer: case-owned series identity The v2 writer, last commit of the W4 block (DX-12/ENG-14/ENG-21): the report job's stamp now writes caseSeriesIdentitiesV2 (case-owned bytes plus HARNESS_VERSION) as series_identity, the full consumed-input fingerprint as series_fingerprint and harness_version as provenance, and EVAL_POLICY.version is 2 (approved 2026-10-04, D1). The v1 touchfile identity is removed. Rollback = revert this commit (and the W4 commits above the reader-compat floor); readers then ignore the v2 records with a printed count. * test: restore pins on paid-harness prompt inputs review-enum-lifecycle and plan-review-report-recording assert the prompt a paid case's harness builds (opts.prompt / call.prompt), a fixture contract, not skill prose. * test(security): check injected safety lines on meaning, not capitals content-security, the canary instruction and the arm judge prompt still must carry their never-follow / never-output / exact-"none" rules; the checks are case-insensitive keyword checks so emphasis can change. The UNTRUSTED envelope markers and "none" value stay exact tokens. (Hidden element stripping fails locally only because Playwright Chromium is not installed; same at base.) * test(plan-ceo): keep the outside-voice scope-summary rule as a meaning check The first codemod pass left an empty if body (TS1313). * chore(windows): re-record the Windows duration seed for the 631-file subset (run 37222668939) * ci(free-tests): summarize every shard's flake ledger in the aggregate job (W7a) The aggregate downloads flake-ledger-* (one subdirectory per shard) and appends lane C's `scripts/test-health-report.ts flake-summary` output to the job summary before strict verification. The runner's own flaky table is dropped so the ledger reader stays the single source; --ci-verify keeps the shard-overrun warning. * feat(evals): free test files anywhere and the stored-zip helper never restore the full PR gate (W1a) The CEO-23 replay found design/test free tests forcing fallback (36485436963). * feat(evals): replay recorded PR pushes through today's selector (CEO-23, ENG-18) scripts/replay-pr-selection.ts reports per push the recorded vs replayed mode and case count, dropped gate cases, retired failed cases and every miss (a failed case that depends on the diff but is no longer selected). * refactor(evals): one case-sharded carve-section-loading file with a case id and touchfiles per skill (W5a) * test(free): hold qa-deadline's full-pipe case on a FIFO, not a paused Bun stream Bun gives a child 'pipe' stdio a unix socketpair, so whether the 2 MB fill actually blocks depends on the parent's reader and the host's socket buffer sizes. CI (run 36761671811, both attempts) saw exit 0 after ~130 ms; with net.core.{w,r}mem_default=4 MiB locally the old case fails 3/3 the same way. A FIFO the test never reads is bounded by the kernel pipe buffer, so the receipt write genuinely blocks and the 5 s settlement deadline must fire. The 7 s bound now starts once the preload has queued the write. * test(free): event-driven waits in pty-output-wake close-releases-observers: the 500 ms wall race becomes an ordering check (the waiter is released while only SIGINT has been sent, before close's escalation finishes), and the Bun.spawn spy fakes only the PTY launch so a shard neighbor's in-process spawn cannot add a stray SIGTERM (CI runs 36015568981, 36036582721, 36074636848). continuous-redraws: the fixture spinner stops once the observer has made five coalesced reads, so the run reaches its work deadline with no redraw in flight. Streaming right up to the deadline raced the helper's screen drain and failed as 'PTY screen parse failed; viewport is incomplete' (5 CI runs 09-30..10-03; 4/40 locally under 8 busy loops, 0/40 after). * test(free): session-runner long-capture handshake waits for the collected Read event The fake CLI's init line arms the runner's work-phase timer, so the worker's handshake (work-ready + a >200 s timer) could pass before the runner's read loop had collected the Read event printed just before work-ready. Advancing the clock then killed the run with the Read lost: readSection false (CI run 37194511124). The worker now also waits for the runner's own progress line for that Read. Forcing the observed ordering (work-ready 200 ms before the Read line) fails the old test deterministically and passes the new one. * test(free): budget-override restores the CI variables it rewrites Its provenance cases set and then deleted CI, GITHUB_ACTIONS, GITHUB_REF_NAME, GITHUB_SHA and the GitLab CI_* names without restoring them, so in a CI shard every later file ran with CI unset. qa-only-cleanup's real browse daemons then launched Chromium without --no-sandbox, the sandbox failed on the runner, and both real-Chromium cases timed out with 'QA cleanup fixture did not become ready' (shard 12 attempt 1 of six runs on 10-04, retry alone green). Retained startup log: 'Chromium sandboxing failed!', child env with no CI. Reproduced 3/3 with CI=true, userns denied and budget-override ahead of qa-only-cleanup in one process; 0/3 after. * test(free): pr-shared-input-selection names its base and package history Every selection now passes EVALS_BASE and packageVersionOnly explicitly, so no case reads the checkout's git state. Before, 12 calls ran detectBaseBranch and the cumulative case diffed package.json against merge-base(origin/main, HEAD) — the ambient input that turned main red in runs 36572856858 and 36594314027. A PATH git spy counts 14 git calls in the old file and 0 now. * feat(evals): derive PR-lane dependents for unmapped files instead of restoring the full gate scripts/pr-dependencies.ts derives, from the checked-out tree, which paid cases consume each repository file: test import closures, repo paths and path.join segments in code/shell/skill text (whole-line comments and selection data excluded), bin script names, compiled binaries by outfile path (scripts/build.sh), package scripts, template placeholders through RESOLVERS, and host configs by host id. The global touchfiles' import closure is global. PR selection: an unmapped file with derived consumers puts the run in the new 'dependents' mode (every dependent gate case and judge, no profile cut); a tracked file under a derivable directory that nothing reaches is exempt; periodic/marathon workflows, .gitignore and tsconfig.test.json never run the PR gate. PR_FULL_GATE_FILES is the fail-closed list (deps, CI image, setup, build, evals.yml and its actions, test preload, tsconfig.json, .gitattributes, the skill generator and preamble resolvers, the eval policy). Untracked or underivable files restore the full gate with fix text. The summary and PR comment list derived dependents. * feat(evals): replay misses count derived dependents; report wave vs ordinary pushes (CEO-23) * test: drop FREE_FIXTURES entries for fixtures deleted in W8b * test: re-shape the census attribution fixture to qa-workflow and carve case shards * test: census fixture case order * ci: 420 s PR-lane slices with per-slice ceilings; periodic tracking issue only on main evals.yml joins the census lanes at the 420 s slice budget (W5c/CEO-15) and indexes each eval slice's job timeout from .plan.sliceCiTimeoutMinutes with the slice-deadline clock as its first step (W2c/ENG-2). evals-periodic.yml's tracking-issue upsert and close run only on refs/heads/main, so a branch dispatch whose gate census is skipped (CEO-04) never opens or closes #2755. * docs: integrate lane docs for the audit wave (eval-bg backends, PR fallback, policy v2, ceilings, evidence) - CLAUDE.md: 'Running evals as an agent' describes scripts/eval-bg.ts's dispatch and local backends, forcing flags, the sentinel and status; eval:select's PR-profile default; prompt-structure helper and value-bar terms. instruction-facts now requires scripts/eval-bg.ts in that section. - CONTRIBUTING.md: decision table gains test:health, eval:select default, eval:bg backends and the include_gate_census / evals_all dispatch lines; 420 s runners and per-runner ceilings; flake ledger, OSV tracking issue and expiry test, weekly test health; prompt-structure guidance. - AGENTS.md: build commands name the eval:bg backend choice and test:health. - docs/TESTING_INTERNALS.md: anchors the fix texts point to (#pr-paid-lane-fallback, #pass-rate-policy-versions, #harness-version, #flake-ledger, #duration-seed), receipts/recovery/debounce, per-slice ceilings, the slice deadline with not_run/hung and checkpoints, the eval-bg cap formula in place of the detach floors, EVAL_POLICY v2 history, carve case shards via --case. - docs/test-audit-2026-10.md: CEO-23 replay (165/168 -> 164/168 full-fallback, 0 misses), ENG-17 slice table (re-measured on this tree), D2 prose-pin metric. - TODOS.md: progress note on 'Let pass-rate history decide the rest'. - test/testing-internals-anchors.test.ts: every TESTING_INTERNALS.md#anchor named in code must match a heading. * test(free): hold qa-deadline's blocked-output-pipe receipts on a FIFO The five 'guard %s receipts survive a blocked output pipe' cases filled a paused Bun child pipe with 2 MiB. That pipe is a unix socketpair, so the fill only blocks while the host's socket buffers are smaller than the write: with net.core.{w,r}mem_default=4 MiB the old cases fail 13 of 15 case runs (the guard finishes before the test resumes the stream). The held stream is now a FIFO the test drains only after resume(), bounded by the kernel pipe buffer; Windows keeps the paused pipe. Bounds and assertions are unchanged. * test(free): qa-deadline waits for a written pid, not just the pid file DRIVER and LEAF write their pid with writeFileSync, so under load ready() can see the file before its digits and fail 'expected > 0, received 0' (seen once in 150 loaded runs). After any early failure the finally SIGKILLs the CLI, which orphans the detached command group; a leaf whose pid file is not yet readable then keeps the output stream open and the case hangs to its 15 s timeout (2/80 old-file runs and 3/310 FIFO-only runs under 8 busy loops; the instrumented hang showed the CLI already SIGKILLed by the finally). ready() now polls within the same 3 s bound until the file holds a positive pid. * fix(evals): a deleted file selects the dependents of the live files that still reference it PR selection treated every path absent from the head tree as an unknown dependency, so any PR that deleted a file forced the full gate. A deleted path is now placed like a live one, against the head tree: a matching touchfile selects its cases; live code, skill text or a full-gate input that still references it (path, extensionless import target, relative import, bin name, ancestor directory, compiled-binary outfile; whole-line comments and data files excluded) contributes its dependents, a full-gate referencer restores the full gate with a 'deleted but still referenced' fix, and an unreferenced deleted path has no dependents. Deleted PR_FULL_GATE_FILES entries still force the full gate. Tests that used absent paths as stand-ins for new unmapped files now use tracked root files (or an explicit tracked set); the CEO-23 replay counts misses through those references too. * docs: integrate free-suite lane docs (seed anchor, Windows lane, native qualification, install fixture, flake rules) - docs/TESTING_INTERNALS.md: '### Free suite duration seed' (the anchor the seed messages link) with DX-13, the 99th-percentile packing, --ci-plan warning, --ci-verify overruns and the CEO-10 seed ratchet; the Windows 6-job lane, its Windows-measured seed and refresh dispatch, curation and WINDOWS_CURATED_FLOOR; native-qualification.yml modes; the merged install fixture; test-only timing knobs; the flake-ledger job summary, Windows ledger names and three free-suite flake rules; zsh in the Ubicloud mirror. - scripts/test-health-report.ts links #free-suite-duration-seed, so the two seed messages share one heading (the anchor test caught the split). - CONTRIBUTING.md: Windows seed refresh and native qualification rows; seed-source pointer. AGENTS.md: Windows six-job lane. - docs/cso-release-qualification.md: the free CSO Docker job runs only cso-docker-integration.test.ts. - docs/test-audit-2026-10.md: free-suite evidence lines. - TODOS.md: CEO-11b progress note on the pre-push hook hang. * test: final duration seeds on the integrated tree; retire the emptied cookie-validation-phases file Linux seed re-recorded on Ubicloud standard-16 and the Windows seed from dispatch run 37227140871, both atc903310. The seed allowlist drops the three entries that are no longer over the ceiling. cookie-validation-phases lost its validation-phase pins with W8f and its Windows/Dia pins moved to windows-native-workflows with native-qualification.yml, so the file is deleted and the curated Windows floor rises to 638. The Windows recorder writes the seed at its default path so GSTACK_FREE_TEST_DURATIONS never leaks into timed tests. * test: re-express stale PR-lane pins against the new evals.yml contract - ci-eval-cache: the profile/fresh expressions now include evals_all=false dispatches and pr_receipts/evals-fresh; the PR store is restored only by recover-receipts (still scoped to repository_id + PR, PR events only), plan-slices takes it as an artifact and never touches the cache, saves stay PR-only, EVALS_CACHE_PR binds pr_receipts dispatches, and the report verdict artifact carries report receipts. - ci-paid-coordination: plan-slices needs only the two ubuntu coordination jobs (never the image); the fork/Dependabot clause is unchanged on plan-slices and also gates debounce and recover-receipts; only recovery holds actions: read. - qa-supervision-selection: auq-consistency and auq-verbose-vs-carved-ab (W1b) reach claude-pty-runner and pty-screen through auq-sdk-capture, exactly as auq-format-gate does. - paid-retry-supervision: the full-validation plan uses a tracked full-gate input (setup); an absent path is now a deletion placed by its references. * ci(windows): record the Windows seed at the default path * chore: HARNESS_VERSION non-behavioral decision for the audit-wave integration * v1.91.20.0 release metadata: test, eval and CI audit wave * test(compare-board): wait for the post-submit state the feedback POST settles The two post-submit assertions read the board once, but the board disables the submit button and shows the message only after its feedback POST resolves. On a loaded Ubicloud shard the read beat the round trip (FLAKY-PASS in the final free-suite run). Delaying the POST by 400 ms reproduces both failures at HEAD; polling for the settled state passes. * test(plan-eng plan-mode): run the seeded STOP-gate smoke in its own fixture project The seeded plan proposes a custom test runner, so the review's Step 0 reads the working tree's test setup. In the gstack checkout that is gstack's own sharded runner, and the case timed out at 300 s in census 37228573062 after this wave reshaped the runner. It now runs in a four-file Bun project with git history, with the product-notice suppression auto-decide-preserved's standalone fixture already uses. Local pass on Bun 1.4.0 + Claude Code 2.1.284: asked in 121 s, seed tokens observed. * test(plan-mode evidence): pin the seeded eng smoke's isolated fixture cwd and env * ci(image): Git >= 2.44 so paid shared-libs cases run the supported safe-git path bin/gstack-safe-git prefixes every /deslop-shared-libs Git read with --no-lazy-fetch, which noble's git 2.43 rejects (exit 129). The eval image therefore ran every paid shared-libs case on the unsupported-Git fallback: local history unavailable, branch HEAD unknown. shared-libs-read-only failed on exactly that in PR run 37230639146 ("Current branch HEAD: unknown. The safe-git probe failed (unknown option: --no-lazy-fetch, git 2.43.0)") and in 2 of 74 earlier PR runs. The image installs current Git from the Ubuntu Git Maintainers PPA and fails its build if --no-lazy-fetch is missing; the deliberate unsupported-Git case keeps its fake Git. * eval: pool matching branch census trials into main's pass-rate series (D1 option b) Approved 2026-10-05. eval:pass-rates also reads completed branch census runs in the weekly window. A branch trial counts only toward a series a main run also has (same case-owned bytes, HARNESS_VERSION, model and CLI), so a branch extends main's history but never starts or becomes a case's current series; weeks for quarantine expiry still count main runs only. The D1 backtest put this at 67 of 210 cases with enough trials for the entry rule, against 0 of 195 on main alone. * C2: /qa defines its outcome filename fields; QA report templates name the measured timing fields Low judge samples in failing panels named both: qa workflow (37186854666) 'an undefined {user} in the outcome filename', and qa-only workflow (37198445662) 'template asks for Duration / durationMs totals while reporting.md mandates Probe budget and Guarded command time'. The qa union stays under its 1.11 size ratio (93292 of 93294 bytes). (cherry picked from commite14c0c1de1) * C2: Browser Quick gets a feasible budget; QA loop names its directory and stop path Judge rationales on the qa-only panels (2 of the last 3) cited the 30 s Browser Quick budget: homepage + 5 targets, each a checkpoint write and a guarded command, left ~5 s per probe. Quick is now 180 s (method text: 3 minutes); the qa-only-no-fix paid check expects the same native deadline. PROBE_DIR points at its source line, a stopped loop goes to the final report steps (annotations, materialize, Markdown), and /review and /ship say that after smoke expiry only plan checks rerun in revalidation. (cherry picked from commit78d4a2a343) * qa: fit the C2 judge fixes under the qa and review size caps Brings GSTA-21's qa.ts trims (3bd625d,ce73af0) for the cherry-picked C2 wording: PROBE_DIR keeps 'owned', the stop line points at §4, and the post-expiry rule is one sentence, so /review fits its cap. qa measures 93,303 / 84,049 = 1.11010 against the same 1.1102 cap GSTA-21 set. The qa-only workflow judge passed 10 of 10 local trials on this wording (7 of 10 before the C2 fixes). * test: qa-only fixture keeps its caller-set 30 s deadline The qa-only-no-fix fixture tells the model to start the guard with 30 s (a caller limit), so its deadline assertion's default stays 30 000 ms; the product's Browser Quick default (180 s) does not apply to it. Reverts the helper and its test to their base. (cherry picked from commite36e622add) * test: qa-quick sets a caller probe limit and drains past its session budget C2 raised Browser Quick's native budget to 180 s. qa-quick is the only gate case on the native budget, and its wall went 194 s (main periodic, pre-C2) -> 254 s and 274 s (this PR's runs) -> 300 s session timeout (run 37242691913): the agent now probes every navigation target with a checkpoint, then needs ~80 s to report. Like qa-only-no-fix, the eval now sets a caller probe limit (30 s), scopes the fixture, and gives bun's outer timeout the session drain margin instead of racing the 300 s session. The product's 180 s budget and the session budget are unchanged. Diagnostic rerun passed in 195 s. (cherry picked from commitf8dc64031e) * chore: restore the audit wave's package scripts after the main merge (version 1.91.23) * qa: keep main's one-sentence post-expiry rule for /review and /ship The C2 variants ("step 4c reruns plan checks only" and "even in 4c") read to the review workflow judge as conflicting with Step 5.8's repeat-pass reruns: 6 of 10 local trials passed with the longer sentence and PR run 37248650879 failed on the shorter one. Main's "Post-expiry smoke rechecks are not-run." passed 10 of 10. qa (9/10), qa-only (10/10) and ship-managed-hook-refresh (10/10) were measured on the same tree. * v1.91.24.0: move off the slot #3036 claims
1045 lines
65 KiB
TypeScript
1045 lines
65 KiB
TypeScript
/**
|
||
* plan-ceo-review carve — static ordering guard (GATE tier, free, deterministic).
|
||
*
|
||
* This is the per-PR mechanical backstop for the v2-plan Phase B carve of
|
||
* plan-ceo-review (Codex outside-voice P2). The periodic real-PTY E2E
|
||
* (skill-e2e-plan-ceo-review-section-loading.test.ts) is the behavioral proof,
|
||
* but it runs weekly and costs money. This file runs on every `bun test` and
|
||
* fails CI the moment the carve's structural invariants break:
|
||
*
|
||
* 1. The skeleton points at the section with a STOP-Read directive, and that
|
||
* directive sits AFTER Step 0 (scope + mode) — so the conversational Step 0
|
||
* stays in the always-loaded skeleton, never stranded in the on-demand file.
|
||
* 2. The heavy review body (Sections 1-11) is NOT in the skeleton — it moved to
|
||
* the section. A regression that inlines it back would re-bloat the skeleton.
|
||
* 3. The review report writer ("GSTACK REVIEW REPORT") lives in the section, and
|
||
* the blocking EXIT PLAN MODE GATE that verifies it lives in the skeleton
|
||
* AFTER the STOP — so the gate fires once the section work returns.
|
||
* 4. Nothing review-governing sits in the skeleton below the STOP (Codex P1):
|
||
* no "Section N", no "## Mode Quick Reference", no "## Formatting Rules".
|
||
*/
|
||
|
||
import { describe, test, expect } from 'bun:test';
|
||
import * as fs from 'fs';
|
||
import * as path from 'path';
|
||
import * as os from 'os';
|
||
import { createHash } from 'node:crypto';
|
||
import { validateCeoReviewCompletion } from './helpers/auq-sdk-capture';
|
||
import { generateAntiShortcutClause, generateSpecReviewLoop } from '../scripts/resolvers/spec-review';
|
||
import { generatePlanFileReviewReport } from '../scripts/resolvers/review-dashboard';
|
||
import { generatePlanReviewApprovalCheck, generateExitPlanModeGate } from '../scripts/resolvers/plan-gates';
|
||
import { generateCodexPlanReview } from '../scripts/resolvers/outside-voice-steps';
|
||
import { HOST_PATHS, type TemplateContext } from '../scripts/resolvers/types';
|
||
import { ALL_HOST_CONFIGS } from '../hosts';
|
||
import { generateTasksSectionEmit } from '../scripts/resolvers/tasks-section';
|
||
import { generateBrainCacheRefresh } from '../scripts/resolvers/gbrain';
|
||
import { runGeneration } from '../scripts/gen-skill-docs';
|
||
import { ASK_QUESTIONS_HEADING } from './helpers/workflow-excerpt';
|
||
import { expectMentions } from './helpers/prompt-structure';
|
||
|
||
const ROOT = path.resolve(import.meta.dir, '..');
|
||
const SKELETON = path.join(ROOT, 'plan-ceo-review', 'SKILL.md');
|
||
const SECTION = path.join(ROOT, 'plan-ceo-review', 'sections', 'review-sections.md');
|
||
|
||
// Prose wrapping is not the contract; keep raw documents for line/heading guards.
|
||
const compactProse = (value: string) => value.replace(/\s+/g, ' ').trim();
|
||
|
||
// Anchors are exact labels, headings, placeholders or table cells (strings) or
|
||
// case-insensitive meaning checks (RegExp); they must all exist, in this order.
|
||
const ordered = (text: string, anchors: Array<string | RegExp>) => {
|
||
const positions = anchors.map(anchor => typeof anchor === 'string' ? text.indexOf(anchor) : text.search(anchor));
|
||
expect(positions.every(position => position >= 0), `missing one of ${anchors.map(String).join(' | ')}`).toBe(true);
|
||
expect(positions).toEqual([...positions].sort((a, b) => a - b));
|
||
};
|
||
|
||
const between = (text: string, start: string, end: string) => {
|
||
const from = text.indexOf(start);
|
||
expect(from, `missing ${start}`).toBeGreaterThan(-1);
|
||
const to = text.indexOf(end, from + start.length);
|
||
expect(to, `missing ${end} after ${start}`).toBeGreaterThan(-1);
|
||
return text.slice(from + start.length, to);
|
||
};
|
||
|
||
const skeletonSource = () => fs.readFileSync(`${SKELETON}.tmpl`, 'utf8');
|
||
const sectionSource = () => fs.readFileSync(`${SECTION}.tmpl`, 'utf8');
|
||
|
||
// Source-order and meaning guards; the selected judge remains the clarity proof.
|
||
test('CEO decision cycle returns to its caller with two complete persistence checkpoints', () => {
|
||
const cycle = compactProse(between(skeletonSource(), '### 0D.', '### 0E.'));
|
||
ordered(cycle, ['**1. Check sources and prior answers.**', '**2. Record the pending choice.**',
|
||
"**3. Compare and save that row's options.**", 'Commitment | Source/approval or pending | Current | A | B | C',
|
||
'**Pre-question checkpoint:**', '**4. Ask, record the answer, and amend.**', '**Post-answer checkpoint:**']);
|
||
expect(cycle.match(/\*\*(?:Pre-question|Post-answer) checkpoint:\*\*/g)).toHaveLength(2);
|
||
expectMentions(cycle, [['not','selection','returns']], 'cycle');
|
||
expectMentions(cycle.split('**Post-answer checkpoint:**')[1], [['do not','calling','return']], 'section');
|
||
});
|
||
|
||
test('CEO mode provenance separates an explicit instruction from asked and automatic question logs', () => {
|
||
const mode = between(skeletonSource(), '### 0E.', '### 0F.');
|
||
const rows = mode.split('\n').filter(line => /^- \*\*(Explicit user choice|Successful preference check|Actual question answer):\*\*/.test(line));
|
||
expect(rows).toHaveLength(3);
|
||
expect(rows[0]).toMatch(/no question log/i);
|
||
expect(rows[1]).toContain('`auto_decided: true`');
|
||
expect(rows[2]).toContain('`auto_decided: false`');
|
||
expect(mode.indexOf('**Explicit user choice:**')).toBeGreaterThan(mode.indexOf('4. **Mode handoff:**'));
|
||
});
|
||
|
||
test('CEO scope destinations preserve deferrals without asking their already answered TODO again', () => {
|
||
const scope = compactProse(between(skeletonSource(), '### 0G.', '### 0H.'));
|
||
ordered(scope, ['**Defer:**', '**Skip / Cut:**']);
|
||
const defer = between(scope, '**Defer:**', '**Skip / Cut:**');
|
||
expect(defer).toContain('TODOS.md');
|
||
expect(defer).toContain('NOT in scope');
|
||
expect(scope.split('**Skip / Cut:**')[1]).toMatch(/NOT in scope[^.]*; no TODO/i);
|
||
const todoMenu = compactProse(between(sectionSource(), '### TODOS.md updates', '{{PLAN_REVIEW_APPROVAL_CHECK}}'));
|
||
expectMentions(todoMenu, [['only', 'unanswered', 'proposals']], 'todoMenu');
|
||
expectMentions(todoMenu, [['do not','deferred','already']], 'todoMenu');
|
||
expect(todoMenu).toMatch(/one per question/i);
|
||
const readiness = compactProse(generatePlanReviewApprovalCheck({ skillName: 'plan-ceo-review' } as TemplateContext));
|
||
expect(readiness).toMatch(/leaves that choice unresolved/i);
|
||
expect(readiness).toMatch(/out of accepted work/i);
|
||
});
|
||
|
||
test('CEO completion facts precede summary and report while publication follows readback', () => {
|
||
const section = sectionSource();
|
||
ordered(section, ['{{PLAN_REVIEW_APPROVAL_CHECK}}', '### Review facts', '### Completion Summary',
|
||
'{{PLAN_FILE_REVIEW_REPORT}}', '**Publish the Completion Summary:**', '## Review Log']);
|
||
});
|
||
|
||
// These source scenarios guard instruction branches, not native execution.
|
||
test('CEO handoff allows no pending choice without inventing an approval', () => {
|
||
const source = skeletonSource();
|
||
const initial = compactProse(between(source, '### 0C.', '### 0D.'));
|
||
const approach = compactProse(between(source, '### 0D.', '### 0E.'));
|
||
const handoff = compactProse(between(source, '**Mode handoff:**', '### 0F.'));
|
||
const noChoice = approach.search(/no new answer needed/i);
|
||
expect(noChoice).toBeGreaterThan(0);
|
||
expect(noChoice).toBeLessThan(approach.indexOf('**2. Record the pending choice.**'));
|
||
expectMentions(approach, [['no','alternatives','approval']], 'approach');
|
||
expectMentions(initial, [['before','unresolved','approaches']], 'initial');
|
||
ordered(initial, ['A) current/requested plan', 'B) smallest scoped alternative', 'C) larger approach/rewrite only with evidence']);
|
||
expectMentions(handoff, [['no', 'approach', 'decision']], 'handoff');
|
||
expect(handoff).toMatch(/preserve 0D approvals/i);
|
||
});
|
||
|
||
test('CEO handoff carries all answered rows instead of one synthetic approach', () => {
|
||
const handoff = between(skeletonSource(), '**Mode handoff:**', '### 0F.');
|
||
expect(handoff).toContain('Auto-decided review mode → <selected mode> (your preference)');
|
||
expect(handoff).toContain('Mode: <selected mode>; approved decisions: <rows or none>');
|
||
expect(handoff).not.toContain('<approved 0D approach>');
|
||
});
|
||
|
||
// Guards the public handoff instruction, not model compliance or posture detection.
|
||
test('CEO mode handoff applies the selected mode before the next question', () => {
|
||
const selection = between(skeletonSource(), '### 0E. Mode Selection', '### 0F.');
|
||
const handoffStart = selection.indexOf('4. **Mode handoff:**');
|
||
const routeStart = selection.indexOf("Follow the selected mode's route:");
|
||
expect(handoffStart).toBeGreaterThan(0);
|
||
expect(routeStart).toBeGreaterThan(handoffStart);
|
||
const handoff = selection.slice(handoffStart, routeStart);
|
||
const instruction = handoff.split('\n')[0]!;
|
||
expect(instruction).toMatch(/before other tools or further questions, run `[^`]*gstack-ceo-mode-handoff/i);
|
||
const selectionSteps = compactProse(selection.slice(0, handoffStart));
|
||
expect(selectionSteps).toMatch(/explicit choice skips steps 2–3/i);
|
||
expect(selectionSteps).toMatch(/exits 0 with `AUTO_DECIDE` selects the recommendation/i);
|
||
expect(selectionSteps).toMatch(/stop for the answer/i);
|
||
const loggingStart = handoff.search(/record mode provenance after the handoff/i);
|
||
expect(loggingStart).toBeGreaterThan(handoff.indexOf('- Other selections:'));
|
||
expect(loggingStart).toBeLessThan(handoff.search(/selecting a mode does not approve changes/i));
|
||
const formats = handoff.split('\n').filter(line => /^- (`plan-ceo-review-mode:|Other selections:)/.test(line));
|
||
expect(formats).toHaveLength(2);
|
||
for (const format of formats) {
|
||
expect(format).toContain('<Application and rationale>');
|
||
expect(format).toContain('<rows or none>');
|
||
}
|
||
});
|
||
|
||
test('SELECTIVE baseline cuts preserve prior answers until their own scope decision', () => {
|
||
const modeWork = between(skeletonSource(), '### 0G.', '### 0H.');
|
||
const holdChecks = between(modeWork, '**For HOLD SCOPE**', '**For SCOPE REDUCTION:**');
|
||
expect([...holdChecks.matchAll(/^\d+\. /gm)]).toHaveLength(3);
|
||
const cuts = compactProse(modeWork.split('**Deferring current scope**')[1]!);
|
||
ordered(cuts, ['**A)** Defer this item to TODOS.md', '**B)** Keep it in scope']);
|
||
ordered(cuts, [/wait for the answer/i, /deferral changes only delivery scope/i, /beside the prior approval/i]);
|
||
});
|
||
|
||
// These check the storage branches and their order, not model compliance.
|
||
test('CEO defines pending choices and storage before its first decision procedure', () => {
|
||
const source = skeletonSource();
|
||
const step0 = compactProse(source.split('## Step 0:')[1]!);
|
||
ordered(step0, ['**Required choice:**', '**Storage policy: choose before writing.**', '### 0D.']);
|
||
expect(step0).toMatch(/retain all current content/i);
|
||
const compare = between(step0, '**3. Compare and save', '**4. Ask, record');
|
||
// Native c6fc D3 saved and asked Effort 0; cdd D3 cited a missing ledger row.
|
||
expect(compare).toMatch(/use effort S[^.]*never effort 0/i);
|
||
expect(compare).toContain('S/M/L/XL effort, low/medium/high risk');
|
||
ordered(compare, ['**Pre-question checkpoint:**', /find exactly one row by its assigned ID/i,
|
||
/one listed value, never a range/i, /correct missing or invalid fields/i, '**Read-back.**']);
|
||
const persistence = between(step0, '### 0H.', '### 0I.');
|
||
ordered(persistence, ['**Save or present both inputs under the storage policy.**',
|
||
'mkdir -p', '**Otherwise:**', '**CEO summary format — use for both saved and chat output:**',
|
||
'# CEO Plan: {Feature Name}', '{{SPEC_REVIEW_LOOP}}']);
|
||
expect(source).toContain('## Reviewer Concerns\n- {unresolved spec-review issues with their owning input, or "None"}');
|
||
});
|
||
|
||
test('CEO chat storage still supplies both spec inputs and the full report without claiming file completion', () => {
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const ctx = { skillName: 'plan-ceo-review', host: host.name, paths: HOST_PATHS[host.name]! } as TemplateContext;
|
||
const spec = compactProse(generateSpecReviewLoop(ctx));
|
||
expect(spec).toMatch(/both complete labeled texts/i);
|
||
expect(spec).toMatch(/all five dimensions/i);
|
||
expectMentions(spec, [['not','successful','reviewer']], 'spec');
|
||
expectMentions(spec, [['stop','completion','required']], 'spec');
|
||
const report = generatePlanFileReviewReport(ctx);
|
||
ordered(report, ['### Generate the report', '### Write to the plan file', '**Read-back gate:**']);
|
||
expect(report).toContain('not persisted');
|
||
expect(report).not.toContain('retry once');
|
||
const writing = compactProse(report);
|
||
expect(writing).toMatch(/Stage 3's blocked chat return/i);
|
||
ordered(writing, [/if the destination file exists, read it now/i, /if the destination file does not exist, use Write/i,
|
||
/keep the report last/i]);
|
||
const tasks = compactProse(generateTasksSectionEmit(ctx, ['ceo-review']));
|
||
expectMentions(tasks, [['do not','implementation','contracts']], 'tasks');
|
||
expect(tasks).toMatch(/"to be determined" and use an empty JSONL files array/i);
|
||
const outside = generateCodexPlanReview(ctx);
|
||
if (outside) expectMentions(compactProse(outside), [['include', 'ceo scope summary']], 'outside voice');
|
||
const gate = compactProse(generateExitPlanModeGate(ctx));
|
||
expectMentions(gate, [['do not','exitplanmode','checks']], 'gate');
|
||
ordered(gate, [/missing plan\/report saves/i, '1. Read the plan file']);
|
||
expectMentions(gate, [['no','forbidden','attempted']], 'gate');
|
||
expect(gate).toContain('**Gate outcome: Blocked**');
|
||
}
|
||
});
|
||
|
||
// The original 660 judge found competing routes. These guards verify the
|
||
// instruction branches and ordering; only the selected judge proves clarity.
|
||
test('CEO routes administrative menus separately while scope and TODO retain the full decision protocol', () => {
|
||
const main = compactProse(skeletonSource());
|
||
const decisions = between(main, '### 0D.', '### 0E.');
|
||
ordered(decisions, ['- **Admin question:**', '- **Plan decision:**', '**1. Check sources']);
|
||
expect(decisions).toMatch(/one column per option/i);
|
||
ordered(main, ["**A)** Add to this plan's scope", '**B)** Defer to TODOS.md', '**C)** Skip']);
|
||
expectMentions(main, [['without','decisions','answered']], 'main');
|
||
const section = compactProse(sectionSource());
|
||
ordered(section, ['**A)** Add to TODOS.md', '**B)** Skip — not valuable enough', '**C)** Keep in the current plan as required work']);
|
||
});
|
||
|
||
test('CEO output stages prepare body before report and publish only after verification with a blocked chat return', () => {
|
||
const raw = sectionSource();
|
||
const section = compactProse(raw);
|
||
ordered(section, ['### Stage 1 — Prepare', '### Review facts', '### Completion Summary',
|
||
'### Stage 2 — Save and verify', '{{PLAN_FILE_REVIEW_REPORT}}', '### Stage 3 — Publish', '## Review Log']);
|
||
expect(section).toMatch(/before the terminal report/i);
|
||
expectMentions(section, [['no','completion','depends']], 'section');
|
||
expectMentions(section, [['do not','append','report']], 'section');
|
||
const chatStart = raw.search(/if no plan\/report write is permitted/i);
|
||
expect(chatStart).toBeGreaterThan(-1);
|
||
const chatEnd = raw.indexOf('\n## ', chatStart);
|
||
expect(chatEnd).toBeGreaterThan(chatStart);
|
||
const chat = compactProse(raw.slice(chatStart, chatEnd));
|
||
expect(chat).toContain('**Gate outcome: Blocked**');
|
||
expect(chat).toMatch(/without claiming saved completion/i);
|
||
const main = compactProse(skeletonSource());
|
||
const blocked = between(main, '- **Blocked:**', '- **Passed with');
|
||
expect(blocked).toContain('**not persisted**');
|
||
expect(blocked).toContain('**completion blocked**');
|
||
expectMentions(blocked, [['do not','verified','missing']], 'blocked');
|
||
expectMentions(blocked, [['without','exitplanmode','telemetry']], 'blocked');
|
||
expectMentions(main, [['block','required','failed']], 'main');
|
||
const gate = compactProse(generateExitPlanModeGate({ skillName: 'plan-ceo-review' } as TemplateContext));
|
||
const precheck = gate.slice(0, gate.indexOf('Verify `Approval readiness: PASS`'));
|
||
expectMentions(precheck, [['block','plan/report','completion']], 'precheck');
|
||
expect(precheck).toMatch(/best-effort history does not/i);
|
||
// Optional task/archive/TODO restrictions cannot block verified required artifacts.
|
||
expect(precheck).not.toMatch(/Forbidden storage|(?:task|archive|TODO).*forbidden/);
|
||
});
|
||
|
||
test('CEO prior unresolved count remains a final report bullet for prior-only and mixed unresolved cases', () => {
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const ctx = { skillName: 'plan-ceo-review', host: host.name, paths: HOST_PATHS[host.name]! } as TemplateContext;
|
||
const report = compactProse(generatePlanFileReviewReport(ctx));
|
||
const status = between(report, '**Unresolved-decisions status', '### Write to the plan file');
|
||
expect(status).toContain('`NO UNRESOLVED DECISIONS`');
|
||
expect(status).toContain('`- + N unresolved from prior reviews`');
|
||
expect(status).toMatch(/not counted twice/i);
|
||
expectMentions(status, [['no','current','items']], 'status');
|
||
expect(status).toMatch(/final non-whitespace line/i);
|
||
expectMentions(status, [['no','separate','append']], 'status');
|
||
}
|
||
});
|
||
|
||
test('CEO saves compared proposals before recording an actual answer in its separate field', () => {
|
||
const procedure = compactProse(between(skeletonSource(), '### 0D.', '### 0E. Mode Selection'));
|
||
ordered(procedure, [/record pending rows before comparisons; never prewrite approval or tasks/i,
|
||
'**3. Compare and save', /in Proposed, compare every commitment/i, '**4. Ask, record',
|
||
/save the answer reference and scope/i, /amend only authorized work/i, /before taking another row/i]);
|
||
});
|
||
|
||
// A topic row alone did not expose the independently selectable test additions.
|
||
// This guards the executable representation/order, not the model's compliance.
|
||
test('CEO value comparisons and decline-all outcomes stay explicit before approval', () => {
|
||
const procedure = compactProse(between(skeletonSource(), '### 0D.', '### 0E.'));
|
||
ordered(procedure, [/record pending rows before comparisons/i, 'Commitment | Source/approval or pending | Current | A | B | C',
|
||
/save\/present the complete current plan, pending rows and comparisons/i, '**4. Ask, record']);
|
||
const comparison = between(procedure, 'Commitment | Source/approval or pending', '**4. Ask, record');
|
||
expectMentions(procedure, [['only','declined','continue']], 'procedure');
|
||
expectMentions(procedure, [['stop','unresolved','direction']], 'procedure');
|
||
});
|
||
|
||
// The paid paired-control skipped its provisional ledger and treated two
|
||
// contracts as one test strategy. Guard drafting before menu synthesis on each
|
||
// host; this is an instruction-order check, not proof of native compliance.
|
||
test('CEO Step 0 drafts provisional contracts before menus and saves their complete comparison on every host', async () => {
|
||
const outputRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'ceo-pending-rows-'));
|
||
try {
|
||
const generated = await runGeneration({ host: 'all', outputRoot, contentLinkRoot: null, log: () => {} });
|
||
expect(generated.exitCode, JSON.stringify(generated.diagnostics)).toBe(0);
|
||
const carriers = generated.artifacts.filter(artifact => artifact.kind === 'skill'
|
||
&& (artifact.relativePath === 'plan-ceo-review/SKILL.md'
|
||
|| artifact.relativePath.endsWith('/gstack-plan-ceo-review/SKILL.md')));
|
||
expect(carriers).toHaveLength(ALL_HOST_CONFIGS.length);
|
||
for (const file of [`${SKELETON}.tmpl`, ...carriers.map(carrier => path.join(outputRoot, carrier.relativePath))]) {
|
||
const source = fs.readFileSync(file, 'utf8');
|
||
const approach = compactProse(source.split('### 0D.')[1]?.split('### 0E. Mode Selection')[0] ?? '');
|
||
ordered(approach, ['**1. Check sources and prior answers.**', '**2. Record the pending choice.**',
|
||
/record owner, behavior, limits, test method and coverage/i, /record pending rows before comparisons/i,
|
||
"**3. Compare and save that row's options.**", 'Build one `currentDecision`', '**Pre-question checkpoint:**',
|
||
'**4. Ask, record the answer, and amend.**', /ask one row per call/i, '**Post-answer checkpoint:**']);
|
||
expectMentions(between(approach, '**4. Ask', '**Post-answer checkpoint:**'), [['stop','actual','answer']], 'section');
|
||
expect(source).toContain('| ID and owner | Contract and evidence | Current | Proposed | Status | Exact approval and scope |');
|
||
expect(source.indexOf('| ID and owner |')).toBeLessThan(source.indexOf('**2. Record the pending choice.**'));
|
||
expect(source.indexOf('### 0D.')).toBeLessThan(source.indexOf('### 0E. Mode Selection'));
|
||
}
|
||
} finally { fs.rmSync(outputRoot, { recursive: true, force: true }); }
|
||
}, 30_000);
|
||
|
||
// A saved topic row did not prevent the captured paired menu from combining
|
||
// separately proposed coverage. Main review also asked about a draft-created
|
||
// recovery promise before reconciling it with the original contract.
|
||
test('CEO decision units and factual reconciliation precede menu synthesis', () => {
|
||
const section = sectionSource();
|
||
expect(compactProse(between(section, '**Analyze.**', '**Resolve.**'))).toMatch(/correct false claims/i);
|
||
ordered(section, ['### Outside Voice Integration Rule', '{{CODEX_PLAN_REVIEW}}']);
|
||
});
|
||
|
||
test('CEO outside findings reuse authority-first decisions without turning unknown facts into policies', () => {
|
||
const section = fs.readFileSync(SECTION, 'utf8');
|
||
const tension = compactProse(between(section, '**Integrate reviewer findings:**', '**Cross-model tension:**'));
|
||
// The outside branch delegates to the original procedure, not a second approval loop.
|
||
expect(tension).toMatch(/same six-column ledger/i);
|
||
expectMentions(tension, [['do not','procedure','second']], 'tension');
|
||
expect(tension).not.toContain('reference | commitment | current value');
|
||
expectMentions(tension, [['without','premises','changing']], 'tension');
|
||
expectMentions(tension, [['not','imagining','behavior']], 'tension');
|
||
expect(tension).toMatch(/need no behavior-change menu/i);
|
||
expect(tension).toMatch(/preserve the requested mode/i);
|
||
expect(tension).toContain('A) Apply this change; B) Keep');
|
||
expect(tension).toContain('A) Include; B) Defer; C) Cut; D) Hold');
|
||
expectMentions(tension, [['never','candidate','silently']], 'tension');
|
||
expectMentions(tension, [['do not','implementation','investigation']], 'tension');
|
||
expectMentions(tension, [['wait','challenges','final']], 'tension');
|
||
expectMentions(tension, [['does not','resolve','pending']], 'tension');
|
||
});
|
||
|
||
// Guard the complete result routes on every host. This verifies instructions,
|
||
// not the quality judge's clarity score or an agent's execution of the workflow.
|
||
test('CEO integrates completed native findings before its external-only comparison', () => {
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const ctx = { skillName: 'plan-ceo-review', host: host.name, paths: HOST_PATHS[host.name]! } as TemplateContext;
|
||
const review = compactProse(generateCodexPlanReview(ctx));
|
||
expect(review.match(/\*\*Outcome routing:\*\*/g)).toHaveLength(1);
|
||
expectMentions(review, [['do not','invocation','restart']], 'review');
|
||
const anchors = [/after a completed external review, go directly/i, '**Native fallback — provider unavailable or execution failed, with reviews enabled:**',
|
||
'**Bounded outside-voice wait', '**Unavailable path:**', '**Integrate reviewer findings:**',
|
||
'**Cross-model tension:**', '**Persist the result:**'];
|
||
ordered(review, anchors);
|
||
const [externalStart, nativeStart, boundedStart, unavailableStart, findingsStart, comparisonStart, persistStart] =
|
||
anchors.map(anchor => typeof anchor === 'string' ? review.indexOf(anchor) : review.search(anchor));
|
||
expect(review.slice(externalStart, nativeStart)).toContain('**Integrate reviewer findings**');
|
||
const bounded = review.slice(boundedStart, unavailableStart);
|
||
expect(bounded).toContain('**Integrate reviewer findings**');
|
||
expect(bounded).toMatch(/follow step 4/i);
|
||
expect(bounded).not.toContain('continue to Cross-model tension');
|
||
const unavailable = review.slice(unavailableStart, findingsStart);
|
||
expect(unavailable).toContain('STATUS = "unavailable", SOURCE = "none", OUTSIDE_STATUS = "unavailable"');
|
||
const findings = review.slice(findingsStart, comparisonStart);
|
||
expect(findings).toMatch(/never as outside coverage/i);
|
||
expectMentions(findings, [['block','unavailable','disabled']], 'findings');
|
||
const comparison = review.slice(comparisonStart, persistStart);
|
||
expectMentions(comparison, [['only','completed','external']], 'comparison');
|
||
expect(comparison).toContain('**Persist the result**');
|
||
expect(comparison).toMatch(/do not write a CROSS-MODEL line/);
|
||
const report = compactProse(generatePlanFileReviewReport(ctx));
|
||
expect(report).toMatch(/\*\*CROSS-MODEL:\*\* only when native and completed external reviews exist/i);
|
||
expect(report).toContain('N findings; R resolved; U unresolved');
|
||
expect(report).toContain('"0 findings — completed review"');
|
||
expect(report).toContain('"no completed external review"');
|
||
expect(report).toMatch(/never imply zero findings/i);
|
||
expect(report).toContain('"finding count not recorded"');
|
||
expect(report).toMatch(/provider and outcome in OUTSIDE COVERAGE/);
|
||
}
|
||
});
|
||
|
||
test('CEO expansion preparation feeds one pending list into the scope decisions', () => {
|
||
const source = compactProse(skeletonSource());
|
||
const framing = between(source, '### 0F.', '### 0G.');
|
||
const decisions = between(source, '### 0G.', '### 0H.');
|
||
expect(framing).toMatch(/prepare 0G candidates/i);
|
||
expect(framing).toContain('S/M/L/XL effort, risk and impact');
|
||
expect(framing).toContain('`(recommended)`');
|
||
expect(decisions).toMatch(/extend 0F's pending list/i);
|
||
expectMentions(decisions, [['ask','separately','addition']], 'decisions');
|
||
});
|
||
|
||
// Repeated public quality feedback identified these missing execution instructions.
|
||
// This guard checks the source contract; native clarity still requires paid evidence.
|
||
test('CEO Step 0 defines the decision record, execution order, and mode approval precedence', () => {
|
||
const source = skeletonSource();
|
||
const step0 = compactProse(source.split('## Step 0:')[1]?.split('### 0F.')[0] ?? '');
|
||
ordered(step0, [/1\. choose the review depth/i, /2\. record 0A–0C evidence/i, /3\. select the mode in 0E/i,
|
||
/4\. complete review sections/i]);
|
||
expectMentions(step0, [['not','unconditional','reusable']], 'step0');
|
||
expectMentions(step0, [['do not','observations','approve']], 'step0');
|
||
ordered(step0, [/follow the preamble's session rules/i, /an explicit choice skips steps 2–3/i]);
|
||
expect(step0).toMatch(/recommend without selecting/i);
|
||
expect(step0).toMatch(/the user's choice wins/i);
|
||
expect(step0).toContain('`question_id=plan-ceo-review-mode`');
|
||
expect(step0).toContain('`gstack-question-preference --check`');
|
||
const reduction = compactProse(source.split('**For SCOPE REDUCTION:**')[1]?.split('### 0H.')[0] ?? '');
|
||
expect(reduction).toMatch(/ask separately per item/i);
|
||
ordered(reduction, ['**A)** Defer this item to TODOS.md', '**B)** Keep it in scope']);
|
||
expect(reduction).not.toContain('Remove it without a follow-up');
|
||
expect(reduction).not.toContain('Accepted items govern');
|
||
});
|
||
|
||
// Verify the source contract reaches the actual evaluated generated carrier;
|
||
// the live AUQ gate still proves model compliance with this instruction.
|
||
test('CEO mode recommendation explains a plan-specific consequence without changing routing', () => {
|
||
for (const file of [`${SKELETON}.tmpl`, SKELETON]) {
|
||
const source = fs.readFileSync(file, 'utf8');
|
||
const mode = between(source, '### 0E. Mode Selection', '### 0F.');
|
||
const recommendation = compactProse(between(mode, '2. Recommend without selecting.', '3. Resolve that recommendation.'));
|
||
expect(recommendation).toMatch(/not just its count\/category/i);
|
||
expectMentions(recommendation, [['>15', 'scope reduction']], 'recommendation');
|
||
expect(recommendation).toContain('(greenfield) → SCOPE EXPANSION');
|
||
expect(recommendation).toContain('added capability → SELECTIVE EXPANSION');
|
||
expect(recommendation).toContain('fix/refactor → HOLD SCOPE');
|
||
expect(recommendation).toContain('recommend HOLD SCOPE');
|
||
const resolve = compactProse(between(mode, '3. Resolve that recommendation.', '4. **Mode handoff:**'));
|
||
expect(resolve).toMatch(/using step 2's recommendation/i);
|
||
expect(resolve).toMatch(/stop for the answer/i);
|
||
}
|
||
});
|
||
|
||
// Boundary checks stay on source templates: generated carriers remain the
|
||
// integration owner's responsibility, and these do not prove model behavior.
|
||
describe('CEO review decision boundaries contract', () => {
|
||
const skeleton = skeletonSource();
|
||
const section = sectionSource();
|
||
const alternatives = compactProse(skeleton.split('### 0D.')[1]?.split('### 0F.')[0] ?? '');
|
||
const temporal = compactProse(skeleton.split('### 0I.')[1]?.split('### 0E.')[0] ?? '');
|
||
const continuity = compactProse(between(section, '### Working review decisions', '### Section 1:'));
|
||
const apply = compactProse(continuity.split('**Apply.**')[1]!);
|
||
|
||
test('every approach comparison preserves approvals and separates independent changes', () => {
|
||
const depth = compactProse(between(skeleton, "**Set review depth from the user's request.**", 'Plain terms:'));
|
||
ordered(depth, ['**A)** Keep this review strategy-only', '**B)** Add implementation design']);
|
||
expectMentions(depth, [['unless','recommend','concrete']], 'depth');
|
||
expect(depth).toMatch(/wait for the answer/i);
|
||
expectMentions(depth, [['only','capability','permits']], 'depth');
|
||
expectMentions(compactProse(section), [['before','depth-expansion','designing']], 'compactProse(section)');
|
||
const outputs = compactProse(section.split('## Required Outputs')[1]!);
|
||
expectMentions(outputs, [['only','strategy-only','capability']], 'outputs');
|
||
expect(outputs).not.toContain('___ methods');
|
||
expect(alternatives).toContain('| Code change and required regressions | Keep together; carry both forward once approved. |');
|
||
expect(alternatives).toContain('| Approved change with open test method/coverage | Decide once;');
|
||
expect(alternatives).toContain('| Tests for existing behavior | Separate independently selectable additions.');
|
||
expect(compactProse(section)).toMatch(/following 0D's test table/i);
|
||
expect(alternatives).toMatch(/preserve unknowns/i);
|
||
});
|
||
|
||
test('settled approach authority resolves the gate while new choices still require approval', () => {
|
||
const approach = alternatives.split('### 0E. Mode Selection')[0]!;
|
||
const reuse = approach.split('**2. Record the pending choice.**')[0]!;
|
||
expectMentions(reuse, [['only','contradictions','instructions']], 'reuse');
|
||
expectMentions(reuse, [['never','speculation','agreement']], 'reuse');
|
||
expect(approach).toMatch(/a recommendation is not approval/i);
|
||
expect(approach).not.toContain('0C-bis');
|
||
expectMentions(compactProse(section), [['before','amending','resolve']], 'compactProse(section)');
|
||
expectMentions(section, [['obvious fix', 'approval']], 'section');
|
||
const generated = fs.readFileSync(SKELETON, 'utf8');
|
||
expect(generated).toContain('### Tool resolution (read first)');
|
||
expect(generated).toContain('SESSION_KIND: spawned');
|
||
const afterAnswer = approach.split('**Post-answer checkpoint:**')[1]!;
|
||
expect(afterAnswer).toContain('"No issues, moving on."');
|
||
});
|
||
|
||
test('coverage scoring is conditional and legitimate early decisions retain their exact approval', () => {
|
||
ordered(alternatives, [/record owner, behavior, limits, test method and coverage/i, /score this row's coverage differences/i]);
|
||
expect(alternatives).toContain('10 = all edge cases');
|
||
expectMentions(alternatives, [['not','completeness','coverage']], 'alternatives');
|
||
// Scoring and recommendation details are reused from the existing preamble.
|
||
const generated = fs.readFileSync(SKELETON, 'utf8');
|
||
expect(generated).toContain('10 = complete, 7 = happy path, 3 = shortcut');
|
||
expect(generated).toContain('(recommended)');
|
||
expect(generated).toContain('AUTO_DECIDE depends on it');
|
||
expectMentions(generated, [['not','completeness','coverage']], 'generated');
|
||
});
|
||
|
||
test('an unresolved section decision is answered before its scoped plan amendment', () => {
|
||
ordered(continuity, ['**Resolve.**', '**Apply.**', /check the saved plan against each answer's exact scope/i,
|
||
/correct discrepancies under the storage policy/i, /record findings and dispositions/i]);
|
||
});
|
||
|
||
test('pending labels authorize only unresolved notes, not an outcome or future review conclusions', () => {
|
||
expectMentions(apply, [['approval','correction','resolve']], 'apply');
|
||
expectMentions(apply, [['do not','conclusions','reviewing']], 'apply');
|
||
expectMentions(apply, [['not','implementation','verification']], 'apply');
|
||
});
|
||
});
|
||
|
||
// These are source-contract checks, not model-behavior evidence. Read the
|
||
// template and resolve its shared clause directly so an old generated carrier
|
||
// cannot conceal conflicting per-section instructions during implementation.
|
||
describe('CEO review decision continuity contract', () => {
|
||
const template = sectionSource();
|
||
const clauses = ALL_HOST_CONFIGS.map(host => generateAntiShortcutClause({
|
||
skillName: 'plan-ceo-review', host: host.name,
|
||
} as TemplateContext));
|
||
const continuity = compactProse(template.split('### Working review decisions')[1]?.split('### Section 1:')[0] ?? '');
|
||
|
||
test('analysis, decision, and approved amendment precede advancing to the next section', () => {
|
||
ordered(continuity, ['**Analyze.**', '**Resolve.**', '**Apply.**']);
|
||
for (const clause of clauses) {
|
||
const text = compactProse(clause);
|
||
expect(text).toContain('Analyze → resolve → apply');
|
||
expectMentions(text, [['do not','remaining','prewrite']], 'text');
|
||
expectMentions(text, [['not','proposed','findings']], 'text');
|
||
}
|
||
});
|
||
|
||
test('every section has one decision gate wired to the shared procedure', () => {
|
||
const declared = Number(template.match(/^## Review Sections \((\d+) sections/m)?.[1]);
|
||
const sections = [...template.matchAll(/^### Section (\d+):([^]*?)(?=^### Section \d+:|^## Closing sequence)/gm)];
|
||
expect(sections.map(section => Number(section[1]))).toEqual(Array.from({ length: declared }, (_, i) => i + 1));
|
||
for (const [, number, body] of sections) {
|
||
expect(body.match(/\*\*Decision gate\.\*\*/g), `Section ${number}`).toHaveLength(1);
|
||
expect(body, `Section ${number}`).toMatch(/\*\*Decision gate\.\*\* Complete Analyze → Resolve → Apply/);
|
||
}
|
||
ordered(template, ['### Working review decisions', '### Section 1:']);
|
||
expect(continuity).toMatch(/at each section's \*\*Decision gate\*\*, follow Analyze → Resolve → Apply/i);
|
||
expectMentions(continuity, [['do not','review','change']], 'continuity');
|
||
expectMentions(compactProse(template), [['"no issues found"', 'only', 'zero']], 'template');
|
||
expect(template).toMatch(/evaluate Sections 1–10 in full/i);
|
||
expect(template).toContain('`SKIPPED (no UI scope)`');
|
||
ordered(template, ['### Completion Summary', '{{PLAN_FILE_REVIEW_REPORT}}']);
|
||
});
|
||
|
||
test('the ledger carries exact approvals and declared contracts without claiming implementation', () => {
|
||
const skeleton = skeletonSource();
|
||
const start = skeleton.search(/keep one decision ledger/i);
|
||
expect(start).toBeGreaterThan(skeleton.indexOf('## Step 0:'));
|
||
expect(start).toBeLessThan(skeleton.indexOf('### 0D.'));
|
||
const earlyLedger = skeleton.slice(start, skeleton.indexOf('### 0A.'));
|
||
expect(earlyLedger).toContain('| ID and owner | Contract and evidence | Current | Proposed | Status | Exact approval and scope |');
|
||
const limits = compactProse(between(skeleton, '**Keep the stated limits.**', '**Storage policy:'));
|
||
expectMentions(limits, [['approval','changing','evidence']], 'limits');
|
||
expect(skeleton.indexOf('**Keep the stated limits.**')).toBeLessThan(skeleton.indexOf('### 0D.'));
|
||
const temporal = compactProse(skeleton.split('### 0I.')[1]?.split('{{SECTION:review-sections}}')[0] ?? '');
|
||
expectMentions(temporal, [['unless','implementation','requested']], 'temporal');
|
||
expectMentions(compactProse(template), [['does not','prioritization','implementation']], 'compactProse(template)');
|
||
});
|
||
|
||
test('ownership never defers a critical risk or merges distinct choices by topic', () => {
|
||
expect(continuity).toMatch(/resolve critical risks now/i);
|
||
const testReview = compactProse(between(template, '### Section 6: Test Review', '### Section 7:'));
|
||
expectMentions(testReview, [['without','requested','re-asking']], 'testReview');
|
||
expectMentions(testReview, [['before','prescribing','resolve']], 'testReview');
|
||
expectMentions(testReview, [['does not','verification','approved']], 'testReview');
|
||
const outside = compactProse(between(template, '### Outside Voice Integration Rule', '{{CODEX_PLAN_REVIEW}}'));
|
||
expectMentions(outside, [['not','agreement','reviewer']], 'outside');
|
||
});
|
||
|
||
test('Design preserves decision gating while DX and fallback retain their existing output on every host', () => {
|
||
// Pin DX's e15ba218 output independently: Eng now has its own wording.
|
||
const originalDevex = '29fe85565c2ea16c0d205a06a2515e30228342078a44bb688489274c157ddba2';
|
||
const fallbacks = new Set<string>();
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const fallback = generateAntiShortcutClause({ skillName: 'review', host: host.name } as TemplateContext);
|
||
fallbacks.add(fallback);
|
||
// Default clause keeps its constraints: the plan cannot replace the interactive review,
|
||
// findings go through AskUserQuestion before the plan, only zero findings skip asking.
|
||
expect(fallback).toMatch(/cannot replace|not a substitute/i);
|
||
expect(fallback).toMatch(/finding[\s\S]*AskUserQuestion/i);
|
||
expect(fallback).toMatch(/zero findings[\s\S]*ExitPlanMode/i);
|
||
const devex = generateAntiShortcutClause({ skillName: 'plan-devex-review', host: host.name } as TemplateContext);
|
||
expect(createHash('sha256').update(devex).digest('hex'), `plan-devex-review/${host.name}`).toBe(originalDevex);
|
||
const design = compactProse(generateAntiShortcutClause({ skillName: 'plan-design-review', host: host.name } as TemplateContext));
|
||
expectMentions(design, [['wait','independent','decision']], 'design');
|
||
expectMentions(design, [['do not','previously','selected']], 'design');
|
||
expectMentions(design, [['does not','independent','remedies']], 'design');
|
||
}
|
||
expect(fallbacks.size, 'default clause is host-independent').toBe(1);
|
||
});
|
||
});
|
||
|
||
test('CEO closing route checks approvals before outputs and verifies artifacts before telemetry without a file bounce', () => {
|
||
const skeleton = skeletonSource();
|
||
const section = sectionSource();
|
||
const route = between(section, '## Closing sequence', '### Outside Voice Integration Rule');
|
||
const routeStages = ['**Outside Voice:**', '**Resolve remaining TODO choices:**', '**Approval readiness:**',
|
||
'**Required Outputs:**', '**Cleanup and history:**', '**Navigation:**', '**Learnings:**'];
|
||
ordered(route, routeStages);
|
||
expect(route.match(/^\d+\. /gm)).toHaveLength(routeStages.length);
|
||
const steps = compactProse(route);
|
||
expectMentions(steps, [['no','unavailable','disabled']], 'steps');
|
||
expectMentions(steps, [['pass', 'before', 'outputs']], 'steps');
|
||
expectMentions(steps, [['only','readiness','approval']], 'steps');
|
||
expectMentions(steps, [['without','settled','choices']], 'steps');
|
||
expect(steps).toMatch(/queue the next skill/i);
|
||
|
||
ordered(section, ['## Closing sequence', '{{CODEX_PLAN_REVIEW}}', '## Resolve remaining TODO choices',
|
||
'### TODOS.md updates', '{{PLAN_REVIEW_APPROVAL_CHECK}}', '## Required Outputs',
|
||
'{{PLAN_FILE_REVIEW_REPORT}}', '## Review Log', '{{REVIEW_DASHBOARD}}', '## Next Steps — Review Chaining',
|
||
'## docs/designs Promotion', '{{LEARNINGS_LOG}}', '{{GBRAIN_SAVE_RESULTS}}', '{{BRAIN_WRITE_BACK}}',
|
||
"Return to this skill's main `SKILL.md`: Section self-check → EXIT PLAN MODE GATE."]);
|
||
expect(section).not.toContain('{{EXIT_PLAN_MODE_GATE}}');
|
||
expect(section).not.toContain('{{BRAIN_CACHE_REFRESH}}');
|
||
expect(section).not.toContain('Run the preamble\'s **Telemetry');
|
||
|
||
const actualGate = skeleton.indexOf('{{EXIT_PLAN_MODE_GATE}}');
|
||
expect(actualGate).toBeGreaterThan(skeleton.indexOf('## Section self-check'));
|
||
const terminal = skeleton.slice(actualGate);
|
||
const success = compactProse(terminal.split('**Passed with a verified persisted report:**')[1]!);
|
||
ordered(success, [/finish the cache refresh below/i, '{{BRAIN_CACHE_REFRESH}}',
|
||
'**Telemetry (run last)** once', /the review is now finished/i, 'Call ExitPlanMode',
|
||
/next-skill handoff starts a separate workflow/i]);
|
||
expect(terminal).not.toMatch(/return to (?:the )?section|Closing hooks/);
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const refresh = compactProse(generateBrainCacheRefresh({ skillName: 'plan-ceo-review', host: host.name, paths: HOST_PATHS[host.name]! } as TemplateContext));
|
||
expectMentions(refresh, [['before','nonblocking','telemetry']], 'refresh');
|
||
expect(refresh).not.toContain('telemetry has logged');
|
||
}
|
||
|
||
const askQuestions = section.search(ASK_QUESTIONS_HEADING);
|
||
const governingStages = [askQuestions, ...['## Formatting Rules',
|
||
'## Mode Quick Reference', '### Working review decisions', '### Section 1:']
|
||
.map(stage => section.indexOf(stage))];
|
||
expect(governingStages.every(position => position >= 0)).toBe(true);
|
||
expect(governingStages).toEqual([...governingStages].sort((a, b) => a - b));
|
||
const questions = section.slice(askQuestions).split('\n').slice(1).join('\n').split('## Mode Quick Reference')[0]!;
|
||
expect(questions).toContain('Use `D<N>` and A/B/C labels');
|
||
const formatting = questions.split('## Formatting Rules')[1]!;
|
||
expect(formatting).toMatch(/0D's exact `currentDecision` question and option descriptions/i);
|
||
expect(questions).not.toMatch(/NUMBER \+ (?:option )?LETTER|"3A"|One sentence max per option/);
|
||
});
|
||
|
||
describe('plan-ceo-review carve — static ordering', () => {
|
||
const skeleton = fs.readFileSync(SKELETON, 'utf-8');
|
||
const section = fs.readFileSync(SECTION, 'utf-8');
|
||
|
||
// Index into the skeleton, -1 if absent.
|
||
const at = (needle: string): number => skeleton.indexOf(needle);
|
||
|
||
const STEP0 = '## Step 0: Nuclear Scope Challenge + Mode Selection';
|
||
const STOP = 'sections/review-sections.md'; // appears in the index row + STOP directive
|
||
const GATE = 'GSTACK REVIEW REPORT';
|
||
|
||
test('the interactive anti-shortcut contract is available before audit or lazy section loading', () => {
|
||
const contract = '**Anti-shortcut clause:**';
|
||
const audit = skeleton.indexOf('## PRE-REVIEW SYSTEM AUDIT');
|
||
expect(skeleton.indexOf(contract)).toBeGreaterThan(-1);
|
||
expect(skeleton.indexOf(contract)).toBeLessThan(audit);
|
||
expect(skeleton.split(contract)).toHaveLength(2);
|
||
expect(section).not.toContain(contract);
|
||
// CEO's shared contract distinguishes unanswered choices from exact prior
|
||
// approvals; the generic fallback's any-finding rule is not its contract.
|
||
expect(skeleton).toContain(generateAntiShortcutClause({ skillName: 'plan-ceo-review' } as TemplateContext));
|
||
expectMentions(skeleton, [['wait','unresolved','reopened']], 'skeleton');
|
||
expectMentions(skeleton, [['never','cross-referencing','decisions']], 'skeleton');
|
||
expectMentions(skeleton, [['never','question','because']], 'skeleton');
|
||
});
|
||
|
||
test('skeleton emits a STOP-Read directive pointing at the section', () => {
|
||
expect(skeleton).toContain('> **STOP.**');
|
||
expect(skeleton).toContain('plan-ceo-review/sections/review-sections.md');
|
||
expect(skeleton).toContain('## Section index — Read each section when its situation applies');
|
||
});
|
||
|
||
test('Step 0 (scope + mode) stays in the skeleton, BEFORE the STOP', () => {
|
||
const step0 = at(STEP0);
|
||
const stop = skeleton.indexOf('> **STOP.**');
|
||
expect(step0).toBeGreaterThan(-1);
|
||
expect(stop).toBeGreaterThan(step0); // STOP fires only after Step 0
|
||
});
|
||
|
||
test('mode selection precedes mode-specific analysis after approach approval', () => {
|
||
const approach = at('### 0D.');
|
||
const mode = at('### 0E. Mode Selection');
|
||
const analysis = at('### 0G. Mode-Specific Analysis');
|
||
expect(approach).toBeGreaterThan(-1);
|
||
expect(mode).toBeGreaterThan(approach);
|
||
expect(analysis).toBeGreaterThan(mode);
|
||
expectMentions(skeleton, [['only','evidence','required']], 'skeleton');
|
||
expect(skeleton).toContain('| SCOPE EXPANSION / SELECTIVE EXPANSION | 0F → 0G → 0H (including its spec review loop) → 0I |');
|
||
expect(skeleton).toContain('| HOLD SCOPE | 0G → 0I |');
|
||
expect(skeleton).toContain('| SCOPE REDUCTION | 0G |');
|
||
expect(skeleton.indexOf('## Continue after Step 0 (all modes)')).toBeGreaterThan(skeleton.indexOf('### 0I.'));
|
||
expect(skeleton.indexOf('## Continue after Step 0 (all modes)')).toBeLessThan(skeleton.indexOf('> **STOP.**'));
|
||
expect(compactProse(skeleton)).toContain('ask separately per item');
|
||
expectMentions(skeleton, [['>15', 'scope reduction']], 'skeleton');
|
||
expect(skeleton).toContain('a new product/system (greenfield) → SCOPE EXPANSION');
|
||
const persist = skeleton.split('### 0H. Persist CEO Plan (EXPANSION and SELECTIVE EXPANSION only)')[1]?.split('### 0I.')[0] ?? '';
|
||
expect(persist).toMatch(/^#### Spec Review Loop$/m);
|
||
const handoff = compactProse(persist.slice(persist.indexOf('After the loop completes or reports unavailable')));
|
||
const approval = ['for final scope-document approval', 'Ask with the preamble question transport:',
|
||
'**A)** Approve these documents and continue to 0I', '**B)** Revise these documents', '**C)** Pause this review',
|
||
'Wait and record the answer', 'For B, resolve the requested changes through 0D',
|
||
'update both inputs and repeat document approval', 'C stops',
|
||
'After A, run 0I before Review Sections'].map(stage => handoff.indexOf(stage));
|
||
expect(approval.every(position => position >= 0)).toBe(true);
|
||
expect(approval).toEqual([...approval].sort((a, b) => a - b));
|
||
expectMentions(handoff, [['only','recommend','decisions']], 'handoff');
|
||
expectMentions(handoff, [['only','implementation','unresolved']], 'handoff');
|
||
});
|
||
|
||
test('the heavy review body (Sections 1-11) is NOT in the skeleton', () => {
|
||
expect(skeleton).not.toContain('### Section 1: Architecture Review');
|
||
expect(skeleton).not.toContain('### Section 11:');
|
||
// ...it lives in the section instead.
|
||
expect(section).toContain('### Section 1: Architecture Review');
|
||
expect(section).toContain('### Section 11:');
|
||
});
|
||
|
||
test('Autoplan CEO uses the loaded Step 0 route before its dual voices and review sections', () => {
|
||
for (const suffix of ['.md.tmpl', '.md']) {
|
||
const phase = fs.readFileSync(path.join(ROOT, 'autoplan/sections/ceo-phase' + suffix), 'utf8');
|
||
const step0 = phase.split('**Required execution checklist (CEO):**')[1]?.split('Step 0.5 (Dual Voices):')[0] ?? '';
|
||
expect(step0).toContain('SELECTIVE EXPANSION');
|
||
expectMentions(step0.replace(/\s+/g, ' '), [['before','document','sections']], 'step0.replace(/\s+/g, )');
|
||
expect(step0).not.toMatch(/^- 0[A-I](?:-bis)?:/m);
|
||
// The headings alone can be ordered while executable reviewer payloads
|
||
// still run ahead of Step 0, or Codex is presented ahead of Claude.
|
||
const prose = phase.replace(/\s+/g, ' ');
|
||
const positions = ['**Required execution checklist (CEO):**', 'Step 0.5 (Dual Voices):',
|
||
'Read `snapshot.json` beside `<CEO_INPUT>`',
|
||
'Send its `nativeDispatchPrompt` verbatim as the Agent prompt', 'Native completion barrier:',
|
||
'Outside prompt: inline the full contents of <CEO_INPUT>',
|
||
suffix === '.md.tmpl' ? '{{OUTSIDE_INVOCATION:autoplan}}' : '_OUTSIDE_EXIT=0',
|
||
'CEO DUAL VOICES — CONSENSUS TABLE:', 'Sections 1-11 —', '**Mandatory outputs from Phase 1:**', '**Close this phase:**',
|
||
suffix === '.md.tmpl' ? '{{SECTION:phase-close}}' : 'Read `~/.claude/skills/gstack/autoplan/sections/phase-close.md` and execute it']
|
||
.map(stage => prose.indexOf(stage));
|
||
expect(positions.every(position => position >= 0)).toBe(true);
|
||
expect(positions).toEqual([...positions].sort((a, b) => a - b));
|
||
}
|
||
});
|
||
|
||
test('nothing review-governing sits in the skeleton below the STOP (Codex P1)', () => {
|
||
// Mode Quick Reference + Formatting Rules govern review-time behavior and must
|
||
// travel with the section, not be stranded below the STOP in the skeleton.
|
||
expect(skeleton).not.toContain('## Mode Quick Reference');
|
||
expect(skeleton).not.toContain('## Formatting Rules');
|
||
expect(section).toContain('## Mode Quick Reference');
|
||
});
|
||
|
||
test('review report writer lives in the section; the EXIT PLAN MODE GATE stays in the skeleton AFTER the STOP', () => {
|
||
// The report itself is produced inside the section work...
|
||
expect(section).toContain(GATE);
|
||
// ...and the blocking gate that verifies it is the last thing the skeleton runs.
|
||
const stop = skeleton.indexOf('> **STOP.**');
|
||
const gate = skeleton.lastIndexOf(GATE);
|
||
expect(gate).toBeGreaterThan(stop);
|
||
});
|
||
|
||
test('the loaded test-review section preserves mandatory behaviors and individual assertion decisions', () => {
|
||
const template = fs.readFileSync(`${SECTION}.tmpl`, 'utf-8');
|
||
for (const document of [template, section]) {
|
||
const testReview = document.split('### Section 6: Test Review')[1]?.split('### Section 7:')[0];
|
||
expect(testReview).toBeDefined();
|
||
const instructions = testReview!.replace(/\s+/g, ' ');
|
||
expectMentions(instructions, [['without','determine','assertion']], 'instructions');
|
||
expectMentions(instructions, [['never','weaken','exact']], 'instructions');
|
||
expectMentions(instructions, [['without','requirements','asking']], 'instructions');
|
||
expectMentions(instructions, [['only','individually','independent']], 'instructions');
|
||
expectMentions(instructions, [['do not','success','labels']], 'instructions');
|
||
expectMentions(instructions, [['does not','scope/approach','individual']], 'instructions');
|
||
expectMentions(instructions, [['does not','coverage','verify']], 'instructions');
|
||
expectMentions(instructions, [['before','requirement','existing']], 'instructions');
|
||
expectMentions(instructions, [['never','behavioral','assertion']], 'instructions');
|
||
expectMentions(instructions, [['unless','explicitly','behaviors']], 'instructions');
|
||
const phases = ['**Map the requirement.**', '**Reuse settled proof.**', '**Resolve actual gaps.**'].map(phase => instructions.indexOf(phase));
|
||
expect(phases.every(position => position >= 0)).toBe(true);
|
||
expect(phases).toEqual([...phases].sort((a, b) => a - b));
|
||
expect(instructions).toContain('**Decision gate.** Complete Analyze → Resolve → Apply above for this section before continuing.');
|
||
const procedure = document.split('### Working review decisions')[1]!.split('### Section 1:')[0]!.replace(/\s+/g, ' ');
|
||
expect(document.indexOf('### Working review decisions')).toBeLessThan(document.indexOf('### Section 6: Test Review'));
|
||
expect(procedure).toContain("At each section's **Decision gate**, follow Analyze → Resolve → Apply below");
|
||
expectMentions(compactProse(document), [['"no issues found"', 'only', 'zero']], 'document');
|
||
}
|
||
});
|
||
|
||
test('the loaded data-flow review requires evidence across interacting operations', () => {
|
||
const template = fs.readFileSync(`${SECTION}.tmpl`, 'utf-8');
|
||
for (const document of [template, section]) {
|
||
const dataFlow = document.split('### Section 4: Data Flow & Interaction Edge Cases')[1]?.split('### Section 5:')[0];
|
||
expect(dataFlow).toBeDefined();
|
||
const instructions = dataFlow!.replace(/\s+/g, ' ');
|
||
expectMentions(instructions, [['do not','single-thread','favorable']], 'instructions');
|
||
expectMentions(instructions, [['only','mechanism','prevents']], 'instructions');
|
||
expectMentions(instructions, [['not','implementation','requirement']], 'instructions');
|
||
const phases = ['**Define the boundary.**', '**Exercise both orders.**', '**Compare the result.**', '**Specify regression proof.**'].map(phase => instructions.indexOf(phase));
|
||
expect(phases.every(position => position >= 0)).toBe(true);
|
||
expect(phases).toEqual([...phases].sort((a, b) => a - b));
|
||
}
|
||
});
|
||
|
||
test('the section is generated, not hand-edited', () => {
|
||
expect(section.slice(0, 120)).toContain('AUTO-GENERATED');
|
||
});
|
||
});
|
||
|
||
describe('section capture completion signal', () => {
|
||
// Isolate the runner mock in its own Bun process. Loading a mock in this free
|
||
// shard would replace the real runner for unrelated tests in the same process.
|
||
function captureFixture(exitReason: string, draft: boolean, options: { seed?: string; output?: string; directoryBefore?: boolean; directoryAfter?: boolean } = {}) {
|
||
const planDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ceo-capture-result-'));
|
||
try {
|
||
if (options.seed !== undefined) fs.writeFileSync(path.join(planDir, 'REPORT.md'), options.seed);
|
||
if (options.directoryBefore) fs.mkdirSync(path.join(planDir, 'REPORT.md'));
|
||
const script = `
|
||
import { mock } from 'bun:test';
|
||
import { writeFileSync, mkdirSync } from 'node:fs';
|
||
let runnerCalls = 0;
|
||
import { captureSectionReads } from ${JSON.stringify(path.join(ROOT, 'test/helpers/auq-sdk-capture.ts'))};
|
||
mock.module(${JSON.stringify(path.join(ROOT, 'test/helpers/session-runner.ts'))}, () => ({
|
||
runSkillTest: async () => {
|
||
runnerCalls++;
|
||
${options.directoryAfter ? `mkdirSync(${JSON.stringify(path.join(planDir, 'REPORT.md'))});` : ''}
|
||
${draft ? `writeFileSync(${JSON.stringify(path.join(planDir, 'REPORT.md'))}, '# Review report\\nIN PROGRESS');` : ''}
|
||
return { exitReason: ${JSON.stringify(exitReason)}, toolCalls: [], output: ${JSON.stringify(options.output ?? 'Final review report from stdout')} };
|
||
},
|
||
}));
|
||
try {
|
||
const capture = await captureSectionReads({
|
||
planDir: ${JSON.stringify(planDir)}, skillName: 'plan-ceo-review',
|
||
scenario: 'fixture', testName: 'capture-result-fixture', reportMarker: /review report/i,
|
||
});
|
||
process.stdout.write(JSON.stringify(capture));
|
||
} catch (error) { process.stdout.write(JSON.stringify({ errorCode: error.code, runnerCalls })); }
|
||
`;
|
||
const child = Bun.spawnSync([process.execPath, '-e', script], { cwd: ROOT, timeout: 10_000 });
|
||
expect(child.exitCode).toBe(0);
|
||
expect(child.stderr.toString()).toBe('');
|
||
return JSON.parse(child.stdout.toString());
|
||
} finally {
|
||
fs.rmSync(planDir, { recursive: true, force: true });
|
||
}
|
||
}
|
||
|
||
test('an unchanged seeded report marker cannot supply attempt completion', () => {
|
||
expect(captureFixture('success', false, { seed: '# Review report\nSeeded plan', output: 'Nothing completed' }))
|
||
.toMatchObject({ exitReason: 'success', reportWritten: false, reportProduced: false, output: 'Nothing completed' });
|
||
});
|
||
test('unchanged seed preserves a valid successful terminal report fallback', () => {
|
||
expect(captureFixture('success', false, { seed: '# Review report\nSeeded plan' }))
|
||
.toMatchObject({ reportWritten: false, reportProduced: true, output: 'Final review report from stdout' });
|
||
});
|
||
test('changed seeded bytes are this attempt artifact, still gated by native success', () => {
|
||
for (const exitReason of ['success', 'timeout']) {
|
||
expect(captureFixture(exitReason, true, { seed: 'Original plan', output: '' }))
|
||
.toMatchObject({ exitReason, reportWritten: true, reportProduced: exitReason === 'success', output: '# Review report\nIN PROGRESS' });
|
||
}
|
||
});
|
||
test('same-byte rewrite is not new report evidence and empty terminal text stays incomplete', () => {
|
||
expect(captureFixture('success', true, { seed: '# Review report\nIN PROGRESS', output: '' }))
|
||
.toMatchObject({ reportWritten: false, reportProduced: false, output: '' });
|
||
});
|
||
test('a newly created report is retained on successful native completion', () => {
|
||
expect(captureFixture('success', true, { output: '' }))
|
||
.toMatchObject({ reportWritten: true, reportProduced: true, output: '# Review report\nIN PROGRESS' });
|
||
});
|
||
test('report snapshot errors retain their cause before the native attempt', () => {
|
||
expect(captureFixture('success', false, { directoryBefore: true }))
|
||
.toEqual({ errorCode: 'EISDIR', runnerCalls: 0 });
|
||
});
|
||
test('report read errors retain their cause after the native attempt', () => {
|
||
expect(captureFixture('success', false, { directoryAfter: true }))
|
||
.toEqual({ errorCode: 'EISDIR', runnerCalls: 1 });
|
||
});
|
||
|
||
test.each(['timeout', 'error_api'])('a draft from %s does not signal shared capture completion', exitReason => {
|
||
const capture = captureFixture(exitReason, true);
|
||
expect(capture).toMatchObject({
|
||
exitReason, reportWritten: true, reportProduced: false, output: '# Review report\nIN PROGRESS',
|
||
});
|
||
});
|
||
|
||
test('successful captures preserve the terminal-output fallback', () => {
|
||
const capture = captureFixture('success', false);
|
||
expect(capture).toMatchObject({
|
||
exitReason: 'success', reportWritten: false, reportProduced: true, output: 'Final review report from stdout',
|
||
});
|
||
});
|
||
});
|
||
|
||
describe('CEO review completion evidence', () => {
|
||
const completeReport = [
|
||
'# CEO review — HOLD SCOPE',
|
||
'Keep the cache process-local and invalidate only after a successful write.',
|
||
'### Completion Summary',
|
||
'| Review area | Outcome |',
|
||
'| --- | --- |',
|
||
'| Section 1 (Arch) | 1 issue: centralize the invalidation boundary |',
|
||
'| Section 2 (Errors) | 3 error paths mapped, 1 fallback gap |',
|
||
'| Section 3 (Security) | 1 issue: include tenant identity in keys |',
|
||
'| Section 4 (Data/UX) | 2 edge cases mapped, 0 unhandled |',
|
||
'| Section 5 (Quality) | No issues found |',
|
||
'| Section 6 (Tests) | Diagram produced, 2 test gaps |',
|
||
'| Section 7 (Perf) | 1 issue: cap cached value size |',
|
||
'| Section 8 (Observ) | 1 gap: missing hit-rate metric |',
|
||
'| Section 9 (Deploy) | 1 risk: warm-up load |',
|
||
'| Section 10 (Future) | Reversibility: 5/5, 0 debt items |',
|
||
'| Section 11 (Design) | SKIPPED (no UI scope) |',
|
||
'## GSTACK REVIEW REPORT',
|
||
'| Review | Runs | Status | Findings |',
|
||
'| CEO | 1 | CLEAR | Cache boundaries reviewed |',
|
||
'**VERDICT:** CEO CLEARED',
|
||
'NO UNRESOLVED DECISIONS',
|
||
].join('\n');
|
||
const completed = { exitReason: 'success', reportWritten: true, output: completeReport };
|
||
|
||
test('accepts a complete compact report, including the documented no-UI skip', () => {
|
||
expect(() => validateCeoReviewCompletion(completed)).not.toThrow();
|
||
});
|
||
|
||
test('accepts bold Markdown section labels', () => {
|
||
const output = completeReport.replace(/\| (Section \d+ \([^|]+\)) \|/g, '| **$1** |');
|
||
expect(() => validateCeoReviewCompletion({ ...completed, output })).not.toThrow();
|
||
});
|
||
|
||
test('accepts a numbered Completion Summary heading', () => {
|
||
const output = completeReport.replace('### Completion Summary', '### 12. Completion Summary');
|
||
expect(() => validateCeoReviewCompletion({ ...completed, output })).not.toThrow();
|
||
});
|
||
|
||
test('does not require the host plan-mode footer in a standalone report', () => {
|
||
const output = completeReport.split('## GSTACK REVIEW REPORT')[0];
|
||
expect(() => validateCeoReviewCompletion({ ...completed, output })).not.toThrow();
|
||
});
|
||
|
||
test('accepts completed reviews with unresolved findings and pending decisions', () => {
|
||
const output = completeReport.replace('No issues found', '2 gaps; approval pending; TODO decisions recorded');
|
||
expect(() => validateCeoReviewCompletion({ ...completed, output })).not.toThrow();
|
||
});
|
||
|
||
test('accepts a concrete finding about skipped work', () => {
|
||
const output = completeReport.replace('1 gap: missing hit-rate metric', '1 gap: audit logging is skipped on failures');
|
||
expect(() => validateCeoReviewCompletion({ ...completed, output })).not.toThrow();
|
||
});
|
||
|
||
test('rejects a timeout even when the report file exists', () => {
|
||
expect(() => validateCeoReviewCompletion({ ...completed, exitReason: 'timeout' }))
|
||
.toThrow('execution failed: timeout');
|
||
});
|
||
|
||
test('rejects report-like stdout when the requested report file is absent', () => {
|
||
expect(() => validateCeoReviewCompletion({ ...completed, reportWritten: false }))
|
||
.toThrow('did not write REPORT.md');
|
||
});
|
||
|
||
test('rejects generic prose mentioning a review and completion summary', () => {
|
||
expect(() => validateCeoReviewCompletion({
|
||
...completed,
|
||
output: 'I reviewed all eleven sections and will produce the Completion Summary and GSTACK REVIEW REPORT. '.repeat(3),
|
||
})).toThrow('missing its Completion Summary');
|
||
});
|
||
|
||
test('requires an actual Completion Summary', () => {
|
||
expect(() => validateCeoReviewCompletion({
|
||
...completed,
|
||
output: completeReport.replace('### Completion Summary', 'The Completion Summary will follow.'),
|
||
})).toThrow('missing its Completion Summary');
|
||
});
|
||
|
||
test.each(Array.from({ length: 11 }, (_, index) => index + 1))('requires the Section %i outcome', section => {
|
||
const output = completeReport.split('\n')
|
||
.filter(line => !line.startsWith(`| Section ${section} (`)).join('\n');
|
||
expect(() => validateCeoReviewCompletion({ ...completed, output }))
|
||
.toThrow(`Section ${section} outcome`);
|
||
});
|
||
|
||
test.each(['___ issues found', 'TBD', 'reviewed', 'SKIPPED'])('rejects unfinished or skipped non-UI outcomes: %s', outcome => {
|
||
const output = completeReport.replace('No issues found', outcome);
|
||
expect(() => validateCeoReviewCompletion({ ...completed, output }))
|
||
.toThrow('Section 5 outcome');
|
||
});
|
||
|
||
test('rejects the original template placeholders as completed outcomes', () => {
|
||
const output = completeReport.replace('SKIPPED (no UI scope)', '___ issues / SKIPPED (no UI scope)');
|
||
expect(() => validateCeoReviewCompletion({ ...completed, output }))
|
||
.toThrow('Section 11 outcome');
|
||
});
|
||
});
|
||
|
||
|
||
// Both b176 paid attempts saved a comparison without their complete native choices.
|
||
// These guard source ordering and branches; captured controls retain those failures.
|
||
describe('CEO complete question persistence before dispatch', () => {
|
||
const source = skeletonSource();
|
||
const cycle = compactProse(between(source, '### 0D.', '### 0E.'));
|
||
const built = between(cycle, 'Build one `currentDecision`', '**Pre-question checkpoint:**');
|
||
const checkpoint = between(cycle, '**Pre-question checkpoint:**', '**4. Ask');
|
||
const dispatch = between(cycle, '**4. Ask', '**Post-answer checkpoint:**');
|
||
|
||
test('CEO constructs and verifies one complete row decision before dispatch', () => {
|
||
ordered(cycle, ['**2. Record the pending choice.**', /cite the source filename\/message/i,
|
||
"**3. Compare and save that row's options.**", 'Build one `currentDecision`', /score this row's coverage differences/i,
|
||
'**Pre-question checkpoint:**', '**Read-back.**', '**4. Ask, record the answer, and amend.**',
|
||
/copy the verified read or chat text into one native arguments object/i,
|
||
/compare its question, header, labels and full descriptions literally/i, /ask one row per call with that object unchanged/i,
|
||
/stop for the actual answer/i, /save the answer reference and scope/i, '**Post-answer checkpoint:**']);
|
||
for (const field of ['| `question` | Full brief:', '| `header` and option labels |', 'Project, ELI10, Stakes, Recommendation',
|
||
'offer 2–3 options', '1–2 sentence summary', 'S/M/L/XL effort', 'low/medium/high risk', 'reuse, verification coverage',
|
||
'Note: options differ in kind, not coverage — no completeness score.']) expect(built).toContain(field);
|
||
for (const field of ['## currentDecision (ROW-ID)', 'Commitment comparison: <complete grid>',
|
||
'Question: <complete currentDecision.question>', 'Header: <exact currentDecision.header>',
|
||
'<full first option description>', '<full second option description; repeat for all offered options>']) expect(checkpoint).toContain(field);
|
||
expectMentions(checkpoint, [['not','summary','pointer']], 'checkpoint');
|
||
expect(checkpoint).toMatch(/verify IDs and fields against `currentDecision`, citations against source/i);
|
||
// Native 043a questions were recomposed after title-only saves; the final
|
||
// Edit ACK also said a Read was unnecessary.
|
||
expectMentions(checkpoint, [['before','mismatches','dispatch']], 'checkpoint');
|
||
// The 6aef retry rewrote its saved fields in the actual call; bind the
|
||
// outgoing object to the saved strings at the dispatch boundary.
|
||
expect(dispatch).toContain('{questions: [{question, header, options: [{label, description}, ...]}]}');
|
||
expectMentions(dispatch, [['only','ignoring','selector']], 'dispatch');
|
||
expect(dispatch).toMatch(/compare strings, not format\/scores/i);
|
||
});
|
||
|
||
test('CEO finishes native field identity and tradeoffs before saving, then reads the entire payload', () => {
|
||
// Native 749df paired retry omitted its row ID; final labels/tradeoffs also drifted.
|
||
expect(built).toContain('`D<N> — <ROW-ID>: <one-line question>`');
|
||
expect(built).toMatch(/D counts questions; ROW-ID identifies the pending choice/i);
|
||
expect(built).toContain('`(recommended)`');
|
||
expect(built).toContain("| Each option's `description` |");
|
||
expect(built).toContain('at least 2 ✅ pros and 1 ❌ con');
|
||
expect(built).toMatch(/within host limits/i);
|
||
expect(built).toMatch(/destructive-choice exception/i);
|
||
expect(checkpoint).toMatch(/fetch continuations/i);
|
||
});
|
||
|
||
test('CEO save verification retains forbidden-write, failed-save, automatic and changed-decision branches', () => {
|
||
const policy = compactProse(between(source, '**Storage policy: choose before writing.**', '| ID and owner |'));
|
||
expectMentions(policy, [['without','forbidden','decisions']], 'policy');
|
||
expect(policy).toContain('**not persisted**');
|
||
expect(policy).toContain('**completion blocked**');
|
||
expectMentions(policy, [['no','exitplanmode','completion']], 'policy');
|
||
expectMentions(policy, [['stop','replace','cannot']], 'policy');
|
||
expectMentions(between(policy, '| 0H spec-review metrics |', '| Review, decision'), [['does not','availability','reviewer']], 'section');
|
||
const history = policy.split('| Review, decision and question history logs |')[1]!;
|
||
expect(cycle).toMatch(/for chat, verify the complete text labeled \*\*not persisted\*\*/i);
|
||
expectMentions(cycle, [['only','auto-decision','authorize']], 'cycle');
|
||
expectMentions(dispatch, [['only','preamble-authorized','auto-decision']], 'dispatch');
|
||
});
|
||
});
|