mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-02 17:40:02 +02:00
test: guard the reduced suite against new test-of-test files
- test/test-of-test-ratchet.test.ts records the 228 free tests that import only test/ code and fails on a new one, naming the owner test to extend instead; a stale baseline entry fails with the remove instruction. - test/helpers/resolve-repo-path.ts is the one specifier/literal resolver for the ratchet and the touchfile closure invariant, with its own unit tests. - CONTRIBUTING "Test tiers" describes the paid-failure workflow (fix, then one row in the detector's owner test) and the ratchet; TEST_PORTFOLIO gains the detector -> owner-test table and no longer claims an Autoplan chain eval. - TODOS: automatic exclusion policy for chronically red periodic files (P3), the deferred native-completion table collapse, the unused CEO payment seeder; the PTY readiness item is narrowed to the paid runner. - docs/test-audit-2026-09.md collects the triage, security mapping, inventories, selection proof, behavior-commit decisions and retained false positives.
This commit is contained in:
garrytan committed
1 parent
67084a5caf
commit
13e4f37201
8 files changed
+848
-26
No files matched your search
+404
-2
@@ -1,8 +1,270 @@
|
||||
# Test audit 2026-09: evidence
|
||||
|
||||
Evidence for the test-reduction branch (plan approved via /autoplan). Sections are added by the commit they support.
|
||||
Evidence for the test-reduction branch (plan approved through /autoplan: "A, approve as-is"; UC1 resolved as
|
||||
delete). Base: 65bfb0c (v1.91.6.0). Wherever the plan asks for a PR-body table or verify item, it resolves here.
|
||||
|
||||
## B8 pre-spend estimate (recorded 2026-09-29, before any B8 paid run)
|
||||
## Commits
|
||||
|
||||
| Commit | Workstream |
|
||||
|---|---|
|
||||
| G | Test-infrastructure dead code |
|
||||
| F | Product tests that fake the product → real-boundary tests |
|
||||
| A | Tests of dead eval code (reachability-driven) |
|
||||
| B-cleanup | Paid lane cleanup (B1–B4, B6, B7) |
|
||||
| C | Retire the never-green finding-count cluster, trim its helpers |
|
||||
| D | Fold per-incident series into detector owners |
|
||||
| H | Startup readiness marker for the plan-count history PTY |
|
||||
| E | Derived touchfile closure invariant (behavior change) |
|
||||
| B5 | Tier-lane skip and census judges (behavior change) |
|
||||
| B8 | Default capture model for eleven paid evals (behavior change) |
|
||||
| Guard | Ratchet, shared resolver, CONTRIBUTING, TODOS, portfolio, this doc |
|
||||
| Release | Durations refresh, CHANGELOG, VERSION, docs sweep |
|
||||
|
||||
## C0 triage (recorded before commit G)
|
||||
|
||||
Recorded 2026-09-29 before commit G. Sources: weekly Periodic Evals runs
|
||||
34812905093 (09-14, sha per run), 35567915613 (09-21, a6b3a575), 36385945043 (09-28, 65bfb0c-era).
|
||||
Preflight: `gh` authenticated (git.capy.ai proxy, account garrytan); `gh run download` 401s but the
|
||||
REST `actions/artifacts/<id>/zip` route works; all three runs' artifacts are retained (not expired).
|
||||
Per-file evidence: 09-28 = uploaded per-shard PTY artifacts (observation.json + terminal logs);
|
||||
09-14/09-21 = per-shard failure tail in the eval-slices job log (bun output + observation dump +
|
||||
last-3KB terminal evidence). Local copies: audit workspace: c0/.
|
||||
|
||||
Classes: product = the skill did not ask per finding; harness = PTY/classifier/timeout/launch;
|
||||
budget = live model still progressing when the deadline hit. Agreement rule applied: harness and
|
||||
budget are both non-product classes; a file is deleted when every artifact is harness or budget
|
||||
(no artifact shows product), kept+excluded otherwise.
|
||||
|
||||
| File | 09-14 | 09-21 | 09-28 | Class | Action |
|
||||
|---|---|---|---|---|---|
|
||||
| skill-e2e-autoplan-chain | harness: session exited in 13 s, no phase marker (launch) | harness: observer saw only the phase-3 marker after context compaction; transcript references CEO, Design and DX methodology files and "Phase 3 complete" | budget: timed out after ordered phase-1, 2, 2.5 hits (2 attempts, 160 min) | harness/budget | delete |
|
||||
| skill-e2e-plan-ceo-finding-count | harness: 5-finding case exited in 11 s; paired case timeout | harness: model asked 8 finding decisions (SQL lookup, Email errors, Orders load, Tests, Sequencing, TODO…), classifier labelled all preReview → `no_review_questions` | harness: classifier threw "Unsupported current CEO decision; cannot exclude it from the 4–7 count" (paired case reached plan_ready, review=2) | harness | delete |
|
||||
| skill-e2e-plan-eng-finding-count | harness: per-finding question D3 rendered; fingerprints are spinner garbage, step0=4 review=0 | harness: retired legacy oracle "mandatory legacy regression coverage absent" with reviewCount=9, then shard wall timeout | harness: model asked D1–D8/D9 per-finding decisions, all labelled preReview → deadline with review=0 | harness | delete |
|
||||
| skill-e2e-plan-design-finding-count | harness: exited in 1.7 s | harness: BAND FAIL above ceiling (review=8 > 7 for 5 findings; asked per finding plus extras) | budget: timeout at review=4 and review=5, still asking | harness/budget | delete |
|
||||
| skill-e2e-plan-devex-finding-count | harness: exited in 2.1 s | harness: seed classifier missed `missing-quickstart` although reviewCount=9 and outcome=plan_ready; retry hit shard wall timeout | pass (plan_ready, review=6) | harness | delete |
|
||||
|
||||
No artifact shows a product failure, so C3's issue is not opened; C0-kept TODO entry not needed.
|
||||
|
||||
## Security mapping (F)
|
||||
|
||||
| design/test/serve.test.ts mirror 'path traversal protection' (5) | design/test/serve.test.ts real serve() reload confinement | removing startsWith(allowedDir) guard in design/src/serve.ts → test 1 fails |
|
||||
| browse/test/terminal-agent-internal-handler.test.ts 1–3 (internalHandler/route source greps; auth gate for grant+revoke) | browse/test/terminal-agent-integration.test.ts "/internal/grant and /internal/revoke bearer auth" (no/wrong/valid × grant/revoke + state effect) | revoke route rewritten without internalHandler (no bearer check) → "revoke: no token…" and "unauthenticated revoke…" fail |
|
||||
| server-security-surface "/health carries no security field and server.ts does not import getStatus" (#2557) | extension-token "GET /health is liveness-only" (real /health body, default + headed/pinned-origin) | injecting `security: 'protected'` into the /health body → 2 fail |
|
||||
| server-security-surface "security.ts no longer exports the unfed status surface" | same /health body check: the only consumer of getStatus was /health.security; an unused export has no user-visible effect | (covered by the row above) |
|
||||
| server-security-surface "the sidepanel shield markup is gone" | same /health body check: the shield's only data source was /health.security, now asserted absent | (covered by the row above) |
|
||||
| server-security-surface:60-66 "server.ts still consumes the sidecar on the inject-scan path" (ENG-OV9) | pty-inject-scan "/pty-inject-scan — L4 sidecar verdict drives the response" (real buildFetchHandler, sidecar client mocked in a child bun test) | replacing `if (sidecarAvail.available && verdict !== 'BLOCK')` with `if (false)` in server.ts → fails |
|
||||
| server-security-surface:68-76 "security.ts keeps the pure combiner + canary exports" | browse/test/security.test.ts imports and exercises THRESHOLDS, combineVerdict, generateCanary, injectCanary, checkCanaryInStructure, extractDomain | un-exporting injectCanary → SyntaxError "Export named 'injectCanary' not found", security.test.ts fails |
|
||||
| server-security-surface "/health stays liveness-only: no token in any mode" | extension-token "GET /health never carries a token (IRON RULE)" (3, existing) + liveness-only test | injecting `token: authToken` → 5 fail |
|
||||
| server-auth "/health never serves a token — no headed-mode or chrome-extension carve-out" | extension-token IRON RULE tests (headed, pinned Origin, both) | injecting `token: authToken` → 5 fail |
|
||||
| server-auth "/health does not expose currentUrl or currentMessage"; security-audit-r2 "/health endpoint security" (2) | extension-token "GET /health is liveness-only" | injecting `currentUrl: 'x'` → 2 fail |
|
||||
| sidebar-tabs "/health no longer surfaces agentStatus or messageQueue length" | extension-token "GET /health is liveness-only" (also asserts terminalPort survives) | injecting `agentStatus: 'idle'` → 2 fail |
|
||||
| security-audit-r2 "Task 1: validateOutputPath uses realpathSync" source greps (4) + behavioral (5) | browse/test/path-validation.test.ts "validateOutputPath — symlink resolution" + "validateOutputPath" allow/deny cases (now importing path-security directly) | replacing both realpathSync resolutions in validateOutputPath with the unresolved path → symlink cases fail |
|
||||
| security-audit-r2 "results.push is present in the loop block"; "viewport case uses rawW/rawH" (identifier greps, not security contracts) | kept siblings: "validateOutputPath appears before page.screenshot() in the loop", "viewport case clamps width and height" | n/a — identifier names only |
|
||||
| test/skill-e2e-brain-privacy-gate.test.ts (paid, never green): privacy question fires once before any artifacts egress | test/gstack-skill-start.test.ts 'artifacts-sync consent is asked before any artifacts egress, and only in interactive sessions' | dropping the sync-mode gate on the daily pull → fails (pull stamp written with consent pending); dropping the interactive-only condition → fails (spawned session gets the gate) |
|
||||
|
||||
## Mixed-file and consolidation inventory
|
||||
|
||||
### F (product tests that fake the product)
|
||||
| File | Block | Decision |
|
||||
|---|---|---|
|
||||
| design/test/serve.test.ts | whole file (16 tests against an inline mirror server) | delete; replaced in place by 2 tests driving the real serve() on an ephemeral port |
|
||||
| test/gbrain-init-rollback.test.ts | 3 tests running a drifted local bash copy | delete; rollback contract moved to test/gbrain-init-voyage-code-3.test.ts executing the template-extracted blocks |
|
||||
| test/gbrain-init-voyage-code-3.test.ts | local-copy voyage cases (4) | move: now execute each template init block (3 sites) |
|
||||
| test/gbrain-init-voyage-code-3.test.ts | "demonstrates the #1798 collision" | delete (tests zsh itself) |
|
||||
| test/gbrain-init-voyage-code-3.test.ts | template-grep count tests (3) | keep (merged into one "template alignment" test) |
|
||||
| browse/test/browser-manager-unit.test.ts | "signature accepts an optional exitCode argument", "server.ts callback forwards exitCode…" | delete (tautologies); real owner: server-factory "buildFetchHandler chains cfgBrowserManager.onDisconnect" |
|
||||
| browse/test/memory-command.test.ts | "12. text mode renders modificationHistory with evicted-count when > 0" | delete (compares two local literals); gap: evicted-count suffix untested at owner |
|
||||
| test/ios-qa-swiftui-tap-regression.test.ts (+2 fixtures, 98 KB) | whole file | delete |
|
||||
| test/memory-ingest-no-put_page.test.ts | whole file | delete; gstack-memory-ingest.test.ts fake gbrain exits 99 on put/put_page |
|
||||
| browse/test/terminal-agent-internal-handler.test.ts | tests 1–3 | delete; replaced by terminal-agent-integration "/internal/grant and /internal/revoke bearer auth" (3×2 + state effect) |
|
||||
| browse/test/terminal-agent-detach-reattach.test.ts | tests 1, 4, 5, 6 | delete (dup of terminal-agent-ring-buffer-runtime) |
|
||||
| browse/test/terminal-agent-detach-reattach.test.ts | tests 2, 3, 7–10 | keep |
|
||||
| browse/test/server-security-surface.test.ts | all 6 | delete; see security mapping |
|
||||
| browse/test/server-auth.test.ts | "/health never serves a token — no headed-mode or chrome-extension carve-out", "/health does not expose currentUrl or currentMessage" | move → extension-token "GET /health is liveness-only" + IRON RULE |
|
||||
| browse/test/security-audit-r2.test.ts | "/health endpoint security" (2) | move → extension-token "GET /health is liveness-only" |
|
||||
| browse/test/security-audit-r2.test.ts | Task 1 block (4 source + 5 behavioral), "results.push is present…", "viewport case uses rawW/rawH…", AGENT_SRC | delete; path-validation owns validateOutputPath |
|
||||
| browse/test/security-audit-r2.test.ts | escapeRegExp behavioral test | keep, imports path-security directly (meta-commands re-export deleted) |
|
||||
| browse/test/security-audit-r2.test.ts | state-load, inbox, responsive, CSS validator ordering greps | keep (only guard) |
|
||||
| browse/test/sidebar-tabs.test.ts | "/health no longer surfaces agentStatus or messageQueue length" | move → extension-token liveness-only (also asserts terminalPort) |
|
||||
| browse/test/sidebar-tabs.test.ts | "browse/src/sidebar-agent.ts is gone", "sidebar-agent test files are gone" | delete |
|
||||
| browse/test/sidebar-ux.test.ts | "stop button style exists", "stop button uses error color", "experimental-banner no longer uses amber…", "tool description uses system font not mono" | delete + dead CSS (.stop-btn, .experimental-banner, .agent-tool, .agent-reasoning; 67 lines) |
|
||||
| browse/test/sidebar-ux.test.ts | "switchTab has bringToFront option" (dup of :50), "shutdown kills the terminal-agent via identity-based kill" (dup of terminal-agent-pid-identity), "quick actions toolbar has cookies button" (dup of sidebar-tabs quick-actions) | delete |
|
||||
| test/skill-validation.test.ts | "Generated SKILL.md freshness" (3) | delete (C14); gen-skill-docs placeholder regex widened to \w+ |
|
||||
| test/gen-skill-docs.test.ts | "generated header is present in SKILL.md", "…in browse/SKILL.md" | delete (C14); "every skill has a generated SKILL.md with auto-generated header" covers both |
|
||||
| test/post-rename-doc-regen.test.ts | "top-level SKILL.md exists and is regenerated" | delete (C15) |
|
||||
| test/static-no-legacy-writes.test.ts | "office-hours/SKILL.md uses --log-session, not raw echo append" | delete (C15); .tmpl sibling + freshness |
|
||||
| make-pdf/test/coverage-gaps.test.ts | all 19 cases | move → diagram-prepass.test.ts (18) and render.test.ts (screenCss) |
|
||||
|
||||
### A (dead eval code)
|
||||
Reachability tool: audit tool reach.ts (ts-morph; roots = every non-helper file importing
|
||||
test/helpers + bin/gstack-model-benchmark + the outside-voice shim; edges = identifier → top-level helper
|
||||
declaration; BFS to a fixed point; --prune removes unreached declarations and unused imports, rerun until 0).
|
||||
Baseline at 65bfb0c: 12 dead declarations = the 10 A4 names + `execGit` (auq-sdk-capture) + `invokeAndObserve`
|
||||
(claude-pty-runner). After A's test deletions: 33 dead (eng-seeded-coverage oracle closure 2,626 lines,
|
||||
autoplan-artifact-permission approvers 474, matchesAutoplanDigestRows 86, the 12 above); pass 2 → 0.
|
||||
|
||||
| File | Block | Decision |
|
||||
|---|---|---|
|
||||
| 25 A1 pure files + eng-native-seed-contract.test.ts | all | delete (all 61 native-seed-contract tests call evaluateEngSeedCoverage; its 3 blocks with live pty-runner asserts replay the plan-eng-finding-count callback; hasNativePlanTerminal / isQuestionlessNativePlanExit / classifyPlanCountFrame keep owners plan-count-pending-exit, plan-count-empty-review, plan-count-completion) |
|
||||
| eng-count-ad-v2 | "first attempt … D9 handoff", "prior successful plan Write…", "closed handoff…", "new task references…", "conditional closure…" | delete (isEngCompletionHandoff) |
|
||||
| eng-count-ad-v2 | "new first-finding and handoff paths…" | keep first-finding half (engFirstReviewAUQ); handoff half deleted |
|
||||
| eng-count-ad-v2 | census() | keep, dead handoff predicate argument removed (retry census unchanged: administrative 0) |
|
||||
| eng-count-ad-v2 | 6 live + touchfile test | keep (touchfile test loses the eng-completion-handoff path line) |
|
||||
| eng-resolution-block-position | tests 1–3 | delete (handoff / seed oracle) |
|
||||
| eng-resolution-block-position | "saved native Header and Options…" (createEngBatchingIssueCounter) | keep |
|
||||
| eng-seeded-completion-ai | "complete native navigation preserves conflicting current states…" | delete (handoff) |
|
||||
| eng-task-pause-navigation-f359 | all check()/handoff tests | delete |
|
||||
| eng-task-pause-navigation-f359 | "handoff alone never supplies a native terminal…" | keep (hasNativePlanTerminal); admin set now the completed call's signature |
|
||||
| eng-next-handoff-ah | 18 handoff tests + parser-ACK replay | delete |
|
||||
| eng-next-handoff-ah | "exact final exit/report replay…", "actual pending ExitPlanMode…" | keep (hasNativePlanTerminal, isCurrentPlanApprovalScreen) |
|
||||
| eng-published-navigation | ~150 handoff checks | delete |
|
||||
| eng-published-navigation | retry, real-completed, D19, investigation, cf74 terminal replays | keep (hasNativePlanTerminal); dead handoff/phase asserts inside removed |
|
||||
| eng-seeded-coverage.test | 23 oracle blocks + 6 describes built on evaluateEngSeedCoverage/isEngSeedDecisionAUQ | delete |
|
||||
| eng-seeded-coverage.test | "Eng semantic native evidence boundary", touchfile test, "batching caller counts…" | keep |
|
||||
| autoplan-edit-digests-al / clipped-suffix-aq / pending-artifact | approver cases (8/8/8) | delete |
|
||||
| same three | recorder/launcher cases | keep |
|
||||
| 11 A2 replay files | all | delete |
|
||||
| autoplan-permission-viewport | 27 tests (autoplan-phase-order + pty-current-screen) | delete; captured settings-overwrite card assertion moved to claude-pty-runner.unit "isPermissionDialogVisible" |
|
||||
| autoplan-phase-observation | 44 tests (all via phase-order helpers) | delete |
|
||||
| eng-finding-fixture.test | seeder + legacy-auth fixture tests (5) | delete |
|
||||
| eng-finding-fixture.test | 2 prompt-builder pins of the paid eng-finding-count file | keep until C (plan said "four prompt-builder tests"; only 2 are) |
|
||||
| ceo-paired-payment-fixture, design-ui-scope, plan-skill-completion, pty-current-screen, required-reads, transcript-section-logger tests | all | delete |
|
||||
| plan-count-fixture | 3 design-ui-captured cases + captured-question fake plumbing | delete |
|
||||
| autoplan-phase-handoff | readPlanSkillCompletion assertion in "captured parent text…" | delete line; test kept |
|
||||
| plan-seed-submission | PtyCurrentScreen decoder | swap to production createPtyScreen (58/58 pass) |
|
||||
| touchfiles.test | plan-skill-completion path in "native completion changes select the Design UI gate" | removed from the each-list |
|
||||
|
||||
### B-cleanup (B1–B4, B6, B7)
|
||||
| File | Block | Decision |
|
||||
|---|---|---|
|
||||
| skill-llm-eval-spec, skill-e2e-spec-execute, gemini-e2e (+ gemini-session-runner + test), skill-e2e-ship-idempotency, 2 overlay opus-4-7 *-sonnet wrappers (+ fixture entries), skill-e2e-conductor-prose, codex-e2e-plan-format, skill-e2e-brain-privacy-gate | all | delete (B1/B7) |
|
||||
| conductor-prose-observation-ao.test.ts + fixture | all (evaluates the deleted paid caller's source) | delete with its paid file |
|
||||
| plan-tune-cathedral-fixture.test.ts | all (evaluates the cathedral file's source under injected fakes) | delete — the cathedral scenarios now run directly in the free suite (B3) |
|
||||
| skill-llm-eval.test.ts | "regression vs baseline" | delete (B2) |
|
||||
| skill-llm-eval.test.ts | "command reference table", "snapshot flags reference", "browse/SKILL.md reference" | collapse → one union judge "browse/SKILL.md reference" (B2) |
|
||||
| skill-llm-eval.test.ts | "baseline score pinning" | fold into the union judge (pins eval-baselines.json browse_skill) |
|
||||
| skill-e2e-opus-47.test.ts | 3 negative routing controls | move → skill-routing-e2e "journey-negatives" (same ≤1-of-3 bound); positives already in skill-routing-e2e |
|
||||
| skill-e2e-ios.test.ts | "ios-qa E2E (with device)" HAS_DEVICE stub | delete (B3) |
|
||||
| gstack-skill-start.test.ts | new "artifacts-sync consent is asked before any artifacts egress…" | add (B7: existing pins did not assert ordering) |
|
||||
| paid census literals (paid-retry-supervision, paid-overlay-scheduling, overlay-lifecycle, overlay-measurement, paid-shards, touchfiles, periodic-fixture-selection, codex-eval-selection, paid-pr-profile) | counts / key lists | updated for the removed files and keys (no assertion removed except ones naming deleted keys) |
|
||||
|
||||
### C (retire finding-count cluster, C2 helper trim)
|
||||
|
||||
Rule: a free test block is deleted when every assertion subject is outside the post-C live closure (the pruned helpers, or a
|
||||
deleted paid file loaded through a registration adapter); a block that only uses dead code as an *input builder* for a live
|
||||
subject is kept and the builder is replaced or restored (rows below). LIVE blocks are kept. Touchfile self-assertions lose
|
||||
only the removed keys (E deletes them).
|
||||
|
||||
| File | Block | Decision |
|
||||
|---|---|---|
|
||||
| test/skill-e2e-autoplan-chain.test.ts | whole file | delete (C1; C0 class harness/budget, see triage) |
|
||||
| test/skill-e2e-plan-ceo-finding-count.test.ts | whole file | delete (C1; C0 class harness/budget, see triage) |
|
||||
| test/skill-e2e-plan-design-finding-count.test.ts | whole file | delete (C1; C0 class harness/budget, see triage) |
|
||||
| test/skill-e2e-plan-devex-finding-count.test.ts | whole file | delete (C1; C0 class harness/budget, see triage) |
|
||||
| test/skill-e2e-plan-eng-finding-count.test.ts | whole file | delete (C1; C0 class harness/budget, see triage) |
|
||||
| 84 free test files (list in commit) | whole file | delete: every block exercised only pruned helpers or deleted paid files |
|
||||
| test/autoplan-eval-budget.test.ts | whole file (AUTOPLAN_CHAIN_BUDGET, dedicated slice) | delete; timer-safe/explicit-override checks moved → eng-finding-retry-budget 'ordinary tiers and registered allocations remain unchanged' |
|
||||
| test/plan-review-native-default.test.ts | 3 tests (omitted multiSelect default) | move → plan-review-decisions 'an omitted native multiSelect receives the false default only in evaluator input' (removal-checked) |
|
||||
| test/autoplan-chain-fixture.test.ts | 'native sequencing config reaches the real CLI reader…' | move → plan-count-fixture.test.ts; other 3 tests delete (chain source pins) |
|
||||
| test/eng-finding-fixture.test.ts, test/design-finding-fixture.test.ts | whole file | delete (read/import the deleted paid files) |
|
||||
| test/ceo-current-decision-record.test.ts (PROD-TOUCH) | all 28 | delete: reads plan-ceo-review template only as input to the retired ceo-payment-findings counter |
|
||||
| test/devex-finding-fixture.test.ts | DX registration (8) + materialized devex-existing-sdk checks (5) | delete (fixture consumed only by the deleted DX count eval); keep 'every host exposes the DX per-call rule…' |
|
||||
| test/ceo-finding-fixture.test.ts | 'native count registration: %s' (11) | delete (imports the deleted paid file); fixture tests keep |
|
||||
| test/eng-semantic-terminal.test.ts | evaluateEngTerminalReview/buildEngSeedDecisionInput blocks (6), registration loops (7) | delete; 'real native Exit…' and 'a late substantive answer…' keep with a direct id callback in place of the dead assessor |
|
||||
| test/eng-seeded-coverage.test.ts | 'Eng semantic native evidence boundary' describe, 2 mixed, touchfile test | delete (buildEngSeedDecisionInput dead; validator owned by plan-review-decisions) |
|
||||
| test/plan-count-fixture.test.ts (PROD-TOUCH) | real PTY children worker | keep; dead design/devex predicates replaced by inline caller policies; dead-classifier assertion removed |
|
||||
| test/plan-count-native-input.test.ts | design outside-voices cases | keep; pickDesignCountOutsideVoices replaced by inline caller policy; autoplan routing test delete |
|
||||
| test/plan-pending-question-pty.test.ts | hook PTY test | keep; autoplanSetupDecision navigation replaced by the fixed native key sequence |
|
||||
| test/helpers/claude-pty-runner.unit.test.ts | findModeOption (7), design/devex Step0 + first-review (14) | delete; 2 prompt-parser tests keep with the dead boundary assertion trimmed |
|
||||
| test/autoplan-method-read-audit.test.ts, autoplan-phase-handoff, autoplan-publication-guard, plan-count-session-cwd, autoplan-preconfigured-onboarding-ar (PROD-TOUCH) | all but chain caller pins | keep; helpers autoplan-method-read-audit.ts / autoplan-preconfigured-fixture.ts restored (they adapt the production phase-publication hook / skill-start) |
|
||||
| test/autoplan-artifact-recorder, autoplan-edit-digests-al, eng-test-plan-edit-approval | recorder tests | keep; readPendingAutoplanArtifact restored (recorder is imported by claude-pty-runner) |
|
||||
| test/carve-guards (helper) | autoplan externalTest | behavioral 'none' (chain was its only section-read proof; TODOS entry) |
|
||||
| 20 replay files (ceo-completion-handoff-m/-o, ceo-handoff-y, ceo-count-ad-v2, design-count-native-8525, …) | MIXED/DEAD blocks | delete; LIVE blocks keep (hasNativePlanTerminal admin exclusion owned by eng-published-navigation / eng-next-handoff-ah) |
|
||||
|
||||
Helpers deleted (11): autoplan-setup-question, ceo-approach-pick, ceo-completion-handoff, ceo-payment-findings,
|
||||
design-artifact-question, design-count-fixture, design-count-outside, design-count-review, devex-count-fixture,
|
||||
devex-seed-coverage, eng-count-question-policy. claude-pty-runner and eng-seeded-coverage trimmed to the paid-root closure.
|
||||
135 fixtures orphaned by these deletions removed (orphans.py diff against fe011e0), plus test/fixtures/devex-existing-sdk/.
|
||||
Known selection effect (not a regression by the plan's definition, E derives the closure): lib/autoplan-phase-publication.ts,
|
||||
bin/gstack-decision-log, lib/gstack-decision.ts and the recorder/dx-navigation helper imports of claude-pty-runner selected
|
||||
only the retired evals and now select none until E.
|
||||
Out of C2 scope, left as is: ceo-finding-fixture seedCeoPaymentProject/pickSuppliedCeoPlanStart and test/fixtures/ceo-existing-payment
|
||||
(no surviving paid consumer; not in the C2 helper list).
|
||||
|
||||
### D (consolidate per-incident series)
|
||||
Mechanism: each incident file is folded verbatim into its detector's owner test as one `describe('<incident>')`
|
||||
block (audit tool merge-into.ts); imports are hoisted and per-incident bindings restored as local consts, so every
|
||||
case runs the identical code against the identical fixture. Dropped only: tests asserting the incident file's own
|
||||
touchfile registration (E-type; the path no longer exists). Accounting per family = owner+incidents before vs
|
||||
owner after, pass count must equal before − dropped with 0 failures (audit tool family.sh). Touchfile lists that named an
|
||||
incident now name the owner (audit tool tfreplace.py). Rows are not rewritten into value tables: a verbatim fold cannot
|
||||
drop an incident-specific control (lane-3 C7 risk note).
|
||||
|
||||
| Detector | Owner | Incident files folded (full paths) | Tests before → after (self-registration dropped) |
|
||||
|---|---|---|---|
|
||||
| hasStaleFillRaceFinding | test/ceo-section-loading-fixture.test.ts | test/sdk-columnar-af, sdk-compact-sequence-aj, sdk-order-b-ag, sdk-ordered-schedule-ar, sdk-ordering-ae, sdk-original-order-ai, sdk-reported-coordination-ar, sdk-schedule-continuation-ah, sdk-stale-table-ad-v3 (.test.ts) | 376 → 368 (8) |
|
||||
| generateModelOverlay / resolveModel | test/model-overlays.test.ts (new) | test/model-overlay-fable-5, -gpt-5.6-sol, -gpt-6-astra, -opus-4-7, -opus-4-8, -sonnet-5 | 37 → 37 (0); every overlay phrase kept |
|
||||
| coverageAuditVerdict / coverageAuditReadEvidence | test/coverage-audit-evidence.test.ts | test/coverage-audit-af, coverage-audit-aw, coverage-audit-shell-legend-at, coverage-checkbox-tail-av, coverage-diagram-legend-as, coverage-shell-display-aq (exercises coverageAuditReadEvidence) | 149 → 145 (4) |
|
||||
| autoplan phase completion | test/autoplan-phase-observer.test.ts | test/autoplan-phase-dash-ao, autoplan-with-result-au (autoplan-final-gate-ao deleted in C) | 82 → 80 (2) |
|
||||
| findNativeAutoDecision | test/native-auto-decide.test.ts | test/auto-decide-current-declaration, -explanatory-mode, -recommendation-scope, -saved-ai, -structured, -target-identity, auto-decision-state (auto-decide-fixture kept: real seeding) | 859 → 858 (1) |
|
||||
| claudeOutsideExecutions | test/outside-voice-evidence.test.ts | test/outside-background-ai, outside-voice-async | 49 → 49 (0) |
|
||||
| engStep0Boundary/engSetupAUQ/engFirstReviewAUQ | test/eng-first-review.test.ts (new) | test/eng-annotated-cache-au, eng-architecture-cache-av, eng-binding-retry-z, eng-binding-z, eng-cache-brief-am, eng-cache-owner-an, eng-cache-writes-as, eng-count-ad-v2, eng-declarative-as, eng-declared-retry-at, eng-first-category-af, eng-first-review-t, eng-injected-export-aq, eng-library-hooks-aq, eng-scope-y | 256 → 247 (9) |
|
||||
| hasNativePlanTerminal (completion/handoff) | test/plan-count-completion.test.ts | test/ceo-completion-handoff-m, ceo-completion-handoff-o, ceo-handoff-y, dx-manual-handoff-ao, plan-count-dx-handoff-o, eng-next-handoff-ah, eng-task-pause-navigation-f359, design-count-native-8525 | 129 → 129 (0) |
|
||||
| createPlanCountPermissionGuard | test/plan-count-file-permission.test.ts | test/batching-permission-at, design-crop-gutter-ap, plan-count-crop-ak, plan-count-permission-ac, plan-count-quoted-frame-ak | 125 → 121 (4) |
|
||||
| ceo-mode-option | test/ceo-mode-option.test.ts | test/ceo-hold-commitment-ar, ceo-hold-posture-ag, ceo-mode-colon-at, ceo-mode-full-ad, ceo-mode-posture-ad, ceo-prerequisite-ad-v2 | 463 → 457 (6) |
|
||||
| plan-scope-selection | test/plan-scope-selection.test.ts | test/design-scope-announcement-ao, design-scope-declaration-ak, design-scope-entry-aq, design-scope-selection-aj, eng-option-b-scope-al, plan-scope-recovery-av | 90 → 84 (6) |
|
||||
| planCountPrerequisitePick | test/plan-count-prerequisite.test.ts (renamed from -n) | test/plan-count-navigation-r, plan-count-prerequisite-n | 38 → 37 (1) |
|
||||
|
||||
Native-completion negative table: after C it survives in 3 files (14 per-incident copies in eng-first-review,
|
||||
2 in plan-count-completion, 1 in dx-selected-navigation-ap), each applied to a different captured call and a
|
||||
different engFirstReviewAUQ branch. Collapsing them to one table is only sound after engFirstReviewAUQ checks
|
||||
native completion once at entry (each branch gates it separately today, claude-pty-runner.ts engFirstReviewAUQ);
|
||||
that is a harness behavior change on a paid verdict, so it is deferred (kept-vs-plan) rather than done here.
|
||||
|
||||
### E (derived touchfile closure)
|
||||
|
||||
Selection regression definition (used by the E proof and the drop rule): a sample edit's `--tier gate --profile pr --list`
|
||||
output after the change is missing a paid case that the before-run selected through any path other than a free `*.test.ts`
|
||||
touchfile entry. Proof computed with computePaidCaseSelection (the function `--list` calls) at 689ef30 vs the E tree:
|
||||
|
||||
| Sample edit | Profile | e2e before → after | judges before → after | lost | gained |
|
||||
|---|---|---|---|---|---|
|
||||
| plan-eng-review/SKILL.md.tmpl | pr | 2 → 2 | 1 → 1 | none | none |
|
||||
| plan-eng-review/SKILL.md.tmpl | full | 28 → 28 | 1 → 1 | none | none |
|
||||
| test/helpers/claude-pty-runner.ts | pr | 0 → 1 | 0 → 0 | none | auq-format-gate |
|
||||
| test/helpers/claude-pty-runner.ts | full | 15 → 20 | 0 → 0 | none | auq-format-gate, carve-section-loading, office-hours-section-loading, plan-ceo-section-loading, ship-section-loading |
|
||||
| test/helpers/plan-count-fixture.ts | pr | 0 → 1 | 0 → 0 | none | auq-format-gate |
|
||||
| test/helpers/plan-count-fixture.ts | full | 10 → 20 | 0 → 0 | none | auq-format-gate, carve-section-loading, office-hours-auto-mode, office-hours-section-loading, plan-ceo-section-loading, plan-design-review-plan-mode, plan-devex-review-plan-mode, plan-eng-review-plan-mode, plan-mode-no-op, ship-section-loading |
|
||||
| bin/gstack-config | pr | 3 → 3 | 0 → 0 | none | none |
|
||||
| bin/gstack-config | full | 9 → 9 | 0 → 0 | none | none |
|
||||
| test/fixtures/plans/autoplan-dashboard.md | pr | 86 → 86 | 23 → 23 | none | none |
|
||||
| test/fixtures/plans/autoplan-dashboard.md | full | 0 → 0 | 0 → 0 | none | none |
|
||||
|
||||
Rewrite: 950 free `*.test.ts` entries removed from E2E/LLM-judge lists; 653 closure paths added (53 distinct helpers/fixtures
|
||||
across 123 keys), all real static imports or literal fixture paths of the key's paid file. Closure traversal stops at
|
||||
GLOBAL_TOUCHFILES modules (an edit there already selects everything) and ignores the selection modules themselves
|
||||
(touchfiles-data/touchfiles/test-selection, map-diffed). Keyless paid files (asserted): codex-e2e-recommendation-substance
|
||||
(census-only, PERIODIC_CI_EXCLUDE), skill-e2e-auq-consistency and skill-e2e-auq-verbose-vs-carved-ab (periodic tier gate only).
|
||||
Deleted: test/periodic-fixture-selection.test.ts (hand-copied inventory), test/fake-impeccable-touchfiles.test.ts,
|
||||
45 per-file selection examples (self-registration / literal selectTests of test/ paths) in 41 files; two emptied files
|
||||
(autoplan-clipped-suffix-aq, codex-eval-selection) and their orphan fixture. Trimmed to non-test paths: 8 tests
|
||||
(CSO each, mode-question capture, live runtime each, mode input each, autoplan-review-discovery, autoplan-snapshot,
|
||||
review-entry-and-design-clarity-au, shared-libs-fixture generation, devex calibration, cookie judge helper exactness).
|
||||
Kept selection-semantics tests (ES-1): matchGlob suite, global touchfile, skill-specific, resolver→consumer equivalence,
|
||||
testing resolver, learnings rendering, browse/aside, gen-skill-docs scoped, unrelated/empty/union, LLM judge, SKILL root,
|
||||
completeness, tiers, dependency-path existence, reverse invariant; eval-cli-family, skill-fixture global, workflow-boundaries F9.
|
||||
Removal check: replacing test/fixtures/fake-impeccable.ts in one key makes the invariant print the paid file, the path, the
|
||||
import/literal chain, the key, the verify command and CONTRIBUTING.md#paid-test-touchfiles plus the lower-bound note.
|
||||
|
||||
## Behavior-changing commits: kept or dropped
|
||||
|
||||
| Commit | Measurement | Decision |
|
||||
|---|---|---|
|
||||
| E | Selection proof above: no lost case for the four sample edits under either profile; growth only from real static dependencies | kept |
|
||||
| B5 | Gate lane 52 → 42 files, weekly gate census 52 → 41 (judges skipped), periodic 77 → 69; PR-profile selection for the sample edits byte-identical before and after | kept |
|
||||
| B8 | Pre-spend estimate net −$3.14/week (below); paid run result below | see paid validation |
|
||||
|
||||
### B8 pre-spend estimate (recorded 2026-09-29, before any B8 paid run)
|
||||
|
||||
Source: latest weekly periodic artifacts (runs 36385945043 = 09-28, 35567915613 = 09-21), per-shard eval JSON cost_usd.
|
||||
Price ratio from test/helpers/pricing.ts: claude-fable-5-1 (default capture, lib/eval-model.ts) $10/$50 per MTok in/out;
|
||||
@@ -21,3 +283,143 @@ if its estimated net weekly dollars after C and B5 savings are above zero. Estim
|
||||
(five retired evals) − B5 savings (18 hollow shards, 23 census judges) < 0 → B8 proceeds to its one paid run.
|
||||
Fallback check: `git log -S claude-sonnet-4-6` on skill-e2e-office-hours and -brain-writeback shows only 636175d / #2264
|
||||
(infra hardening), no cost rationale → both re-pinned.
|
||||
|
||||
## Paid validation and fallbacks
|
||||
|
||||
- B2 union judge "browse/SKILL.md reference": PASS (clarity 4, completeness 4, actionability 4), $0.02. Fallback not
|
||||
taken; the three original browse judges are deleted.
|
||||
- B6 folded journey negatives in `skill-routing-e2e`: 3/3 unrouted, $0.36. Fallback not taken; `skill-e2e-opus-47` deleted.
|
||||
- B8 re-pin run and the full gate census: recorded in the release commit.
|
||||
|
||||
## Before metrics (65bfb0c)
|
||||
|
||||
- `bun run test:ubicloud --record-durations` (standard-16, 2026-09-29 04:34Z): EXIT 0, 1065 files one-per-shard,
|
||||
wall 142 s; recorded serial sum 1,888.2 s (committed durations file at 65bfb0c: see release commit diff).
|
||||
Raw copy: audit workspace: metrics/before-durations.json; log audit workspace: ubi-before.log
|
||||
- File/LOC counts: audit workspace: metrics/before-counts.txt
|
||||
- Paid --list: before-gate-list.txt (gate 58/119 files, 216 tests selected), before-periodic-list.txt (periodic 100/119)
|
||||
tracked test files: 1187
|
||||
under test/: 982
|
||||
test/ LOC (ts): 274208
|
||||
test/helpers LOC: 51390
|
||||
test/fixtures bytes: 16289330 total
|
||||
all test-file LOC: 279898
|
||||
- free tests: 27,331 passed, 0 failed (1065 shards)
|
||||
|
||||
## Retained false positives (lane reports §4)
|
||||
|
||||
### Lane 1
|
||||
|
||||
- `skill-e2e-hermetic-canary.test.ts` — paid test of test infrastructure, but it is the only falsifiable proof the
|
||||
child env/auth/config is hermetic ($0.02, 5–8s, gate + PR profile). Keep.
|
||||
- `paid-*.test.ts` (8 files, 1,948 LOC, ~3.5s free) — they test `scripts/test-paid-shards.ts` (1,866 LOC) and
|
||||
`test-pr-profile.ts`, the real paid runner. Legit tooling tests. Minor smell only: `paid-free-boundary.test.ts`
|
||||
pins a sha256 of `test/helpers/test-selection.ts` and has incident-named tests (`as at 06ed920`, `PR 2956`).
|
||||
- `llm-judge-abort.test.ts`, `llm-judge-frontier.test.ts` (287 LOC, 74ms) — unit tests of the shared judge client
|
||||
every judge uses. Keep.
|
||||
- `llm-judge-recommendation.test.ts` — fixture-based negative coverage for `judgeRecommendation` (~$0.04). Keep.
|
||||
- `make-pdf/test/e2e/*` — run in the Linux free suite and again in `make-pdf-gate.yml` on macOS: different
|
||||
platform, so not a duplicate. `ci-prereqs.test.ts` is the anti-silent-skip tripwire. Keep.
|
||||
- Carve / overlay per-case wrappers — see C4. Keep.
|
||||
- `overlay-harness-claude-dedicated-tools-vs-bash-sonnet` — applies `claude.md` to Sonnet 4.6, a pairing production
|
||||
does render. Keep (unlike D5).
|
||||
- `skill-e2e-office-hours` posture judges, `skill-e2e-benchmark-providers` ($0.001) — quality benchmarks that
|
||||
CLAUDE.md explicitly classifies periodic. Keep.
|
||||
- `codex-e2e-sol-scope.test.ts` — never runs in CI, but it pins the current `gpt-5.6-sol` overlay behavior;
|
||||
move to manual lane (C2), do not delete.
|
||||
### Lane 2
|
||||
|
||||
- `autoplan-overwrite-progress-ax` (70 LOC): tests `autoplanPermissionProgressKey`, used live at `skill-e2e-autoplan-chain.test.ts:205`. KEEP (could merge into a recorder/progress test).
|
||||
- `autoplan-artifact-recorder.test.ts` (7.5 s): owner of the live hook approval. KEEP; it's the proof that makes candidate 1 safe.
|
||||
- `auq-format-always-loaded`: greps generated SKILL.md for the AskUserQuestion format and per-skill cadence rules. This is a prompt-byte contract (retention bar). KEEP.
|
||||
- `auq-error-fallback-hook`, `autoplan-publication-guard/-hook/-generation`, `autoplan-snapshot/-init/-obligations/-methodology-names/-phase-order`,
|
||||
`outside-voice-provenance`, `outside-voice-invocation/-preflight/-routing`: exercise production hooks, bins, and resolvers. KEEP.
|
||||
- `autoplan-review-discovery` (14 s): real copy/symlink install layouts per host plus `bin/gstack-autoplan-snapshot`. KEEP. Most of the
|
||||
cost is one `gen-skill-docs --host all` into tmp, a candidate for sharing generated output across tests (perf, not deletion).
|
||||
- `auq-parallel` (17.4 s): tests the paid AUQ harness's concurrency, deadline, and cleanup through a mocked SDK. It's test-of-harness, but it
|
||||
guards paid-run cost and timeout behavior. KEEP; maybe reduce scenarios.
|
||||
- `carve-guard-completeness`, `carve-section-ordering`, `carve-guards-negative`: generated-output structure guards for carved skills,
|
||||
plus a negative control proving the guard fires. KEEP. `carve-section-sharding` and `autoplan-eval-budget` test paid-runner
|
||||
scheduling; they could MOVE next to the `test-paid-shards` tests but are fine as is.
|
||||
- `outside-voice-fixture`, `carve-plan-fixture`, `autoplan-chain-fixture`: tests of fixture builders used by paid evals. They're cheap and guard
|
||||
paid-run validity. KEEP (low priority).
|
||||
- Not audited in depth: `autoplan-amend-input`, `autoplan-method-read-audit`, `autoplan-phase-handoff`, `autoplan-dual-voice-*`,
|
||||
`autoplan-owned-state`, `autoplan-preconfigured-onboarding-ar`, `autoplan-pending-question`, `auq-native-capture`,
|
||||
`batching-permission-at` (its helper `plan-count-file-permission.ts` is live in the runner; its siblings `plan-count-crop-ak`,
|
||||
`plan-count-permission-ac`, and `design-crop-gutter-ap` are outside this lane).
|
||||
### Lane 3
|
||||
|
||||
- `plan-count-transcript.test.ts` (252 LOC): looks like harness, but `helpers/plan-count-transcript.ts` is a
|
||||
re-export of `lib/claude-public-transcript.ts`, which production uses (`lib/autoplan-phase-publication.ts`,
|
||||
`autoplan/bin/phase-publication-hook.ts`). Keep; consider renaming to the lib owner.
|
||||
Same for `plan-count-session-cwd` and `plan-count-cross-cwd-ancestry` (import the lib directly).
|
||||
- `design-checklist-sync.test.ts`: generated-file drift contract (`review/design-checklist.md` from
|
||||
`lib/design-catalog.ts`), named in CLAUDE.md. Keep.
|
||||
- `design-catalog`, `design-md`, `design-detect-contract`, `design-flag-utils`, `review-log`,
|
||||
`review-start-evidence` (13.3 s, `lib/review-evidence`), `office-hours-review`, `plan-tune`,
|
||||
`ship-version-sync`, `ship-template-redaction`, `ship-test-detection-markers`, `spec-quality-gate-secret-sink`:
|
||||
test production modules/bins. Keep.
|
||||
- `ship-review-loop.test.ts` test 1 (no `**STOP** … run /ship again` across rendered hosts) is a real #2391
|
||||
regression guard; tests 2–3 are exact-sentence pins ("stay in this invocation and loop") and could be
|
||||
loosened, but prose here is the skill's instruction, so not a deletion candidate.
|
||||
- `ship-apple-gate.test.ts`: ordering assertion (Apple adapter before branch gate) is behavior in prose. Keep.
|
||||
- `spec-template-invariants`, `ship-workflow-clarity`, `ship-plan-completion-invariants`, `eng-scope-entry-ap`,
|
||||
`review-entry-and-design-clarity-au`, `design-scope-entry-aq`, `plan-scope-recovery-av`,
|
||||
`ceo-mode-preference-al` (~400 `toContain`/`toMatch` on skill prose): mixed ordering checks (keep) and
|
||||
exact-sentence pins (fragile). Needs a per-assertion pass; not a deletion batch.
|
||||
- `plan-skill-questions.test.ts` (2,439 LOC, 22.6 s, 430 tests): `helpers/plan-skill-questions.ts` is used by
|
||||
9 helpers and 2 fixture modules on the paid path; large but live. Candidate for C7-style consolidation later.
|
||||
- `claude-pty-runner.ts` exports: only 4 top-level declarations (111 LOC) unreachable from paid/helper code
|
||||
(`isTrustDialogVisible`, `findModeOption`, `PLAN_SKILL_COUNT_FINALIZE_MS`, `isUnknownSlashCommandVisible`);
|
||||
small, not pursued.
|
||||
### Lane 4
|
||||
|
||||
- **`test/setup-codex-scope*.test.ts` (5 files, 160 s, the biggest time sink in the lane):**
|
||||
they spawn the real `setup` twice per case and assert no mutation of global or foreign
|
||||
skills. These are data-loss safety contracts, and each case covers a distinct layout, alias,
|
||||
or ownership shape. One exception: the first test in `setup-codex-scope.test.ts`, "fixture
|
||||
writes reject physical escapes…", tests the fixture guard, which is test infra. AGENTS.md
|
||||
mandates that guard, so it goes to lane 5 rather than being deleted.
|
||||
- **`test/cso-cli*.test.ts` (73 s) and `cso-scanner-cli`:** each test is a distinct CLI
|
||||
contract (recheck resolution, launcher trust, redaction, deadlines). They're slow because
|
||||
they go through the compiled launcher, which is the real boundary.
|
||||
- **`gstack-memory-ingest.test.ts` (75 s):** behavioral CLI tests with a fake gbrain. Only the
|
||||
"probes the gbrain executable directly…" source grep is weak; it's a minor candidate.
|
||||
- **browse/test cookie-* cluster (14 files, ~65 s):** behavioral security tests (decryption,
|
||||
origin policy, Keychain denial, isolated Chromium auth). There's no duplication beyond the
|
||||
different layers they cover.
|
||||
- **browse xvfb (43 s), handoff (55 s), commands (36 s):** real behavior. The `expect(true)
|
||||
.toBe(false)` calls in commands.test.ts sit inside try/catch "should not reach" blocks whose
|
||||
catch asserts the error message, so they aren't tautologies.
|
||||
- **`design/test/feedback-roundtrip.test.ts`:** its server is also a mirror, but the thing
|
||||
under test is the generated board JS in a real browser. The daemon file owns the server
|
||||
contract. Suggestion: point the browser at the real daemon to remove the mirror.
|
||||
- **`setup-gbrain-path4-structure.test.ts`:** a grep of template prose, but the prose (token
|
||||
never in argv or CLAUDE.md, STOP gates) is the prompt contract itself.
|
||||
- **`terminal-agent-pid-identity` test 1 (repo-wide no `pkill -f terminal-agent`):** the
|
||||
cheapest independent guard for a cross-session kill bug.
|
||||
- **security-audit-r2 ordering greps (state load, inbox, responsive, CSS validator):** the only
|
||||
guard today. Convert them, don't delete them.
|
||||
- **sidebar-ux "welcome page has left-aligned text":** it encodes a stated user design
|
||||
preference, and it's cheap.
|
||||
### Lane 5
|
||||
|
||||
- **Meta-tests guarding real CI contracts (KEEP):**
|
||||
- `ci-image-tag-binding` (three-way hashFiles drift causes silent rebuilds)
|
||||
- `ci-image-cli-pin` (unpinned CLI broke the PTY harness 3×)
|
||||
- `workflow-concurrency` (the generic loop)
|
||||
- `free-tests-workflow-wiring` (secretless, no `pull_request_target`, least privilege)
|
||||
- `evals-workflow-wiring`, `ci-eval-cache`, `e2e-tier-alignment` (inert-demotion class)
|
||||
- `paid-orphan-tripwire`, `eval-budgets-policy`, `eval-detach-timeout-floor`, `periodic-exclude-policy`, `gate-secret-scan`
|
||||
- `strict-output*`, `test-free-shards*`, `paid-shards`/`paid-retry-supervision`/`paid-run-manifest`/`paid-selection-propagation`/`paid-overlay-scheduling`/`ci-paid-coordination` (they test the code that decides CI verdicts)
|
||||
- `touchfiles-map-diff` (real fail-closed selection logic)
|
||||
- `hermetic-wiring` (source grep that is brittle by design, retention bar)
|
||||
- `spawnsync-timeout-tripwire` and `parity-suite` (named by AGENTS.md)
|
||||
- `llm-judge-frontier` (judge parsing decides eval verdicts)
|
||||
- `secret-sink-harness.test` (negative controls run real setup-gbrain bins)
|
||||
- `paid-free-boundary`
|
||||
- **Weak literal pins worth trimming later (not candidates):** `free-tests-workflow-wiring` `max-parallel: 20` + the exact matrix string. `workflow-concurrency` hard-pins `actionlint.yml`/`skill-docs.yml`. `test-free-shards-sandbox-knobs` "Linux caps at 16". `parity-baseline-integrity` pins CHANGELOG headline numbers, a docs-consistency check rather than behavior.
|
||||
- **Paid-callback replays (26 files, for example `review-n-plus-one-contract`):** tests of tests, but AGENTS.md step 4 explicitly requires them, and they catch broken pass predicates that would otherwise waste paid runs.
|
||||
- **`test/helpers/claude-pty-runner.unit.test.ts` (3,694 LOC, 223 tests, 0.1 s):** a large self-test of the 5,885-LOC PTY harness. The classifiers it covers (`classifyVisible`, `parseNumberedOptions`, `Step0BoundaryPredicate`…) are live in paid runs, so keep it. The per-capture blocks (`captured F`, `captured G`) belong to the per-incident consolidation lane.
|
||||
- **Zero-importer helpers** `auq-parallel-worker`, `setup-gbrain-fixture-command`, `emulate-bun-windows-eexist` are loaded by path (preload / generated import). `benchmark-judge` has a production caller (dynamic import in `bin/gstack-model-benchmark`).
|
||||
- **`browse/src` `__reset*`/`reset*ForTests` exports (12):** standard singleton-reset seams, keep.
|
||||
Reference in new issue
Block a user