gstack/docs/TEST_PORTFOLIO.md

Test portfolio: coverage ownership and repeated work

Audit source: 06ed920a974809ebedc6bcbbe402fb81f5944598, September 24, 2026. This is a coverage inventory and a focused refactor, not a claim that every test assertion is interchangeable with another assertion about the same skill.

One owner for each kind of evidence

Tests can share setup or inspect the same public capture. They must not count one capture twice when the contract requires independent trials. In particular, free parser tests, a prompt-quality score, and a completed live workflow prove different things even when they mention the same skill.

Responsibility	Owner	What this evidence does not replace
Selection, budgets, process ownership, cleanup, native-event parsing and recorded failure controls	Free runner and fixture regressions	A live model choosing or completing the right workflow
Generated files, host parity, section manifests and complete input identity	Free generation and structural tests	An agent actually loading the required section
Prompt clarity and rubric quality	The registered quality judge for that complete prompt	Tool execution, native acknowledgments or a finished report
Native first-question format and substance	Live SDK/PTY question capture	An answered question or a completed workflow; receipts explicitly say workflowCompleted: false
Stochastic consistency and verbose/carved comparison	Independent captures, with separate stability and A/B oracles	One successful sample reused as three trials, or one prompt version standing in for the other
Decisions, findings and report completion	Per-skill native workflow fixtures	The first question alone, screen text without native evidence, or a generic question count
Offline deployment and canary report construction	The explicitly simulated workflow fixtures	A real GitHub merge, deployment, rollback or production health check
Multi-phase ordering and hand-offs	One uninterrupted Autoplan chain	Four independent successful skill sessions
External reviewers, other model providers, browser engines and platform behavior	Their respective live integration fixtures	Prompt parity or a mock transport

Overlay efficacy experiments retain their full fixture/model/arm/trial matrix. Security cases retain their source, path, socket, process and lease identities. These are distinct scenario dimensions, not repeated work to delete.

Complete inventory, not just the fast subset

At the audited revision, all 1,124 tracked Bun test files partition into 1,010 free files and 114 paid files. The paid inventory contains 207 registered E2E selection IDs (84 gate and 123 periodic) and 25 main quality-judge IDs. IDs, files, model calls, samples, attempts and executed Bun tests are different counts; use the run manifest and receipts rather than substituting one for another.

The full gate and periodic manifests plan 153 file processes across 112 unique files. Forty-one files appear in both manifests because they contain mixed-tier cases; that does not establish 41 duplicated scenarios. Brain privacy and ship idempotency have existing explicit exclusions, iOS needs hardware, and the spec quality file contains a TODO. The iOS fixture's native Swift smoke test is outside this Bun-file census and requires its platform runtime. Unavailable or excluded work is not coverage.

Existing sources remain authoritative:

• test/helpers/paid-test-set.ts owns the free/paid file boundary.
• test/helpers/touchfiles-data.ts owns registered dependency and tier data.
• scripts/test-pr-profile.ts owns the deliberately partial PR profile.
• test/helpers/eval-budgets.ts owns supervision and retry exceptions.
• The free and paid shard runners own inventory, scheduling and reconciliation.

test:quick and test:pr are feedback lanes, not full release acceptance. This refactor does not remove cases, shrink samples, change tiers, lower thresholds, shorten production deadlines, or move checks to a later cadence.

Repeated work removed

Synthetic terminal startup. The 42 fake-terminal scenarios in nine files emit their existing readiness marker only after installing handlers. They no longer spend the real CLI's eight-second grace waiting for an already-ready fake. Native CLI grace, terminal geometry, observation delays, scenario inputs and all existing assertions stay intact.

Publication polling. Invalid/unavailable-journal cases still call the real hook and transcript reader through all 40 polling intervals. A scoped serial fake clock removes wall-clock sleeping, while controls verify the full logical deadline, late arrival, and restoration after success and failure. The real delayed-journal and shell-transport cases still use real time.

Native watchdog setup. Compile the identical source once per test file and copy the executable into each case's isolated directory. Every watchdog still executes; only duplicate checks of the same compiler result are consolidated. Mutable work directories and cleanup ownership are never pooled.

Independent live captures. Start all three consistency captures through the existing three-query semaphore and both A/B arms concurrently. Await every settlement before cleanup, retain sibling failures, and attempt cleanup for every owned directory. Consistency judging remains sequential; A/B has at most two simultaneous judges. The free regression exercises the actual registered callbacks, native capture receipts, semaphore and judge request builder.

Runner ownership. Path normalization belongs to the existing shared strict output utility, not the free runner. The paid runner no longer imports the free runner to use it. Exact free-only exemptions cover the free runner and the free AUQ replay worker, with paid import-closure and selection controls. Mapped dependencies still win; unknown dependencies retain the broad fallback. No directory-wide exemption is introduced.

Local scheduling. Default free workers use available CPU affinity, with a floor of one and the existing cap of six. Each shard stays serial internally, and explicit GSTACK_FREE_JOBS overrides keep their previous meaning. The separate 20-machine CI plan is unchanged. Windows free-test CI explicitly retains its two-worker budget rather than inheriting the local default; local worker gains are not CI gains.

Measurement contract

Compare original and edited code on the same machine, runtime, launch environment and workload. Preserve failure and retry records. Count the full case/sample inventory and report skips and unavailable platforms separately. Do not subtract failures from elapsed time or use a smaller selection as proof that the complete suite got faster.

Measured component comparisons:

Workload	Before	After	Coverage retained
Nine synthetic-terminal files, serial aggregate	165.87s	72.98s	81 tests, 1,593 assertions, 42 PTY scenarios
Publication guard and watchdog files, serial aggregate	56.57s	9.63s	245 original tests; three additional clock controls
Two live periodic AUQ files, same machine and runtime	325.09s	136.40s	Two tests, five independent captures, all original grading rules

Exact selectors for the synthetic-terminal comparison:

test/plan-count-fixture.test.ts
test/plan-count-design-ui-recovery.test.ts
test/plan-count-native-input.test.ts
test/plan-count-empty-review.test.ts
test/plan-count-owned-permission.test.ts
test/plan-count-quoted-frame-ak.test.ts
test/plan-count-truncated-question.test.ts
test/plan-count-preview-footer.test.ts
test/eng-test-plan-edit-approval.test.ts

The publication/watchdog pair is test/autoplan-publication-guard.test.ts and test/cso-watchdog.test.ts. The live pair is test/skill-e2e-auq-consistency.test.ts and test/skill-e2e-auq-verbose-vs-carved-ab.test.ts, using the default three consistency samples plus the two A/B captures.

These are separate comparisons; do not add their percentages or call them a complete paid-suite result. The terminal aggregate is 56% faster, the two publication/watchdog files are 83% faster, and the live AUQ pair is 58% faster. Watchdog assertion counts fall only because identical compilation is checked once; all behavioral and security assertions remain.

The matched complete local free-suite comparison used the same four-CPU Linux machine, Bun 1.4.0, Node 24.18.0, built artifacts, display and isolated Git configuration. Neither run set a worker override: the original default used two workers and the changed default used four.

Complete free-suite result	Original	Performance refactor
Wall time	887.82s	413.78s
Passing cases	25,504	25,547
Failing cases	0	0
Skipped cases	60	60
Assertions	199,132	200,351
Test files	1,010	1,012

This is a 53.4% local reduction. Case-level reconciliation retained every original passing identity except the intentionally renamed worker-default policy test, added 43 cases, and retained the exact same 60 skipped identities. This comparison predates the additional fixture and deployment-workflow regression cases; final acceptance must cover those too. It is not a measurement of the separate CI matrix or the complete paid census.

The exact historical PR #2956 diff selected 84 gate IDs plus 25 judges because the free runner was treated as an unknown paid dependency. Replaying that diff after the boundary repair selects the intended 26 fast IDs plus the same 25 judges. A free-runner-only diff selects no paid work. That is routing accuracy, not a fresh full-census runtime improvement.

Evidence validity

Check the executable actually used by each SDK, print-mode and terminal launcher. A CLI version cached during preflight does not prove the version used by later sessions if PATH contents change. Use native session-init versions, terminal startup captures or process witnesses, and keep runtime controls effective after the hermetic environment is constructed.

Captured-event regressions prove the validator accepts valid evidence and rejects invalid evidence. They do not turn an old failed model run into a pass. Preserve every configured retry attempt and distinguish actual registrations from filtered or out-of-tier placeholders. A retained passing judge is reusable only when its complete request, rubric, parameters and relevant dependencies match; native behavioral acceptance follows its separate freshness contract.

Remaining work, not claimed savings

Report final acceptance with the delivery revision and its actual case inventory. Complete fresh gate/periodic and cross-platform performance comparisons are not established by the measurements above.

The longest indivisible live workflow limits the benefit of extra workers. Historical paid-duration replay suggests better scheduling alone cannot halve the full lane. A follow-up should unify executable case ownership/counts before sharing captures between judges or splitting long files: keep each oracle, scenario, retry and independent-trial requirement explicit. The ordered Autoplan chain, host integrations and security boundary cases must not be replaced with cheaper look-alikes.