gstack/docs/TEST_PORTFOLIO.md

Test portfolio: coverage ownership and repeated work

Audit source: 06ed920a974809ebedc6bcbbe402fb81f5944598, September 24, 2026. This is a coverage inventory and a focused refactor, not a claim that every test assertion is interchangeable with another assertion about the same skill.

One owner for each kind of evidence

Tests can share setup or inspect the same public capture. They must not count one capture twice when the contract requires independent trials. In particular, free parser tests, a prompt-quality score, and a completed live workflow prove different things even when they mention the same skill.

Responsibility	Owner	What this evidence does not replace
Selection, budgets, process ownership, cleanup, native-event parsing and recorded failure controls	Free runner and fixture regressions	A live model choosing or completing the right workflow
Generated files, host parity, section manifests and complete input identity	Free generation and structural tests	An agent actually loading the required section
Prompt clarity and rubric quality	The registered quality judge for that complete prompt	Tool execution, native acknowledgments or a finished report
Native first-question format and substance	Live SDK/PTY question capture	An answered question or a completed workflow; receipts explicitly say workflowCompleted: false
Stochastic consistency and verbose/carved comparison	Independent captures, with separate stability and A/B oracles	One successful sample reused as three trials, or one prompt version standing in for the other
Decisions, findings and report completion	Per-skill native workflow fixtures	The first question alone, screen text without native evidence, or a generic question count
Offline deployment and canary report construction	The explicitly simulated workflow fixtures	A real GitHub merge, deployment, rollback or production health check
Multi-phase ordering and hand-offs	The production phase-publication hook, pinned by the free test/autoplan-publication-guard.test.ts; no paid chain eval since the 2026-09 audit (TODOS: "No paid eval runs the full /autoplan chain")	A live model completing CEO → Design → DX → Eng
External reviewers, other model providers, browser engines and platform behavior	Their respective live integration fixtures	Prompt parity or a mock transport

Overlay efficacy experiments retain their full fixture/model/arm/trial matrix. Security cases retain their source, path, socket, process and lease identities. These are distinct scenario dimensions, not repeated work to delete.

Detector owner tests

A captured paid failure becomes one row (a describe block or table entry) in its detector's owner test, never a new per-incident file; test/test-of-test-ratchet.test.ts enforces this. Owners after the 2026-09 audit (evidence):

Detector	Owner test
hasStaleFillRaceFinding	test/ceo-section-loading-fixture.test.ts
generateModelOverlay / resolveModel (overlay phrases)	test/model-overlays.test.ts
coverageAuditVerdict / coverageAuditReadEvidence	test/coverage-audit-evidence.test.ts
Autoplan phase completion (autoplanPhaseCompletions)	test/autoplan-phase-observer.test.ts
findNativeAutoDecision and auto-decision state	test/native-auto-decide.test.ts
claudeOutsideExecutions	test/outside-voice-evidence.test.ts
engStep0Boundary / engSetupAUQ / engFirstReviewAUQ	test/eng-first-review.test.ts
hasNativePlanTerminal (completion and hand-off)	test/plan-count-completion.test.ts
createPlanCountPermissionGuard	test/plan-count-file-permission.test.ts
CEO mode option parsing (ceo-mode-option)	test/ceo-mode-option.test.ts
Plan scope selection (plan-scope-selection)	test/plan-scope-selection.test.ts
planCountPrerequisitePick	test/plan-count-prerequisite.test.ts

Functional QA contract map

The deterministic owners below protect the failure boundary; their live partners prove that an agent follows it. A shared fixture or captured event does not replace an independent live trial. All free owners run in bun run test; quick eligibility depends on measured duration or an explicit QUICK_CORE entry, not this table.

Contract	Deterministic owner	Necessary live boundary	Host and lane
CLI/API/webhook QA without browser setup	qa-functional-fixture, qa-functional-evidence, qa-lazy-sections	skill-e2e-qa-functional: CLI and webhook report sessions	Linux/macOS free; selected PR gate; Windows only where curated
Report-only preserves local and remote authority	qa-only-capability, qa-functional-observer, qa-functional-observer-atomic, qa-caller-authority	Independent report-only sessions with synthetic owned endpoints/auth	Linux kernel observation; free callback controls plus selected PR gate
Repair reproduces the defect, adds a failing regression and rechecks adjacent behavior	qa-fix-loop-fixture, qa-functional-evidence	skill-e2e-qa-functional-fix: CLI and webhook repair sessions	Free controls plus selected PR gate
Review and Ship actually explore	qa-exploratory-callers, qa-caller-report-observer, qa-checkpoint-evidence	skill-e2e-qa-callers: actual Review/Ship callers	Free captures plus selected PR gate
Smoke expiry preserves required plan checks	qa-deadline, qa-deadline-selection, qa-browser-deadline-evidence	ship-exploratory-plan-checks	Free deadline/dispatch controls plus selected PR gate
Late changes invalidate affected results	qa-caller-freshness-order, qa-deadline-publication-observer, shared-libs-revalidation-prompt	ship-exploratory-late-input and the existing late-input documentation handoff	Free stale-input controls plus selected PR gate
Documentation completes before publication and respects protected files	docsync-authority, docsync-atomic-writes, docsync-report-interface, docsync-lifecycle-interface	skill-e2e-ship-docsync, skill-e2e-docsync-spawned	Free state/permission controls; registered gate/periodic scenarios retain their tiers
Cancellation drains owned work before another attempt	shared-libs-cancellation, session-runner-stream-lifecycle, agent-sdk-runner, paid-shard-settlement	Existing actual shared-library/SDK caller scenarios	Free real-callback/process controls; registered live gate/periodic trials remain independent
Missing tools or incomplete results never become verified coverage	qa-probe-gates, qa-supervision-selection, test-free-shards, test-free-shards-capture, paid-shards	ship-exploratory-unavailable and existing reporting-boundary sessions	Free negative controls plus selected PR gate; unsupported hosts remain unexecuted

Names without a suffix refer to test/<name>.test.ts. Keep missing, stale, duplicate, selected-but-unstarted, malformed/truncated and observer-overflow controls distinct from legitimate empty selections. File restoration cannot replace write observation, and a clean local tree cannot prove that an external request made no mutation. Fixture endpoints and credentials must be synthetic and owned; specifically authorized functional requests remain permitted.

Functional fixtures register their existing closed command policy as a native PreToolUse hook, so an unsupported request is refused before execution. The callback regression invokes the registered command with native hook input, observes an isolated mutation target and permits the owned webhook positive control. This is a command boundary, not a sandbox for arbitrary target code. Its private CLI configuration is outside the observed product tree, and the fixture's existing cleanup owns both directories.

Review/Ship observations now use the same strict native event decoder as QA checkpoints and documentation. Caller-specific handoff/freshness interpretation stays separate. Original missing, orphaned and duplicate-call controls were run before replacing three incidental error-wording assertions with rejection checks; the existing positive attribution case still runs, and a completed-ID reuse negative control prevents incomplete evidence from becoming green.

Complete inventory, not just the fast subset

At the audited revision, all 1,124 tracked Bun test files partition into 1,010 free files and 114 paid files. The paid inventory contains 207 registered E2E selection IDs (84 gate and 123 periodic) and 25 main quality-judge IDs. IDs, files, model calls, samples, attempts and executed Bun tests are different counts; use the run manifest and receipts rather than substituting one for another.

The full gate and periodic manifests plan 153 file processes across 112 unique files. Forty-one files appear in both manifests because they contain mixed-tier cases; that does not establish 41 duplicated scenarios. Brain privacy and ship idempotency have existing explicit exclusions, iOS needs hardware, and the spec quality file contains a TODO. The iOS fixture's native Swift smoke test is outside this Bun-file census and requires its platform runtime. Unavailable or excluded work is not coverage.

Existing sources remain authoritative:

• test/helpers/paid-test-set.ts owns the free/paid file boundary.
• test/helpers/touchfiles-data.ts owns registered dependency and tier data.
• scripts/test-pr-profile.ts owns the deliberately partial PR profile.
• test/helpers/eval-budgets.ts owns supervision and retry exceptions.
• The free and paid shard runners own inventory, scheduling and reconciliation.

test:quick and test:pr are feedback lanes, not full release acceptance. This refactor does not remove cases, shrink samples, change tiers, lower thresholds, shorten production deadlines, or move checks to a later cadence.

Repeated work removed

Synthetic terminal startup. The 42 fake-terminal scenarios in nine files emit their existing readiness marker only after installing handlers. They no longer spend the real CLI's eight-second grace waiting for an already-ready fake. Native CLI grace, terminal geometry, observation delays, scenario inputs and all existing assertions stay intact.

Publication polling. Invalid/unavailable-journal cases still call the real hook and transcript reader through all 40 polling intervals. A scoped serial fake clock removes wall-clock sleeping, while controls verify the full logical deadline, late arrival, and restoration after success and failure. The real delayed-journal and shell-transport cases still use real time.

Native watchdog setup. Compile the identical source once per test file and copy the executable into each case's isolated directory. Every watchdog still executes; only duplicate checks of the same compiler result are consolidated. Mutable work directories and cleanup ownership are never pooled.

Independent live captures. Start all three consistency captures through the existing three-query semaphore and both A/B arms concurrently. Await every settlement before cleanup, retain sibling failures, and attempt cleanup for every owned directory. Consistency judging remains sequential; A/B has at most two simultaneous judges. The free regression exercises the actual registered callbacks, native capture receipts, semaphore and judge request builder.

Runner ownership. Path normalization belongs to the existing shared strict output utility, not the free runner. The paid runner no longer imports the free runner to use it. Exact free-only exemptions cover the free runner and the free AUQ replay worker, with paid import-closure and selection controls. Mapped dependencies still win; unknown dependencies retain the broad fallback. No directory-wide exemption is introduced.

Local scheduling. Default free workers use available CPU affinity, with a floor of one and the existing cap of six. Each shard stays serial internally, and explicit GSTACK_FREE_JOBS overrides keep their previous meaning. The separate 20-machine CI plan is unchanged. Windows free-test CI explicitly retains its two-worker budget rather than inheriting the local default; local worker gains are not CI gains.

Measurement contract

Compare original and edited code on the same machine, runtime, launch environment and workload. Preserve failure and retry records. Count the full case/sample inventory and report skips and unavailable platforms separately. Do not subtract failures from elapsed time or use a smaller selection as proof that the complete suite got faster.

Functional-QA cleanup measurement — September 28, 2026

On the same four-CPU Linux machine, using Bun 1.4.0, Node 22.20.0 and Claude Code 2.1.251, the existing duration recorder measured all 1,113 free files. The refreshed seed selects 931 files for quick feedback: 90 newly included and 20 newly excluded by measured cost, a net increase of 70. No files remain unclassified. All 182 slow files remain in the complete suite. The functional command observer, checkpoint decoder and log-capture controls are explicit quick-core cases; each measured under two seconds.

Existing command / attempt	Executed scope	Result	Wall time
bun run test:free --record-durations	1,113 files	29,175 pass, 5 fail, 131 skip	680.64s
bun run test:quick, first measured attempt	931 files	22,160 pass, 2 fail, 100 skip	125.39s
bun run test:quick, repaired attempt	The same 931 files	22,162 pass, 0 fail, 100 skip	52.43s

The profile's five failures came from the machine's Git identity wrapper overwriting synthetic fixture authors. Running the two affected files with native Git in the isolated test environment passed all 59 tests in 75.32s; normal checkout commits retained the configured identity. Both quick attempts used that corrected environment. Their two telemetry timeouts used Bun's synchronous piped-input path; the repair reuses the existing file-backed command capture helper without changing commands, assertions or deadlines. The seed retains observed costs, including failed attempts; it is a scheduling hint, not a passing receipt.

Cold dependency installation took 0.477s and the integrated build took 3.84s, separate from warm test execution; CLI installation was not independently timed. An earlier 63.37s profile was cancelled for a decoder repair, with an additional scoped browser cleanup, and earns no completion credit. Failed, cancelled and repair runs are costs, not time removed from the workflow. The quick target of one minute was met on this machine, but these measurements establish neither a cross-environment speedup nor full release, live-model or Windows acceptance.

Earlier component comparisons

Measured component comparisons:

Workload	Before	After	Coverage retained
Nine synthetic-terminal files, serial aggregate	165.87s	72.98s	81 tests, 1,593 assertions, 42 PTY scenarios
Publication guard and watchdog files, serial aggregate	56.57s	9.63s	245 original tests; three additional clock controls
Two live periodic AUQ files, same machine and runtime	325.09s	136.40s	Two tests, five independent captures, all original grading rules

Exact selectors for the synthetic-terminal comparison:

test/plan-count-fixture.test.ts
test/plan-count-design-ui-recovery.test.ts
test/plan-count-native-input.test.ts
test/plan-count-empty-review.test.ts
test/plan-count-owned-permission.test.ts
test/plan-count-file-permission.test.ts
test/plan-count-truncated-question.test.ts
test/plan-count-preview-footer.test.ts
test/eng-test-plan-edit-approval.test.ts

The quoted-frame selector was folded into test/plan-count-file-permission.test.ts in the 2026-09 audit.

The publication/watchdog pair is test/autoplan-publication-guard.test.ts and test/cso-watchdog.test.ts. The live pair is test/skill-e2e-auq-consistency.test.ts and test/skill-e2e-auq-verbose-vs-carved-ab.test.ts, using the default three consistency samples plus the two A/B captures.

These are separate comparisons; do not add their percentages or call them a complete paid-suite result. The terminal aggregate is 56% faster, the two publication/watchdog files are 83% faster, and the live AUQ pair is 58% faster. Watchdog assertion counts fall only because identical compilation is checked once; all behavioral and security assertions remain.

The matched complete local free-suite comparison used the same four-CPU Linux machine, Bun 1.4.0, Node 24.18.0, built artifacts, display and isolated Git configuration. Neither run set a worker override: the original default used two workers and the changed default used four.

Complete free-suite result	Original	Performance refactor
Wall time	887.82s	413.78s
Passing cases	25,504	25,547
Failing cases	0	0
Skipped cases	60	60
Assertions	199,132	200,351
Test files	1,010	1,012

This is a 53.4% local reduction. Case-level reconciliation retained every original passing identity except the intentionally renamed worker-default policy test, added 43 cases, and retained the exact same 60 skipped identities. This comparison predates the additional fixture and deployment-workflow regression cases; final acceptance must cover those too. It is not a measurement of the separate CI matrix or the complete paid census.

The exact historical PR #2956 diff selected 84 gate IDs plus 25 judges because the free runner was treated as an unknown paid dependency. Replaying that diff after the boundary repair selects the intended 26 fast IDs plus the same 25 judges. A free-runner-only diff selects no paid work. That is routing accuracy, not a fresh full-census runtime improvement.

Evidence validity

After integrating main's September 28 Ubicloud improvements, the scheduling seed uses upstream's CI-environment timings for shared files and preserves the 52 previously measured branch-only entries. These are scheduling hints from two machines, not a matched performance comparison or acceptance result. Refresh the whole seed with bun run test:ubicloud --record-durations when measuring a new common baseline; do not infer a speedup by adding these measurements.

Check the executable actually used by each SDK, print-mode and terminal launcher. A CLI version cached during preflight does not prove the version used by later sessions if PATH contents change. Use native session-init versions, terminal startup captures or process witnesses, and keep runtime controls effective after the hermetic environment is constructed.

Captured-event regressions prove the validator accepts valid evidence and rejects invalid evidence. They do not turn an old failed model run into a pass. Preserve every configured retry attempt and distinguish actual registrations from filtered or out-of-tier placeholders. A retained passing judge is reusable only when its complete request, rubric, parameters and relevant dependencies match; native behavioral acceptance follows its separate freshness contract.

Remaining work, not claimed savings

Report final acceptance with the delivery revision and its actual case inventory. Complete fresh gate/periodic and cross-platform performance comparisons are not established by the measurements above.

The longest indivisible live workflow limits the benefit of extra workers. Historical paid-duration replay suggests better scheduling alone cannot halve the full lane. A follow-up should unify executable case ownership/counts before sharing captures between judges or splitting long files: keep each oracle, scenario, retry and independent-trial requirement explicit. Host integrations and security boundary cases must not be replaced with cheaper look-alikes; the retired Autoplan chain eval needs a replacement that fits the ordinary tiers.