test: plan significant pre-feature quality program

.aiwg/testing/master-test-plan.md

@@ -6,21 +6,28 @@ Phase: construction
 
 ## Current verified baseline
 
-Baseline commit: `9b93e20276eac0ab3217ef33e719fc160a323fc3`
+Baseline commit: `340173e203757757451893347445ac8bd5fdc816`
 
-The mandatory offline CPU selection currently completes in 78.89 seconds with
-1,144 passing tests, 9 conditionally deselected tests, no skips, and no
-warnings. Branch-enabled coverage is 66.36% line and 51.92% branch. The
-repository also enforces 90% changed-line coverage, 80% line / 75% branch for
-the documented mature CPU-testable scope, a 70% selective mutation score, and
-zero unexpected warnings.
+The mandatory offline CPU selection currently completes locally in 72.55
+seconds with 1,195 passing tests, 9 conditionally deselected tests, no skips,
+and no warnings. Branch-enabled coverage is 66.44% line and 52.08% branch. The
+documented mature CPU-testable scope is 89.36% line and 75.41% branch. The
+repository also enforces 90% changed-line coverage, per-touched-module line and
+branch no-regression, a 70% selective mutation score, and zero unexpected
+warnings. Hosted CI is green for Python 3.10, 3.11, and 3.12, packaging,
+quality depth, supply chain, and the current exact `main` head.
+The latest conditional run is green for its selected pinned-model,
+external-evaluation, loopback-network, and operator-UI gates. CUDA,
+bitsandbytes, MPS, MLX, and remote execution were not selected and therefore
+have no fresh support evidence.
 
-The first six milestones in this plan established a sane baseline. The next
-program is deliberately risk-weighted: it improves contract durability and
-failure-path coverage before raising repository-wide percentages. Ordinary new
-feature work remains paused through Wave A below. The outstanding contributor
-merge train may resume after the Wave A gate, one exact-head PR at a time, with
-the per-PR test contract in this plan applied to every integration.
+The first six milestones and Wave A established a sane, evidence-producing
+baseline. The next program is deliberately risk-weighted: it improves contract
+durability, failure behavior, integration confidence, and real-environment
+evidence before raising blanket percentages. The legacy contributor merge
+train is parked until Gate 1 below. Ordinary new feature work is parked until
+Gate 2. Planning, test-only changes, defect fixes discovered by the program,
+and the minimum refactors needed to expose deterministic seams remain allowed.
 
 ## Reasoning
 
@@ -132,73 +139,104 @@ module may not lose line or branch coverage unless the PR documents why the
 measurement is misleading and adds equivalent contract or conditional evidence.
 No flaky-test rerun may turn a required failure into success.
 
-## Second-generation improvement program
+## Testing-first improvement program
 
-### Wave A — policy and evidence lock (merge-train entry gate)
+### Completed foundation — policy and evidence lock
 
-1. Retain the hermetic CPU/offline matrix on Python 3.10–3.12 as the mandatory
-   PR gate, including installed wheel/sdist smoke, lint, package policy, JUnit,
-   line and branch coverage, mutation, and warning evidence.
-2. Add a touched-module no-regression check for both line and branch coverage;
-   retain the 90% changed-line floor.
-3. Publish a source-to-test risk map for the critical interfaces named in the
-   per-PR contract and link each conditional exclusion to its runnable gate.
-4. Record coverage and flaky-test trend data as retained CI artifacts. A test
-   that flakes twice in 30 days blocks the train until repaired or quarantined
-   with an owner, reason, and expiry.
-5. Re-run the full hosted matrix at the exact baseline head. All required jobs
-   must pass with zero unexpected warnings and no unresolved review thread.
+The hermetic Python 3.10–3.12 matrix, installed wheel/sdist smoke, exact-base
+coverage comparison, 90% changed-line floor, touched-module no-regression gate,
+risk map, quality policy, repeat and mutation evidence, conditional workflows,
+and supply-chain job are merged and green. These controls are the floor for all
+subsequent waves and may only be tightened.
 
-Wave A exit permits the existing contributor PR train to resume. It does not
-permit unrelated new feature scope.
+### Wave B1 — high-consequence CPU contracts (Gate 1)
 
-### Wave B — high-risk boundary campaign
+Deliver small, reviewable test or test-enabling PRs in this order:
 
-Deliver small, reviewable PRs in this order:
+1. **Research and numerical correctness** — refusal direction selection,
+   activation aggregation, empty/singleton behavior, dtype/tolerance handling,
+   deterministic metric serialization, and error signaling. Prefer invariant
+   and property assertions over frozen implementation details.
+2. **Loader and architecture boundaries** — device maps, memory budgets,
+   quantization configuration, cache behavior, tied/shared weights, missing
+   projections, unsupported architectures, and CPU fallback behavior.
+3. **Persistence and destructive operations** — checkpoint atomicity,
+   interrupted/partial writes, corrupt input, overwrite refusal, cleanup,
+   save/reload round trips, and idempotent retry.
+4. **Public and remote contracts** — CLI/config defaults and invalid values,
+   local/remote propagation, shell-safe command construction, cancellation,
+   result synchronization, malformed provider responses, and timeout paths.
+5. **Evaluation and research outputs** — causal/classifier routing, dataset
+   bounds, report schema, telemetry aggregation, provenance fields, deterministic
+   ordering, and explicit failure rather than silent fallback.
 
-1. CLI/config propagation and validation;
-2. loader, device, dtype, quantization, and architecture selection;
-3. evaluation, reporting, telemetry, and research-output contracts;
-4. checkpoint atomicity, save/reload, and destructive-operation safeguards;
-5. pure decision logic currently embedded in model, remote, watchtower, and
-   orchestration modules.
+Tests must exercise negative, boundary, and cross-component behavior and must
+fail when the protected contract is deliberately broken. External model,
+service, filesystem, clock, process, or hardware edges may be faked; the
+decision logic and schemas under test may not be replaced by mocks. Minimum
+production refactors are allowed only to extract deterministic seams, and each
+extracted helper becomes part of the mutation/property-test scope.
 
-The target is not a blanket test count. Each PR must add negative, boundary,
-and cross-component assertions that keep real pure functions and schemas under
-test while faking only external model, service, filesystem, or hardware edges.
+Gate 1 requires all of the following on exact-head hosted CI:
+
+- repository coverage at least 70% line and 55% branch;
+- mature CPU scope at least 90% line and 78% branch;
+- the critical-module targets below met or exceeded;
+- selective mutation expanded beyond configuration/policy scripts to at least
+  three high-consequence pure contract surfaces, with at least 75% killed;
+- zero unexpected warnings, active flaky quarantines, P0/P1 defects, or
+  unresolved review threads;
+- focused contributor tests complete in under two minutes and the complete
+  mandatory lane remains under ten minutes per Python version.
 
 | Surface | Verified baseline | Wave B target |
 |---|---:|---:|
-| Repository | 66.36% line / 51.92% branch | 70% line / 55% branch |
-| `cli.py` | 97.87% / 81.06% | preserve ≥97% / raise to ≥85% |
-| `config.py` | 98.71% / 81.25% | preserve ≥98% / raise to ≥90% |
+| Repository | 66.44% line / 52.08% branch | ≥70% / ≥55% |
+| Mature CPU scope | 89.36% / 75.41% | ≥90% / ≥78% |
+| `cli.py` | 97.91% / 81.43% | preserve ≥97% / raise to ≥85% |
+| `config.py` | 98.73% / 81.25% | preserve ≥98% / raise to ≥90% |
 | `device.py` | 96.47% / 92.42% | preserve ≥96% / ≥92% |
 | `architecture_profiles.py` | 88.62% / 82.95% | ≥92% / ≥85% |
-| `models/loader.py` | 84.78% / 81.96% | ≥90% / ≥85% |
+| `models/loader.py` | 85.79% / 84.09% | ≥90% / ≥87% |
 | `evaluation/evaluator.py` | 72.00% / 71.05% | ≥85% / ≥80% |
 | `reporting/report.py` | 82.94% / 64.81% | ≥90% / ≥80% |
 | `telemetry.py` | 71.07% / 62.40% | ≥80% / ≥75% |
 
-Large runtime-bound modules are split conceptually into CPU-testable decision
-logic and conditional execution. CPU-testable helpers extracted from
+Large runtime-bound modules must be split conceptually into CPU-testable
+decision logic and conditional execution. CPU-testable helpers extracted from
 `abliterate.py`, `informed_pipeline.py`, `remote.py`, `watchtower.py`, and
-related orchestrators target at least 85% line / 80% branch. Real model,
+related orchestrators target at least 90% line / 85% branch. Real model,
 accelerator, service, UI, and remote execution remains covered by the mapped
 conditional gates instead of being disguised by mocks.
 
-Wave B exit permits ordinary feature work to resume under the per-PR contract.
+Gate 1 permits the legacy contributor merge train to resume, one exact-head PR
+at a time with courtesy tests for the already-open queue. It does not permit
+ordinary new feature work.
 
-### Wave C — offline integration and failure atomicity
+### Wave B2 — offline vertical slices and reliability (Gate 2)
 
-1. Extend the tiny offline model path through CLI/config → loader/device →
-   pipeline → checkpoint → evaluator → report.
-2. Add interruption, partial-write, corrupted-state, and retry tests for
-   checkpoint and report creation.
-3. Add deterministic community contribution and evaluation/report round trips.
-4. Keep the complete mandatory lane under ten minutes per Python version and
-   preserve a sub-two-minute focused contributor loop.
+1. Add a deterministic tiny-model vertical slice through CLI/config →
+   loader/device → pipeline → checkpoint → evaluator → report, exercising the
+   installed package rather than only source-tree imports.
+2. Add a second vertical slice for failure recovery: interrupted mutation,
+   partial/corrupt checkpoint, retry, atomic replacement, and cleanup.
+3. Add deterministic community-contribution and evaluation/report round trips,
+   including malformed and forward-compatible schema cases.
+4. Add concurrency/idempotency tests for filesystem writes and cancellation
+   boundaries that can be exercised without external services.
+5. Make the risk map exhaustive for production modules and map every public
+   option, persisted schema, research metric, and environment-bound path to at
+   least one mandatory or conditional test owner.
 
-### Wave D — property, mutation, and regression depth
+Gate 2 requires repository coverage of at least 75% line and 60% branch,
+mature CPU coverage of at least 92% line and 80% branch, 100% coverage of the
+documented critical vertical slices, and no regression in the Gate 1 targets.
+The complete lane must remain below ten minutes per Python version and produce
+JUnit, coverage, package, repeat, mutation, and normalized trend evidence.
+
+Gate 2 permits ordinary new feature work to resume under the per-PR contract.
+
+### Wave C — property, mutation, determinism, and regression depth
 
 1. Add property tests for metric bounds, permutation/label invariance,
    serialization round trips, architecture selection, and configuration
@@ -207,10 +245,12 @@ Wave B exit permits ordinary feature work to resume under the per-PR contract.
    parser, selection, checkpoint, reporting, and evaluation helpers.
 3. Require at least 80% mutation score for the selected mature scope, with
    changed contract code killing every applicable generated mutant.
-4. Run deterministic ordering and hash-seed repeats; publish trends without
-   silently retrying failures.
+4. Run deterministic ordering, fixed-seed replay, and multiple hash-seed
+   repeats; publish trends without silently retrying failures.
+5. Track duration by test and marker, fail on unowned slow-test growth, and
+   keep quarantines time-bounded, issue-linked, and absent from critical paths.
 
-### Wave E — conditional and release evidence
+### Wave D — conditional and release evidence
 
 1. Require fresh successful model-download, external-evaluation, network, and
    operator-UI evidence for releases and for PRs that modify those mappings.
@@ -220,12 +260,34 @@ Wave B exit permits ordinary feature work to resume under the per-PR contract.
    an explicit time-bounded waiver; do not weaken the mandatory CPU gate.
 4. Retain exact commit, dependency lock, environment, result, and artifact
    provenance for every conditional run.
+5. Replace closed umbrella-issue links in exclusion records with durable gate
+   documentation plus an active owner/tracker item only when work or a waiver
+   remains outstanding.
 
-## Merge-train resumption gate
+### Delivery sequence
 
-The paused contributor merge train resumes only when:
+The program should be delivered as bounded PRs, each green and exact-head
+audited before the next begins:
 
-- Wave A is merged and green on the current `main` head;
+1. policy truth reset and exhaustive traceability schema;
+2. research/numerical property contracts;
+3. loader/device/architecture negative and mutation contracts;
+4. checkpoint and destructive-operation reliability;
+5. remote/CLI/provider safety and failure behavior;
+6. evaluation/report/telemetry research-output contracts;
+7. installed-package offline vertical slices;
+8. conditional evidence freshness and runner/waiver reconciliation.
+
+Each PR must publish before/after module coverage, changed-line coverage, killed
+mutants where applicable, runtime delta, and the exact tests that demonstrate
+the protected behavior. Percentage-only changes without a named contract do
+not advance a gate.
+
+## Merge-train and new-work gates
+
+The paused contributor merge train resumes only when Gate 1 is met and:
+
+- the completed foundation remains green on the current `main` head;
 - required CPU, package, quality-depth, and supply-chain jobs pass;
 - repository, mature-scope, changed-line, mutation, and warning floors pass;
 - no unresolved flaky-test quarantine affects the touched module;
@@ -234,6 +296,11 @@ The paused contributor merge train resumes only when:
 - the next PR is re-audited at its exact head and includes the per-PR evidence
   above, with maintainer courtesy tests added for the current legacy queue.
 
+Ordinary new feature work resumes only when Gate 2 is met on `main`. Security,
+correctness, data-loss, and test-infrastructure defects discovered during the
+program may be fixed before that point, but they must remain narrowly scoped
+and include a reproducing regression test.
+
 ## Milestones
 
 Completed baseline:
@@ -247,11 +314,11 @@ Completed baseline:
 
 Next program:
 
-7. Wave A policy/evidence lock and merge-train entry gate.
-8. Wave B high-risk boundary campaign and ordinary-work entry gate.
-9. Wave C offline integration and failure atomicity.
-10. Wave D property, mutation, repeat, and regression depth.
-11. Wave E conditional environment and release-evidence freshness.
+7. Completed foundation policy/evidence lock.
+8. Wave B1 high-consequence CPU contracts and legacy-train entry Gate 1.
+9. Wave B2 offline vertical slices and ordinary-work entry Gate 2.
+10. Wave C property, mutation, determinism, and regression depth.
+11. Wave D conditional environment and release-evidence freshness.
 
 ## Metrics and reporting