test: plan second-generation quality program

.aiwg/testing/master-test-plan.md

@@ -4,6 +4,24 @@ Date: 2026-08-14
 Owner: maintainers
 Phase: construction
 
+## Current verified baseline
+
+Baseline commit: `9b93e20276eac0ab3217ef33e719fc160a323fc3`
+
+The mandatory offline CPU selection currently completes in 78.89 seconds with
+1,144 passing tests, 9 conditionally deselected tests, no skips, and no
+warnings. Branch-enabled coverage is 66.36% line and 51.92% branch. The
+repository also enforces 90% changed-line coverage, 80% line / 75% branch for
+the documented mature CPU-testable scope, a 70% selective mutation score, and
+zero unexpected warnings.
+
+The first six milestones in this plan established a sane baseline. The next
+program is deliberately risk-weighted: it improves contract durability and
+failure-path coverage before raising repository-wide percentages. Ordinary new
+feature work remains paused through Wave A below. The outstanding contributor
+merge train may resume after the Wave A gate, one exact-head PR at a time, with
+the per-PR test contract in this plan applied to every integration.
+
 ## Reasoning
 
 1. **Scope** — package/public CLI behavior, core ablation and analysis logic,
@@ -92,8 +110,134 @@ Program exit:
 - CPU-testable code reaches the mature 80% line / 75% branch target, or every
   remaining exclusion has a documented environment-bound rationale.
 
+## Per-PR test contract
+
+Every behavior-changing PR must include tests that fail without the change and
+exercise the smallest meaningful production boundary. Coverage-only assertions
+or tests that merely inspect source text do not satisfy this contract.
+
+| Change | Required evidence |
+|---|---|
+| Bug fix | focused regression reproducing the original failure plus the relevant suite |
+| Public option or configuration | default, explicit value, invalid/boundary value, propagation, and runtime effect |
+| CLI change | parser contract, dispatch contract, and local/remote propagation when both exist |
+| Model/architecture adapter | projection discovery, shape contract, missing/unsupported structure, and tied/shared-weight behavior where relevant |
+| Evaluation or reporting | numerical invariant, empty/singleton boundary, output/schema contract, and deterministic serialization |
+| Checkpoint or persistence | success, interruption/failure atomicity, reload/round-trip, and cleanup behavior |
+| Hardware, network, or remote path | deterministic fake-based contract test plus the mapped conditional gate |
+| Research claim or metric | provenance/interpretation contract and a test preventing silent semantic drift |
+
+New or modified code must maintain at least 90% changed-line coverage. A touched
+module may not lose line or branch coverage unless the PR documents why the
+measurement is misleading and adds equivalent contract or conditional evidence.
+No flaky-test rerun may turn a required failure into success.
+
+## Second-generation improvement program
+
+### Wave A — policy and evidence lock (merge-train entry gate)
+
+1. Retain the hermetic CPU/offline matrix on Python 3.10–3.12 as the mandatory
+   PR gate, including installed wheel/sdist smoke, lint, package policy, JUnit,
+   line and branch coverage, mutation, and warning evidence.
+2. Add a touched-module no-regression check for both line and branch coverage;
+   retain the 90% changed-line floor.
+3. Publish a source-to-test risk map for the critical interfaces named in the
+   per-PR contract and link each conditional exclusion to its runnable gate.
+4. Record coverage and flaky-test trend data as retained CI artifacts. A test
+   that flakes twice in 30 days blocks the train until repaired or quarantined
+   with an owner, reason, and expiry.
+5. Re-run the full hosted matrix at the exact baseline head. All required jobs
+   must pass with zero unexpected warnings and no unresolved review thread.
+
+Wave A exit permits the existing contributor PR train to resume. It does not
+permit unrelated new feature scope.
+
+### Wave B — high-risk boundary campaign
+
+Deliver small, reviewable PRs in this order:
+
+1. CLI/config propagation and validation;
+2. loader, device, dtype, quantization, and architecture selection;
+3. evaluation, reporting, telemetry, and research-output contracts;
+4. checkpoint atomicity, save/reload, and destructive-operation safeguards;
+5. pure decision logic currently embedded in model, remote, watchtower, and
+   orchestration modules.
+
+The target is not a blanket test count. Each PR must add negative, boundary,
+and cross-component assertions that keep real pure functions and schemas under
+test while faking only external model, service, filesystem, or hardware edges.
+
+| Surface | Verified baseline | Wave B target |
+|---|---:|---:|
+| Repository | 66.36% line / 51.92% branch | 70% line / 55% branch |
+| `cli.py` | 97.87% / 81.06% | preserve ≥97% / raise to ≥85% |
+| `config.py` | 98.71% / 81.25% | preserve ≥98% / raise to ≥90% |
+| `device.py` | 96.47% / 92.42% | preserve ≥96% / ≥92% |
+| `architecture_profiles.py` | 88.62% / 82.95% | ≥92% / ≥85% |
+| `models/loader.py` | 84.78% / 81.96% | ≥90% / ≥85% |
+| `evaluation/evaluator.py` | 72.00% / 71.05% | ≥85% / ≥80% |
+| `reporting/report.py` | 82.94% / 64.81% | ≥90% / ≥80% |
+| `telemetry.py` | 71.07% / 62.40% | ≥80% / ≥75% |
+
+Large runtime-bound modules are split conceptually into CPU-testable decision
+logic and conditional execution. CPU-testable helpers extracted from
+`abliterate.py`, `informed_pipeline.py`, `remote.py`, `watchtower.py`, and
+related orchestrators target at least 85% line / 80% branch. Real model,
+accelerator, service, UI, and remote execution remains covered by the mapped
+conditional gates instead of being disguised by mocks.
+
+Wave B exit permits ordinary feature work to resume under the per-PR contract.
+
+### Wave C — offline integration and failure atomicity
+
+1. Extend the tiny offline model path through CLI/config → loader/device →
+   pipeline → checkpoint → evaluator → report.
+2. Add interruption, partial-write, corrupted-state, and retry tests for
+   checkpoint and report creation.
+3. Add deterministic community contribution and evaluation/report round trips.
+4. Keep the complete mandatory lane under ten minutes per Python version and
+   preserve a sub-two-minute focused contributor loop.
+
+### Wave D — property, mutation, and regression depth
+
+1. Add property tests for metric bounds, permutation/label invariance,
+   serialization round trips, architecture selection, and configuration
+   normalization.
+2. Expand selective mutation from configuration and policy scripts to pure
+   parser, selection, checkpoint, reporting, and evaluation helpers.
+3. Require at least 80% mutation score for the selected mature scope, with
+   changed contract code killing every applicable generated mutant.
+4. Run deterministic ordering and hash-seed repeats; publish trends without
+   silently retrying failures.
+
+### Wave E — conditional and release evidence
+
+1. Require fresh successful model-download, external-evaluation, network, and
+   operator-UI evidence for releases and for PRs that modify those mappings.
+2. Require CUDA/bitsandbytes, MPS, MLX, or remote evidence when the matching
+   runner is configured and the affected surface changes.
+3. If a runner is unavailable, block claims about that environment and record
+   an explicit time-bounded waiver; do not weaken the mandatory CPU gate.
+4. Retain exact commit, dependency lock, environment, result, and artifact
+   provenance for every conditional run.
+
+## Merge-train resumption gate
+
+The paused contributor merge train resumes only when:
+
+- Wave A is merged and green on the current `main` head;
+- required CPU, package, quality-depth, and supply-chain jobs pass;
+- repository, mature-scope, changed-line, mutation, and warning floors pass;
+- no unresolved flaky-test quarantine affects the touched module;
+- conditional mappings remain valid, with unavailable untouched environments
+  explicitly non-blocking;
+- the next PR is re-audited at its exact head and includes the per-PR evidence
+  above, with maintainer courtesy tests added for the current legacy queue.
+
 ## Milestones
 
+Completed baseline:
+
 1. Evidence, strict pytest policy, workflow validation.
 2. Installed distributions and supply-chain evidence.
 3. Boundary contracts and 55% repository line coverage.
@@ -101,6 +245,14 @@ Program exit:
 5. Research-integrity, property, repeat, and selected mutation gates.
 6. Conditional backend/service workflows and mature CPU-testable targets.
 
+Next program:
+
+7. Wave A policy/evidence lock and merge-train entry gate.
+8. Wave B high-risk boundary campaign and ordinary-work entry gate.
+9. Wave C offline integration and failure atomicity.
+10. Wave D property, mutation, repeat, and regression depth.
+11. Wave E conditional environment and release-evidence freshness.
+
 ## Metrics and reporting
 
 - tests passed/failed/skipped by marker and Python version;