diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index 6364d38..022c077 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -4,6 +4,24 @@ Date: 2026-08-14 Owner: maintainers Phase: construction +## Current verified baseline + +Baseline commit: `9b93e20276eac0ab3217ef33e719fc160a323fc3` + +The mandatory offline CPU selection currently completes in 78.89 seconds with +1,144 passing tests, 9 conditionally deselected tests, no skips, and no +warnings. Branch-enabled coverage is 66.36% line and 51.92% branch. The +repository also enforces 90% changed-line coverage, 80% line / 75% branch for +the documented mature CPU-testable scope, a 70% selective mutation score, and +zero unexpected warnings. + +The first six milestones in this plan established a sane baseline. The next +program is deliberately risk-weighted: it improves contract durability and +failure-path coverage before raising repository-wide percentages. Ordinary new +feature work remains paused through Wave A below. The outstanding contributor +merge train may resume after the Wave A gate, one exact-head PR at a time, with +the per-PR test contract in this plan applied to every integration. + ## Reasoning 1. **Scope** — package/public CLI behavior, core ablation and analysis logic, @@ -92,8 +110,134 @@ Program exit: - CPU-testable code reaches the mature 80% line / 75% branch target, or every remaining exclusion has a documented environment-bound rationale. +## Per-PR test contract + +Every behavior-changing PR must include tests that fail without the change and +exercise the smallest meaningful production boundary. Coverage-only assertions +or tests that merely inspect source text do not satisfy this contract. + +| Change | Required evidence | +|---|---| +| Bug fix | focused regression reproducing the original failure plus the relevant suite | +| Public option or configuration | default, explicit value, invalid/boundary value, propagation, and runtime effect | +| CLI change | parser contract, dispatch contract, and local/remote propagation when both exist | +| Model/architecture adapter | projection discovery, shape contract, missing/unsupported structure, and tied/shared-weight behavior where relevant | +| Evaluation or reporting | numerical invariant, empty/singleton boundary, output/schema contract, and deterministic serialization | +| Checkpoint or persistence | success, interruption/failure atomicity, reload/round-trip, and cleanup behavior | +| Hardware, network, or remote path | deterministic fake-based contract test plus the mapped conditional gate | +| Research claim or metric | provenance/interpretation contract and a test preventing silent semantic drift | + +New or modified code must maintain at least 90% changed-line coverage. A touched +module may not lose line or branch coverage unless the PR documents why the +measurement is misleading and adds equivalent contract or conditional evidence. +No flaky-test rerun may turn a required failure into success. + +## Second-generation improvement program + +### Wave A — policy and evidence lock (merge-train entry gate) + +1. Retain the hermetic CPU/offline matrix on Python 3.10–3.12 as the mandatory + PR gate, including installed wheel/sdist smoke, lint, package policy, JUnit, + line and branch coverage, mutation, and warning evidence. +2. Add a touched-module no-regression check for both line and branch coverage; + retain the 90% changed-line floor. +3. Publish a source-to-test risk map for the critical interfaces named in the + per-PR contract and link each conditional exclusion to its runnable gate. +4. Record coverage and flaky-test trend data as retained CI artifacts. A test + that flakes twice in 30 days blocks the train until repaired or quarantined + with an owner, reason, and expiry. +5. Re-run the full hosted matrix at the exact baseline head. All required jobs + must pass with zero unexpected warnings and no unresolved review thread. + +Wave A exit permits the existing contributor PR train to resume. It does not +permit unrelated new feature scope. + +### Wave B — high-risk boundary campaign + +Deliver small, reviewable PRs in this order: + +1. CLI/config propagation and validation; +2. loader, device, dtype, quantization, and architecture selection; +3. evaluation, reporting, telemetry, and research-output contracts; +4. checkpoint atomicity, save/reload, and destructive-operation safeguards; +5. pure decision logic currently embedded in model, remote, watchtower, and + orchestration modules. + +The target is not a blanket test count. Each PR must add negative, boundary, +and cross-component assertions that keep real pure functions and schemas under +test while faking only external model, service, filesystem, or hardware edges. + +| Surface | Verified baseline | Wave B target | +|---|---:|---:| +| Repository | 66.36% line / 51.92% branch | 70% line / 55% branch | +| `cli.py` | 97.87% / 81.06% | preserve ≥97% / raise to ≥85% | +| `config.py` | 98.71% / 81.25% | preserve ≥98% / raise to ≥90% | +| `device.py` | 96.47% / 92.42% | preserve ≥96% / ≥92% | +| `architecture_profiles.py` | 88.62% / 82.95% | ≥92% / ≥85% | +| `models/loader.py` | 84.78% / 81.96% | ≥90% / ≥85% | +| `evaluation/evaluator.py` | 72.00% / 71.05% | ≥85% / ≥80% | +| `reporting/report.py` | 82.94% / 64.81% | ≥90% / ≥80% | +| `telemetry.py` | 71.07% / 62.40% | ≥80% / ≥75% | + +Large runtime-bound modules are split conceptually into CPU-testable decision +logic and conditional execution. CPU-testable helpers extracted from +`abliterate.py`, `informed_pipeline.py`, `remote.py`, `watchtower.py`, and +related orchestrators target at least 85% line / 80% branch. Real model, +accelerator, service, UI, and remote execution remains covered by the mapped +conditional gates instead of being disguised by mocks. + +Wave B exit permits ordinary feature work to resume under the per-PR contract. + +### Wave C — offline integration and failure atomicity + +1. Extend the tiny offline model path through CLI/config → loader/device → + pipeline → checkpoint → evaluator → report. +2. Add interruption, partial-write, corrupted-state, and retry tests for + checkpoint and report creation. +3. Add deterministic community contribution and evaluation/report round trips. +4. Keep the complete mandatory lane under ten minutes per Python version and + preserve a sub-two-minute focused contributor loop. + +### Wave D — property, mutation, and regression depth + +1. Add property tests for metric bounds, permutation/label invariance, + serialization round trips, architecture selection, and configuration + normalization. +2. Expand selective mutation from configuration and policy scripts to pure + parser, selection, checkpoint, reporting, and evaluation helpers. +3. Require at least 80% mutation score for the selected mature scope, with + changed contract code killing every applicable generated mutant. +4. Run deterministic ordering and hash-seed repeats; publish trends without + silently retrying failures. + +### Wave E — conditional and release evidence + +1. Require fresh successful model-download, external-evaluation, network, and + operator-UI evidence for releases and for PRs that modify those mappings. +2. Require CUDA/bitsandbytes, MPS, MLX, or remote evidence when the matching + runner is configured and the affected surface changes. +3. If a runner is unavailable, block claims about that environment and record + an explicit time-bounded waiver; do not weaken the mandatory CPU gate. +4. Retain exact commit, dependency lock, environment, result, and artifact + provenance for every conditional run. + +## Merge-train resumption gate + +The paused contributor merge train resumes only when: + +- Wave A is merged and green on the current `main` head; +- required CPU, package, quality-depth, and supply-chain jobs pass; +- repository, mature-scope, changed-line, mutation, and warning floors pass; +- no unresolved flaky-test quarantine affects the touched module; +- conditional mappings remain valid, with unavailable untouched environments + explicitly non-blocking; +- the next PR is re-audited at its exact head and includes the per-PR evidence + above, with maintainer courtesy tests added for the current legacy queue. + ## Milestones +Completed baseline: + 1. Evidence, strict pytest policy, workflow validation. 2. Installed distributions and supply-chain evidence. 3. Boundary contracts and 55% repository line coverage. @@ -101,6 +245,14 @@ Program exit: 5. Research-integrity, property, repeat, and selected mutation gates. 6. Conditional backend/service workflows and mature CPU-testable targets. +Next program: + +7. Wave A policy/evidence lock and merge-train entry gate. +8. Wave B high-risk boundary campaign and ordinary-work entry gate. +9. Wave C offline integration and failure atomicity. +10. Wave D property, mutation, repeat, and regression depth. +11. Wave E conditional environment and release-evidence freshness. + ## Metrics and reporting - tests passed/failed/skipped by marker and Python version;