diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index 022c077..e70a654 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -6,21 +6,28 @@ Phase: construction ## Current verified baseline -Baseline commit: `9b93e20276eac0ab3217ef33e719fc160a323fc3` +Baseline commit: `340173e203757757451893347445ac8bd5fdc816` -The mandatory offline CPU selection currently completes in 78.89 seconds with -1,144 passing tests, 9 conditionally deselected tests, no skips, and no -warnings. Branch-enabled coverage is 66.36% line and 51.92% branch. The -repository also enforces 90% changed-line coverage, 80% line / 75% branch for -the documented mature CPU-testable scope, a 70% selective mutation score, and -zero unexpected warnings. +The mandatory offline CPU selection currently completes locally in 72.55 +seconds with 1,195 passing tests, 9 conditionally deselected tests, no skips, +and no warnings. Branch-enabled coverage is 66.44% line and 52.08% branch. The +documented mature CPU-testable scope is 89.36% line and 75.41% branch. The +repository also enforces 90% changed-line coverage, per-touched-module line and +branch no-regression, a 70% selective mutation score, and zero unexpected +warnings. Hosted CI is green for Python 3.10, 3.11, and 3.12, packaging, +quality depth, supply chain, and the current exact `main` head. +The latest conditional run is green for its selected pinned-model, +external-evaluation, loopback-network, and operator-UI gates. CUDA, +bitsandbytes, MPS, MLX, and remote execution were not selected and therefore +have no fresh support evidence. -The first six milestones in this plan established a sane baseline. The next -program is deliberately risk-weighted: it improves contract durability and -failure-path coverage before raising repository-wide percentages. Ordinary new -feature work remains paused through Wave A below. The outstanding contributor -merge train may resume after the Wave A gate, one exact-head PR at a time, with -the per-PR test contract in this plan applied to every integration. +The first six milestones and Wave A established a sane, evidence-producing +baseline. The next program is deliberately risk-weighted: it improves contract +durability, failure behavior, integration confidence, and real-environment +evidence before raising blanket percentages. The legacy contributor merge +train is parked until Gate 1 below. Ordinary new feature work is parked until +Gate 2. Planning, test-only changes, defect fixes discovered by the program, +and the minimum refactors needed to expose deterministic seams remain allowed. ## Reasoning @@ -132,73 +139,104 @@ module may not lose line or branch coverage unless the PR documents why the measurement is misleading and adds equivalent contract or conditional evidence. No flaky-test rerun may turn a required failure into success. -## Second-generation improvement program +## Testing-first improvement program -### Wave A — policy and evidence lock (merge-train entry gate) +### Completed foundation — policy and evidence lock -1. Retain the hermetic CPU/offline matrix on Python 3.10–3.12 as the mandatory - PR gate, including installed wheel/sdist smoke, lint, package policy, JUnit, - line and branch coverage, mutation, and warning evidence. -2. Add a touched-module no-regression check for both line and branch coverage; - retain the 90% changed-line floor. -3. Publish a source-to-test risk map for the critical interfaces named in the - per-PR contract and link each conditional exclusion to its runnable gate. -4. Record coverage and flaky-test trend data as retained CI artifacts. A test - that flakes twice in 30 days blocks the train until repaired or quarantined - with an owner, reason, and expiry. -5. Re-run the full hosted matrix at the exact baseline head. All required jobs - must pass with zero unexpected warnings and no unresolved review thread. +The hermetic Python 3.10–3.12 matrix, installed wheel/sdist smoke, exact-base +coverage comparison, 90% changed-line floor, touched-module no-regression gate, +risk map, quality policy, repeat and mutation evidence, conditional workflows, +and supply-chain job are merged and green. These controls are the floor for all +subsequent waves and may only be tightened. -Wave A exit permits the existing contributor PR train to resume. It does not -permit unrelated new feature scope. +### Wave B1 — high-consequence CPU contracts (Gate 1) -### Wave B — high-risk boundary campaign +Deliver small, reviewable test or test-enabling PRs in this order: -Deliver small, reviewable PRs in this order: +1. **Research and numerical correctness** — refusal direction selection, + activation aggregation, empty/singleton behavior, dtype/tolerance handling, + deterministic metric serialization, and error signaling. Prefer invariant + and property assertions over frozen implementation details. +2. **Loader and architecture boundaries** — device maps, memory budgets, + quantization configuration, cache behavior, tied/shared weights, missing + projections, unsupported architectures, and CPU fallback behavior. +3. **Persistence and destructive operations** — checkpoint atomicity, + interrupted/partial writes, corrupt input, overwrite refusal, cleanup, + save/reload round trips, and idempotent retry. +4. **Public and remote contracts** — CLI/config defaults and invalid values, + local/remote propagation, shell-safe command construction, cancellation, + result synchronization, malformed provider responses, and timeout paths. +5. **Evaluation and research outputs** — causal/classifier routing, dataset + bounds, report schema, telemetry aggregation, provenance fields, deterministic + ordering, and explicit failure rather than silent fallback. -1. CLI/config propagation and validation; -2. loader, device, dtype, quantization, and architecture selection; -3. evaluation, reporting, telemetry, and research-output contracts; -4. checkpoint atomicity, save/reload, and destructive-operation safeguards; -5. pure decision logic currently embedded in model, remote, watchtower, and - orchestration modules. +Tests must exercise negative, boundary, and cross-component behavior and must +fail when the protected contract is deliberately broken. External model, +service, filesystem, clock, process, or hardware edges may be faked; the +decision logic and schemas under test may not be replaced by mocks. Minimum +production refactors are allowed only to extract deterministic seams, and each +extracted helper becomes part of the mutation/property-test scope. -The target is not a blanket test count. Each PR must add negative, boundary, -and cross-component assertions that keep real pure functions and schemas under -test while faking only external model, service, filesystem, or hardware edges. +Gate 1 requires all of the following on exact-head hosted CI: + +- repository coverage at least 70% line and 55% branch; +- mature CPU scope at least 90% line and 78% branch; +- the critical-module targets below met or exceeded; +- selective mutation expanded beyond configuration/policy scripts to at least + three high-consequence pure contract surfaces, with at least 75% killed; +- zero unexpected warnings, active flaky quarantines, P0/P1 defects, or + unresolved review threads; +- focused contributor tests complete in under two minutes and the complete + mandatory lane remains under ten minutes per Python version. | Surface | Verified baseline | Wave B target | |---|---:|---:| -| Repository | 66.36% line / 51.92% branch | 70% line / 55% branch | -| `cli.py` | 97.87% / 81.06% | preserve ≥97% / raise to ≥85% | -| `config.py` | 98.71% / 81.25% | preserve ≥98% / raise to ≥90% | +| Repository | 66.44% line / 52.08% branch | ≥70% / ≥55% | +| Mature CPU scope | 89.36% / 75.41% | ≥90% / ≥78% | +| `cli.py` | 97.91% / 81.43% | preserve ≥97% / raise to ≥85% | +| `config.py` | 98.73% / 81.25% | preserve ≥98% / raise to ≥90% | | `device.py` | 96.47% / 92.42% | preserve ≥96% / ≥92% | | `architecture_profiles.py` | 88.62% / 82.95% | ≥92% / ≥85% | -| `models/loader.py` | 84.78% / 81.96% | ≥90% / ≥85% | +| `models/loader.py` | 85.79% / 84.09% | ≥90% / ≥87% | | `evaluation/evaluator.py` | 72.00% / 71.05% | ≥85% / ≥80% | | `reporting/report.py` | 82.94% / 64.81% | ≥90% / ≥80% | | `telemetry.py` | 71.07% / 62.40% | ≥80% / ≥75% | -Large runtime-bound modules are split conceptually into CPU-testable decision -logic and conditional execution. CPU-testable helpers extracted from +Large runtime-bound modules must be split conceptually into CPU-testable +decision logic and conditional execution. CPU-testable helpers extracted from `abliterate.py`, `informed_pipeline.py`, `remote.py`, `watchtower.py`, and -related orchestrators target at least 85% line / 80% branch. Real model, +related orchestrators target at least 90% line / 85% branch. Real model, accelerator, service, UI, and remote execution remains covered by the mapped conditional gates instead of being disguised by mocks. -Wave B exit permits ordinary feature work to resume under the per-PR contract. +Gate 1 permits the legacy contributor merge train to resume, one exact-head PR +at a time with courtesy tests for the already-open queue. It does not permit +ordinary new feature work. -### Wave C — offline integration and failure atomicity +### Wave B2 — offline vertical slices and reliability (Gate 2) -1. Extend the tiny offline model path through CLI/config → loader/device → - pipeline → checkpoint → evaluator → report. -2. Add interruption, partial-write, corrupted-state, and retry tests for - checkpoint and report creation. -3. Add deterministic community contribution and evaluation/report round trips. -4. Keep the complete mandatory lane under ten minutes per Python version and - preserve a sub-two-minute focused contributor loop. +1. Add a deterministic tiny-model vertical slice through CLI/config → + loader/device → pipeline → checkpoint → evaluator → report, exercising the + installed package rather than only source-tree imports. +2. Add a second vertical slice for failure recovery: interrupted mutation, + partial/corrupt checkpoint, retry, atomic replacement, and cleanup. +3. Add deterministic community-contribution and evaluation/report round trips, + including malformed and forward-compatible schema cases. +4. Add concurrency/idempotency tests for filesystem writes and cancellation + boundaries that can be exercised without external services. +5. Make the risk map exhaustive for production modules and map every public + option, persisted schema, research metric, and environment-bound path to at + least one mandatory or conditional test owner. -### Wave D — property, mutation, and regression depth +Gate 2 requires repository coverage of at least 75% line and 60% branch, +mature CPU coverage of at least 92% line and 80% branch, 100% coverage of the +documented critical vertical slices, and no regression in the Gate 1 targets. +The complete lane must remain below ten minutes per Python version and produce +JUnit, coverage, package, repeat, mutation, and normalized trend evidence. + +Gate 2 permits ordinary new feature work to resume under the per-PR contract. + +### Wave C — property, mutation, determinism, and regression depth 1. Add property tests for metric bounds, permutation/label invariance, serialization round trips, architecture selection, and configuration @@ -207,10 +245,12 @@ Wave B exit permits ordinary feature work to resume under the per-PR contract. parser, selection, checkpoint, reporting, and evaluation helpers. 3. Require at least 80% mutation score for the selected mature scope, with changed contract code killing every applicable generated mutant. -4. Run deterministic ordering and hash-seed repeats; publish trends without - silently retrying failures. +4. Run deterministic ordering, fixed-seed replay, and multiple hash-seed + repeats; publish trends without silently retrying failures. +5. Track duration by test and marker, fail on unowned slow-test growth, and + keep quarantines time-bounded, issue-linked, and absent from critical paths. -### Wave E — conditional and release evidence +### Wave D — conditional and release evidence 1. Require fresh successful model-download, external-evaluation, network, and operator-UI evidence for releases and for PRs that modify those mappings. @@ -220,12 +260,34 @@ Wave B exit permits ordinary feature work to resume under the per-PR contract. an explicit time-bounded waiver; do not weaken the mandatory CPU gate. 4. Retain exact commit, dependency lock, environment, result, and artifact provenance for every conditional run. +5. Replace closed umbrella-issue links in exclusion records with durable gate + documentation plus an active owner/tracker item only when work or a waiver + remains outstanding. -## Merge-train resumption gate +### Delivery sequence -The paused contributor merge train resumes only when: +The program should be delivered as bounded PRs, each green and exact-head +audited before the next begins: -- Wave A is merged and green on the current `main` head; +1. policy truth reset and exhaustive traceability schema; +2. research/numerical property contracts; +3. loader/device/architecture negative and mutation contracts; +4. checkpoint and destructive-operation reliability; +5. remote/CLI/provider safety and failure behavior; +6. evaluation/report/telemetry research-output contracts; +7. installed-package offline vertical slices; +8. conditional evidence freshness and runner/waiver reconciliation. + +Each PR must publish before/after module coverage, changed-line coverage, killed +mutants where applicable, runtime delta, and the exact tests that demonstrate +the protected behavior. Percentage-only changes without a named contract do +not advance a gate. + +## Merge-train and new-work gates + +The paused contributor merge train resumes only when Gate 1 is met and: + +- the completed foundation remains green on the current `main` head; - required CPU, package, quality-depth, and supply-chain jobs pass; - repository, mature-scope, changed-line, mutation, and warning floors pass; - no unresolved flaky-test quarantine affects the touched module; @@ -234,6 +296,11 @@ The paused contributor merge train resumes only when: - the next PR is re-audited at its exact head and includes the per-PR evidence above, with maintainer courtesy tests added for the current legacy queue. +Ordinary new feature work resumes only when Gate 2 is met on `main`. Security, +correctness, data-loss, and test-infrastructure defects discovered during the +program may be fixed before that point, but they must remain narrowly scoped +and include a reproducing regression test. + ## Milestones Completed baseline: @@ -247,11 +314,11 @@ Completed baseline: Next program: -7. Wave A policy/evidence lock and merge-train entry gate. -8. Wave B high-risk boundary campaign and ordinary-work entry gate. -9. Wave C offline integration and failure atomicity. -10. Wave D property, mutation, repeat, and regression depth. -11. Wave E conditional environment and release-evidence freshness. +7. Completed foundation policy/evidence lock. +8. Wave B1 high-consequence CPU contracts and legacy-train entry Gate 1. +9. Wave B2 offline vertical slices and ordinary-work entry Gate 2. +10. Wave C property, mutation, determinism, and regression depth. +11. Wave D conditional environment and release-evidence freshness. ## Metrics and reporting