diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index 5c06010..2a569e3 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -8,6 +8,8 @@ Phase: construction Gate 2 canonical code commit: `fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56` +Current post-merge baseline commit: `eab67c993d53b88ed9168096d4de20e8ee43a374` + The mandatory offline CPU selection passes on Python 3.10-3.12 with 1,541 tests, 9 conditionally deselected tests, no skips, and no warnings. The slowest hosted pytest selection is Python 3.12 at 136.33 seconds; its complete CI job is @@ -28,11 +30,25 @@ https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039. CUDA, bitsandbytes, MPS, MLX, and remote execution were not selected and therefore have no fresh support evidence or support claim. -The foundation, Wave A, Gate 1, and Gate 2 now establish a sane, -evidence-producing baseline. The contributor merge train and ordinary feature -work may proceed one exact-head PR at a time under the per-PR test contract. -Wave C and Wave D continue to deepen property, mutation, determinism, and real -environment evidence without weakening the mandatory CPU baseline. +The foundation, Wave A, Gate 1, and Gate 2 establish a sane, +evidence-producing baseline. The legacy contributor merge train proceeded one +exact-head PR at a time under the per-PR test contract. Wave C and Wave D deepen +property, mutation, determinism, and real-environment evidence without weakening +the mandatory CPU baseline. + +After the Gate 2 contributor integrations, the mandatory suite contains 1,585 +passing tests, 9 conditional deselections, and 2 documented expected failures. +Repository coverage is 75.80% statements and 61.89% branches; mature CPU scope +is 92.89% statements and 81.72% branches. All seven jobs at the current +post-merge commit passed: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31872157242. + +The operator has now placed ordinary feature work back behind a testing-depth +gate. Gate 2 remains the verified floor, but it is no longer sufficient by +itself to start unrelated feature work. The Gate 3 plan below must first be +adopted and its required work completed. Correctness, security, data-loss, and +test-infrastructure repairs remain permitted when narrowly scoped and paired +with reproducing tests. ## Reasoning @@ -239,7 +255,9 @@ documented critical vertical slices, and no regression in the Gate 1 targets. The complete lane must remain below ten minutes per Python version and produce JUnit, coverage, package, repeat, mutation, and normalized trend evidence. -Gate 2 permits ordinary new feature work to resume under the per-PR contract. +Gate 2 originally permitted ordinary new feature work to resume under the +per-PR contract. The later operator Gate 3 below supersedes that release +decision. ### Wave C — property, mutation, determinism, and regression depth @@ -255,6 +273,139 @@ Gate 2 permits ordinary new feature work to resume under the per-PR contract. 5. Track duration by test and marker, fail on unowned slow-test growth, and keep quarantines time-bounded, issue-linked, and absent from critical paths. +### Gate 3 — significant testing depth before new feature work + +Gate 3 converts Wave C from an open-ended improvement direction into the next +mandatory delivery gate. Test count is not a success metric. Each increment +must protect a named behavior, fail under a deliberate implementation defect, +and improve at least one of oracle strength, failure-path coverage, mutation +resistance, environment evidence, or deterministic replay. + +#### Quantitative exit criteria + +| Measure | Current | Gate 3 minimum | +|---|---:|---:| +| Repository statements | 75.80% | 80.00% | +| Repository branches | 61.89% | 65.00% | +| Mature CPU statements | 92.89% | 94.00% | +| Mature CPU branches | 81.72% | 84.00% | +| Changed executable lines | 90.00% floor | 95.00% floor | +| Selected mutation score | 91.31% on narrow scope | at least 85% on expanded scope | +| Unexpected warnings | 0 | 0 | +| Active flaky quarantines | 0 | 0 | +| Software conditional evidence age | at most 8 days | at most 8 days | + +The expanded mutation scope must include core mutation math, architecture and +loader decisions, persistence/atomicity, evaluation/report serialization, and +at least one service-orchestration state machine. Every applicable mutant on +new or changed pure contract logic must be killed; the aggregate 85% floor is +not permission to leave changed behavior weakly specified. + +The complete mandatory lane must remain under ten minutes per Python version. +If deeper campaigns exceed that budget, deterministic PR selections remain +blocking while the complete mutation, repeat, and conditional campaigns run as +required exact-head or scheduled gates. A failed required campaign may not be +converted to green by a retry. + +#### Risk-ranked work packages + +1. **Numerical oracles and research invariants.** Add property and metamorphic + tests for projection idempotence, orthogonality, norm bounds, permutation and + label invariance, dtype/tolerance behavior, singular and non-finite inputs, + deterministic seeds, and serialization stability. Use independent reference + calculations for small tensors instead of asserting against the production + helper's own intermediate values. +2. **Mutation and architecture decisions.** Expand selective mutation into + refusal-direction selection, architecture traversal, device/dtype and + quantization decisions, evaluation routing, and report construction. Add + missing negative cases before accepting a surviving mutant as equivalent. +3. **Failure injection and destructive operations.** Exercise partial writes, + fsync/replace failures, corrupt or truncated checkpoints, cancellation at + each state transition, cleanup failures, idempotent retry, concurrent writers, + and save/reload invariants. No test may risk operator data; use isolated + temporary filesystems and fault-injected adapters. +4. **Orchestration, services, and UI decision seams.** Extract only the minimal + clock, transport, process, filesystem, and input/output seams needed to test + BESTIARY sync, model catalog resolution, watchtower scheduling, tournament + state, interactive flows, and UI construction deterministically. Real + networking and UI launch remain in conditional gates. +5. **Tiny-runtime and quantized semantics.** Extend the pinned tiny-model slice + through model mutation, norm restoration, checkpoint round trip, evaluation, + and report generation. Add integer/quantized storage semantics, tied/shared + weights, unsupported layouts, and explicit numerical-loss expectations before + integrating FP8/NVFP4 or large-model offload work. +6. **Evidence integrity and duration ownership.** Record per-test duration, + marker, seed/order, exact candidate SHA, dependency lock, coverage delta, + mutation result, and conditional environment. Add policy tests that reject + stale, empty, skipped, or wrong-SHA evidence and unowned slow-test growth. + +#### Coverage targets for the largest current gaps + +These targets apply to CPU-testable decision logic, not to real accelerator, +model, service, or browser execution. Where a module mixes both, extract a +small pure seam and measure it rather than mocking away the behavior under test. + +| Surface | Current statements / branches | Gate 3 intent | +|---|---:|---| +| `abliterate.py` | 45.78% / 37.31% | at least 52% / 45% with mutation/math contracts | +| `lora_ablation.py` | 7.19% / 0% | at least 70% / 55% for CPU-testable decisions | +| `tourney.py` | 9.28% / 0% | at least 55% / 40% for lifecycle/state transitions | +| `bestiary_sync.py` | 0% / 0% | at least 80% / 65% using a loopback/fake transport | +| `models_client.py` | 19.54% / 3.57% | at least 80% / 65% for resolution/error contracts | +| `watchtower.py` | 19.19% / 0% | at least 70% / 55% with injected clock/client seams | +| `interactive.py` | 8.47% / 0% | at least 70% / 55% with scripted I/O contracts | +| `local_ui.py` | 40.87% / 15.22% | at least 70% / 55% for launch decisions and failures | +| `ui_watchtower.py` | 0% / 0% | cover construction/callback contracts; real UI stays conditional | +| `evaluation/heretic_eval.py` | 42.46% / 30.51% | at least 70% / 55% for CPU-testable evaluation logic | + +#### Bounded delivery sequence + +Each item is an independently reviewable PR and must be exact-head audited +before merge. Later items may be split further; they may not be combined into a +single repository-wide coverage PR. + +1. Adopt Gate 3 policy, refresh current measurements, and add duration-budget + enforcement without changing production behavior. +2. Add numerical/reference-oracle and metamorphic contracts; expand mutation to + the corresponding pure math. +3. Add loader, architecture, dtype, quantization, and shared-weight decision + contracts with mutation evidence. +4. Add checkpoint failure injection, concurrency, atomicity, and retry contracts. +5. Add BESTIARY/model-client/watchtower transport and scheduler state contracts. +6. Add tournament, interactive, and UI decision-seam contracts. +7. Extend the installed tiny-model vertical slice and quantized-storage + semantics, including the exact Float-to-integer restoration regression. +8. Reconcile conditional software evidence, then run or explicitly waive each + unavailable hardware/remote environment under the claim policy. +9. Publish the Gate 3 execution, coverage, mutation, repeat, duration, and + conditional-evidence reports at one canonical `main` commit. + +#### Per-PR acceptance evidence + +Every Gate 3 PR must publish: + +- the exact base and head commits and a clean exact-head audit; +- named contracts and the defect/mutant each new test detects; +- focused and mandatory-suite results with warning and skip counts; +- before/after repository and touched-module line/branch coverage; +- changed-line coverage and exact-base no-regression results; +- mutation results when a selected surface changes; +- runtime delta and slowest affected tests; +- conditional evidence or an explicit statement that no mapped environment was + touched; +- all required hosted checks green and no unresolved review thread. + +#### New-work release decision + +Ordinary feature work may resume only after items 1–7 are merged, all Gate 3 +quantitative criteria pass on one canonical post-merge commit, and the final +reports are reviewed. Item 8 may use a time-bounded waiver only when a required +runner or credential is unavailable; that waiver blocks support claims for the +environment and may not weaken CPU gates. Item 9 records the decision and exact +evidence. Outstanding feature PRs may be audited meanwhile, but only correctness, +security, data-loss, CI/test-infrastructure fixes, and the Gate 3 testing PRs are +merge candidates during the pause. + ### Wave D — conditional and release evidence 1. Require fresh successful model-download, external-evaluation, network, and