OBLITERATUS/.aiwg/testing/master-test-plan.md

OBLITERATUS master test plan

Date: 2026-08-16 Owner: maintainers Phase: construction

Current verified baseline

Gate 3 increment 1 canonical commit: 256c39ea6a492749a4db44146830e9d78fd3fed8

Gate 3 increment 2 canonical commit: 2c2b38b501f94af46ffac3a772f7bf9475154dc5 (PR #102, rebase merged)

Gate 3 increment 3 canonical commit: c1b34503ddd3cb797e5d70671c47afcabfe73832 (PR #103, rebase merged)

Gate 3 increment 4 canonical commit: d7d2ad566af778b1d315616bbea8b8a3c4dd2191 (PR #104, rebase merged)

Gate 3 increment 5 canonical commit: aa182cc44883890c019c393f17602e2e2702e7d7 (PR #105, rebase merged)

Gate 3 increment 6 canonical commit: 9683e0be4d892e6a6a134b45a065e591e139da3c (PR #108, signed fast-forward)

Gate 3 increment 7 canonical commit: b961623513a0e137b959f608ce31511d59e85888 (PR #109, signed fast-forward)

Gate 3 increment 8 canonical commit: 42b30f7e5b8ee596b3b0b039b10af16f01deec1e (PR #111, signed fast-forward)

Current canonical AIWG and CI-contract baseline: e8ac3b65670d696ed5b68f06adf14706bc2ff865 (PR #107, rebase merged)

The latest canonical mandatory offline CPU lane passes 2,119 tests on Python 3.12 with 9 conditional tests deselected by policy, no failures, errors, skips, or unexpected warnings, and 83.49% statement / 71.04% branch coverage. Mature CPU-testable coverage is 94.02% / 84.54%. The canonical mutation campaign kills 1,844 of 2,057 mutants (89.65%), with 208 survivors and 5 timeouts, while 702 tests pass in each of three file orders and hash seeds with no flakes. Package, Windows checkpoint, supply-chain, and all other hosted jobs passed on attempt 1: https://github.com/elder-plinius/OBLITERATUS/actions/runs/31952112615.

Gate 3 increment 8 is canonical at signed head 42b30f7e5b8ee596b3b0b039b10af16f01deec1e. Fresh pinned-model download, external-evaluation, loopback-network, operator-UI, and final-summary evidence passes at that exact head in runs 31950226930 and 31952128889. Titan CUDA/bitsandbytes and Mutsu MPS/MLX operator probes pass at signed implementation head fa233fd8c97d2a1463141535f2a288b264abe93b. GitHub runner registration and remote execution remain time-bounded no-support/no-correctness/no-performance waivers through 2026-09-15 under issue #110. Details are in gate3-increment-8-report.md and gate3-final-report.md.

The foundation, Wave A, Gate 1, and Gate 2 establish a sane, evidence-producing baseline. The legacy contributor merge train proceeded one exact-head PR at a time under the per-PR test contract. Wave C and Wave D deepen property, mutation, determinism, and real-environment evidence without weakening the mandatory CPU baseline.

The operator has now placed ordinary feature work back behind a testing-depth gate. Gate 3 is adopted and items 1–8 are canonical; the final report must pass exact-head review and canonical reconciliation before unrelated feature work resumes. Correctness, security, data-loss, and test-infrastructure repairs remain permitted when narrowly scoped and paired with reproducing tests.

Reasoning

1. Scope — package/public CLI behavior, core ablation and analysis logic, loader/device/quantization/architecture boundaries, offline pipeline and checkpoint flows, evaluation/reporting/telemetry, packaging, and CI policy.
2. Risk priority — research-metric correctness, destructive checkpoint operations, device/dtype selection, model architecture adaptation, refusal decisions, installed-distribution behavior, and error handling.
3. Coverage strategy — risk-weighted unit/boundary tests, tiny offline integration fixtures, changed-line gates, branch coverage, property tests, selective mutation, and separate conditional backend gates.
4. Resources — Python 3.10–3.12 CPU runners for mandatory CI; optional accelerator/backend runners and least-privileged service credentials only for explicit conditional jobs.
5. Quality criteria — all mandatory checks green, no unexpected warnings, retained machine-readable evidence, installed artifacts verified, documented coverage thresholds met, and no unresolved review threads.

Objectives and items

The plan validates the source package, both distribution formats, supported Python versions, public CLI, scripts named in the canonical gate, and the test and workflow configuration itself. It aims to expose behavioral regressions and unsupported environment assumptions before merge.

In scope

• deterministic pure/unit and boundary-contract tests;
• negative and error-path behavior;
• offline tiny-model integration and save/reload contracts;
• evaluation, reporting, telemetry, and research-metric invariants;
• packaging metadata and installed CLI/import behavior;
• CI workflow policy, evidence, supply-chain checks, and explicit markers;
• conditional accelerator, optional backend, network, and remote workflows.

Out of scope for the mandatory PR job

• large model downloads or benchmark-quality model runs;
• credentials, remote execution, or production services;
• CUDA, MPS, MLX, and bitsandbytes availability;
• performance claims requiring dedicated hardware.

These remain in scope for conditional gates and release evidence.

Approach and deliverables

Layer	Required timing	Deliverables
CPU unit/boundary	every PR	pytest/JUnit, line+branch coverage, warning result
package	every PR	sdist/wheel, metadata check, clean install smoke
offline integration	every PR after stabilization	tiny-model pipeline, checkpoint/save/reload, eval/report
quality depth	PR or scheduled by cost	property/repeat tests, selected mutation report
conditional environment	manual/scheduled/release	backend-specific JUnit/log evidence

Environment and data

The mandatory environment uses a clean checkout, an isolated Python environment, CPU-only execution, no service credentials, no network/model downloads, and synthetic deterministic fixtures. Integration fixtures must be small enough for the repository and may not embed third-party model weights without documented license/provenance.

Entry and exit criteria

Entry:

• clean base commit and dependency set;
• test scope and marker declared;
• acceptance criteria mapped to tests;
• external/hardware prerequisites separated from default CI.

Exit for each issue:

• implementation and relevant negative/boundary tests merged;
• complete required suite and hosted CI green;
• branch/line/changed-line thresholds for that wave met;
• generated evidence retained and reviewed;
• no unresolved issue/PR feedback;
• documentation and tracker state reconciled.

Program exit:

• Phases 0–2 are merged before ordinary feature work resumes;
• all planned waves are delivered or explicitly accepted as conditional release gates with a runnable workflow and owner;
• CPU-testable code reaches the mature 90% line / 78% branch target, or every remaining exclusion has a documented environment-bound rationale.

Per-PR test contract

Every behavior-changing PR must include tests that fail without the change and exercise the smallest meaningful production boundary. Coverage-only assertions or tests that merely inspect source text do not satisfy this contract.

Change	Required evidence
Bug fix	focused regression reproducing the original failure plus the relevant suite
Public option or configuration	default, explicit value, invalid/boundary value, propagation, and runtime effect
CLI change	parser contract, dispatch contract, and local/remote propagation when both exist
Model/architecture adapter	projection discovery, shape contract, missing/unsupported structure, and tied/shared-weight behavior where relevant
Evaluation or reporting	numerical invariant, empty/singleton boundary, output/schema contract, and deterministic serialization
Checkpoint or persistence	success, interruption/failure atomicity, reload/round-trip, and cleanup behavior
Hardware, network, or remote path	deterministic fake-based contract test plus the mapped conditional gate
Research claim or metric	provenance/interpretation contract and a test preventing silent semantic drift

New or modified code must maintain at least 95% changed-line coverage. A touched module may not lose line or branch coverage unless the PR documents why the measurement is misleading and adds equivalent contract or conditional evidence. No flaky-test rerun may turn a required failure into success.

Testing-first improvement program

Completed foundation — policy and evidence lock

The hermetic Python 3.10–3.12 matrix, installed wheel/sdist smoke, exact-base coverage comparison, 90% changed-line floor, touched-module no-regression gate, risk map, quality policy, repeat and mutation evidence, conditional workflows, and supply-chain job are merged and green. These controls are the floor for all subsequent waves and may only be tightened.

Wave B1 — high-consequence CPU contracts (Gate 1)

Deliver small, reviewable test or test-enabling PRs in this order:

1. Research and numerical correctness — refusal direction selection, activation aggregation, empty/singleton behavior, dtype/tolerance handling, deterministic metric serialization, and error signaling. Prefer invariant and property assertions over frozen implementation details.
2. Loader and architecture boundaries — device maps, memory budgets, quantization configuration, cache behavior, tied/shared weights, missing projections, unsupported architectures, and CPU fallback behavior.
3. Persistence and destructive operations — checkpoint atomicity, interrupted/partial writes, corrupt input, overwrite refusal, cleanup, save/reload round trips, and idempotent retry.
4. Public and remote contracts — CLI/config defaults and invalid values, local/remote propagation, shell-safe command construction, cancellation, result synchronization, malformed provider responses, and timeout paths.
5. Evaluation and research outputs — causal/classifier routing, dataset bounds, report schema, telemetry aggregation, provenance fields, deterministic ordering, and explicit failure rather than silent fallback.

Tests must exercise negative, boundary, and cross-component behavior and must fail when the protected contract is deliberately broken. External model, service, filesystem, clock, process, or hardware edges may be faked; the decision logic and schemas under test may not be replaced by mocks. Minimum production refactors are allowed only to extract deterministic seams, and each extracted helper becomes part of the mutation/property-test scope.

Gate 1 requires all of the following on exact-head hosted CI:

• repository coverage at least 70% line and 55% branch;
• mature CPU scope at least 90% line and 78% branch;
• the critical-module targets below met or exceeded;
• selective mutation expanded beyond configuration/policy scripts to at least three high-consequence pure contract surfaces, with at least 75% killed;
• zero unexpected warnings, active flaky quarantines, P0/P1 defects, or unresolved review threads;
• focused contributor tests complete in under two minutes and the complete mandatory lane remains under ten minutes per Python version.

Surface	Verified baseline	Wave B target
Repository	70.20% line / 57.41% branch	≥70% / ≥55%
Mature CPU scope	92.23% / 80.36%	≥90% / ≥78%
cli.py	97.93% / 89.15%	preserve ≥97% / raise to ≥85%
config.py	100.00% / 90.00%	preserve ≥98% / raise to ≥90%
device.py	100.00% / 95.00%	preserve ≥96% / ≥92%
architecture_profiles.py	97.52% / 92.31%	≥92% / ≥85%
models/loader.py	91.01% / 88.14%	≥90% / ≥87%
evaluation/evaluator.py	100.00% / 97.37%	≥85% / ≥80%
reporting/report.py	100.00% / 98.21%	≥90% / ≥80%
telemetry.py	91.62% / 81.88%	≥80% / ≥75%

Large runtime-bound modules must be split conceptually into CPU-testable decision logic and conditional execution. CPU-testable helpers extracted from abliterate.py, informed_pipeline.py, remote.py, watchtower.py, and related orchestrators target at least 90% line / 85% branch. Real model, accelerator, service, UI, and remote execution remains covered by the mapped conditional gates instead of being disguised by mocks.

Gate 1 permits the legacy contributor merge train to resume, one exact-head PR at a time with courtesy tests for the already-open queue. It does not permit ordinary new feature work.

Wave B2 — offline vertical slices and reliability (Gate 2)

1. Add a deterministic tiny-model vertical slice through CLI/config → loader/device → pipeline → checkpoint → evaluator → report, exercising the installed package rather than only source-tree imports.
2. Add a second vertical slice for failure recovery: interrupted mutation, partial/corrupt checkpoint, retry, atomic replacement, and cleanup.
3. Add deterministic community-contribution and evaluation/report round trips, including malformed and forward-compatible schema cases.
4. Add concurrency/idempotency tests for filesystem writes and cancellation boundaries that can be exercised without external services.
5. Make the risk map exhaustive for production modules and map every public option, persisted schema, research metric, and environment-bound path to at least one mandatory or conditional test owner.

Gate 2 requires repository coverage of at least 75% line and 60% branch, mature CPU coverage of at least 92% line and 80% branch, 100% coverage of the documented critical vertical slices, and no regression in the Gate 1 targets. The complete lane must remain below ten minutes per Python version and produce JUnit, coverage, package, repeat, mutation, and normalized trend evidence.

Gate 2 originally permitted ordinary new feature work to resume under the per-PR contract. The later operator Gate 3 below supersedes that release decision.

Wave C — property, mutation, determinism, and regression depth

1. Add property tests for metric bounds, permutation/label invariance, serialization round trips, architecture selection, and configuration normalization.
2. Expand selective mutation from configuration and policy scripts to pure parser, selection, checkpoint, reporting, and evaluation helpers.
3. Require at least 80% mutation score for the selected mature scope, with changed contract code killing every applicable generated mutant.
4. Run deterministic ordering, fixed-seed replay, and multiple hash-seed repeats; publish trends without silently retrying failures.
5. Track duration by test and marker, fail on unowned slow-test growth, and keep quarantines time-bounded, issue-linked, and absent from critical paths.

Gate 3 — significant testing depth before new feature work

Gate 3 converts Wave C from an open-ended improvement direction into the next mandatory delivery gate. Test count is not a success metric. Each increment must protect a named behavior, fail under a deliberate implementation defect, and improve at least one of oracle strength, failure-path coverage, mutation resistance, environment evidence, or deterministic replay.

Quantitative exit criteria

Measure	Current	Gate 3 minimum
Repository statements	83.49% item-8 canonical	80.00%
Repository branches	71.04% item-8 canonical	65.00%
Mature CPU statements	94.02% item-8 canonical	94.00%
Mature CPU branches	84.54% item-8 canonical	84.00%
Changed executable lines	95.00% floor	95.00% floor
Selected mutation score	89.65% item-8 canonical; 1,844/2,057 killed	at least 85% on expanded scope
Unexpected warnings	0	0
Active flaky quarantines	0	0
Software conditional evidence age	same-day item-8 canonical evidence	at most 8 days

The expanded mutation scope must include core mutation math, architecture and loader decisions, persistence/atomicity, evaluation/report serialization, and at least one service-orchestration state machine. Every applicable mutant on new or changed pure contract logic must be killed; the aggregate 85% floor is not permission to leave changed behavior weakly specified.

The complete mandatory lane must remain under ten minutes per Python version. If deeper campaigns exceed that budget, deterministic PR selections remain blocking while the complete mutation, repeat, and conditional campaigns run as required exact-head or scheduled gates. A failed required campaign may not be converted to green by a retry.

Risk-ranked work packages

1. Numerical oracles and research invariants. Add property and metamorphic tests for projection idempotence, orthogonality, norm bounds, permutation and label invariance, dtype/tolerance behavior, singular and non-finite inputs, deterministic seeds, and serialization stability. Use independent reference calculations for small tensors instead of asserting against the production helper's own intermediate values.
2. Mutation and architecture decisions. Expand selective mutation into refusal-direction selection, architecture traversal, device/dtype and quantization decisions, evaluation routing, and report construction. Add missing negative cases before accepting a surviving mutant as equivalent.
3. Failure injection and destructive operations. Exercise partial writes, fsync/replace failures, corrupt or truncated checkpoints, cancellation at each state transition, cleanup failures, idempotent retry, concurrent writers, and save/reload invariants. No test may risk operator data; use isolated temporary filesystems and fault-injected adapters.
4. Orchestration, services, and UI decision seams. Extract only the minimal clock, transport, process, filesystem, and input/output seams needed to test BESTIARY sync, model catalog resolution, watchtower scheduling, tournament state, interactive flows, and UI construction deterministically. Real networking and UI launch remain in conditional gates.
5. Tiny-runtime and quantized semantics. Extend the pinned tiny-model slice through model mutation, norm restoration, checkpoint round trip, evaluation, and report generation. Add integer/quantized storage semantics, tied/shared weights, unsupported layouts, and explicit numerical-loss expectations before integrating FP8/NVFP4 or large-model offload work.
6. Evidence integrity and duration ownership. Record per-test duration, marker, seed/order, exact candidate SHA, dependency lock, coverage delta, mutation result, and conditional environment. Add policy tests that reject stale, empty, skipped, or wrong-SHA evidence and unowned slow-test growth.

Coverage targets for the largest current gaps

These targets apply to CPU-testable decision logic, not to real accelerator, model, service, or browser execution. Where a module mixes both, extract a small pure seam and measure it rather than mocking away the behavior under test.

Surface	Current statements / branches	Gate 3 intent
abliterate.py	50.91% / 38.03% candidate	at least 52% / 45% with mutation/math contracts
lora_ablation.py	9.57% / 0%	at least 70% / 55% for CPU-testable decisions
tourney.py	12.01% / 0%	at least 55% / 40% for lifecycle/state transitions
bestiary_sync.py	96.55% / 100% item-5 candidate	at least 80% / 65% using a loopback/fake transport
models_client.py	92.86% / 92.00% item-5 candidate	at least 80% / 65% for resolution/error contracts
watchtower.py	96.99% / 92.22% item-5 candidate	at least 70% / 55% with injected clock/client seams
interactive.py	10.49% / 0%	at least 70% / 55% with scripted I/O contracts
local_ui.py	48.15% / 15.22%	at least 70% / 55% for launch decisions and failures
ui_watchtower.py	0% / 0%	cover construction/callback contracts; real UI stays conditional
evaluation/heretic_eval.py	45.11% / 27.12%	at least 70% / 55% for CPU-testable evaluation logic

Bounded delivery sequence

Each item is an independently reviewable PR and must be exact-head audited before merge. Later items may be split further; they may not be combined into a single repository-wide coverage PR.

1. Completed at 256c39e. Adopt Gate 3 policy, refresh current measurements, and add duration-budget enforcement without changing production behavior. Exact canonical evidence: https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085.
2. Completed at 2c2b38b (PR #102). Add numerical/reference-oracle and metamorphic contracts; expand mutation to the corresponding pure math. Canonical evidence is recorded in gate3-increment-2-report.md.
3. Completed at c1b3450 (PR #103). Add loader, architecture, dtype, quantization, and shared-weight decision contracts with mutation evidence. Canonical evidence is recorded in gate3-increment-3-report.md.
4. Completed at d7d2ad5 (PR #104). Add checkpoint failure injection, concurrency, atomicity, retry, cleanup-ownership, and Windows portability contracts. Canonical evidence is recorded in gate3-increment-4-report.md.
5. Completed at aa182cc (PR #105). Add BESTIARY/model-client/watchtower transport and scheduler state contracts. Exact evidence is recorded in gate3-increment-5-report.md and the canonical PR follow-up.
6. Completed at 9683e0b (PR #108). Add tournament, interactive, and UI decision-seam contracts. Canonical run 31941334488 passed all eight required checks; exact evidence is recorded in gate3-increment-6-report.md and the canonical PR reconciliation comment.
7. Completed at b961623 (PR #109). Extend the installed tiny-model vertical slice and quantized-storage semantics, including the exact Float-to-integer restoration regression. The canonical head reaches 94.02% mature CPU statements and 84.54% branches, raises the immutable floors to 94% / 84%, and passes all eight hosted checks.
8. Completed at 42b30f7 (PR #111). Reconcile fresh software evidence, run exact-head Titan/Mutsu operator probes, and explicitly waive unavailable scheduled runner/remote lanes under the claim policy. The signed commits were preserved by fast-forward integration.
9. In progress. Publish the Gate 3 execution, coverage, mutation, repeat, duration, and conditional-evidence report, then exact-head audit, signed merge, and canonically reconcile the report-only commit.

Per-PR acceptance evidence

Every Gate 3 PR must publish:

• the exact base and head commits and a clean exact-head audit;
• named contracts and the defect/mutant each new test detects;
• focused and mandatory-suite results with warning and skip counts;
• before/after repository and touched-module line/branch coverage;
• changed-line coverage and exact-base no-regression results;
• mutation results when a selected surface changes;
• runtime delta and slowest affected tests;
• conditional evidence or an explicit statement that no mapped environment was touched;
• all required hosted checks green and no unresolved review thread.

New-work release decision

Ordinary feature work may resume only after items 1–7 are merged, all Gate 3 quantitative criteria pass on one canonical post-merge commit, and the final reports are reviewed. Item 8 may use a time-bounded waiver only when a required runner or credential is unavailable; that waiver blocks support claims for the environment and may not weaken CPU gates. Item 9 records the decision and exact evidence. Outstanding feature PRs may be audited meanwhile, but only correctness, security, data-loss, CI/test-infrastructure fixes, and the Gate 3 testing PRs are merge candidates during the pause.

Wave D — conditional and release evidence

1. Require fresh successful model-download, external-evaluation, network, and operator-UI evidence for releases and for PRs that modify those mappings.
2. Require CUDA/bitsandbytes, MPS, MLX, or remote evidence when the matching runner is configured and the affected surface changes.
3. If a runner is unavailable, block claims about that environment and record an explicit time-bounded waiver; do not weaken the mandatory CPU gate.
4. Retain exact commit, dependency lock, environment, result, and artifact provenance for every conditional run.
5. Replace closed umbrella-issue links in exclusion records with durable gate documentation plus an active owner/tracker item only when work or a waiver remains outstanding.

Delivery sequence

The program should be delivered as bounded PRs, each green and exact-head audited before the next begins:

1. policy truth reset and exhaustive traceability schema;
2. research/numerical property contracts;
3. loader/device/architecture negative and mutation contracts;
4. checkpoint and destructive-operation reliability;
5. remote/CLI/provider safety and failure behavior;
6. evaluation/report/telemetry research-output contracts;
7. installed-package offline vertical slices;
8. conditional evidence freshness and runner/waiver reconciliation.

Each PR must publish before/after module coverage, changed-line coverage, killed mutants where applicable, runtime delta, and the exact tests that demonstrate the protected behavior. Percentage-only changes without a named contract do not advance a gate.

Merge-train and new-work gates

The paused contributor merge train resumes only when Gate 1 is met and:

• the completed foundation remains green on the current main head;
• required CPU, package, quality-depth, and supply-chain jobs pass;
• repository, mature-scope, changed-line, mutation, and warning floors pass;
• no unresolved flaky-test quarantine affects the touched module;
• conditional mappings remain valid, with unavailable untouched environments explicitly non-blocking;
• the next PR is re-audited at its exact head and includes the per-PR evidence above, with maintainer courtesy tests added for the current legacy queue.

Ordinary new feature work resumes only when Gate 2 is met on main. Security, correctness, data-loss, and test-infrastructure defects discovered during the program may be fixed before that point, but they must remain narrowly scoped and include a reproducing regression test.

Milestones

Completed baseline:

1. Evidence, strict pytest policy, workflow validation.
2. Installed distributions and supply-chain evidence.
3. Boundary contracts and 55% repository line coverage.
4. Offline integration and 60% repository line coverage.
5. Research-integrity, property, repeat, and selected mutation gates.
6. Conditional backend/service workflows and mature CPU-testable targets.

Completed testing-first gates:

7. Completed foundation policy/evidence lock.
8. Wave B1 high-consequence CPU contracts and legacy-train entry Gate 1.
9. Wave B2 offline vertical slices and ordinary-work entry Gate 2.

Next program:

10. Wave C property, mutation, determinism, and regression depth.
11. Wave D conditional environment and release-evidence freshness.

Metrics and reporting

• tests passed/failed/skipped by marker and Python version;
• line and branch coverage overall and by critical module;
• changed-line coverage;
• warning count/category;
• distribution validation/install results;
• repeat/flaky failures and mutation score for selected modules;
• conditional workflow freshness and result.

References

• @.aiwg/requirements/UC-testing-quality-program.md
• @.aiwg/architecture/sketch-testing-quality-program.md
• @.aiwg/risks/risks-testing-quality-program.md
• @.aiwg/security/screening-testing-quality-program.md
• @.aiwg/working/issue-planner/research-synthesis.md
• @WORKSPACE.md
• @pyproject.toml
• @.github/workflows/ci.yml
• @tests/