# OBLITERATUS master test plan

Date: 2026-08-14
Owner: maintainers
Phase: construction

## Reasoning

1. **Scope** — package/public CLI behavior, core ablation and analysis logic,
   loader/device/quantization/architecture boundaries, offline pipeline and
   checkpoint flows, evaluation/reporting/telemetry, packaging, and CI policy.
2. **Risk priority** — research-metric correctness, destructive checkpoint
   operations, device/dtype selection, model architecture adaptation, refusal
   decisions, installed-distribution behavior, and error handling.
3. **Coverage strategy** — risk-weighted unit/boundary tests, tiny offline
   integration fixtures, changed-line gates, branch coverage, property tests,
   selective mutation, and separate conditional backend gates.
4. **Resources** — Python 3.10–3.12 CPU runners for mandatory CI; optional
   accelerator/backend runners and least-privileged service credentials only for
   explicit conditional jobs.
5. **Quality criteria** — all mandatory checks green, no unexpected warnings,
   retained machine-readable evidence, installed artifacts verified, documented
   coverage thresholds met, and no unresolved review threads.

## Objectives and items

The plan validates the source package, both distribution formats, supported
Python versions, public CLI, scripts named in the canonical gate, and the test
and workflow configuration itself. It aims to expose behavioral regressions and
unsupported environment assumptions before merge.

## In scope

- deterministic pure/unit and boundary-contract tests;
- negative and error-path behavior;
- offline tiny-model integration and save/reload contracts;
- evaluation, reporting, telemetry, and research-metric invariants;
- packaging metadata and installed CLI/import behavior;
- CI workflow policy, evidence, supply-chain checks, and explicit markers;
- conditional accelerator, optional backend, network, and remote workflows.

## Out of scope for the mandatory PR job

- large model downloads or benchmark-quality model runs;
- credentials, remote execution, or production services;
- CUDA, MPS, MLX, and bitsandbytes availability;
- performance claims requiring dedicated hardware.

These remain in scope for conditional gates and release evidence.

## Approach and deliverables

| Layer | Required timing | Deliverables |
|---|---|---|
| CPU unit/boundary | every PR | pytest/JUnit, line+branch coverage, warning result |
| package | every PR | sdist/wheel, metadata check, clean install smoke |
| offline integration | every PR after stabilization | tiny-model pipeline, checkpoint/save/reload, eval/report |
| quality depth | PR or scheduled by cost | property/repeat tests, selected mutation report |
| conditional environment | manual/scheduled/release | backend-specific JUnit/log evidence |

## Environment and data

The mandatory environment uses a clean checkout, an isolated Python environment,
CPU-only execution, no service credentials, no network/model downloads, and
synthetic deterministic fixtures. Integration fixtures must be small enough for
the repository and may not embed third-party model weights without documented
license/provenance.

## Entry and exit criteria

Entry:

- clean base commit and dependency set;
- test scope and marker declared;
- acceptance criteria mapped to tests;
- external/hardware prerequisites separated from default CI.

Exit for each issue:

- implementation and relevant negative/boundary tests merged;
- complete required suite and hosted CI green;
- branch/line/changed-line thresholds for that wave met;
- generated evidence retained and reviewed;
- no unresolved issue/PR feedback;
- documentation and tracker state reconciled.

Program exit:

- Phases 0–2 are merged before ordinary feature work resumes;
- all planned waves are delivered or explicitly accepted as conditional release
  gates with a runnable workflow and owner;
- CPU-testable code reaches the mature 80% line / 75% branch target, or every
  remaining exclusion has a documented environment-bound rationale.

## Milestones

1. Evidence, strict pytest policy, workflow validation.
2. Installed distributions and supply-chain evidence.
3. Boundary contracts and 55% repository line coverage.
4. Offline integration and 60% repository line coverage.
5. Research-integrity, property, repeat, and selected mutation gates.
6. Conditional backend/service workflows and mature CPU-testable targets.

## Metrics and reporting

- tests passed/failed/skipped by marker and Python version;
- line and branch coverage overall and by critical module;
- changed-line coverage;
- warning count/category;
- distribution validation/install results;
- repeat/flaky failures and mutation score for selected modules;
- conditional workflow freshness and result.

## References

- @.aiwg/requirements/UC-testing-quality-program.md
- @.aiwg/architecture/sketch-testing-quality-program.md
- @.aiwg/risks/risks-testing-quality-program.md
- @.aiwg/security/screening-testing-quality-program.md
- @.aiwg/working/issue-planner/research-synthesis.md
- @WORKSPACE.md
- @pyproject.toml
- @.github/workflows/ci.yml
- @tests/
