diff --git a/.aiwg/architecture/sketch-testing-quality-program.md b/.aiwg/architecture/sketch-testing-quality-program.md index 59341db..ae596cb 100644 --- a/.aiwg/architecture/sketch-testing-quality-program.md +++ b/.aiwg/architecture/sketch-testing-quality-program.md @@ -16,14 +16,18 @@ produce retained evidence and never silently convert a required failure into a success. Tool versions remain project-controlled. The default gate must not access external services, inherit user model caches, or require credentials. -The initial coverage waves are complete: +The initial coverage waves and Gate 2 are complete: - measurement wave: preserve the 49% line floor and establish branch baseline; - boundary wave: repository line coverage at least 55%, changed lines at least 90%, and touched critical modules at least 70%; - integration wave: repository line coverage at least 60%; -- mature CPU-testable target: at least 80% line and 75% branch coverage, with - exclusions limited to documented conditional environment code. +- Gate 1: at least 70% repository line and 55% branch coverage, with at least + 90% line and 78% branch coverage for the mature CPU-testable scope; +- Gate 2: at least 75% repository line and 60% branch coverage, with at least + 92% line and 80% branch coverage for the mature CPU-testable scope and an + installed-package CLI/config-to-report vertical slice; +- exclusions remain limited to documented conditional environment code. Wave A locks that baseline by generating exact-base coverage in the same CI run, rejecting per-module line or branch regressions, validating a machine-readable @@ -31,3 +35,6 @@ source-to-test and conditional-gate graph, retaining normalized trend evidence for 90 days, and enforcing owner/issue/expiry requirements for repeated flaky tests. The exact thresholds and exclusion graph remain owned by `ci/test-quality-policy.json`; `ci/test-risk-map.json` owns test responsibility. +Software-only conditional evidence must match the candidate commit. A stale +evidence exception requires a reason, a canonical numeric OBLITERATUS issue, +and an expiry no more than 30 days away. diff --git a/.aiwg/requirements/UC-testing-quality-program.md b/.aiwg/requirements/UC-testing-quality-program.md index 0b76130..64a84cf 100644 --- a/.aiwg/requirements/UC-testing-quality-program.md +++ b/.aiwg/requirements/UC-testing-quality-program.md @@ -27,4 +27,13 @@ and a tiny offline model fixture. An operator can intentionally run clearly documented CUDA, bitsandbytes, MPS, MLX, model-download, network, and remote checks without weakening or surprising -the default pull-request gate. +the default pull-request gate. Software-only evidence records the exact +candidate commit; any stale-evidence exception is reasoned, issue-linked, and +expires within 30 days. + +## UC-TQ-06 — Maintainer verifies the installed vertical contract + +A maintainer can exercise a non-editable installed distribution with a local +deterministic tiny model through CLI/config parsing, loader/device selection, +pipeline execution, checkpoint save/reload, evaluation, and JSON/CSV report +generation without external services or accelerator hardware. diff --git a/.aiwg/risks/risks-testing-quality-program.md b/.aiwg/risks/risks-testing-quality-program.md index cf8a4c7..1d97c8b 100644 --- a/.aiwg/risks/risks-testing-quality-program.md +++ b/.aiwg/risks/risks-testing-quality-program.md @@ -10,3 +10,4 @@ | TQ-R6 | Mutation/property jobs exceed useful feedback time | medium | medium | target small pure modules and run depth gates separately | | TQ-R7 | Installed package behavior differs from checkout | medium | high | clean wheel and sdist installation smoke tests | | TQ-R8 | Numerical tests are device/dtype brittle | medium | high | invariant/tolerance contracts and separate backend-specific evidence | +| TQ-R9 | Conditional evidence is reused for a different commit | medium | high | candidate-SHA validation; exceptions require a reason, canonical issue, and expiry within 30 days | diff --git a/.aiwg/testing/coverage-report.md b/.aiwg/testing/coverage-report.md index e56de19..5e879ff 100644 --- a/.aiwg/testing/coverage-report.md +++ b/.aiwg/testing/coverage-report.md @@ -1,21 +1,25 @@ -# Gate 1 coverage report +# Gate 2 coverage report Date: 2026-08-15 Coverage format: coverage.py branch JSON v3 | Scope | Measured | Enforced floor | Status | |---|---:|---:|---| -| Repository statements | 70.20% | 70% | PASS | -| Repository branches | 57.41% | 55% | PASS | -| Mature CPU statements | 92.23% | 90% | PASS | -| Mature CPU branches | 80.36% | 78% | PASS | -| Changed executable lines | 100.00% | 90% | PASS | +| Repository statements | 75.35% | 75% | PASS | +| Repository branches | 61.49% | 60% | PASS | +| Mature CPU statements | 92.69% | 92% | PASS | +| Mature CPU branches | 81.47% | 80% | PASS | +| Changed executable lines | Not applicable (test/policy-only change) | 90% when production changes | PASS | -Critical changed surfaces are fully or strongly covered: evaluation baselines -100%/90.91%, evaluator 100%/97.37%, lm-eval adapter 100%/100%, report -100%/98.21%, and telemetry 91.62%/81.88% (statements/branches). +Gate 1 critical surfaces remain fully or strongly covered: evaluation +baselines 100%/90.91%, evaluator 100%/97.37%, lm-eval adapter 100%/100%, report +100%/98.21%, and telemetry 91.62%/81.88% (statements/branches). Gate 2 adds +96.53%/81.94% for Bayesian optimization, 94.90%/76.73% for informed pipeline, +and 100%/100% for sweep orchestration. The immutable floors are encoded in `ci/test-quality-policy.json`, pytest, the CI workflow, and `scripts/check_quality_policy.py`. Exact-base -touched-module no-regression passed for all four changed production modules at -the audited PR head and was independently regenerated by hosted CI before merge. +touched-module no-regression remains mandatory for production changes. Gate 2 +changed only tests and policy, so hosted CI correctly treated the changed-line +and touched-production-module comparison as not applicable while still +regenerating exact-base repository and risk-module evidence. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index d539446..5c06010 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -6,34 +6,33 @@ Phase: construction ## Current verified baseline -Gate 1 canonical commit: `37d008d462a6c3b2f04bd9046593e728870227a2` +Gate 2 canonical code commit: `fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56` -The mandatory offline CPU selection currently completes from a clean, -non-editable Python 3.12 environment in 68.22 seconds with 1,468 passing tests, -9 conditionally deselected tests, no skips, and no warnings. Branch-enabled -coverage is 70.20% statements and 57.41% branches. The documented mature -CPU-testable scope is 92.23% statements and 80.36% branches. The repository -enforces 90% changed-line coverage, per-touched-module line and branch -no-regression, a 75% selective mutation floor, and zero unexpected warnings. -The current bounded campaign kills 809 of 886 mutants (91.31%), and 316 -high-consequence tests pass in each of three file orders and hash seeds with no -flake candidates. Package, installed wheel/sdist, vulnerability, secret, -license, and SBOM checks are green locally and in hosted CI. All seven jobs in -the exact canonical-commit run passed: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31866177503. -The latest conditional run is green for its selected pinned-model, -external-evaluation, loopback-network, and operator-UI gates. CUDA, +The mandatory offline CPU selection passes on Python 3.10-3.12 with 1,541 +tests, 9 conditionally deselected tests, no skips, and no warnings. The slowest +hosted pytest selection is Python 3.12 at 136.33 seconds; its complete CI job is +4 minutes 52 seconds. Branch-enabled coverage is 75.35% statements and 61.49% +branches. The documented mature CPU-testable scope is 92.69% statements and +81.47% branches. The repository enforces 90% changed-line coverage, +per-touched-module line and branch no-regression, a 75% selective mutation +floor, and zero unexpected warnings. The current bounded campaign kills 809 of +886 mutants (91.31%), and 388 high-consequence tests pass in each of three file +orders and hash seeds with no flake candidates. Package, installed wheel/sdist, +vulnerability, secret, license, and SBOM checks are green locally and in hosted +CI. All seven jobs in the exact canonical-commit run passed: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123. + +The current-head conditional run is green for pinned-model download and +evaluation, loopback network, operator UI, and evidence-freshness policy: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039. CUDA, bitsandbytes, MPS, MLX, and remote execution were not selected and therefore -have no fresh support evidence. +have no fresh support evidence or support claim. -The first six milestones and Wave A established a sane, evidence-producing -baseline. The next program is deliberately risk-weighted: it improves contract -durability, failure behavior, integration confidence, and real-environment -evidence before raising blanket percentages. The legacy contributor merge -train is eligible to resume one exact-head PR at a time now that Gate 1 passes. -Ordinary new feature work is parked until Gate 2. Planning, test-only changes, -defect fixes discovered by the program, and the minimum refactors needed to -expose deterministic seams remain allowed. +The foundation, Wave A, Gate 1, and Gate 2 now establish a sane, +evidence-producing baseline. The contributor merge train and ordinary feature +work may proceed one exact-head PR at a time under the per-PR test contract. +Wave C and Wave D continue to deepen property, mutation, determinism, and real +environment evidence without weakening the mandatory CPU baseline. ## Reasoning @@ -318,11 +317,14 @@ Completed baseline: 5. Research-integrity, property, repeat, and selected mutation gates. 6. Conditional backend/service workflows and mature CPU-testable targets. -Next program: +Completed testing-first gates: 7. Completed foundation policy/evidence lock. 8. Wave B1 high-consequence CPU contracts and legacy-train entry Gate 1. 9. Wave B2 offline vertical slices and ordinary-work entry Gate 2. + +Next program: + 10. Wave C property, mutation, determinism, and regression depth. 11. Wave D conditional environment and release-evidence freshness. diff --git a/.aiwg/testing/regression-report.md b/.aiwg/testing/regression-report.md index 3505452..6c77618 100644 --- a/.aiwg/testing/regression-report.md +++ b/.aiwg/testing/regression-report.md @@ -1,16 +1,20 @@ -# Gate 1 regression report +# Gate 2 regression report Date: 2026-08-15 -Base: `62a480fd6a8f9ef091e60ba8dbf6b0ca94a8b785` +Base: `dadd42648a05620298ec3614fa23864684c0b201` -- Mandatory CPU regressions: none; 1,468 tests pass with no warnings. +- Mandatory CPU regressions: none; 1,541 tests pass with no warnings on Python 3.10-3.12. - Determinism regressions: none; three orders and hash seeds are green. - Packaging regressions: none in wheel, sdist, metadata, import, or CLI contracts. - Supply-chain regressions: none in vulnerability, secret, license, lock, or SBOM checks. -- Coverage direction: repository and mature-scope measures exceed every Gate 1 target. +- Conditional software regressions: none in model/evaluation, loopback network, + operator UI, or current-SHA freshness validation. +- Coverage direction: +5.27 statement points and +4.25 branch points over the + exact base; repository and mature-scope measures exceed every Gate 2 target. -The exact committed-head comparison passed with 100% changed-line coverage and -no line or branch regression across all four touched production modules. Hosted -CI independently regenerated coverage for the exact base SHA before merge, and -all seven post-merge jobs passed at -`37d008d462a6c3b2f04bd9046593e728870227a2`. +Gate 2 changed tests and policy only, so no production module required a +changed-line comparison. Hosted CI independently regenerated exact-base +coverage, verified the absence of touched production modules, and passed all +seven post-merge jobs at +`fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123. diff --git a/.aiwg/testing/test-execution-report.md b/.aiwg/testing/test-execution-report.md index 06c7fad..8a6ba3d 100644 --- a/.aiwg/testing/test-execution-report.md +++ b/.aiwg/testing/test-execution-report.md @@ -1,4 +1,4 @@ -# Gate 1 test execution report +# Gate 2 test execution report Date: 2026-08-15 Scope: mandatory offline CPU baseline, quality depth, packaging, and supply chain @@ -8,38 +8,42 @@ Status: PASS | Gate | Result | |---|---| -| Mandatory CPU | 1,468 passed, 9 conditional deselections, 0 failures/skips/warnings | -| Repository coverage | 70.20% statements, 57.41% branches | -| Mature CPU scope | 92.23% statements, 80.36% branches | -| Changed executable lines | 100% at audited PR head; four touched modules without regression | -| Repeat | 316 tests × 3 orders/seeds; no flakes or consistent failures | +| Mandatory CPU | 1,541 passed, 9 conditional deselections, 0 failures/skips/warnings on Python 3.10-3.12 | +| Repository coverage | 75.35% statements, 61.49% branches | +| Mature CPU scope | 92.69% statements, 81.47% branches | +| Changed executable lines | Not applicable; Gate 2 changed tests and policy only | +| Repeat | 388 tests x 3 orders/seeds; no flakes or consistent failures | | Mutation | 809/886 killed (91.31%); 72 survived; 5 timed out | | Package | wheel and sdist build, metadata, isolated imports, and both CLI paths pass | | Supply chain | 0 vulnerabilities, 0 secret findings, 100 licenses accepted, SBOM bound | -| Hosted CI | all seven PR and post-merge jobs pass on Python 3.10–3.12 | +| Installed vertical slice | Local tiny model through CLI/config, checkpoint reload, evaluation, and JSON/CSV report passes from a non-editable install | +| Conditional software gates | Current-head model/evaluation, loopback network, operator UI, and freshness policy pass | +| Hosted CI | all seven post-merge jobs pass on Python 3.10-3.12 | -The mandatory lane ran in a clean non-editable Python 3.12 environment with -offline Hugging Face and accelerator settings. Wall time was 68.22 seconds and -maximum RSS was 791,476 KB. The mutation lane ran under the CI-pinned Python -3.12/tool configuration in 69.79 seconds with maximum RSS of 1,399,824 KB. -The exact post-merge run is -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31866177503 at -`37d008d462a6c3b2f04bd9046593e728870227a2`. +The hosted pytest selections complete in 67.42 seconds on Python 3.10, 127.69 +seconds on Python 3.11, and 136.33 seconds on Python 3.12. The slowest complete +test job is 4 minutes 52 seconds, below the ten-minute Gate 2 budget. The repeat +campaign completes in 47.15 seconds and the selective mutation score remains +91.31%. The exact post-merge run is +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123 at +`fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`. -## Defects found and corrected +## Gate 2 contracts added -1. The lm-eval adapter treated a measured accuracy of `0.0` as missing. -2. Public sanitizers did not reliably remove Windows paths on non-Windows hosts. -3. Report heatmap generation surfaced a dependency deprecation as an unexpected warning. -4. The evaluation package eagerly loaded heavyweight model dependencies for pure adapters; - it now preserves the public API through lazy imports. - -Each correction has a focused regression contract. Evaluation/reporting and -telemetry failure contracts are also part of the deterministic repeat gate; -the pure lm-eval adapter is part of bounded mutation testing. +1. Bayesian optimizer, informed pipeline, strategy navigation, model profile, + sweep, package-export, evaluation, and retry boundaries have deterministic + positive, negative, and malformed-input contracts. +2. A non-editable installed package executes a local tiny-model CLI path from + obliteration through checkpoint reload, YAML-configured evaluation, and + JSON/CSV reporting. +3. Software-only conditional evidence must match the candidate SHA. A stale + exception requires a reason, a canonical numeric OBLITERATUS issue URL, and + an expiry within 30 days. +4. The risk map and deterministic repeat selection own every new contract. ## Remaining conditional evidence -CUDA, bitsandbytes, MPS, MLX, remote execution, real model downloads, and live -external evaluation remain owned by their conditional gates. They are not -required for this offline pull-request baseline and were not rerun here. +CUDA, bitsandbytes, MPS, MLX, and remote execution remain owned by their +conditional gates and were not selected. Pinned-model download/evaluation, +loopback network, and operator UI were rerun successfully at the canonical SHA: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039. diff --git a/WORKSPACE.md b/WORKSPACE.md index 939f009..ba4be8e 100644 --- a/WORKSPACE.md +++ b/WORKSPACE.md @@ -43,11 +43,11 @@ accelerator, or remote-execution credentials. Canonical required checks: - the exact Ruff F and actionlint command set in [.github/workflows/ci.yml](.github/workflows/ci.yml); -- `python -m pytest` with at least 70% repository line coverage and 55% branch +- `python -m pytest` with at least 75% repository line coverage and 60% branch coverage; - at least 90% changed-line coverage plus no line or branch regression in any touched production module, compared with coverage from the exact base commit; -- at least 90% line and 78% branch coverage for the documented mature +- at least 92% line and 80% branch coverage for the documented mature CPU-testable scope, plus a 75% selective mutation score and zero unexpected warnings; - `python -m build --sdist --wheel`