diff --git a/.aiwg/testing/coverage-report.md b/.aiwg/testing/coverage-report.md index 5e879ff..48fa860 100644 --- a/.aiwg/testing/coverage-report.md +++ b/.aiwg/testing/coverage-report.md @@ -1,25 +1,34 @@ -# Gate 2 coverage report +# Gate 3 increment 1 coverage report Date: 2026-08-15 +Canonical commit: `256c39ea6a492749a4db44146830e9d78fd3fed8` +Exact base: `8456e52bf8604a72ec6cfcc1ab71cbf46c90e1e7` Coverage format: coverage.py branch JSON v3 -| Scope | Measured | Enforced floor | Status | -|---|---:|---:|---| -| Repository statements | 75.35% | 75% | PASS | -| Repository branches | 61.49% | 60% | PASS | -| Mature CPU statements | 92.69% | 92% | PASS | -| Mature CPU branches | 81.47% | 80% | PASS | -| Changed executable lines | Not applicable (test/policy-only change) | 90% when production changes | PASS | +| Scope | Measured on Python 3.12 | Enforced floor | Gate 3 exit | Status | +|---|---:|---:|---:|---| +| Repository statements | 75.68% | 75% | 80% | PASS floor; exit open | +| Repository branches | 61.72% | 60% | 65% | PASS floor; exit open | +| Mature CPU statements | 92.71% | 92% | 94% | PASS floor; exit open | +| Mature CPU branches | 81.55% | 80% | 84% | PASS floor; exit open | +| Changed production lines | Not applicable (policy/test/docs only) | 95% | 95% | PASS | +| Changed policy-script lines | 100% (242/242 focused local audit) | 95% | 95% | PASS | -Gate 1 critical surfaces remain fully or strongly covered: evaluation -baselines 100%/90.91%, evaluator 100%/97.37%, lm-eval adapter 100%/100%, report -100%/98.21%, and telemetry 91.62%/81.88% (statements/branches). Gate 2 adds -96.53%/81.94% for Bayesian optimization, 94.90%/76.73% for informed pipeline, -and 100%/100% for sweep orchestration. +The Python-version measurements vary slightly because version-specific branches +are exercised: repository coverage spans 75.68–75.78% statements and +61.72–61.81% branches. Exact-base replay on Python 3.12 reports a zero-point +line and branch delta and zero touched production modules, as expected for this +test-infrastructure-only increment. -The immutable floors are encoded in `ci/test-quality-policy.json`, pytest, -the CI workflow, and `scripts/check_quality_policy.py`. Exact-base -touched-module no-regression remains mandatory for production changes. Gate 2 -changed only tests and policy, so hosted CI correctly treated the changed-line -and touched-production-module comparison as not applicable while still -regenerating exact-base repository and risk-module evidence. +Gate 3 increment 1 raises the immutable changed-line floor from 90% to 95%. +The value is consistent in `ci/test-quality-policy.json`, +`scripts/check_quality_policy.py`, `.github/workflows/ci.yml`, tests, and +operator documentation. The normal repository coverage gate measures +`obliteratus`; the audit therefore also measured the changed executable lines +in the two modified policy/evidence scripts directly and covered all 242. + +The quantitative Gate 3 exit remains open. Later bounded increments must add +named production contracts to reach 80% repository statements, 65% repository +branches, 94% mature CPU statements, and 84% mature CPU branches without +weakening these floors. Canonical evidence is retained by CI run +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index 558cd50..89a2350 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -6,29 +6,31 @@ Phase: construction ## Current verified baseline -Gate 2 canonical code commit: `fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56` +Gate 3 increment 1 canonical commit: +`256c39ea6a492749a4db44146830e9d78fd3fed8` -Current post-merge baseline commit: `eab67c993d53b88ed9168096d4de20e8ee43a374` - -The mandatory offline CPU selection passes on Python 3.10-3.12 with 1,541 +The mandatory offline CPU selection passes on Python 3.10–3.12 with 1,644 tests, 9 conditionally deselected tests, no skips, and no warnings. The slowest -hosted pytest selection is Python 3.12 at 136.33 seconds; its complete CI job is -4 minutes 52 seconds. Branch-enabled coverage is 75.35% statements and 61.49% -branches. The documented mature CPU-testable scope is 92.69% statements and -81.47% branches. The repository enforces 95% changed-line coverage, -per-touched-module line and branch no-regression, a 75% selective mutation -floor, and zero unexpected warnings. The current bounded campaign kills 809 of -886 mutants (91.31%), and 388 high-consequence tests pass in each of three file -orders and hash seeds with no flake candidates. Package, installed wheel/sdist, -vulnerability, secret, license, and SBOM checks are green locally and in hosted -CI. All seven jobs in the exact canonical-commit run passed: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123. +hosted pytest selection is Python 3.12 at 158.989 seconds; its complete CI job +is 5 minutes 41 seconds because it also regenerates exact-base coverage. +Branch-enabled Python 3.12 coverage is 75.68% statements and 61.72% branches. +The documented mature CPU-testable scope is 92.71% statements and 81.55% +branches. The repository enforces 95% changed-line coverage, per-touched-module +line and branch no-regression, a 75% selective mutation floor, zero unexpected +warnings, and owned suite/test/marker/repeat duration budgets. The bounded +campaign kills 809 of 886 mutants (91.31%), and 391 high-consequence tests pass +in each of three file orders and hash seeds with no flake candidates. Package, +installed wheel/sdist, vulnerability, secret, license, and SBOM checks are +green. All seven jobs in the exact canonical-commit run passed: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085. -The current-head conditional run is green for pinned-model download and -evaluation, loopback network, operator UI, and evidence-freshness policy: +The latest conditional run is green for pinned-model download and evaluation, +loopback network, operator UI, and evidence-freshness policy at the earlier +Gate 2 commit: https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039. CUDA, bitsandbytes, MPS, MLX, and remote execution were not selected and therefore -have no fresh support evidence or support claim. +have no fresh support evidence or support claim. Gate 3 item 8 must reconcile +all conditional evidence at the eventual release candidate. The foundation, Wave A, Gate 1, and Gate 2 establish a sane, evidence-producing baseline. The legacy contributor merge train proceeded one @@ -36,19 +38,11 @@ exact-head PR at a time under the per-PR test contract. Wave C and Wave D deepen property, mutation, determinism, and real-environment evidence without weakening the mandatory CPU baseline. -After the Gate 2 contributor integrations, the mandatory suite contains 1,585 -passing tests, 9 conditional deselections, and 2 documented expected failures. -Repository coverage is 75.80% statements and 61.89% branches; mature CPU scope -is 92.89% statements and 81.72% branches. All seven jobs at the current -post-merge commit passed: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31872157242. - The operator has now placed ordinary feature work back behind a testing-depth -gate. Gate 2 remains the verified floor, but it is no longer sufficient by -itself to start unrelated feature work. The Gate 3 plan below must first be -adopted and its required work completed. Correctness, security, data-loss, and -test-infrastructure repairs remain permitted when narrowly scoped and paired -with reproducing tests. +gate. Gate 3 is adopted and item 1 is complete, but items 2–7 and the +quantitative exit criteria must pass before unrelated feature work resumes. +Correctness, security, data-loss, and test-infrastructure repairs remain +permitted when narrowly scoped and paired with reproducing tests. ## Reasoning @@ -285,11 +279,11 @@ resistance, environment evidence, or deterministic replay. | Measure | Current | Gate 3 minimum | |---|---:|---:| -| Repository statements | 75.80% | 80.00% | -| Repository branches | 61.89% | 65.00% | -| Mature CPU statements | 92.89% | 94.00% | -| Mature CPU branches | 81.72% | 84.00% | -| Changed executable lines | 90.00% floor | 95.00% floor | +| Repository statements | 75.68% | 80.00% | +| Repository branches | 61.72% | 65.00% | +| Mature CPU statements | 92.71% | 94.00% | +| Mature CPU branches | 81.55% | 84.00% | +| Changed executable lines | 95.00% floor | 95.00% floor | | Selected mutation score | 91.31% on narrow scope | at least 85% on expanded scope | | Unexpected warnings | 0 | 0 | | Active flaky quarantines | 0 | 0 | @@ -347,16 +341,16 @@ small pure seam and measure it rather than mocking away the behavior under test. | Surface | Current statements / branches | Gate 3 intent | |---|---:|---| -| `abliterate.py` | 45.78% / 37.31% | at least 52% / 45% with mutation/math contracts | -| `lora_ablation.py` | 7.19% / 0% | at least 70% / 55% for CPU-testable decisions | -| `tourney.py` | 9.28% / 0% | at least 55% / 40% for lifecycle/state transitions | +| `abliterate.py` | 49.63% / 37.31% | at least 52% / 45% with mutation/math contracts | +| `lora_ablation.py` | 9.57% / 0% | at least 70% / 55% for CPU-testable decisions | +| `tourney.py` | 12.01% / 0% | at least 55% / 40% for lifecycle/state transitions | | `bestiary_sync.py` | 0% / 0% | at least 80% / 65% using a loopback/fake transport | -| `models_client.py` | 19.54% / 3.57% | at least 80% / 65% for resolution/error contracts | -| `watchtower.py` | 19.19% / 0% | at least 70% / 55% with injected clock/client seams | -| `interactive.py` | 8.47% / 0% | at least 70% / 55% with scripted I/O contracts | -| `local_ui.py` | 40.87% / 15.22% | at least 70% / 55% for launch decisions and failures | +| `models_client.py` | 27.12% / 3.57% | at least 80% / 65% for resolution/error contracts | +| `watchtower.py` | 24.63% / 0% | at least 70% / 55% with injected clock/client seams | +| `interactive.py` | 10.49% / 0% | at least 70% / 55% with scripted I/O contracts | +| `local_ui.py` | 48.15% / 15.22% | at least 70% / 55% for launch decisions and failures | | `ui_watchtower.py` | 0% / 0% | cover construction/callback contracts; real UI stays conditional | -| `evaluation/heretic_eval.py` | 42.46% / 30.51% | at least 70% / 55% for CPU-testable evaluation logic | +| `evaluation/heretic_eval.py` | 45.11% / 27.12% | at least 70% / 55% for CPU-testable evaluation logic | #### Bounded delivery sequence @@ -364,8 +358,10 @@ Each item is an independently reviewable PR and must be exact-head audited before merge. Later items may be split further; they may not be combined into a single repository-wide coverage PR. -1. Adopt Gate 3 policy, refresh current measurements, and add duration-budget - enforcement without changing production behavior. +1. **Completed at `256c39e`.** Adopt Gate 3 policy, refresh current + measurements, and add duration-budget enforcement without changing + production behavior. Exact canonical evidence: + https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085. 2. Add numerical/reference-oracle and metamorphic contracts; expand mutation to the corresponding pure math. 3. Add loader, architecture, dtype, quantization, and shared-weight decision diff --git a/.aiwg/testing/regression-report.md b/.aiwg/testing/regression-report.md index 6c77618..c8bf329 100644 --- a/.aiwg/testing/regression-report.md +++ b/.aiwg/testing/regression-report.md @@ -1,20 +1,28 @@ -# Gate 2 regression report +# Gate 3 increment 1 regression report Date: 2026-08-15 -Base: `dadd42648a05620298ec3614fa23864684c0b201` +Base: `8456e52bf8604a72ec6cfcc1ab71cbf46c90e1e7` +Canonical head: `256c39ea6a492749a4db44146830e9d78fd3fed8` -- Mandatory CPU regressions: none; 1,541 tests pass with no warnings on Python 3.10-3.12. -- Determinism regressions: none; three orders and hash seeds are green. -- Packaging regressions: none in wheel, sdist, metadata, import, or CLI contracts. -- Supply-chain regressions: none in vulnerability, secret, license, lock, or SBOM checks. -- Conditional software regressions: none in model/evaluation, loopback network, - operator UI, or current-SHA freshness validation. -- Coverage direction: +5.27 statement points and +4.25 branch points over the - exact base; repository and mature-scope measures exceed every Gate 2 target. +- Mandatory CPU regressions: none; 1,644 tests pass with no warnings on Python + 3.10–3.12, versus 1,587 selected base tests in the Python 3.12 exact-base + replay. +- Production regressions: none; the increment changes policy, evidence, tests, + CI, and documentation without changing `obliteratus` behavior. +- Coverage regressions: none; exact-base Python 3.12 evidence reports 0.00 line + and branch percentage-point delta and zero touched production modules. +- Determinism regressions: none; 391 tests pass in all three fixed orders and + hash seeds with no flake candidates or consistent failures. +- Duration regressions: none unowned. Every suite, registered marker, testcase, + and repeat duration is within policy. The known installed-package cost is + tracked by issue #100 with a 120s ceiling and 2026-11-13 expiry. +- Mutation regressions: none; the bounded campaign remains 809/886 killed + (91.31%), above the current 75% floor. +- Packaging and supply-chain regressions: none in wheel, sdist, metadata, + installed import/CLI, lock, vulnerability, secret, license, or SBOM checks. -Gate 2 changed tests and policy only, so no production module required a -changed-line comparison. Hosted CI independently regenerated exact-base -coverage, verified the absence of touched production modules, and passed all -seven post-merge jobs at -`fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123. +All seven canonical post-merge jobs passed at the head above: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085. + +This report closes bounded Gate 3 item 1 only. The overall Gate 3 release +decision remains blocked on items 2–7 and the quantitative exit criteria. diff --git a/.aiwg/testing/test-execution-report.md b/.aiwg/testing/test-execution-report.md index 8a6ba3d..9f53db0 100644 --- a/.aiwg/testing/test-execution-report.md +++ b/.aiwg/testing/test-execution-report.md @@ -1,49 +1,55 @@ -# Gate 2 test execution report +# Gate 3 increment 1 test execution report Date: 2026-08-15 -Scope: mandatory offline CPU baseline, quality depth, packaging, and supply chain +Canonical commit: `256c39ea6a492749a4db44146830e9d78fd3fed8` +Scope: mandatory offline CPU baseline, duration policy, quality depth, packaging, +and supply chain Status: PASS ## Results | Gate | Result | |---|---| -| Mandatory CPU | 1,541 passed, 9 conditional deselections, 0 failures/skips/warnings on Python 3.10-3.12 | -| Repository coverage | 75.35% statements, 61.49% branches | -| Mature CPU scope | 92.69% statements, 81.47% branches | -| Changed executable lines | Not applicable; Gate 2 changed tests and policy only | -| Repeat | 388 tests x 3 orders/seeds; no flakes or consistent failures | +| Mandatory CPU | 1,644 passed, 9 conditional deselections, 0 failures/skips/warnings on Python 3.10–3.12 | +| Repository coverage | Python 3.12: 75.68% statements, 61.72% branches | +| Mature CPU scope | Python 3.12: 92.71% statements, 81.55% branches | +| Changed executable lines | Production: not applicable; focused policy/evidence scripts: 242/242 (100%) | +| Repeat | 391 tests x 3 orders/seeds; 51.085s total; no flakes or consistent failures | | Mutation | 809/886 killed (91.31%); 72 survived; 5 timed out | -| Package | wheel and sdist build, metadata, isolated imports, and both CLI paths pass | -| Supply chain | 0 vulnerabilities, 0 secret findings, 100 licenses accepted, SBOM bound | -| Installed vertical slice | Local tiny model through CLI/config, checkpoint reload, evaluation, and JSON/CSV report passes from a non-editable install | -| Conditional software gates | Current-head model/evaluation, loopback network, operator UI, and freshness policy pass | -| Hosted CI | all seven post-merge jobs pass on Python 3.10-3.12 | +| Duration evidence | Complete marker metadata; every testcase retained; all suite, marker, testcase, and repeat budgets pass | +| Package | Wheel and sdist build, metadata, isolated imports, and both CLI paths pass | +| Supply chain | Vulnerability, secret, license, lock, and bound-SBOM policies pass | +| Hosted CI | All seven post-merge jobs pass on Python 3.10–3.12 | -The hosted pytest selections complete in 67.42 seconds on Python 3.10, 127.69 -seconds on Python 3.11, and 136.33 seconds on Python 3.12. The slowest complete -test job is 4 minutes 52 seconds, below the ten-minute Gate 2 budget. The repeat -campaign completes in 47.15 seconds and the selective mutation score remains -91.31%. The exact post-merge run is -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123 at -`fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`. +## Hosted duration evidence -## Gate 2 contracts added +| Python | Pytest suite | Complete job | CPU marker | Integration marker | Unmarked | +|---|---:|---:|---:|---:|---:| +| 3.10 | 154.051s | 2m55s | 94.259s | 93.778s | 24.187s | +| 3.11 | 129.534s | 2m37s | 76.812s | 76.395s | 19.360s | +| 3.12 | 158.989s | 5m41s | 93.549s | 93.087s | 24.709s | -1. Bayesian optimizer, informed pipeline, strategy navigation, model profile, - sweep, package-export, evaluation, and retry boundaries have deterministic - positive, negative, and malformed-input contracts. -2. A non-editable installed package executes a local tiny-model CLI path from - obliteration through checkpoint reload, YAML-configured evaluation, and - JSON/CSV reporting. -3. Software-only conditional evidence must match the candidate SHA. A stale - exception requires a reason, a canonical numeric OBLITERATUS issue URL, and - an expiry within 30 days. -4. The risk map and deterministic repeat selection own every new contract. +Python 3.12 is the slowest complete job because it also regenerates coverage at +the exact base commit. It remains 4m19s inside the ten-minute job cap. The +slowest individual testcase is the installed-package offline +checkpoint-to-report slice at 80.119s. It retains the general 15s testcase +floor through a narrow 120s owner budget linked to issue #100 and expiring +2026-11-13; no permanent or implicit exception was introduced. -## Remaining conditional evidence +## Contracts added -CUDA, bitsandbytes, MPS, MLX, and remote execution remain owned by their -conditional gates and were not selected. Pinned-model download/evaluation, -loopback network, and operator UI were rerun successfully at the canonical SHA: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039. +1. The changed-line floor is immutable at 95% unless the existing reviewed, + issue-linked, time-bounded threshold exception contract is satisfied. +2. JUnit normalization retains suite wall time, every testcase duration, + registered test-layer markers, marker aggregates, and missing-marker state. +3. Policy enforcement rejects malformed or inconsistent evidence, missing + marker metadata, unbudgeted markers, suite/test/marker overruns, and repeat + total or pass overruns. +4. Deliberately slow tests require an owner, reason, repository issue, explicit + ceiling, and expiring review window. + +The exact post-merge run is +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085. Conditional +hardware, remote, model-download, network, and operator-UI environments were +not changed or rerun by this increment; their freshness and support-claim +reconciliation remains Gate 3 item 8.