CalvinBackup/OBLITERATUS
1
0
Fork 0
mirror of https://github.com/elder-plinius/OBLITERATUS.git synced 2026-08-18 00:47:23 +02:00
Code Issues Packages Projects Releases Wiki Activity

Record Gate 3 canonical evidence

Browse Source
This commit is contained in:
Joseph Magly
2026-08-15 04:48:59 -04:00
parent 256c39ea6a
commit 369417fbb4
4 changed files with 134 additions and 115 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.aiwg/testing/coverage-report.md
+28 -19
View File
@@ -1,25 +1,34 @@
# Gate 2 coverage report
# Gate 3 increment 1 coverage report
Date: 2026-08-15
Canonical commit: `256c39ea6a492749a4db44146830e9d78fd3fed8`
Exact base: `8456e52bf8604a72ec6cfcc1ab71cbf46c90e1e7`
Coverage format: coverage.py branch JSON v3
| Scope | Measured | Enforced floor | Status |
|---|---:|---:|---|
| Repository statements | 75.35% | 75% | PASS |
| Repository branches | 61.49% | 60% | PASS |
| Mature CPU statements | 92.69% | 92% | PASS |
| Mature CPU branches | 81.47% | 80% | PASS |
| Changed executable lines | Not applicable (test/policy-only change) | 90% when production changes | PASS |
| Scope | Measured on Python 3.12 | Enforced floor | Gate 3 exit | Status |
|---|---:|---:|---:|---|
| Repository statements | 75.68% | 75% | 80% | PASS floor; exit open |
| Repository branches | 61.72% | 60% | 65% | PASS floor; exit open |
| Mature CPU statements | 92.71% | 92% | 94% | PASS floor; exit open |
| Mature CPU branches | 81.55% | 80% | 84% | PASS floor; exit open |
| Changed production lines | Not applicable (policy/test/docs only) | 95% | 95% | PASS |
| Changed policy-script lines | 100% (242/242 focused local audit) | 95% | 95% | PASS |
Gate 1 critical surfaces remain fully or strongly covered: evaluation
baselines 100%/90.91%, evaluator 100%/97.37%, lm-eval adapter 100%/100%, report
100%/98.21%, and telemetry 91.62%/81.88% (statements/branches). Gate 2 adds
96.53%/81.94% for Bayesian optimization, 94.90%/76.73% for informed pipeline,
and 100%/100% for sweep orchestration.
The Python-version measurements vary slightly because version-specific branches
are exercised: repository coverage spans 75.68–75.78% statements and
61.72–61.81% branches. Exact-base replay on Python 3.12 reports a zero-point
line and branch delta and zero touched production modules, as expected for this
test-infrastructure-only increment.
The immutable floors are encoded in `ci/test-quality-policy.json`, pytest,
the CI workflow, and `scripts/check_quality_policy.py`. Exact-base
touched-module no-regression remains mandatory for production changes. Gate 2
changed only tests and policy, so hosted CI correctly treated the changed-line
and touched-production-module comparison as not applicable while still
regenerating exact-base repository and risk-module evidence.
Gate 3 increment 1 raises the immutable changed-line floor from 90% to 95%.
The value is consistent in `ci/test-quality-policy.json`,
`scripts/check_quality_policy.py`, `.github/workflows/ci.yml`, tests, and
operator documentation. The normal repository coverage gate measures
`obliteratus`; the audit therefore also measured the changed executable lines
in the two modified policy/evidence scripts directly and covered all 242.
The quantitative Gate 3 exit remains open. Later bounded increments must add
named production contracts to reach 80% repository statements, 65% repository
branches, 94% mature CPU statements, and 84% mature CPU branches without
weakening these floors. Canonical evidence is retained by CI run
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085.
.aiwg/testing/master-test-plan.md
+41 -45
View File
@@ -6,29 +6,31 @@ Phase: construction
## Current verified baseline
Gate 2 canonical code commit: `fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`
Gate 3 increment 1 canonical commit:
`256c39ea6a492749a4db44146830e9d78fd3fed8`
Current post-merge baseline commit: `eab67c993d53b88ed9168096d4de20e8ee43a374`
The mandatory offline CPU selection passes on Python 3.10-3.12 with 1,541
The mandatory offline CPU selection passes on Python 3.10–3.12 with 1,644
tests, 9 conditionally deselected tests, no skips, and no warnings. The slowest
hosted pytest selection is Python 3.12 at 136.33 seconds; its complete CI job is
4 minutes 52 seconds. Branch-enabled coverage is 75.35% statements and 61.49%
branches. The documented mature CPU-testable scope is 92.69% statements and
81.47% branches. The repository enforces 95% changed-line coverage,
per-touched-module line and branch no-regression, a 75% selective mutation
floor, and zero unexpected warnings. The current bounded campaign kills 809 of
886 mutants (91.31%), and 388 high-consequence tests pass in each of three file
orders and hash seeds with no flake candidates. Package, installed wheel/sdist,
vulnerability, secret, license, and SBOM checks are green locally and in hosted
CI. All seven jobs in the exact canonical-commit run passed:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123.
hosted pytest selection is Python 3.12 at 158.989 seconds; its complete CI job
is 5 minutes 41 seconds because it also regenerates exact-base coverage.
Branch-enabled Python 3.12 coverage is 75.68% statements and 61.72% branches.
The documented mature CPU-testable scope is 92.71% statements and 81.55%
branches. The repository enforces 95% changed-line coverage, per-touched-module
line and branch no-regression, a 75% selective mutation floor, zero unexpected
warnings, and owned suite/test/marker/repeat duration budgets. The bounded
campaign kills 809 of 886 mutants (91.31%), and 391 high-consequence tests pass
in each of three file orders and hash seeds with no flake candidates. Package,
installed wheel/sdist, vulnerability, secret, license, and SBOM checks are
green. All seven jobs in the exact canonical-commit run passed:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085.
The current-head conditional run is green for pinned-model download and
evaluation, loopback network, operator UI, and evidence-freshness policy:
The latest conditional run is green for pinned-model download and evaluation,
loopback network, operator UI, and evidence-freshness policy at the earlier
Gate 2 commit:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039. CUDA,
bitsandbytes, MPS, MLX, and remote execution were not selected and therefore
have no fresh support evidence or support claim.
have no fresh support evidence or support claim. Gate 3 item 8 must reconcile
all conditional evidence at the eventual release candidate.
The foundation, Wave A, Gate 1, and Gate 2 establish a sane,
evidence-producing baseline. The legacy contributor merge train proceeded one
@@ -36,19 +38,11 @@ exact-head PR at a time under the per-PR test contract. Wave C and Wave D deepen
property, mutation, determinism, and real-environment evidence without weakening
the mandatory CPU baseline.
After the Gate 2 contributor integrations, the mandatory suite contains 1,585
passing tests, 9 conditional deselections, and 2 documented expected failures.
Repository coverage is 75.80% statements and 61.89% branches; mature CPU scope
is 92.89% statements and 81.72% branches. All seven jobs at the current
post-merge commit passed:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31872157242.
The operator has now placed ordinary feature work back behind a testing-depth
gate. Gate 2 remains the verified floor, but it is no longer sufficient by
itself to start unrelated feature work. The Gate 3 plan below must first be
adopted and its required work completed. Correctness, security, data-loss, and
test-infrastructure repairs remain permitted when narrowly scoped and paired
with reproducing tests.
gate. Gate 3 is adopted and item 1 is complete, but items 2–7 and the
quantitative exit criteria must pass before unrelated feature work resumes.
Correctness, security, data-loss, and test-infrastructure repairs remain
permitted when narrowly scoped and paired with reproducing tests.
## Reasoning
@@ -285,11 +279,11 @@ resistance, environment evidence, or deterministic replay.
| Measure | Current | Gate 3 minimum |
|---|---:|---:|
| Repository statements | 75.80% | 80.00% |
| Repository branches | 61.89% | 65.00% |
| Mature CPU statements | 92.89% | 94.00% |
| Mature CPU branches | 81.72% | 84.00% |
| Changed executable lines | 90.00% floor | 95.00% floor |
| Repository statements | 75.68% | 80.00% |
| Repository branches | 61.72% | 65.00% |
| Mature CPU statements | 92.71% | 94.00% |
| Mature CPU branches | 81.55% | 84.00% |
| Changed executable lines | 95.00% floor | 95.00% floor |
| Selected mutation score | 91.31% on narrow scope | at least 85% on expanded scope |
| Unexpected warnings | 0 | 0 |
| Active flaky quarantines | 0 | 0 |
@@ -347,16 +341,16 @@ small pure seam and measure it rather than mocking away the behavior under test.
| Surface | Current statements / branches | Gate 3 intent |
|---|---:|---|
| `abliterate.py` | 45.78% / 37.31% | at least 52% / 45% with mutation/math contracts |
| `lora_ablation.py` | 7.19% / 0% | at least 70% / 55% for CPU-testable decisions |
| `tourney.py` | 9.28% / 0% | at least 55% / 40% for lifecycle/state transitions |
| `abliterate.py` | 49.63% / 37.31% | at least 52% / 45% with mutation/math contracts |
| `lora_ablation.py` | 9.57% / 0% | at least 70% / 55% for CPU-testable decisions |
| `tourney.py` | 12.01% / 0% | at least 55% / 40% for lifecycle/state transitions |
| `bestiary_sync.py` | 0% / 0% | at least 80% / 65% using a loopback/fake transport |
| `models_client.py` | 19.54% / 3.57% | at least 80% / 65% for resolution/error contracts |
| `watchtower.py` | 19.19% / 0% | at least 70% / 55% with injected clock/client seams |
| `interactive.py` | 8.47% / 0% | at least 70% / 55% with scripted I/O contracts |
| `local_ui.py` | 40.87% / 15.22% | at least 70% / 55% for launch decisions and failures |
| `models_client.py` | 27.12% / 3.57% | at least 80% / 65% for resolution/error contracts |
| `watchtower.py` | 24.63% / 0% | at least 70% / 55% with injected clock/client seams |
| `interactive.py` | 10.49% / 0% | at least 70% / 55% with scripted I/O contracts |
| `local_ui.py` | 48.15% / 15.22% | at least 70% / 55% for launch decisions and failures |
| `ui_watchtower.py` | 0% / 0% | cover construction/callback contracts; real UI stays conditional |
| `evaluation/heretic_eval.py` | 42.46% / 30.51% | at least 70% / 55% for CPU-testable evaluation logic |
| `evaluation/heretic_eval.py` | 45.11% / 27.12% | at least 70% / 55% for CPU-testable evaluation logic |
#### Bounded delivery sequence
@@ -364,8 +358,10 @@ Each item is an independently reviewable PR and must be exact-head audited
before merge. Later items may be split further; they may not be combined into a
single repository-wide coverage PR.
1. Adopt Gate 3 policy, refresh current measurements, and add duration-budget
enforcement without changing production behavior.
1. **Completed at `256c39e`.** Adopt Gate 3 policy, refresh current
measurements, and add duration-budget enforcement without changing
production behavior. Exact canonical evidence:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085.
2. Add numerical/reference-oracle and metamorphic contracts; expand mutation to
the corresponding pure math.
3. Add loader, architecture, dtype, quantization, and shared-weight decision
.aiwg/testing/regression-report.md
+24 -16
View File
@@ -1,20 +1,28 @@
# Gate 2 regression report
# Gate 3 increment 1 regression report
Date: 2026-08-15
Base: `dadd42648a05620298ec3614fa23864684c0b201`
Base: `8456e52bf8604a72ec6cfcc1ab71cbf46c90e1e7`
Canonical head: `256c39ea6a492749a4db44146830e9d78fd3fed8`
- Mandatory CPU regressions: none; 1,541 tests pass with no warnings on Python 3.10-3.12.
- Determinism regressions: none; three orders and hash seeds are green.
- Packaging regressions: none in wheel, sdist, metadata, import, or CLI contracts.
- Supply-chain regressions: none in vulnerability, secret, license, lock, or SBOM checks.
- Conditional software regressions: none in model/evaluation, loopback network,
operator UI, or current-SHA freshness validation.
- Coverage direction: +5.27 statement points and +4.25 branch points over the
exact base; repository and mature-scope measures exceed every Gate 2 target.
- Mandatory CPU regressions: none; 1,644 tests pass with no warnings on Python
3.10–3.12, versus 1,587 selected base tests in the Python 3.12 exact-base
replay.
- Production regressions: none; the increment changes policy, evidence, tests,
CI, and documentation without changing `obliteratus` behavior.
- Coverage regressions: none; exact-base Python 3.12 evidence reports 0.00 line
and branch percentage-point delta and zero touched production modules.
- Determinism regressions: none; 391 tests pass in all three fixed orders and
hash seeds with no flake candidates or consistent failures.
- Duration regressions: none unowned. Every suite, registered marker, testcase,
and repeat duration is within policy. The known installed-package cost is
tracked by issue #100 with a 120s ceiling and 2026-11-13 expiry.
- Mutation regressions: none; the bounded campaign remains 809/886 killed
(91.31%), above the current 75% floor.
- Packaging and supply-chain regressions: none in wheel, sdist, metadata,
installed import/CLI, lock, vulnerability, secret, license, or SBOM checks.
Gate 2 changed tests and policy only, so no production module required a
changed-line comparison. Hosted CI independently regenerated exact-base
coverage, verified the absence of touched production modules, and passed all
seven post-merge jobs at
`fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123.
All seven canonical post-merge jobs passed at the head above:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085.
This report closes bounded Gate 3 item 1 only. The overall Gate 3 release
decision remains blocked on items 2–7 and the quantitative exit criteria.
.aiwg/testing/test-execution-report.md
+41 -35
View File
@@ -1,49 +1,55 @@
# Gate 2 test execution report
# Gate 3 increment 1 test execution report
Date: 2026-08-15
Scope: mandatory offline CPU baseline, quality depth, packaging, and supply chain
Canonical commit: `256c39ea6a492749a4db44146830e9d78fd3fed8`
Scope: mandatory offline CPU baseline, duration policy, quality depth, packaging,
and supply chain
Status: PASS
## Results
| Gate | Result |
|---|---|
| Mandatory CPU | 1,541 passed, 9 conditional deselections, 0 failures/skips/warnings on Python 3.10-3.12 |
| Repository coverage | 75.35% statements, 61.49% branches |
| Mature CPU scope | 92.69% statements, 81.47% branches |
| Changed executable lines | Not applicable; Gate 2 changed tests and policy only |
| Repeat | 388 tests x 3 orders/seeds; no flakes or consistent failures |
| Mandatory CPU | 1,644 passed, 9 conditional deselections, 0 failures/skips/warnings on Python 3.10–3.12 |
| Repository coverage | Python 3.12: 75.68% statements, 61.72% branches |
| Mature CPU scope | Python 3.12: 92.71% statements, 81.55% branches |
| Changed executable lines | Production: not applicable; focused policy/evidence scripts: 242/242 (100%) |
| Repeat | 391 tests x 3 orders/seeds; 51.085s total; no flakes or consistent failures |
| Mutation | 809/886 killed (91.31%); 72 survived; 5 timed out |
| Package | wheel and sdist build, metadata, isolated imports, and both CLI paths pass |
| Supply chain | 0 vulnerabilities, 0 secret findings, 100 licenses accepted, SBOM bound |
| Installed vertical slice | Local tiny model through CLI/config, checkpoint reload, evaluation, and JSON/CSV report passes from a non-editable install |
| Conditional software gates | Current-head model/evaluation, loopback network, operator UI, and freshness policy pass |
| Hosted CI | all seven post-merge jobs pass on Python 3.10-3.12 |
| Duration evidence | Complete marker metadata; every testcase retained; all suite, marker, testcase, and repeat budgets pass |
| Package | Wheel and sdist build, metadata, isolated imports, and both CLI paths pass |
| Supply chain | Vulnerability, secret, license, lock, and bound-SBOM policies pass |
| Hosted CI | All seven post-merge jobs pass on Python 3.10–3.12 |
The hosted pytest selections complete in 67.42 seconds on Python 3.10, 127.69
seconds on Python 3.11, and 136.33 seconds on Python 3.12. The slowest complete
test job is 4 minutes 52 seconds, below the ten-minute Gate 2 budget. The repeat
campaign completes in 47.15 seconds and the selective mutation score remains
91.31%. The exact post-merge run is
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123 at
`fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`.
## Hosted duration evidence
## Gate 2 contracts added
| Python | Pytest suite | Complete job | CPU marker | Integration marker | Unmarked |
|---|---:|---:|---:|---:|---:|
| 3.10 | 154.051s | 2m55s | 94.259s | 93.778s | 24.187s |
| 3.11 | 129.534s | 2m37s | 76.812s | 76.395s | 19.360s |
| 3.12 | 158.989s | 5m41s | 93.549s | 93.087s | 24.709s |
1. Bayesian optimizer, informed pipeline, strategy navigation, model profile,
sweep, package-export, evaluation, and retry boundaries have deterministic
positive, negative, and malformed-input contracts.
2. A non-editable installed package executes a local tiny-model CLI path from
obliteration through checkpoint reload, YAML-configured evaluation, and
JSON/CSV reporting.
3. Software-only conditional evidence must match the candidate SHA. A stale
exception requires a reason, a canonical numeric OBLITERATUS issue URL, and
an expiry within 30 days.
4. The risk map and deterministic repeat selection own every new contract.
Python 3.12 is the slowest complete job because it also regenerates coverage at
the exact base commit. It remains 4m19s inside the ten-minute job cap. The
slowest individual testcase is the installed-package offline
checkpoint-to-report slice at 80.119s. It retains the general 15s testcase
floor through a narrow 120s owner budget linked to issue #100 and expiring
2026-11-13; no permanent or implicit exception was introduced.
## Remaining conditional evidence
## Contracts added
CUDA, bitsandbytes, MPS, MLX, and remote execution remain owned by their
conditional gates and were not selected. Pinned-model download/evaluation,
loopback network, and operator UI were rerun successfully at the canonical SHA:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039.
1. The changed-line floor is immutable at 95% unless the existing reviewed,
issue-linked, time-bounded threshold exception contract is satisfied.
2. JUnit normalization retains suite wall time, every testcase duration,
registered test-layer markers, marker aggregates, and missing-marker state.
3. Policy enforcement rejects malformed or inconsistent evidence, missing
marker metadata, unbudgeted markers, suite/test/marker overruns, and repeat
total or pass overruns.
4. Deliberately slow tests require an owner, reason, repository issue, explicit
ceiling, and expiring review window.
The exact post-merge run is
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085. Conditional
hardware, remote, model-download, network, and operator-UI environments were
not changed or rerun by this increment; their freshness and support-claim
reconciliation remains Gate 3 item 8.