mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-08-17 16:37:30 +02:00
docs: finalize Gate 2 evidence
This commit is contained in:
Joseph Magly
@@ -16,14 +16,18 @@ produce retained evidence and never silently convert a required failure into a
|
||||
success. Tool versions remain project-controlled. The default gate must not
|
||||
access external services, inherit user model caches, or require credentials.
|
||||
|
||||
The initial coverage waves are complete:
|
||||
The initial coverage waves and Gate 2 are complete:
|
||||
|
||||
- measurement wave: preserve the 49% line floor and establish branch baseline;
|
||||
- boundary wave: repository line coverage at least 55%, changed lines at least
|
||||
90%, and touched critical modules at least 70%;
|
||||
- integration wave: repository line coverage at least 60%;
|
||||
- mature CPU-testable target: at least 80% line and 75% branch coverage, with
|
||||
exclusions limited to documented conditional environment code.
|
||||
- Gate 1: at least 70% repository line and 55% branch coverage, with at least
|
||||
90% line and 78% branch coverage for the mature CPU-testable scope;
|
||||
- Gate 2: at least 75% repository line and 60% branch coverage, with at least
|
||||
92% line and 80% branch coverage for the mature CPU-testable scope and an
|
||||
installed-package CLI/config-to-report vertical slice;
|
||||
- exclusions remain limited to documented conditional environment code.
|
||||
|
||||
Wave A locks that baseline by generating exact-base coverage in the same CI run,
|
||||
rejecting per-module line or branch regressions, validating a machine-readable
|
||||
@@ -31,3 +35,6 @@ source-to-test and conditional-gate graph, retaining normalized trend evidence
|
||||
for 90 days, and enforcing owner/issue/expiry requirements for repeated flaky
|
||||
tests. The exact thresholds and exclusion graph remain owned by
|
||||
`ci/test-quality-policy.json`; `ci/test-risk-map.json` owns test responsibility.
|
||||
Software-only conditional evidence must match the candidate commit. A stale
|
||||
evidence exception requires a reason, a canonical numeric OBLITERATUS issue,
|
||||
and an expiry no more than 30 days away.
|
||||
|
||||
@@ -27,4 +27,13 @@ and a tiny offline model fixture.
|
||||
|
||||
An operator can intentionally run clearly documented CUDA, bitsandbytes, MPS,
|
||||
MLX, model-download, network, and remote checks without weakening or surprising
|
||||
the default pull-request gate.
|
||||
the default pull-request gate. Software-only evidence records the exact
|
||||
candidate commit; any stale-evidence exception is reasoned, issue-linked, and
|
||||
expires within 30 days.
|
||||
|
||||
## UC-TQ-06 — Maintainer verifies the installed vertical contract
|
||||
|
||||
A maintainer can exercise a non-editable installed distribution with a local
|
||||
deterministic tiny model through CLI/config parsing, loader/device selection,
|
||||
pipeline execution, checkpoint save/reload, evaluation, and JSON/CSV report
|
||||
generation without external services or accelerator hardware.
|
||||
|
||||
@@ -10,3 +10,4 @@
|
||||
| TQ-R6 | Mutation/property jobs exceed useful feedback time | medium | medium | target small pure modules and run depth gates separately |
|
||||
| TQ-R7 | Installed package behavior differs from checkout | medium | high | clean wheel and sdist installation smoke tests |
|
||||
| TQ-R8 | Numerical tests are device/dtype brittle | medium | high | invariant/tolerance contracts and separate backend-specific evidence |
|
||||
| TQ-R9 | Conditional evidence is reused for a different commit | medium | high | candidate-SHA validation; exceptions require a reason, canonical issue, and expiry within 30 days |
|
||||
|
||||
@@ -1,21 +1,25 @@
|
||||
# Gate 1 coverage report
|
||||
# Gate 2 coverage report
|
||||
|
||||
Date: 2026-08-15
|
||||
Coverage format: coverage.py branch JSON v3
|
||||
|
||||
| Scope | Measured | Enforced floor | Status |
|
||||
|---|---:|---:|---|
|
||||
| Repository statements | 70.20% | 70% | PASS |
|
||||
| Repository branches | 57.41% | 55% | PASS |
|
||||
| Mature CPU statements | 92.23% | 90% | PASS |
|
||||
| Mature CPU branches | 80.36% | 78% | PASS |
|
||||
| Changed executable lines | 100.00% | 90% | PASS |
|
||||
| Repository statements | 75.35% | 75% | PASS |
|
||||
| Repository branches | 61.49% | 60% | PASS |
|
||||
| Mature CPU statements | 92.69% | 92% | PASS |
|
||||
| Mature CPU branches | 81.47% | 80% | PASS |
|
||||
| Changed executable lines | Not applicable (test/policy-only change) | 90% when production changes | PASS |
|
||||
|
||||
Critical changed surfaces are fully or strongly covered: evaluation baselines
|
||||
100%/90.91%, evaluator 100%/97.37%, lm-eval adapter 100%/100%, report
|
||||
100%/98.21%, and telemetry 91.62%/81.88% (statements/branches).
|
||||
Gate 1 critical surfaces remain fully or strongly covered: evaluation
|
||||
baselines 100%/90.91%, evaluator 100%/97.37%, lm-eval adapter 100%/100%, report
|
||||
100%/98.21%, and telemetry 91.62%/81.88% (statements/branches). Gate 2 adds
|
||||
96.53%/81.94% for Bayesian optimization, 94.90%/76.73% for informed pipeline,
|
||||
and 100%/100% for sweep orchestration.
|
||||
|
||||
The immutable floors are encoded in `ci/test-quality-policy.json`, pytest,
|
||||
the CI workflow, and `scripts/check_quality_policy.py`. Exact-base
|
||||
touched-module no-regression passed for all four changed production modules at
|
||||
the audited PR head and was independently regenerated by hosted CI before merge.
|
||||
touched-module no-regression remains mandatory for production changes. Gate 2
|
||||
changed only tests and policy, so hosted CI correctly treated the changed-line
|
||||
and touched-production-module comparison as not applicable while still
|
||||
regenerating exact-base repository and risk-module evidence.
|
||||
|
||||
@@ -6,34 +6,33 @@ Phase: construction
|
||||
|
||||
## Current verified baseline
|
||||
|
||||
Gate 1 canonical commit: `37d008d462a6c3b2f04bd9046593e728870227a2`
|
||||
Gate 2 canonical code commit: `fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`
|
||||
|
||||
The mandatory offline CPU selection currently completes from a clean,
|
||||
non-editable Python 3.12 environment in 68.22 seconds with 1,468 passing tests,
|
||||
9 conditionally deselected tests, no skips, and no warnings. Branch-enabled
|
||||
coverage is 70.20% statements and 57.41% branches. The documented mature
|
||||
CPU-testable scope is 92.23% statements and 80.36% branches. The repository
|
||||
enforces 90% changed-line coverage, per-touched-module line and branch
|
||||
no-regression, a 75% selective mutation floor, and zero unexpected warnings.
|
||||
The current bounded campaign kills 809 of 886 mutants (91.31%), and 316
|
||||
high-consequence tests pass in each of three file orders and hash seeds with no
|
||||
flake candidates. Package, installed wheel/sdist, vulnerability, secret,
|
||||
license, and SBOM checks are green locally and in hosted CI. All seven jobs in
|
||||
the exact canonical-commit run passed:
|
||||
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31866177503.
|
||||
The latest conditional run is green for its selected pinned-model,
|
||||
external-evaluation, loopback-network, and operator-UI gates. CUDA,
|
||||
The mandatory offline CPU selection passes on Python 3.10-3.12 with 1,541
|
||||
tests, 9 conditionally deselected tests, no skips, and no warnings. The slowest
|
||||
hosted pytest selection is Python 3.12 at 136.33 seconds; its complete CI job is
|
||||
4 minutes 52 seconds. Branch-enabled coverage is 75.35% statements and 61.49%
|
||||
branches. The documented mature CPU-testable scope is 92.69% statements and
|
||||
81.47% branches. The repository enforces 90% changed-line coverage,
|
||||
per-touched-module line and branch no-regression, a 75% selective mutation
|
||||
floor, and zero unexpected warnings. The current bounded campaign kills 809 of
|
||||
886 mutants (91.31%), and 388 high-consequence tests pass in each of three file
|
||||
orders and hash seeds with no flake candidates. Package, installed wheel/sdist,
|
||||
vulnerability, secret, license, and SBOM checks are green locally and in hosted
|
||||
CI. All seven jobs in the exact canonical-commit run passed:
|
||||
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123.
|
||||
|
||||
The current-head conditional run is green for pinned-model download and
|
||||
evaluation, loopback network, operator UI, and evidence-freshness policy:
|
||||
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039. CUDA,
|
||||
bitsandbytes, MPS, MLX, and remote execution were not selected and therefore
|
||||
have no fresh support evidence.
|
||||
have no fresh support evidence or support claim.
|
||||
|
||||
The first six milestones and Wave A established a sane, evidence-producing
|
||||
baseline. The next program is deliberately risk-weighted: it improves contract
|
||||
durability, failure behavior, integration confidence, and real-environment
|
||||
evidence before raising blanket percentages. The legacy contributor merge
|
||||
train is eligible to resume one exact-head PR at a time now that Gate 1 passes.
|
||||
Ordinary new feature work is parked until Gate 2. Planning, test-only changes,
|
||||
defect fixes discovered by the program, and the minimum refactors needed to
|
||||
expose deterministic seams remain allowed.
|
||||
The foundation, Wave A, Gate 1, and Gate 2 now establish a sane,
|
||||
evidence-producing baseline. The contributor merge train and ordinary feature
|
||||
work may proceed one exact-head PR at a time under the per-PR test contract.
|
||||
Wave C and Wave D continue to deepen property, mutation, determinism, and real
|
||||
environment evidence without weakening the mandatory CPU baseline.
|
||||
|
||||
## Reasoning
|
||||
|
||||
@@ -318,11 +317,14 @@ Completed baseline:
|
||||
5. Research-integrity, property, repeat, and selected mutation gates.
|
||||
6. Conditional backend/service workflows and mature CPU-testable targets.
|
||||
|
||||
Next program:
|
||||
Completed testing-first gates:
|
||||
|
||||
7. Completed foundation policy/evidence lock.
|
||||
8. Wave B1 high-consequence CPU contracts and legacy-train entry Gate 1.
|
||||
9. Wave B2 offline vertical slices and ordinary-work entry Gate 2.
|
||||
|
||||
Next program:
|
||||
|
||||
10. Wave C property, mutation, determinism, and regression depth.
|
||||
11. Wave D conditional environment and release-evidence freshness.
|
||||
|
||||
|
||||
@@ -1,16 +1,20 @@
|
||||
# Gate 1 regression report
|
||||
# Gate 2 regression report
|
||||
|
||||
Date: 2026-08-15
|
||||
Base: `62a480fd6a8f9ef091e60ba8dbf6b0ca94a8b785`
|
||||
Base: `dadd42648a05620298ec3614fa23864684c0b201`
|
||||
|
||||
- Mandatory CPU regressions: none; 1,468 tests pass with no warnings.
|
||||
- Mandatory CPU regressions: none; 1,541 tests pass with no warnings on Python 3.10-3.12.
|
||||
- Determinism regressions: none; three orders and hash seeds are green.
|
||||
- Packaging regressions: none in wheel, sdist, metadata, import, or CLI contracts.
|
||||
- Supply-chain regressions: none in vulnerability, secret, license, lock, or SBOM checks.
|
||||
- Coverage direction: repository and mature-scope measures exceed every Gate 1 target.
|
||||
- Conditional software regressions: none in model/evaluation, loopback network,
|
||||
operator UI, or current-SHA freshness validation.
|
||||
- Coverage direction: +5.27 statement points and +4.25 branch points over the
|
||||
exact base; repository and mature-scope measures exceed every Gate 2 target.
|
||||
|
||||
The exact committed-head comparison passed with 100% changed-line coverage and
|
||||
no line or branch regression across all four touched production modules. Hosted
|
||||
CI independently regenerated coverage for the exact base SHA before merge, and
|
||||
all seven post-merge jobs passed at
|
||||
`37d008d462a6c3b2f04bd9046593e728870227a2`.
|
||||
Gate 2 changed tests and policy only, so no production module required a
|
||||
changed-line comparison. Hosted CI independently regenerated exact-base
|
||||
coverage, verified the absence of touched production modules, and passed all
|
||||
seven post-merge jobs at
|
||||
`fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`:
|
||||
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Gate 1 test execution report
|
||||
# Gate 2 test execution report
|
||||
|
||||
Date: 2026-08-15
|
||||
Scope: mandatory offline CPU baseline, quality depth, packaging, and supply chain
|
||||
@@ -8,38 +8,42 @@ Status: PASS
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| Mandatory CPU | 1,468 passed, 9 conditional deselections, 0 failures/skips/warnings |
|
||||
| Repository coverage | 70.20% statements, 57.41% branches |
|
||||
| Mature CPU scope | 92.23% statements, 80.36% branches |
|
||||
| Changed executable lines | 100% at audited PR head; four touched modules without regression |
|
||||
| Repeat | 316 tests × 3 orders/seeds; no flakes or consistent failures |
|
||||
| Mandatory CPU | 1,541 passed, 9 conditional deselections, 0 failures/skips/warnings on Python 3.10-3.12 |
|
||||
| Repository coverage | 75.35% statements, 61.49% branches |
|
||||
| Mature CPU scope | 92.69% statements, 81.47% branches |
|
||||
| Changed executable lines | Not applicable; Gate 2 changed tests and policy only |
|
||||
| Repeat | 388 tests x 3 orders/seeds; no flakes or consistent failures |
|
||||
| Mutation | 809/886 killed (91.31%); 72 survived; 5 timed out |
|
||||
| Package | wheel and sdist build, metadata, isolated imports, and both CLI paths pass |
|
||||
| Supply chain | 0 vulnerabilities, 0 secret findings, 100 licenses accepted, SBOM bound |
|
||||
| Hosted CI | all seven PR and post-merge jobs pass on Python 3.10–3.12 |
|
||||
| Installed vertical slice | Local tiny model through CLI/config, checkpoint reload, evaluation, and JSON/CSV report passes from a non-editable install |
|
||||
| Conditional software gates | Current-head model/evaluation, loopback network, operator UI, and freshness policy pass |
|
||||
| Hosted CI | all seven post-merge jobs pass on Python 3.10-3.12 |
|
||||
|
||||
The mandatory lane ran in a clean non-editable Python 3.12 environment with
|
||||
offline Hugging Face and accelerator settings. Wall time was 68.22 seconds and
|
||||
maximum RSS was 791,476 KB. The mutation lane ran under the CI-pinned Python
|
||||
3.12/tool configuration in 69.79 seconds with maximum RSS of 1,399,824 KB.
|
||||
The exact post-merge run is
|
||||
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31866177503 at
|
||||
`37d008d462a6c3b2f04bd9046593e728870227a2`.
|
||||
The hosted pytest selections complete in 67.42 seconds on Python 3.10, 127.69
|
||||
seconds on Python 3.11, and 136.33 seconds on Python 3.12. The slowest complete
|
||||
test job is 4 minutes 52 seconds, below the ten-minute Gate 2 budget. The repeat
|
||||
campaign completes in 47.15 seconds and the selective mutation score remains
|
||||
91.31%. The exact post-merge run is
|
||||
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868380123 at
|
||||
`fa99d5a424f8725f6e7e0a97af4f8abd7bcf1f56`.
|
||||
|
||||
## Defects found and corrected
|
||||
## Gate 2 contracts added
|
||||
|
||||
1. The lm-eval adapter treated a measured accuracy of `0.0` as missing.
|
||||
2. Public sanitizers did not reliably remove Windows paths on non-Windows hosts.
|
||||
3. Report heatmap generation surfaced a dependency deprecation as an unexpected warning.
|
||||
4. The evaluation package eagerly loaded heavyweight model dependencies for pure adapters;
|
||||
it now preserves the public API through lazy imports.
|
||||
|
||||
Each correction has a focused regression contract. Evaluation/reporting and
|
||||
telemetry failure contracts are also part of the deterministic repeat gate;
|
||||
the pure lm-eval adapter is part of bounded mutation testing.
|
||||
1. Bayesian optimizer, informed pipeline, strategy navigation, model profile,
|
||||
sweep, package-export, evaluation, and retry boundaries have deterministic
|
||||
positive, negative, and malformed-input contracts.
|
||||
2. A non-editable installed package executes a local tiny-model CLI path from
|
||||
obliteration through checkpoint reload, YAML-configured evaluation, and
|
||||
JSON/CSV reporting.
|
||||
3. Software-only conditional evidence must match the candidate SHA. A stale
|
||||
exception requires a reason, a canonical numeric OBLITERATUS issue URL, and
|
||||
an expiry within 30 days.
|
||||
4. The risk map and deterministic repeat selection own every new contract.
|
||||
|
||||
## Remaining conditional evidence
|
||||
|
||||
CUDA, bitsandbytes, MPS, MLX, remote execution, real model downloads, and live
|
||||
external evaluation remain owned by their conditional gates. They are not
|
||||
required for this offline pull-request baseline and were not rerun here.
|
||||
CUDA, bitsandbytes, MPS, MLX, and remote execution remain owned by their
|
||||
conditional gates and were not selected. Pinned-model download/evaluation,
|
||||
loopback network, and operator UI were rerun successfully at the canonical SHA:
|
||||
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039.
|
||||
|
||||
+2
-2
@@ -43,11 +43,11 @@ accelerator, or remote-execution credentials.
|
||||
Canonical required checks:
|
||||
|
||||
- the exact Ruff F and actionlint command set in [.github/workflows/ci.yml](.github/workflows/ci.yml);
|
||||
- `python -m pytest` with at least 70% repository line coverage and 55% branch
|
||||
- `python -m pytest` with at least 75% repository line coverage and 60% branch
|
||||
coverage;
|
||||
- at least 90% changed-line coverage plus no line or branch regression in any
|
||||
touched production module, compared with coverage from the exact base commit;
|
||||
- at least 90% line and 78% branch coverage for the documented mature
|
||||
- at least 92% line and 80% branch coverage for the documented mature
|
||||
CPU-testable scope, plus a 75% selective mutation score and zero unexpected
|
||||
warnings;
|
||||
- `python -m build --sdist --wheel`
|
||||
|
||||
Reference in New Issue
Block a user