From c1b34503ddd3cb797e5d70671c47afcabfe73832 Mon Sep 17 00:00:00 2001 From: Joseph Magly <1159087+jmagly@users.noreply.github.com> Date: Sat, 15 Aug 2026 21:40:04 -0400 Subject: [PATCH] docs: record Gate 3 loader decision evidence --- .aiwg/testing/gate3-increment-2-report.md | 11 +-- .aiwg/testing/gate3-increment-3-report.md | 93 +++++++++++++++++++++++ .aiwg/testing/master-test-plan.md | 28 ++++--- 3 files changed, 115 insertions(+), 17 deletions(-) create mode 100644 .aiwg/testing/gate3-increment-3-report.md diff --git a/.aiwg/testing/gate3-increment-2-report.md b/.aiwg/testing/gate3-increment-2-report.md index ffaa129..4ab300c 100644 --- a/.aiwg/testing/gate3-increment-2-report.md +++ b/.aiwg/testing/gate3-increment-2-report.md @@ -3,8 +3,10 @@ Date: 2026-08-15 Exact base: `369417fbb45d25e544a72246abb05e21d1ac00e0` Evidence scope: the containing commit, resolved with `git rev-parse HEAD` -Canonical merge: pending -Status: PASS locally; hosted exact-head audit and merge pending +Canonical merge: `2c2b38b501f94af46ffac3a772f7bf9475154dc5` (PR #102, +rebase merged) +Status: PASS; exact-head audit, rebase merge, post-merge CI, and issue #100 +reconciliation complete ## Results @@ -55,7 +57,6 @@ final canonical Gate 3 publication remain open. CUDA, MPS, MLX, network, download, remote, and operator-UI mappings were not changed; their six local skips remain governed by item 8 rather than being treated as CPU-gate failures. -The mutation artifacts are preserved in +The local mutation artifacts were preserved in `/tmp/obliteratus-gate3-repair-mutation-CJRWT2/quality-evidence/` for the local -audit. Hosted CI artifacts become authoritative after exact-head review and -merge. +audit. The hosted PR and post-merge CI artifacts are the canonical evidence. diff --git a/.aiwg/testing/gate3-increment-3-report.md b/.aiwg/testing/gate3-increment-3-report.md new file mode 100644 index 0000000..d40cdfe --- /dev/null +++ b/.aiwg/testing/gate3-increment-3-report.md @@ -0,0 +1,93 @@ +# Gate 3 increment 3 loader/runtime decision report + +Date: 2026-08-15 +Exact base: `2c2b38b501f94af46ffac3a772f7bf9475154dc5` +Exact implementation/evidence head: `e2e120c211adf3ec9a7909a401f5e7a34f17984f` +Delivery head: the containing documentation commit, resolved with +`git rev-parse HEAD` +Canonical merge: pending +Status: PASS locally; hosted exact-head audit and merge pending + +## Results + +| Gate | Result | +|---|---| +| Focused decision contracts | 228 passed in 18.08s; 0 failures | +| Mandatory CPU matrix | Python 3.10: 1,832 passed in 157.58s; Python 3.11: 1,832 passed in 111.93s; Python 3.12: 1,830 passed, 2 expected xfails in 61.09s; 9 conditional deselections on each interpreter | +| Repository coverage | 76.26% statements / 62.67% branches before; 76.60% / 63.25% after | +| Mature CPU scope | 92.87% statements / 82.24% branches before; 93.12% / 82.60% after | +| Changed executable lines | 77/77 (100%) against the exact base | +| Touched production modules | 3/3 pass exact-base line and branch no-regression | +| Runtime decision seam | `runtime_contracts.py`: 100% statements and 100% branches (95 statements, 50 branches) | +| Loader | 91.15% statements / 88.71% branches before; 92.14% / 90.00% after | +| Abliteration runtime | 50.22% statements / 37.38% branches before; 50.82% / 38.03% after | +| Selective mutation | 1,424/1,637 killed (86.99%); 208 survived; 5 timed out; 0 no-test, suspicious, interrupted, or segfault results | +| Mutation runtime | 910.99s; 878,908 KiB maximum parent RSS; required-target guard passed | +| Repeatability | 431 tests passed in each of three file orders/hash seeds; 35.735s total; no flake candidates or consistent failures | +| Package | sdist and wheel built; metadata, contents, isolated import, version, module entry point, and console entry point passed | +| Supply chain | Lock, all-extras import, Gitleaks, Python 3.10–3.12 vulnerability audits, bound CycloneDX SBOM, and license policy passed | + +The mandatory Python lanes remain below the 240-second policy budget. The +Python 3.12 wall-clock test time improved from the exact-base 62.73s to 61.09s +while adding three mandatory tests. The slowest focused test was the explicit +4-bit quantization configuration contract at 1.91s. + +## Contracts added + +1. `ModelLoadPolicy` and `resolve_model_load_policy` define one precedence + contract for dtype selection, native quantization, bitsandbytes modes, + device-map behavior, and post-load device movement. Conflicting or unsupported + combinations fail with exact diagnostic contracts rather than silently + changing execution mode. +2. `attention_projection_names` defines shared-KV ownership: ordinary layouts + return all projections, output-only mode returns output projections, the + shared owner projects all eligible weights, borrowers omit shared K/V and + K-normalization weights, and impossible ownership layouts fail closed. +3. `is_quantized_parameter` and `classify_weight_storage` distinguish packed + modules, quantized parameters, integer storage, and floating storage before + dequantization or replacement. Tests protect float/integer/packed read paths, + safe packed failures, repacker use, and compatibility temporary-directory + cleanup. +4. Loader and abliteration execution now consume the pure decision seams. The + tests assert propagation into the production wrappers, not only helper return + values. +5. The first exact-SHA mutation campaign exposed eight new seam survivors: one + diagnostic gap, six killable shared-KV boundary/message gaps, and one + logically redundant layer-count condition. The repair added the missing + boundaries and exact errors and removed the redundant condition. The repaired + full campaign has zero survivors in `runtime_contracts.py`. +6. `runtime_contracts.py` is now a required mutation target. A fresh exact-head + campaign generated 1,637 mutants, passed the required-target guard, and + exceeded the immutable 85% score floor without no-test results. + +## Verification and evidence + +- Python 3.12 coverage/JUnit/trend/repeat evidence: + `/tmp/obliteratus-gate3-item3-final-py312-98XhzU/` +- Python 3.10 and 3.11 coverage/JUnit/trend evidence: + `/tmp/obliteratus-gate3-item3-matrix-oKcE2R/` +- Exact-head mutation evidence: + `/tmp/obliteratus-gate3-item3-repair-mutation-PV8ArC/quality-evidence/` +- Package evidence: + `/tmp/obliteratus-gate3-item3-package-Z22lEz/` +- Supply-chain evidence: + `/tmp/obliteratus-gate3-item3-supply-uzr6r7/` + +The canonical Ruff F gate, checksum-verified `actionlint`, `uv lock --check`, +conditional-policy validation, risk-map validation, changed-line gate, +touched-module gate, immutable coverage policy, normalized duration policy, +repeat gate, mutation score, and supply-chain decisions all pass. The worktree +was clean before this documentation update. + +## Scope and release decision + +This increment implements Gate 3 bounded delivery item 3. It does not reopen +ordinary feature work: items 4–7, conditional-evidence reconciliation, and the +final canonical Gate 3 publication remain open. It changes deterministic CPU +decision logic and its wrappers only; CUDA, bitsandbytes hardware execution, +MPS, MLX, network, download, remote, and operator-UI mappings are unchanged, so +no new environment support claim is made. + +Delivery requires the containing documentation commit to be pushed, audited at +its final SHA, pass all hosted checks with no unresolved review thread, and be +rebase merged before item 4 begins. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index ef685a5..ce250de 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -9,6 +9,9 @@ Phase: construction Gate 3 increment 1 canonical commit: `256c39ea6a492749a4db44146830e9d78fd3fed8` +Gate 3 increment 2 canonical commit: +`2c2b38b501f94af46ffac3a772f7bf9475154dc5` (PR #102, rebase merged) + The mandatory offline CPU selection passes on Python 3.10–3.12 with 1,644 tests, 9 conditionally deselected tests, no skips, and no warnings. The slowest hosted pytest selection is Python 3.12 at 158.989 seconds; its complete CI job @@ -39,7 +42,7 @@ property, mutation, determinism, and real-environment evidence without weakening the mandatory CPU baseline. The operator has now placed ordinary feature work back behind a testing-depth -gate. Gate 3 is adopted and item 1 is complete, but items 2–7 and the +gate. Gate 3 is adopted and items 1–2 are complete, but items 3–7 and the quantitative exit criteria must pass before unrelated feature work resumes. Correctness, security, data-loss, and test-infrastructure repairs remain permitted when narrowly scoped and paired with reproducing tests. @@ -279,12 +282,12 @@ resistance, environment evidence, or deterministic replay. | Measure | Current | Gate 3 minimum | |---|---:|---:| -| Repository statements | 75.68% | 80.00% | -| Repository branches | 61.72% | 65.00% | -| Mature CPU statements | 92.71% | 94.00% | -| Mature CPU branches | 81.55% | 84.00% | +| Repository statements | 76.26% | 80.00% | +| Repository branches | 62.67% | 65.00% | +| Mature CPU statements | 92.87% | 94.00% | +| Mature CPU branches | 82.24% | 84.00% | | Changed executable lines | 95.00% floor | 95.00% floor | -| Selected mutation score | 91.31% on narrow scope | at least 85% on expanded scope | +| Selected mutation score | 86.15% on expanded numerical scope | at least 85% on expanded scope | | Unexpected warnings | 0 | 0 | | Active flaky quarantines | 0 | 0 | | Software conditional evidence age | at most 8 days | at most 8 days | @@ -362,12 +365,13 @@ single repository-wide coverage PR. measurements, and add duration-budget enforcement without changing production behavior. Exact canonical evidence: https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085. -2. **Implemented in Gate 3 increment 2; canonical merge pending.** Add - numerical/reference-oracle and metamorphic contracts; expand mutation to the - corresponding pure math. Local evidence is recorded in - `gate3-increment-2-report.md`. -3. Add loader, architecture, dtype, quantization, and shared-weight decision - contracts with mutation evidence. +2. **Completed at `2c2b38b` (PR #102).** Add numerical/reference-oracle and + metamorphic contracts; expand mutation to the corresponding pure math. + Canonical evidence is recorded in `gate3-increment-2-report.md`. +3. **Implemented in Gate 3 increment 3; exact-head PR audit and canonical merge + pending.** Add loader, architecture, dtype, quantization, and shared-weight + decision contracts with mutation evidence. Candidate evidence is recorded in + `gate3-increment-3-report.md`. 4. Add checkpoint failure injection, concurrency, atomicity, and retry contracts. 5. Add BESTIARY/model-client/watchtower transport and scheduler state contracts. 6. Add tournament, interactive, and UI decision-seam contracts.