From 04e757eda8ecc5b3fb876d5059f16f1015fb4e74 Mon Sep 17 00:00:00 2001 From: Joseph Magly <1159087+jmagly@users.noreply.github.com> Date: Sun, 16 Aug 2026 07:53:41 -0400 Subject: [PATCH] docs: record Gate 3 increment 7 evidence --- .aiwg/testing/gate3-increment-6-report.md | 26 +++-- .aiwg/testing/gate3-increment-7-report.md | 120 ++++++++++++++++++++++ .aiwg/testing/master-test-plan.md | 25 +++-- 3 files changed, 150 insertions(+), 21 deletions(-) create mode 100644 .aiwg/testing/gate3-increment-7-report.md diff --git a/.aiwg/testing/gate3-increment-6-report.md b/.aiwg/testing/gate3-increment-6-report.md index 4964471..d95bcc4 100644 --- a/.aiwg/testing/gate3-increment-6-report.md +++ b/.aiwg/testing/gate3-increment-6-report.md @@ -4,8 +4,9 @@ Date: 2026-08-16 Exact base: `e8ac3b65670d696ed5b68f06adf14706bc2ff865` Exact core implementation and local evidence head: `77adf119f871ba820f1f02d55fe2760aa24c84b2` -Status: PASS locally; exact-head PR audit, hosted matrix, and canonical merge -remain pending +Exact canonical head: `9683e0be4d892e6a6a134b45a065e591e139da3c` +Canonical run: https://github.com/elder-plinius/OBLITERATUS/actions/runs/31941334488 +Status: PASS; PR #108 was exact-head audited and canonically merged ## Results @@ -26,9 +27,12 @@ remain pending | Repeatability | 669 tests passed in each of three file orders/hash seeds; 35.353s total; no flake candidates or consistent failures | | Policy | Ruff F, conditional policy, source-to-test risk map, mutation-target guard, global/mature coverage, changed-line, and touched-module gates passed | -The repository statement and branch exits now both exceed the Gate 3 targets. -The remaining new-work hold is bounded item 7, followed by conditional-evidence -reconciliation and final canonical reporting. +The repository statement and branch exits both exceed the Gate 3 targets. The +canonical run passed all eight required checks, including 2,059 tests on each +of Python 3.10, 3.11, and 3.12; 89.50% selected mutation score (1,816/2,029); +three repeat orders with 669 tests each and no flakes; Windows checkpoint, +package, and supply-chain evidence. The remaining new-work hold is bounded item +7, followed by conditional-evidence reconciliation and final reporting. ## Contracts added @@ -101,12 +105,12 @@ root/model-directory resolution, raising the final changed-line result to - Mutation metadata and prepared selection evidence: ignored worktree-local `mutants/` artifacts at the exact implementation head -The exact implementation head passes the enforced Ruff F scope, immutable -quality policy, conditional-policy validation, risk-map validation, mutation -required-target guard, 75% statement and 60% branch floors, 95% changed-line -floor, and touched-module no-regression gate. The full 2,029-mutant campaign, -package matrix, supply-chain checks, exact-head audit, and canonical evidence -remain hosted PR/merge gates. +The exact implementation and canonical heads pass the enforced Ruff F scope, +immutable quality policy, conditional-policy validation, risk-map validation, +mutation required-target guard, 75% statement and 60% branch floors, 95% +changed-line floor, and touched-module no-regression gate. The hosted +2,029-mutant campaign, package matrix, supply-chain checks, exact-head audit, +and canonical reconciliation are complete. ## Scope and release decision diff --git a/.aiwg/testing/gate3-increment-7-report.md b/.aiwg/testing/gate3-increment-7-report.md new file mode 100644 index 0000000..f274b52 --- /dev/null +++ b/.aiwg/testing/gate3-increment-7-report.md @@ -0,0 +1,120 @@ +# Gate 3 increment 7 tiny-runtime and quantized-semantics report + +Date: 2026-08-16 +Exact base: `9683e0be4d892e6a6a134b45a065e591e139da3c` +Exact implementation head: `acc6b3b254e6d358ddabe495b1ca79295dd36d70` +Status: PASS locally; signed exact-head PR audit, hosted matrix, and canonical +merge remain pending + +## Results + +| Gate | Result | +|---|---| +| Focused contracts | 267 restoration, runtime, offline-model, and policy tests passed; the final runtime set is 111/111 green | +| Mandatory CPU lane | Python 3.12: 2,112 passed, 9 conditional tests deselected, 0 failed, and 0 skipped in 87.70s | +| Repository coverage | 82.76% statements / 69.55% branches before; 83.50% / 71.08% after | +| Mature CPU scope | 93.27% / 83.06% before; 94.02% statements / 84.54% branches after | +| Immutable mature floor | Raised from 92% / 80% to 94% / 84% in policy and validator | +| Changed executable lines | 58/58 (100%) against the exact base before documentation-only changes | +| Core pipeline | `abliterate.py`: 50.91% / 38.03% before; 52.63% statements / 40.73% branches after | +| Adaptive defaults | 93.59% / 81.65% before; 100% statements / 89.87% branches after | +| Advanced metrics | 91.73% / 82.19% before; 100% statements / 98.61% branches after | +| Loader decisions | 92.14% / 90.00% before; 100% statements / 96.67% branches after | +| Runtime contracts | 100% statements / 100% branches, including the new restoration policy | +| Selected mutation | 1,844/2,057 killed (89.65%); 208 survived and 5 timed out outside changed logic; every restoration-helper mutant killed | +| Repeatability | 702 tests passed in each of three file orders/hash seeds in 36.857s; no flakes, skips, or consistent failures | +| Package | Wheel and sdist built at version 0.1.2; both pass Twine metadata validation | +| Policy | Ruff, immutable quality policy, conditional policy, risk map, mutation-target guard, global/mature coverage, and changed-line gates passed | + +The item-7 candidate reaches every quantitative Gate 3 repository and mature +CPU exit criterion. Ordinary feature work remains held until this increment is +merged and the conditional-evidence and final-reporting items are reconciled. + +## Contracts added + +1. Multi-direction norm restoration measures logical dequantized weights, + restores once after the full subspace projection, caps amplification, and + fails closed for non-finite or degenerate norms. +2. Integer storage is promoted to a logical float parameter instead of casting + a projected float back to Byte/uint8 and erasing the result. The exact + Float-to-integer failure reported in issue #11 is covered. +3. Float and integer restoration preserve `Parameter` identity. Quantized + replacement rebinds every module alias to the same replacement parameter, + preserving tied/shared-weight semantics. +4. Packed modules use their repacker when available, explicitly materialize a + float weight with a warning when safe, and fail closed when they expose + neither a repacker nor writable logical weight. +5. A deterministic, generated GPT-2 fixture executes the real advanced + two-direction pipeline, saves and reloads the checkpoint, proves weights + changed, and independently verifies every transformer-layer matrix norm. +6. Adaptive-default knowledge lookup and merge boundaries, advanced-metric + tensor/refusal/KL/CKA boundaries, and loader import-compatibility failures + now have executable negative contracts rather than uncovered decision paths. +7. The source-to-test risk map now makes projection-math contracts mandatory + whenever the core abliteration pipeline changes. + +## Defects detected and corrected + +The pre-change norm restoration iterated raw parameters, converted integer +weights to float for arithmetic, and copied the result back to the original +integer dtype. That reproduces the reported PyTorch Float-to-Byte failure or, +where a cast succeeds, silently destroys fractional projection and rescaling +values. It also measured packed bytes instead of logical dequantized weights. + +The corrected path classifies storage before restoration, uses the model's +dequantization and replacement seams, and keeps integer values in float storage +when no quantization scale/zero-point contract exists. The first tied-weight +review then exposed a second defect: replacing one quantized parameter could +break aliases. Alias rebinding now preserves the tie after replacement. + +The changed-line gate initially reported 43/47 covered lines. Packed fallback, +degenerate no-op, and in-place float restoration contracts raised the final +production result to 58/58. Coverage review also found a duplicate unreachable +empty-sequence check in first-token KL validation; the shared tensor validator +already rejects that input, so the dead branch was removed. + +## Verification and evidence + +- Mandatory Python 3.12 coverage: + `/tmp/obliteratus-item7-final2-coverage-py3.12.json` +- Mandatory Python 3.12 JUnit: + `/tmp/obliteratus-item7-final2-junit-py3.12.xml` +- Exact-base Python 3.12 coverage: + `/tmp/obliteratus-main-9683e0b-zvr8hF/test-evidence-py3.12/coverage-py3.12.json` +- Clean non-editable Python 3.12 test environment: + `/tmp/obliteratus-item7-py312-env-AxUcVD` +- Mutation artifacts: + ignored worktree-local `mutants/` evidence prepared from the exact candidate; + final score 89.65% (1,844/2,057) with no changed-helper survivor +- Repeat evidence: `/tmp/obliteratus-item7-repeat-gate.json` +- Package artifacts: `/tmp/obliteratus-item7-package-QBOhVU/`; SHA-256 + `34dcd464df0661bf63da3a9c4e406b7acbfa73e6fd09127aef6211d7a30a249e` + (wheel) and + `f487391c98e4931896238b38b5232fb2f7c08147ec90767328734bf04c5c5c69` + (sdist) + +The first isolated sync selected unsupported Python 3.13 and was not used as +evidence. A separate reused Python 3.12 environment correctly failed two +installed-artifact provenance assertions because it imported a different +worktree. The final environment was recreated non-editably from this worktree, +its isolated import resolved to its own `site-packages`, and all 2,112 selected +tests passed. + +## Signing and integration policy + +Every candidate commit is signed with the configured repository publish key +fingerprint `62297562B1C7053088F405DB0117DAAA677A5BF2` through the approved host +GPG wrapper and verified locally with `git verify-commit`. Integration must +preserve that signature by exact fast-forward or create a locally signed merge +commit. GitHub rebase merge is not acceptable because it recreates unsigned +canonical commits. + +## Scope and release decision + +This increment uses a synthetic random-initialized offline model and isolated +temporary filesystems. It makes no network, external-model, accelerator, +credential, or publication claim. The exact-head PR must still pass all hosted +CPU, repeat, mutation, package, Windows, and supply-chain gates before a +signature-preserving merge. Conditional evidence reconciliation and final Gate +3 reporting remain open after merge, so ordinary feature work does not resume +yet. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index da5c3a4..902fc90 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -299,12 +299,12 @@ resistance, environment evidence, or deterministic replay. | Measure | Current | Gate 3 minimum | |---|---:|---:| -| Repository statements | 82.78% item-6 candidate | 80.00% | -| Repository branches | 69.62% item-6 candidate | 65.00% | -| Mature CPU statements | 93.27% item-6 candidate; 73 lines remain | 94.00% | -| Mature CPU branches | 83.06% item-6 candidate; 29 branches remain | 84.00% | +| Repository statements | 83.50% item-7 candidate | 80.00% | +| Repository branches | 71.08% item-7 candidate | 65.00% | +| Mature CPU statements | 94.02% item-7 candidate | 94.00% | +| Mature CPU branches | 84.54% item-7 candidate | 84.00% | | Changed executable lines | 95.00% floor | 95.00% floor | -| Selected mutation score | 89.46% item-5 canonical; item-6 campaign expands to 2,029 mutants | at least 85% on expanded scope | +| Selected mutation score | 89.65% item-7 candidate; 1,844/2,057 killed | at least 85% on expanded scope | | Unexpected warnings | 0 | 0 | | Active flaky quarantines | 0 | 0 | | Software conditional evidence age | 1 day at the item-6 candidate date | at most 8 days | @@ -395,11 +395,16 @@ single repository-wide coverage PR. 5. **Completed at `aa182cc` (PR #105).** Add BESTIARY/model-client/watchtower transport and scheduler state contracts. Exact evidence is recorded in `gate3-increment-5-report.md` and the canonical PR follow-up. -6. **Implemented locally at `77adf11`.** Add tournament, interactive, and UI - decision-seam contracts. Exact local evidence is recorded in - `gate3-increment-6-report.md`; audit and canonical merge remain pending. -7. Extend the installed tiny-model vertical slice and quantized-storage - semantics, including the exact Float-to-integer restoration regression. +6. **Completed at `9683e0b` (PR #108).** Add tournament, interactive, and UI + decision-seam contracts. Canonical run 31941334488 passed all eight required + checks; exact evidence is recorded in `gate3-increment-6-report.md` and the + canonical PR reconciliation comment. +7. **Implemented locally at `acc6b3b` against `9683e0b`.** Extend the installed + tiny-model vertical slice and quantized-storage semantics, including the + exact Float-to-integer restoration regression. The candidate reaches 94.02% + mature CPU statements and 84.54% branches and raises the immutable floors + to 94% / 84%; signed commit, exact-head audit, hosted checks, and canonical + merge remain pending. 8. Reconcile conditional software evidence, then run or explicitly waive each unavailable hardware/remote environment under the claim policy. 9. Publish the Gate 3 execution, coverage, mutation, repeat, duration, and