diff --git a/.aiwg/testing/gate3-final-report.md b/.aiwg/testing/gate3-final-report.md new file mode 100644 index 0000000..f4ee8e7 --- /dev/null +++ b/.aiwg/testing/gate3-final-report.md @@ -0,0 +1,110 @@ +# Gate 3 final baseline and release report + +Date: 2026-08-16 +Exact measured canonical commit: +`42b30f7e5b8ee596b3b0b039b10af16f01deec1e` +Integration: PR #111, signature-preserving fast-forward +Status: measured criteria PASS; final report PR audit and canonical +reconciliation remain required before ordinary feature work resumes + +## Exit-criterion results + +| Gate | Canonical result | +|---|---| +| Mandatory Python 3.12 lane | 2,119 selected tests passed, 9 conditional tests deselected by policy, and 0 failed, errored, or skipped in 123.069s | +| Repository coverage | 83.49% statements / 71.04% branches; floors are 80% / 65% | +| Mature CPU coverage | 94.02% statements / 84.54% branches; immutable floors are 94% / 84% | +| Changed executable lines | 95% blocking floor; Gate 3 production increments met the floor, including 58/58 on item 7 | +| Selected mutation | 1,844/2,057 killed (89.65%); 208 survived and 5 timed out; blocking floor is 85% | +| Repeatability | 702 tests passed in each of three order/hash-seed runs; 0 flakes, failures, or skips; 50.907s total | +| Duration | Complete Python 3.12 suite 123.069s, repeat gate 50.907s, mutation 1,872.35s; all within their 10m/5m/45m budgets | +| Portability and delivery | Python 3.10/3.11/3.12, Windows checkpoint contracts, package, Ruff/actionlint, and supply chain all passed | +| Software conditional evidence | Exact-head run 31950226930 and canonical run 31952128889 passed policy, pinned model, external evaluation, loopback network, operator UI, and final freshness summary | +| Accelerator operator evidence | Titan RTX 4090 passed CUDA and bitsandbytes; Mutsu Apple M4 / 16 GB passed MPS and MLX at signed implementation head `fa233fd8c97d2a1463141535f2a288b264abe93b` | +| Unscheduled environments | GitHub accelerator-runner registration and remote execution remain explicit no-support/no-correctness/no-performance waivers through 2026-09-15 under issue #110 | +| Review integrity | PR #111 was audited at immutable head, had no findings or review threads, passed all eight mandatory jobs, and preserved four publish-key signatures on `main` | + +The canonical mandatory run for the measured commit is +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31952112615. +It passed all eight mandatory jobs on the first attempt. The result above also +matches the successful exact-merge candidate run +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31950217640. + +## Delivered increments + +| Item | Canonical commit | Delivery and protected behavior | +|---:|---|---| +| 1 | `256c39ea6a492749a4db44146830e9d78fd3fed8` | Adopted quantitative Gate 3 policy, duration budgets, and initial measurements | +| 2 | `2c2b38b501f94af46ffac3a772f7bf9475154dc5` | Added independent numerical, property, and metamorphic oracles (PR #102) | +| 3 | `c1b34503ddd3cb797e5d70671c47afcabfe73832` | Added loader, architecture, dtype, quantization, and shared-weight decision contracts (PR #103) | +| 4 | `d7d2ad566af778b1d315616bbea8b8a3c4dd2191` | Added checkpoint failure injection, atomicity, retry, cleanup, concurrency, and Windows contracts (PR #104) | +| 5 | `aa182cc44883890c019c393f17602e2e2702e7d7` | Added BESTIARY, model-client, transport, and watchtower state contracts (PR #105) | +| 6 | `9683e0be4d892e6a6a134b45a065e591e139da3c` | Added tournament, interactive, and UI decision-seam contracts (PR #108) | +| 7 | `b961623513a0e137b959f608ce31511d59e85888` | Added the tiny-model vertical slice and safe quantized/tied-weight restoration semantics (PR #109) | +| 8 | `42b30f7e5b8ee596b3b0b039b10af16f01deec1e` | Added conditional evidence governance, expiring claim waivers, and a functional same-version CUDA Torch overlay (PR #111) | + +The AIWG workspace and CI contract were also refreshed and reconciled at +`e8ac3b65670d696ed5b68f06adf14706bc2ff865` (PR #107). The deployed AIWG +version is 2026.8.11, and the generated provider context continues to route +tracker and delivery authority through `.aiwg/aiwg.config`. + +## Evidence integrity and claim boundary + +The mandatory CI run tests the GitHub pull-request merge candidate; separate +manual conditional runs bind their JSON and JUnit evidence to the exact source +commit. Every selected software conditional record for item 8 names +`42b30f7e5b8ee596b3b0b039b10af16f01deec1e`, reports zero failures, errors, +or skips, and the summary has an empty failure list. + +The Titan and Mutsu probes used isolated temporary directories, +checksum-verified uv 0.12.4 binaries, managed Python 3.12.13, the exact signed +source archive, and the committed dependency lock. Their temporary remote +directories were removed after evidence retrieval. These probes establish only +the recorded operations. They do not replace scheduled GitHub lanes or create +a broad accelerator support, compatibility, correctness, or performance claim. + +Issue https://github.com/elder-plinius/OBLITERATUS/issues/110 remains open for +durable GitHub runner registration and least-privileged remote-execution +coverage. Waivers fail closed if malformed, expired, longer than 30 days, or +detached from that canonical tracker. + +## Evidence locations + +- PR #111 audit: + https://github.com/elder-plinius/OBLITERATUS/pull/111#issuecomment-5307860827 +- Exact-head conditional run: + https://github.com/elder-plinius/OBLITERATUS/actions/runs/31950226930 +- Canonical conditional run: + https://github.com/elder-plinius/OBLITERATUS/actions/runs/31952128889 +- Downloaded exact-head conditional artifacts: + `/tmp/obliteratus-item8-final-head-eJ79m7` +- Downloaded exact-head quality artifacts: + `/tmp/obliteratus-item8-quality-au2hsV` +- Downloaded canonical CI and conditional artifacts: + `/tmp/obliteratus-item8-canonical-b7MJu4` +- Exact-head hardware JSON/JUnit evidence: + `/tmp/obliteratus-item8-hardware-evidence-GqqBSm` +- Hardware evidence and waiver tracker comment: + https://github.com/elder-plinius/OBLITERATUS/issues/110#issuecomment-5307675226 +- Per-increment detail: + `gate3-increment-2-report.md` through `gate3-increment-8-report.md` + +Canonical SHA-256 anchors are `143c42b579c91f5dc6d1d82fd74d10c0295056dd913986da178eb22b4e491275` +for mutation statistics, `11038dbd3f27c152cf057a3ae806a845cefab8dec9b55fbd26b5fe94c4cf21bb` +for the quality trend, `3b69fe70763b0c8fc82875a4bc9febca203a7b4c384be4605c7d1746ae0ab230` +for Python 3.12 coverage, and +`bc72408d88b4d2c480ebfaeae958a4ad0db0d13790adaa1ac52aee1f92e48671` +for the conditional summary. + +## Release decision + +The measured canonical commit satisfies every quantitative Gate 3 CPU, +coverage, mutation, repeatability, portability, delivery, and fresh-software +conditional criterion. There are no active flaky quarantines and no selected +gate was converted to green through a skip or retry. + +The remaining accelerator and remote gaps are visible, owned, expiring, and +claim-blocking rather than silently treated as support evidence. Subject to a +clean exact-head audit, mandatory CI, signed integration, and post-merge +reconciliation for this report-only item 9, the testing-depth pause may end and +the contributor merge train may resume one immutable PR head at a time. diff --git a/.aiwg/testing/gate3-increment-8-report.md b/.aiwg/testing/gate3-increment-8-report.md index 5570549..b97ddfd 100644 --- a/.aiwg/testing/gate3-increment-8-report.md +++ b/.aiwg/testing/gate3-increment-8-report.md @@ -3,8 +3,13 @@ Date: 2026-08-16 Exact base: `b961623513a0e137b959f608ce31511d59e85888` Exact policy implementation head: `fa233fd8c97d2a1463141535f2a288b264abe93b` -Status: PASS locally, on exact-head software infrastructure, and on Titan/Mutsu -operator hardware; hosted PR verification and canonical merge remain pending +Exact canonical head: `42b30f7e5b8ee596b3b0b039b10af16f01deec1e` +Canonical mandatory run: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31952112615 +Canonical conditional run: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31952128889 +Status: PASS; PR #111 was exact-head audited, signature-preserving merged, and +canonically reconciled ## Results @@ -14,13 +19,14 @@ operator hardware; hosted PR verification and canonical merge remain pending | Policy regression set | 110 policy and CI-contract tests passed | | Mandatory CPU lane | Python 3.12: 2,122 passed and 6 environment-dependent tests skipped in 81.44s; repository coverage 83.51% statements | | Policy validation | Committed conditional policy passes direct CLI validation on 2026-08-16 | -| Software conditional gates | Exact-head run 31949881557 passed pinned-model runtime, external evaluation, loopback network, operator UI, policy, and final summary | +| Software conditional gates | Exact final-head run 31950226930 and canonical run 31952128889 passed pinned-model runtime, external evaluation, loopback network, operator UI, policy, and final summary | +| Canonical mandatory gates | Run 31952112615 passed Python 3.10/3.11/3.12, Windows, package, Ruff/actionlint, supply chain, repeatability, and mutation depth | | CUDA and bitsandbytes | Titan RTX 4090: Torch 2.13.0+cu130 / CUDA 13.0; both gate invocations passed 2/2 at exact head; GitHub runner registration remains waived through 2026-09-15 under issue #110 | | Apple MPS | Mutsu Apple M4 / 16 GB: native Torch 2.13.0 reports MPS built and available; gate passed 1/1 at exact head; GitHub runner registration remains waived through 2026-09-15 | | Apple MLX | Mutsu Apple M4 / 16 GB: MLX 0.32.0 and MLX-LM 0.31.3; gate passed 1/1 at exact head; GitHub runner registration remains waived through 2026-09-15 | | Remote execution | Required repository variables and secrets are unset; waived through 2026-09-15 under issue #110; no support, compatibility, correctness, or performance claim | | Runner inventory | Titan exposes Gitea GPU/host runners only; Mutsu exposes a Colima builder but no runner process; the GitHub token also receives HTTP 403 from the repository-runner inventory API | -| Signing | Policy commit is signed and verified with publish-key fingerprint `62297562B1C7053088F405DB0117DAAA677A5BF2` | +| Signing | All four item-8 commits are signed and verified with publish-key fingerprint `62297562B1C7053088F405DB0117DAAA677A5BF2` | ## Contracts added @@ -74,9 +80,15 @@ register maintainable GitHub runners. ## Verification and evidence - Exact-head software conditional run: - https://github.com/elder-plinius/OBLITERATUS/actions/runs/31949881557 + https://github.com/elder-plinius/OBLITERATUS/actions/runs/31950226930 +- Canonical software conditional run: + https://github.com/elder-plinius/OBLITERATUS/actions/runs/31952128889 +- Canonical mandatory run: + https://github.com/elder-plinius/OBLITERATUS/actions/runs/31952112615 - Downloaded software JSON/JUnit evidence: - `/tmp/obliteratus-item8-software-final-GMGS0x` + `/tmp/obliteratus-item8-final-head-eJ79m7` +- Downloaded canonical CI and conditional evidence: + `/tmp/obliteratus-item8-canonical-b7MJu4` - Exact-head hardware JSON/JUnit and SHA-256 evidence: `/tmp/obliteratus-item8-hardware-evidence-GqqBSm` - Focused local command: @@ -89,11 +101,13 @@ register maintainable GitHub runners. https://github.com/elder-plinius/OBLITERATUS/issues/110 - Hardware-evidence tracker comment: https://github.com/elder-plinius/OBLITERATUS/issues/110#issuecomment-5307675226 +- Exact-head audit: + https://github.com/elder-plinius/OBLITERATUS/pull/111#issuecomment-5307860827 ## Scope and release decision This increment changes conditional-test governance and CUDA environment setup. It does not alter production behavior or weaken mandatory CPU, package, coverage, repeat, mutation, Windows, or supply-chain gates. Gate 3 item 9 and -the final exact-canonical report remain required before the contributor merge -train resumes. +its final report audit and canonical reconciliation remain required before the +contributor merge train resumes. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index 0e80c81..15e2a81 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -27,26 +27,30 @@ Gate 3 increment 6 canonical commit: Gate 3 increment 7 canonical commit: `b961623513a0e137b959f608ce31511d59e85888` (PR #109, signed fast-forward) +Gate 3 increment 8 canonical commit: +`42b30f7e5b8ee596b3b0b039b10af16f01deec1e` (PR #111, signed fast-forward) + Current canonical AIWG and CI-contract baseline: `e8ac3b65670d696ed5b68f06adf14706bc2ff865` (PR #107, rebase merged) -The latest canonical mandatory offline CPU lane passes 2,112 tests on Python +The latest canonical mandatory offline CPU lane passes 2,119 tests on Python 3.12 with 9 conditional tests deselected by policy, no failures, errors, skips, -or unexpected warnings, and 83.50% statement / 71.08% branch coverage. Mature +or unexpected warnings, and 83.49% statement / 71.04% branch coverage. Mature CPU-testable coverage is 94.02% / 84.54%. The canonical mutation campaign kills 1,844 of 2,057 mutants (89.65%), with 208 survivors and 5 timeouts, while 702 tests pass in each of three file orders and hash seeds with no flakes. Package, Windows checkpoint, supply-chain, and all other hosted jobs passed on attempt 1: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31947837466. +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31952112615. -Gate 3 increment 8 is implemented at signed policy head -`fa233fd8c97d2a1463141535f2a288b264abe93b`. Fresh pinned-model download, -external-evaluation, loopback-network, and operator-UI evidence passes at that -exact head in run 31949881557. Titan CUDA/bitsandbytes and Mutsu MPS/MLX operator -probes also pass at that exact head. GitHub runner registration and remote -execution remain time-bounded no-support/no-correctness/no-performance waivers -through 2026-09-15 under issue #110. Details are in -`gate3-increment-8-report.md`. +Gate 3 increment 8 is canonical at signed head +`42b30f7e5b8ee596b3b0b039b10af16f01deec1e`. Fresh pinned-model download, +external-evaluation, loopback-network, operator-UI, and final-summary evidence +passes at that exact head in runs 31950226930 and 31952128889. Titan +CUDA/bitsandbytes and Mutsu MPS/MLX operator probes pass at signed implementation +head `fa233fd8c97d2a1463141535f2a288b264abe93b`. GitHub runner registration and +remote execution remain time-bounded no-support/no-correctness/no-performance +waivers through 2026-09-15 under issue #110. Details are in +`gate3-increment-8-report.md` and `gate3-final-report.md`. The foundation, Wave A, Gate 1, and Gate 2 establish a sane, evidence-producing baseline. The legacy contributor merge train proceeded one @@ -55,8 +59,9 @@ property, mutation, determinism, and real-environment evidence without weakening the mandatory CPU baseline. The operator has now placed ordinary feature work back behind a testing-depth -gate. Gate 3 is adopted and items 1–7 are canonical; conditional reconciliation -and the final canonical report must pass before unrelated feature work resumes. +gate. Gate 3 is adopted and items 1–8 are canonical; the final report must pass +exact-head review and canonical reconciliation before unrelated feature work +resumes. Correctness, security, data-loss, and test-infrastructure repairs remain permitted when narrowly scoped and paired with reproducing tests. @@ -295,15 +300,15 @@ resistance, environment evidence, or deterministic replay. | Measure | Current | Gate 3 minimum | |---|---:|---:| -| Repository statements | 83.50% item-7 candidate | 80.00% | -| Repository branches | 71.08% item-7 candidate | 65.00% | -| Mature CPU statements | 94.02% item-7 candidate | 94.00% | -| Mature CPU branches | 84.54% item-7 candidate | 84.00% | +| Repository statements | 83.49% item-8 canonical | 80.00% | +| Repository branches | 71.04% item-8 canonical | 65.00% | +| Mature CPU statements | 94.02% item-8 canonical | 94.00% | +| Mature CPU branches | 84.54% item-8 canonical | 84.00% | | Changed executable lines | 95.00% floor | 95.00% floor | -| Selected mutation score | 89.65% item-7 candidate; 1,844/2,057 killed | at least 85% on expanded scope | +| Selected mutation score | 89.65% item-8 canonical; 1,844/2,057 killed | at least 85% on expanded scope | | Unexpected warnings | 0 | 0 | | Active flaky quarantines | 0 | 0 | -| Software conditional evidence age | 1 day at the item-6 candidate date | at most 8 days | +| Software conditional evidence age | same-day item-8 canonical evidence | at most 8 days | The expanded mutation scope must include core mutation math, architecture and loader decisions, persistence/atomicity, evaluation/report serialization, and @@ -400,12 +405,13 @@ single repository-wide coverage PR. Float-to-integer restoration regression. The canonical head reaches 94.02% mature CPU statements and 84.54% branches, raises the immutable floors to 94% / 84%, and passes all eight hosted checks. -8. **Implemented at signed policy head `fa233fd`.** Reconcile fresh software - evidence and explicitly waive each unavailable hardware/remote environment - under the claim policy. Exact-head conditional evidence, PR audit, and - canonical merge remain pending. -9. Publish the Gate 3 execution, coverage, mutation, repeat, duration, and - conditional-evidence reports at one canonical `main` commit. +8. **Completed at `42b30f7` (PR #111).** Reconcile fresh software evidence, + run exact-head Titan/Mutsu operator probes, and explicitly waive unavailable + scheduled runner/remote lanes under the claim policy. The signed commits + were preserved by fast-forward integration. +9. **In progress.** Publish the Gate 3 execution, coverage, mutation, repeat, + duration, and conditional-evidence report, then exact-head audit, signed + merge, and canonically reconcile the report-only commit. #### Per-PR acceptance evidence