diff --git a/.aiwg/testing/gate3-increment-7-report.md b/.aiwg/testing/gate3-increment-7-report.md index 0526f11..876a9a2 100644 --- a/.aiwg/testing/gate3-increment-7-report.md +++ b/.aiwg/testing/gate3-increment-7-report.md @@ -2,9 +2,11 @@ Date: 2026-08-16 Exact base: `9683e0be4d892e6a6a134b45a065e591e139da3c` -Exact implementation head: `acc6b3b254e6d358ddabe495b1ca79295dd36d70` -Status: PASS locally after the portability correction; hosted exact-head rerun -and canonical merge remain pending +Exact core implementation head: `acc6b3b254e6d358ddabe495b1ca79295dd36d70` +Exact canonical head: `b961623513a0e137b959f608ce31511d59e85888` +Canonical run: https://github.com/elder-plinius/OBLITERATUS/actions/runs/31947837466 +Status: PASS; PR #109 was exact-head audited, signature-preserving merged, and +canonically reconciled ## Results @@ -27,9 +29,11 @@ and canonical merge remain pending | Package | Wheel and sdist built at version 0.1.2; both pass Twine metadata validation | | Policy | Ruff, immutable quality policy, conditional policy, risk map, mutation-target guard, global/mature coverage, and changed-line gates passed | -The item-7 candidate reaches every quantitative Gate 3 repository and mature -CPU exit criterion. Ordinary feature work remains held until this increment is -merged and the conditional-evidence and final-reporting items are reconciled. +The canonical item-7 head reaches every quantitative Gate 3 repository and +mature CPU exit criterion. Its post-merge run passed all eight required jobs; +the quality-depth lane completed the repeat and 2,057-mutant campaign in +28m26s. Ordinary feature work remains held until the conditional-evidence and +final-reporting items are reconciled. ## Contracts added @@ -105,6 +109,8 @@ policy at 94.04% statements and 84.61% branches. (wheel) and `f487391c98e4931896238b38b5232fb2f7c08147ec90767328734bf04c5c5c69` (sdist) +- Canonical reconciliation comment: + https://github.com/elder-plinius/OBLITERATUS/pull/109#issuecomment-5307612606 The first isolated sync selected unsupported Python 3.13 and was not used as evidence. A separate reused Python 3.12 environment correctly failed two @@ -126,8 +132,8 @@ canonical commits. This increment uses a synthetic random-initialized offline model and isolated temporary filesystems. It makes no network, external-model, accelerator, -credential, or publication claim. The exact-head PR must still pass all hosted -CPU, repeat, mutation, package, Windows, and supply-chain gates before a -signature-preserving merge. Conditional evidence reconciliation and final Gate -3 reporting remain open after merge, so ordinary feature work does not resume -yet. +credential, or publication claim. The exact-head and canonical runs passed all +hosted CPU, repeat, mutation, package, Windows, and supply-chain gates, and the +signed commits were preserved on `main`. Conditional evidence reconciliation +and final Gate 3 reporting remain open, so ordinary feature work does not +resume yet. diff --git a/.aiwg/testing/gate3-increment-8-report.md b/.aiwg/testing/gate3-increment-8-report.md new file mode 100644 index 0000000..5570549 --- /dev/null +++ b/.aiwg/testing/gate3-increment-8-report.md @@ -0,0 +1,99 @@ +# Gate 3 increment 8 conditional-evidence and waiver report + +Date: 2026-08-16 +Exact base: `b961623513a0e137b959f608ce31511d59e85888` +Exact policy implementation head: `fa233fd8c97d2a1463141535f2a288b264abe93b` +Status: PASS locally, on exact-head software infrastructure, and on Titan/Mutsu +operator hardware; hosted PR verification and canonical merge remain pending + +## Results + +| Gate | Result | +|---|---| +| Focused policy contracts | 17 conditional-policy, summary, CUDA-overlay, and evidence-freshness tests passed | +| Policy regression set | 110 policy and CI-contract tests passed | +| Mandatory CPU lane | Python 3.12: 2,122 passed and 6 environment-dependent tests skipped in 81.44s; repository coverage 83.51% statements | +| Policy validation | Committed conditional policy passes direct CLI validation on 2026-08-16 | +| Software conditional gates | Exact-head run 31949881557 passed pinned-model runtime, external evaluation, loopback network, operator UI, policy, and final summary | +| CUDA and bitsandbytes | Titan RTX 4090: Torch 2.13.0+cu130 / CUDA 13.0; both gate invocations passed 2/2 at exact head; GitHub runner registration remains waived through 2026-09-15 under issue #110 | +| Apple MPS | Mutsu Apple M4 / 16 GB: native Torch 2.13.0 reports MPS built and available; gate passed 1/1 at exact head; GitHub runner registration remains waived through 2026-09-15 | +| Apple MLX | Mutsu Apple M4 / 16 GB: MLX 0.32.0 and MLX-LM 0.31.3; gate passed 1/1 at exact head; GitHub runner registration remains waived through 2026-09-15 | +| Remote execution | Required repository variables and secrets are unset; waived through 2026-09-15 under issue #110; no support, compatibility, correctness, or performance claim | +| Runner inventory | Titan exposes Gitea GPU/host runners only; Mutsu exposes a Colima builder but no runner process; the GitHub token also receives HTTP 403 from the repository-runner inventory API | +| Signing | Policy commit is signed and verified with publish-key fingerprint `62297562B1C7053088F405DB0117DAAA677A5BF2` | + +## Contracts added + +1. Every unavailable conditional lane has one + versioned waiver naming its gate, canonical issue, reason, opening date, + expiry date, and the exact claim that is blocked. +2. A waiver may last at most 30 days. Missing fields, non-object records, + duplicate gates, unknown gates, software-only gates, invalid issue URLs, + invalid or future dates, reversed dates, overlong lifetimes, and expired + records fail policy validation. +3. An unselected gate with an active waiver is emitted as + `waived_no_support_claim` with its tracker, expiry, and blocked claim. + Unselected gates without a waiver remain + `not_selected_no_fresh_evidence`. +4. Selecting and successfully executing a waived gate produces `success` and + therefore uses real evidence instead of the waiver. Selected failures remain + failures, including shared CUDA/bitsandbytes job failures. +5. An expired or malformed waiver also fails the final summary, preventing a + stale record from becoming a silent green result after the policy job. +6. Mandatory Linux CI remains CPU-only. The selected CUDA lane derives the + exact locked Torch base version, replaces only Torch with the same-version + official `cu130` build, asserts that the resulting build exposes CUDA, and + runs `uv pip check` before the hardware probe. + +## Availability findings and claim boundary + +`gh variable list` and `gh secret list` for the canonical repository returned +no configured entries on 2026-08-16. The runner inventory endpoint returned +HTTP 403 because the current maintainer token lacks repository-runner read +permission. Live read-only inspection then confirmed that Titan has an RTX +4090 and active Gitea runners, while Mutsu is a 16 GB Apple M4 builder. Neither +host exposes a GitHub Actions runner process, so the waivers now cover GitHub +runner registration and scheduled automation rather than hardware absence. + +Equivalent operator runs used isolated temporary directories, checksum-verified +uv 0.12.4 binaries, a managed Python 3.12.13 runtime, the exact signed source +archive, and the committed dependency lock. Titan installed the official +same-version CUDA overlay and passed CUDA/bitsandbytes. Mutsu installed the +native macOS lock plus the MLX group and passed MPS/MLX. The evidence was copied +off-host and hashed before both remote temporary directories were removed. + +The waivers are tracked by +https://github.com/elder-plinius/OBLITERATUS/issues/110. They expire on +2026-09-15, 30 days after opening. Before expiry, maintainers must either +configure and run the matching gate or review and replace the waiver under the +tracker. The fresh probes establish only the recorded operations on Titan and +Mutsu at the exact candidate. They do not create a broad support, compatibility, +correctness, or performance claim, and they do not replace the open work to +register maintainable GitHub runners. + +## Verification and evidence + +- Exact-head software conditional run: + https://github.com/elder-plinius/OBLITERATUS/actions/runs/31949881557 +- Downloaded software JSON/JUnit evidence: + `/tmp/obliteratus-item8-software-final-GMGS0x` +- Exact-head hardware JSON/JUnit and SHA-256 evidence: + `/tmp/obliteratus-item8-hardware-evidence-GqqBSm` +- Focused local command: + `/tmp/obliteratus-item8-py312-env/bin/python -m pytest -q --no-cov + tests/test_conditional_gate_scripts.py + tests/test_conditional_evidence_freshness.py` +- Mandatory local environment: `/tmp/obliteratus-item8-py312-env` +- Local generated summary: `/tmp/obliteratus-item8-summary.json` +- Waiver tracker: + https://github.com/elder-plinius/OBLITERATUS/issues/110 +- Hardware-evidence tracker comment: + https://github.com/elder-plinius/OBLITERATUS/issues/110#issuecomment-5307675226 + +## Scope and release decision + +This increment changes conditional-test governance and CUDA environment setup. +It does not alter production behavior or weaken mandatory CPU, package, +coverage, repeat, mutation, Windows, or supply-chain gates. Gate 3 item 9 and +the final exact-canonical report remain required before the contributor merge +train resumes. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index 902fc90..0e80c81 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -21,35 +21,32 @@ Gate 3 increment 4 canonical commit: Gate 3 increment 5 canonical commit: `aa182cc44883890c019c393f17602e2e2702e7d7` (PR #105, rebase merged) +Gate 3 increment 6 canonical commit: +`9683e0be4d892e6a6a134b45a065e591e139da3c` (PR #108, signed fast-forward) + +Gate 3 increment 7 canonical commit: +`b961623513a0e137b959f608ce31511d59e85888` (PR #109, signed fast-forward) + Current canonical AIWG and CI-contract baseline: `e8ac3b65670d696ed5b68f06adf14706bc2ff865` (PR #107, rebase merged) -The latest canonical mandatory offline CPU lane passes 1,985 tests on Python +The latest canonical mandatory offline CPU lane passes 2,112 tests on Python 3.12 with 9 conditional tests deselected by policy, no failures, errors, skips, -or unexpected warnings, and 78.57% statement / 65.89% branch coverage. Mature -CPU-testable coverage is 93.09% / 82.89%. The canonical mutation campaign kills -1,807 of 2,020 mutants (89.46%), with 208 survivors and 5 timeouts, while 595 +or unexpected warnings, and 83.50% statement / 71.08% branch coverage. Mature +CPU-testable coverage is 94.02% / 84.54%. The canonical mutation campaign kills +1,844 of 2,057 mutants (89.65%), with 208 survivors and 5 timeouts, while 702 tests pass in each of three file orders and hash seeds with no flakes. Package, Windows checkpoint, supply-chain, and all other hosted jobs passed on attempt 1: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31937799938. +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31947837466. -Gate 3 increment 6 is implemented and locally verified at exact core evidence -head `77adf119f871ba820f1f02d55fe2760aa24c84b2`. Python 3.12 passes 2,057 -mandatory tests with 2 expected source-shadow xfails and 9 conditional tests -deselected in 56.15s; coverage is 82.78% statements / 69.62% branches; mature -CPU coverage is 93.27% / 83.06%; 62/62 changed executable lines and all five -touched modules pass. All 9 new checkpoint-parser mutants are killed, the full -campaign expands to 2,029 mutants for hosted verification, and 669 tests pass -in each repeat order with no flakes. Exact-head PR audit, hosted matrix, and -canonical merge remain pending; details are in `gate3-increment-6-report.md`. - -The latest conditional run is green for pinned-model download and evaluation, -loopback network, operator UI, and evidence-freshness policy at the earlier -Gate 2 commit: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31868394039. CUDA, -bitsandbytes, MPS, MLX, and remote execution were not selected and therefore -have no fresh support evidence or support claim. Gate 3 item 8 must reconcile -all conditional evidence at the eventual release candidate. +Gate 3 increment 8 is implemented at signed policy head +`fa233fd8c97d2a1463141535f2a288b264abe93b`. Fresh pinned-model download, +external-evaluation, loopback-network, and operator-UI evidence passes at that +exact head in run 31949881557. Titan CUDA/bitsandbytes and Mutsu MPS/MLX operator +probes also pass at that exact head. GitHub runner registration and remote +execution remain time-bounded no-support/no-correctness/no-performance waivers +through 2026-09-15 under issue #110. Details are in +`gate3-increment-8-report.md`. The foundation, Wave A, Gate 1, and Gate 2 establish a sane, evidence-producing baseline. The legacy contributor merge train proceeded one @@ -58,9 +55,8 @@ property, mutation, determinism, and real-environment evidence without weakening the mandatory CPU baseline. The operator has now placed ordinary feature work back behind a testing-depth -gate. Gate 3 is adopted and items 1–5 are canonical; item 6 is locally complete -but not yet canonical; item 7 and the remaining quantitative exit criteria must -pass before unrelated feature work resumes. +gate. Gate 3 is adopted and items 1–7 are canonical; conditional reconciliation +and the final canonical report must pass before unrelated feature work resumes. Correctness, security, data-loss, and test-infrastructure repairs remain permitted when narrowly scoped and paired with reproducing tests. @@ -399,14 +395,15 @@ single repository-wide coverage PR. decision-seam contracts. Canonical run 31941334488 passed all eight required checks; exact evidence is recorded in `gate3-increment-6-report.md` and the canonical PR reconciliation comment. -7. **Implemented locally at `acc6b3b` against `9683e0b`.** Extend the installed - tiny-model vertical slice and quantized-storage semantics, including the - exact Float-to-integer restoration regression. The candidate reaches 94.02% - mature CPU statements and 84.54% branches and raises the immutable floors - to 94% / 84%; signed commit, exact-head audit, hosted checks, and canonical - merge remain pending. -8. Reconcile conditional software evidence, then run or explicitly waive each - unavailable hardware/remote environment under the claim policy. +7. **Completed at `b961623` (PR #109).** Extend the installed tiny-model + vertical slice and quantized-storage semantics, including the exact + Float-to-integer restoration regression. The canonical head reaches 94.02% + mature CPU statements and 84.54% branches, raises the immutable floors to + 94% / 84%, and passes all eight hosted checks. +8. **Implemented at signed policy head `fa233fd`.** Reconcile fresh software + evidence and explicitly waive each unavailable hardware/remote environment + under the claim policy. Exact-head conditional evidence, PR audit, and + canonical merge remain pending. 9. Publish the Gate 3 execution, coverage, mutation, repeat, duration, and conditional-evidence reports at one canonical `main` commit. diff --git a/docs/conditional-testing.md b/docs/conditional-testing.md index 974cd90..bc38504 100644 --- a/docs/conditional-testing.md +++ b/docs/conditional-testing.md @@ -21,6 +21,12 @@ and expire on 2026-09-15; they are not evidence that any waived backend works. They block support, compatibility, correctness, and performance claims for the waived environment. +The current accelerator waivers cover missing GitHub runner registration, not +missing hardware. Titan has an RTX 4090 behind Gitea runners, and Mutsu is a 16 GB +Apple M4 builder. Exact-head operator probes may be attached to issue #110, but +they do not make the scheduled GitHub lanes runnable or establish a broad backend +support claim. + ## Hosted gates The model gate downloads only