From 659ff63f1ebe7b86cd59fcec038a59ea38d718ed Mon Sep 17 00:00:00 2001 From: Joseph Magly <1159087+jmagly@users.noreply.github.com> Date: Sun, 16 Aug 2026 04:52:25 -0400 Subject: [PATCH] docs: record Gate 3 operator interface evidence --- .aiwg/testing/gate3-increment-5-report.md | 14 ++- .aiwg/testing/gate3-increment-6-report.md | 108 ++++++++++++++++++++++ .aiwg/testing/master-test-plan.md | 65 +++++++------ 3 files changed, 155 insertions(+), 32 deletions(-) create mode 100644 .aiwg/testing/gate3-increment-6-report.md diff --git a/.aiwg/testing/gate3-increment-5-report.md b/.aiwg/testing/gate3-increment-5-report.md index cdd0822..53bb1db 100644 --- a/.aiwg/testing/gate3-increment-5-report.md +++ b/.aiwg/testing/gate3-increment-5-report.md @@ -4,8 +4,8 @@ Date: 2026-08-16 Exact base: `d7d2ad566af778b1d315616bbea8b8a3c4dd2191` Exact core implementation and evidence head: `2a7b3818c89a017c152f9f787c876891cc89e648` -Status: PASS locally; exact-head PR audit, hosted matrix, and canonical merge -remain pending +Canonical commit: `aa182cc44883890c019c393f17602e2e2702e7d7` +Status: PASS canonical (PR #105, rebase merged) ## Results @@ -109,13 +109,19 @@ background threads, and lost ownership after stop timeout. `/tmp/obliteratus-item5-package-2a7b381/` - Installed-wheel verification environment: `/tmp/obliteratus-item5-wheel-env-2a7b381/` +- Canonical 8/8 hosted evidence: + https://github.com/elder-plinius/OBLITERATUS/actions/runs/31933066525 The exact-head Ruff F gate, recovered pinned actionlint, `uv lock --check`, conditional-policy validation, risk-map validation, changed-line gate, touched-module gate, immutable quality policy, duration policy, repeat gate, mutation score and required-target gates, package checks, and supply-chain -policy all pass. The original operator checkout was not modified; work and -evidence remain isolated in the dedicated item-5 worktree and temporary paths. +policy all pass. The canonical run completed all eight jobs successfully; its +1,805/2,020 mutation result differed from the PR/local 1,807/2,020 result only +because two simple mutants timed out on that runner and were killed on the +identical tree in both other campaigns. The original operator checkout was not +modified; work and evidence remained isolated in the dedicated item-5 worktree +and temporary paths. ## Scope and release decision diff --git a/.aiwg/testing/gate3-increment-6-report.md b/.aiwg/testing/gate3-increment-6-report.md new file mode 100644 index 0000000..1b9ab0b --- /dev/null +++ b/.aiwg/testing/gate3-increment-6-report.md @@ -0,0 +1,108 @@ +# Gate 3 increment 6 tournament and operator-interface report + +Date: 2026-08-16 +Exact base: `e8ac3b65670d696ed5b68f06adf14706bc2ff865` +Exact core implementation and local evidence head: +`8a13ea9965c0c455b42262c3b81e4ea3bf9ae798` +Status: PASS locally; exact-head PR audit, hosted matrix, and canonical merge +remain pending + +## Results + +| Gate | Result | +|---|---| +| Focused contracts | 70 new tournament, interactive, local-launcher, and Watchtower-UI tests passed | +| Mandatory CPU lane | Python 3.12: 2,053 passed, 2 expected source-shadow xfails, and 9 conditional tests deselected in 56.34s | +| Repository coverage | 78.69% statements / 66.05% branches before; 82.77% / 69.60% after | +| Mature CPU scope | 93.27% statements / 83.06% branches | +| Changed executable lines | 51/51 (100%) against the exact base | +| Touched production modules | 5/5 pass exact-base line and branch no-regression | +| Tournament lifecycle | `tourney.py`: 12.01% / 0% before; 83.09% statements / 77.78% branches after | +| Interactive decisions | `interactive.py`: 10.49% / 0% before; 76.92% / 73.53% after | +| Local launcher | `local_ui.py`: 48.15% / 15.22% before; 85.19% / 63.04% after | +| Watchtower UI handlers | `ui_watchtower.py`: 0% / 0% before; 70.89% / 83.33% after | +| Pure checkpoint seam | `tourney_contracts.py`: 100% statements / 100% branches | +| Changed-seam mutation | 9/9 checkpoint-parser mutants killed; configured campaign expands from 2,020 to 2,029 mutants | +| Repeatability | 665 tests passed in each of three file orders/hash seeds; 34.300s total; no flake candidates or consistent failures | +| Policy | Ruff F, conditional policy, source-to-test risk map, mutation-target guard, global/mature coverage, changed-line, and touched-module gates passed | + +The repository statement and branch exits now both exceed the Gate 3 targets. +The remaining new-work hold is bounded item 7, followed by conditional-evidence +reconciliation and final canonical reporting. + +## Contracts added + +1. Tournament scoring, result ordering, bracket/model-card rendering, round + ranking, loser cleanup, no-winner behavior, and callback sequencing are + deterministic without loading a model. +2. Checkpoints reject malformed, non-object, unreadable, unsupported-version, + and invalid-encoding documents. Completed and interrupted rounds preserve + direction and spectral-certification metadata, and quota interruption resumes + at the exact remaining method without repeating completed work. +3. Guided terminal choices map hardware boundaries, custom models, presets, + strategies, sample sizes, cancellation, and quantized execution into an exact + study configuration. Remote repository code is disabled by default, and the + quantized branch returns its study result. +4. The local launcher masks passwords, reports network listeners, forwards the + full Gradio server contract, avoids hardware work in quiet mode, fails clearly + when the optional UI runtime is absent, and degrades to CPU when a GPU driver + probe raises a runtime error. +5. Watchtower pure handlers are mandatory even without Gradio. Actual tab + construction still requires the optional UI extra, while update mappings, + scan/history/scheduler callbacks, zero-valued metrics, and failure rendering + remain CPU-testable. +6. One-click result selection orders numeric iteration identifiers correctly, + ignores malformed names and unreadable entries, and refuses symlink targets + outside its managed result root. +7. Dynamic Watchtower status and scan errors are HTML-escaped, and an exact + `0.0` refusal rate is reported as measured zero rather than `unknown`. + +## Defects detected and corrected + +The first focused tournament run found that non-object JSON roots raised +`AttributeError` and that interrupted contenders lost direction/certification +metadata on resume. Scripted interactive tests then exposed unconditional +`trust_remote_code=True` and a discarded quantized-study return value. Launcher +tests reproduced a driver-probe exception that prevented CPU fallback. + +Watchtower handler tests detected lexicographic iteration ordering (`iter_2` +beating `iter_10`), symlink escape from the managed download root, unescaped +HTML in scan errors and invalid timestamps, and false `unknown` reporting for a +measured zero refusal rate. The initial mandatory run also proved that a +collection-time Gradio stub contaminated conditional test selection; pure +handlers now have a real optional-dependency seam instead. + +The first changed-line comparison reported 35/40 covered executable changes. +Negative tests for the installed-Gradio update path, empty model IDs, malformed +iteration names, and unreadable iteration directories raised the final gate to +51/51 without weakening the 95% policy. + +## Verification and evidence + +- Mandatory Python 3.12 coverage: + `/tmp/obliteratus-item6-rebased-coverage.json` +- Mandatory Python 3.12 JUnit: + `/tmp/obliteratus-item6-rebased-junit.xml` +- Exact-base Python 3.12 coverage: + `/tmp/obliteratus-main-e8ac3b6-UGNOIz/coverage-py3.12.json` +- Final repeat evidence: + `/tmp/obliteratus-item6-rebased-repeat-vU1gQh/` +- Exact-head package artifacts: + `/tmp/obliteratus-item6-rebased-package-tRW7rz/` +- Mutation metadata and prepared selection evidence: + ignored worktree-local `mutants/` artifacts at the exact implementation head + +The exact implementation head passes the enforced Ruff F scope, immutable +quality policy, conditional-policy validation, risk-map validation, mutation +required-target guard, 75% statement and 60% branch floors, 95% changed-line +floor, and touched-module no-regression gate. The full 2,029-mutant campaign, +package matrix, supply-chain checks, exact-head audit, and canonical evidence +remain hosted PR/merge gates. + +## Scope and release decision + +This increment implements bounded delivery item 6. It makes no live model, +network, browser, accelerator, credential, or publication claim. All model and +service behavior uses injected fakes and isolated temporary filesystems. Item 7 +and the final Gate 3 release decision remain open, so ordinary feature work does +not resume yet. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index c1d0933..983a620 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -18,23 +18,30 @@ Gate 3 increment 3 canonical commit: Gate 3 increment 4 canonical commit: `d7d2ad566af778b1d315616bbea8b8a3c4dd2191` (PR #104, rebase merged) -The latest canonical mandatory offline CPU lane passes 1,883 tests on Python -3.12 with 9 conditional tests deselected by policy, no failures or warnings, -and 76.76% statement / 63.49% branch coverage. Mature CPU-testable coverage is -93.20% / 82.86%. The canonical mutation campaign kills 1,618 of 1,831 mutants -(88.37%), including every changed persistence mutant, while 497 tests pass in -each of three file orders and hash seeds. Package, Windows checkpoint, -supply-chain, and all other hosted jobs passed on attempt 1: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31926709748. +Gate 3 increment 5 canonical commit: +`aa182cc44883890c019c393f17602e2e2702e7d7` (PR #105, rebase merged) -Gate 3 increment 5 is implemented and locally verified at exact core evidence -head `2a7b3818c89a017c152f9f787c876891cc89e648`. Python 3.12 passes 1,981 -mandatory tests in 79.28s; coverage is 78.69% statements / 66.05% branches; -mature CPU coverage is 93.26% / 83.05%; 328/328 changed executable lines and -all four touched modules pass. The expanded campaign kills 1,807 of 2,020 -mutants (89.46%), including all 189 new service-state mutants, and 595 tests -pass in each repeat order. Exact-head PR audit, hosted matrix, and canonical -merge remain pending; details are in `gate3-increment-5-report.md`. +Current canonical AIWG and CI-contract baseline: +`e8ac3b65670d696ed5b68f06adf14706bc2ff865` (PR #107, rebase merged) + +The latest canonical mandatory offline CPU lane passes 1,985 tests on Python +3.12 with 9 conditional tests deselected by policy, no failures, errors, skips, +or unexpected warnings, and 78.57% statement / 65.89% branch coverage. Mature +CPU-testable coverage is 93.09% / 82.89%. The canonical mutation campaign kills +1,807 of 2,020 mutants (89.46%), with 208 survivors and 5 timeouts, while 595 +tests pass in each of three file orders and hash seeds with no flakes. Package, +Windows checkpoint, supply-chain, and all other hosted jobs passed on attempt 1: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31937799938. + +Gate 3 increment 6 is implemented and locally verified at exact core evidence +head `8a13ea9965c0c455b42262c3b81e4ea3bf9ae798`. Python 3.12 passes 2,053 +mandatory tests with 2 expected source-shadow xfails and 9 conditional tests +deselected in 56.34s; coverage is 82.77% statements / 69.60% branches; mature +CPU coverage is 93.27% / 83.06%; 51/51 changed executable lines and all five +touched modules pass. All 9 new checkpoint-parser mutants are killed, the full +campaign expands to 2,029 mutants for hosted verification, and 665 tests pass +in each repeat order with no flakes. Exact-head PR audit, hosted matrix, and +canonical merge remain pending; details are in `gate3-increment-6-report.md`. The latest conditional run is green for pinned-model download and evaluation, loopback network, operator UI, and evidence-freshness policy at the earlier @@ -51,9 +58,9 @@ property, mutation, determinism, and real-environment evidence without weakening the mandatory CPU baseline. The operator has now placed ordinary feature work back behind a testing-depth -gate. Gate 3 is adopted and items 1–4 are canonical; item 5 is locally complete -but not yet canonical; items 6–7 and the remaining quantitative exit criteria -must pass before unrelated feature work resumes. +gate. Gate 3 is adopted and items 1–5 are canonical; item 6 is locally complete +but not yet canonical; item 7 and the remaining quantitative exit criteria must +pass before unrelated feature work resumes. Correctness, security, data-loss, and test-infrastructure repairs remain permitted when narrowly scoped and paired with reproducing tests. @@ -292,15 +299,15 @@ resistance, environment evidence, or deterministic replay. | Measure | Current | Gate 3 minimum | |---|---:|---:| -| Repository statements | 78.69% item-5 candidate | 80.00% | -| Repository branches | 66.05% item-5 candidate | 65.00% | -| Mature CPU statements | 93.26% item-5 candidate | 94.00% | -| Mature CPU branches | 83.05% item-5 candidate | 84.00% | +| Repository statements | 82.77% item-6 candidate | 80.00% | +| Repository branches | 69.60% item-6 candidate | 65.00% | +| Mature CPU statements | 93.27% item-6 candidate; 73 lines remain | 94.00% | +| Mature CPU branches | 83.06% item-6 candidate; 29 branches remain | 84.00% | | Changed executable lines | 95.00% floor | 95.00% floor | -| Selected mutation score | 89.46% item-5 candidate on expanded service scope | at least 85% on expanded scope | +| Selected mutation score | 89.46% item-5 canonical; item-6 campaign expands to 2,029 mutants | at least 85% on expanded scope | | Unexpected warnings | 0 | 0 | | Active flaky quarantines | 0 | 0 | -| Software conditional evidence age | at most 8 days | at most 8 days | +| Software conditional evidence age | 1 day at the item-6 candidate date | at most 8 days | The expanded mutation scope must include core mutation math, architecture and loader decisions, persistence/atomicity, evaluation/report serialization, and @@ -385,10 +392,12 @@ single repository-wide coverage PR. concurrency, atomicity, retry, cleanup-ownership, and Windows portability contracts. Canonical evidence is recorded in `gate3-increment-4-report.md`. -5. **Implemented locally at `2a7b381`.** Add BESTIARY/model-client/watchtower +5. **Completed at `aa182cc` (PR #105).** Add BESTIARY/model-client/watchtower transport and scheduler state contracts. Exact evidence is recorded in - `gate3-increment-5-report.md`; audit and canonical merge remain pending. -6. Add tournament, interactive, and UI decision-seam contracts. + `gate3-increment-5-report.md` and the canonical PR follow-up. +6. **Implemented locally at `8a13ea9`.** Add tournament, interactive, and UI + decision-seam contracts. Exact local evidence is recorded in + `gate3-increment-6-report.md`; audit and canonical merge remain pending. 7. Extend the installed tiny-model vertical slice and quantized-storage semantics, including the exact Float-to-integer restoration regression. 8. Reconcile conditional software evidence, then run or explicitly waive each