From 98b0b78e9046e8a54bc4a451151a820eb94af4f0 Mon Sep 17 00:00:00 2001 From: Joseph Magly <1159087+jmagly@users.noreply.github.com> Date: Sun, 16 Aug 2026 02:06:06 -0400 Subject: [PATCH] docs: publish Gate 3 item 5 evidence --- .aiwg/testing/gate3-increment-5-report.md | 119 ++++++++++++++++++++++ .aiwg/testing/master-test-plan.md | 67 ++++++------ 2 files changed, 153 insertions(+), 33 deletions(-) create mode 100644 .aiwg/testing/gate3-increment-5-report.md diff --git a/.aiwg/testing/gate3-increment-5-report.md b/.aiwg/testing/gate3-increment-5-report.md new file mode 100644 index 0000000..8f43407 --- /dev/null +++ b/.aiwg/testing/gate3-increment-5-report.md @@ -0,0 +1,119 @@ +# Gate 3 increment 5 service-orchestration report + +Date: 2026-08-16 +Exact base: `d7d2ad566af778b1d315616bbea8b8a3c4dd2191` +Exact core implementation and evidence head: +`f00878768e00ea02f5b4c9d1bb4bd9d8a9cef730` +Status: PASS locally; exact-head PR audit, hosted matrix, and canonical merge +remain pending + +## Results + +| Gate | Result | +|---|---| +| Focused service contracts | 97 BESTIARY, model-client, and Watchtower tests; slowest affected test 0.009s and combined affected-test time 0.526s | +| Mandatory CPU lane | Python 3.12: 1,980 passed in 82.63s; 9 conditional tests deselected by policy; 0 failures, errors, skips, or unexpected warnings | +| Repository coverage | 76.76% statements / 63.49% branches before; 78.69% / 66.05% after | +| Mature CPU scope | 93.20% statements / 82.86% branches before; 93.26% / 83.05% after | +| Changed executable lines | 324/324 (100%) against the exact base | +| Touched production modules | 4/4 pass exact-base line and branch no-regression | +| BESTIARY adapter | `bestiary_sync.py`: 96.55% statements / 100% branches, from 0% / 0% | +| Catalog client | `models_client.py`: 92.86% statements / 92.00% branches, from 27.12% / 3.57% | +| Watchtower | `watchtower.py`: 96.97% statements / 92.22% branches, from 24.63% / 0% | +| Pure service seam | `service_contracts.py`: 100% statements / 95.65% branches | +| Selective mutation | 1,807/2,020 killed (89.46%); 208 survived; 5 timed out; 0 no-test, skipped, suspicious, interrupted, or segfault results | +| Changed service mutation | 189/189 killed; 0 survived or timed out; required-target guard passed | +| Repeatability | 594 tests passed in each of three file orders/hash seeds; 33.504s total; no flake candidates or consistent failures | +| Package | sdist and wheel built; required modules and console entry point present; isolated Python 3.12 wheel import/version contract passed | +| Policy and workflow | Locked dependency graph, Ruff F, actionlint 1.7.12, conditional policy, risk map, supply-chain policy, and 158 CI/policy contract tests passed | + +The final mandatory lane added 97 passing tests relative to the canonical +increment-4 lane while completing 2.21 seconds faster (82.63s versus 84.84s). +The affected service tests consume 0.526s in aggregate, and no affected test +exceeds 0.009s. The repository branch-coverage Gate 3 exit of 65% is now met; +repository statements and mature-scope exits remain assigned to increments +6–7. + +## Contracts added + +1. An explicitly configured catalog path or URL is authoritative. Missing, + oversized, non-UTF-8, malformed JSON, and invalid catalog envelopes fail + closed instead of silently falling through to an unrelated local catalog. +2. Local and URL catalog reads are bounded at 8 MiB, URL reads use a finite + timeout, and the stable catalog envelope and records are schema-checked. +3. Channel, vendor, and capability filters normalize case and whitespace; + aliases remain stable; malformed list, vendor, identifier, date, and + `open_weight` values are rejected rather than coerced. +4. Newest-model selection uses an injected UTC clock, validates its day window, + falls back from `first_seen` to `released`, skips invalid dates, and sorts by + the parsed effective date. +5. BESTIARY-derived presets preserve curated-ID authority and deterministic + de-duplication, validate record fields, infer parameter tiers including MoE + and unknown-size boundaries, and fail safe as one empty extension on a + malformed catalog. +6. The Hugging Face adapter is injectable and deterministic. Per-organization + failures are isolated, minimum-download filtering is enforced, malformed + list/candidate responses are ignored or rejected at explicit boundaries, + and no test performs a live service call. +7. License matching respects identifier boundaries, and instruction tuning is + recognized by name tokens or exact tags rather than unsafe substrings. +8. Persisted Watchtower state loads transactionally: a malformed envelope, + key/ID mismatch, field type, count, status, or metrics object cannot leave a + partially accepted in-memory state. Failed temporary writes are cleaned up. +9. Scheduler interval and status inputs are validated by pure contracts. Start, + restart, stop, timeout, and start-failure transitions are explicit; a stuck + thread remains owned and blocks a duplicate scheduler. +10. The scheduler runs immediately, stops through an injected event without + sleep races, and safely defers confirmation when a callback requests stop + from the scheduler's own thread, avoiding self-join and orphaning. + +## Defects and mutants detected + +The first focused campaign killed 166 of 194 service-contract mutants. The 28 +survivors exposed weak boundaries around catalog field types, day/interval +validation, candidate defaults, state transitions, license matching, and exact +diagnostics. Survivor-driven tests and simplification of equivalent defaults +reduced the final pure seam to 189 meaningful mutants and killed every one. + +Manual pre-delivery review then found two uncovered behaviors: string/integer +`open_weight` values could be truth-coerced, and a discovery callback could ask +the scheduler thread to join itself. Both now have reproducing tests and +fail-closed implementations. The fresh mutation prepass also rejected a stale +generated sandbox copy after these tests changed; rebuilding only the ignored +sandbox artifacts restored the exact 2,020-mutant campaign without touching +repository source. + +The contracts additionally detect unbounded or ambiguous catalog resolution, +malformed records, case-sensitive filter drift, curated-preset replacement, +license/instruction substring false positives, non-deterministic organization +order, partial state acceptance, invalid lifecycle transitions, duplicate +background threads, and lost ownership after stop timeout. + +## Verification and evidence + +- Exact-head normalized JUnit, coverage, repeat, mutation, and trend evidence: + `/tmp/obliteratus-item5-final-evidence-f008787/` +- Exact-head Python 3.12 raw coverage/JUnit evidence: + `/tmp/obliteratus-item5-final-py312-f008787/` +- Exact-base canonical Python 3.12 evidence: + `/tmp/obliteratus-postmerge-item4-py312-CH4T03/` +- Exact-head package artifacts and hashes: + `/tmp/obliteratus-item5-package-f008787/` +- Installed-wheel verification environment: + `/tmp/obliteratus-item5-wheel-env-f008787/` + +The exact-head Ruff F gate, recovered pinned actionlint, `uv lock --check`, +conditional-policy validation, risk-map validation, changed-line gate, +touched-module gate, immutable quality policy, duration policy, repeat gate, +mutation score and required-target gates, package checks, and supply-chain +policy all pass. The original operator checkout was not modified; work and +evidence remain isolated in the dedicated item-5 worktree and temporary paths. + +## Scope and release decision + +This increment implements Gate 3 bounded delivery item 5. It does not reopen +ordinary feature work: items 6–7, conditional-evidence reconciliation, and the +final canonical Gate 3 publication remain open. All catalog and Hugging Face +service behavior in this increment uses injected adapters, operator-provided +files, or loopback-style fakes. It makes no new live-network, remote, +accelerator, model-download, browser, or operator-UI support claim. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index 9f7b621..dad2f3b 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -1,6 +1,6 @@ # OBLITERATUS master test plan -Date: 2026-08-15 +Date: 2026-08-16 Owner: maintainers Phase: construction @@ -15,27 +15,26 @@ Gate 3 increment 2 canonical commit: Gate 3 increment 3 canonical commit: `c1b34503ddd3cb797e5d70671c47afcabfe73832` (PR #103, rebase merged) -The latest canonical mandatory offline CPU selection passes on Python 3.10–3.12 -with 1,830–1,832 passing tests, 9 conditionally deselected tests, 2 expected -Python 3.12 xfails, no unexpected skips, and no warnings. Branch-enabled Python -3.12 coverage is 76.60% statements and 63.25% branches; the documented mature -CPU-testable scope is 93.12% statements and 82.60% branches. The repository -enforces 95% changed-line coverage, per-touched-module line and branch -no-regression, an 85% selective mutation floor, zero unexpected warnings, and -owned suite/test/marker/repeat duration budgets. The canonical increment-3 -campaign killed 1,424 of 1,637 mutants (86.99%), and 431 high-consequence tests -passed in each of three file orders and hash seeds. Package, installed -wheel/sdist, vulnerability, secret, license, and SBOM checks are green. All -seven jobs in the canonical post-merge run passed: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31920984688. +Gate 3 increment 4 canonical commit: +`d7d2ad566af778b1d315616bbea8b8a3c4dd2191` (PR #104, rebase merged) -Gate 3 increment 4 is implemented and locally verified at core evidence head -`9da022e6f3bc73d17fd31d4da70b42e3ca052ba2`, with delivery evidence recorded in -`gate3-increment-4-report.md`. Candidate coverage is 76.77% statements and -63.55% branches; mature CPU coverage is 93.19% statements and 82.85% branches; -the complete campaign kills 1,614 of 1,827 mutants (88.34%), including all -229 persistence mutants. Exact-head PR audit, the first hosted Windows -checkpoint contract, and canonical merge remain pending. +The latest canonical mandatory offline CPU lane passes 1,883 tests on Python +3.12 with 9 conditional tests deselected by policy, no failures or warnings, +and 76.76% statement / 63.49% branch coverage. Mature CPU-testable coverage is +93.20% / 82.86%. The canonical mutation campaign kills 1,618 of 1,831 mutants +(88.37%), including every changed persistence mutant, while 497 tests pass in +each of three file orders and hash seeds. Package, Windows checkpoint, +supply-chain, and all other hosted jobs passed on attempt 1: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31926709748. + +Gate 3 increment 5 is implemented and locally verified at exact core evidence +head `f00878768e00ea02f5b4c9d1bb4bd9d8a9cef730`. Python 3.12 passes 1,980 +mandatory tests in 82.63s; coverage is 78.69% statements / 66.05% branches; +mature CPU coverage is 93.26% / 83.05%; 324/324 changed executable lines and +all four touched modules pass. The expanded campaign kills 1,807 of 2,020 +mutants (89.46%), including all 189 new service-state mutants, and 594 tests +pass in each repeat order. Exact-head PR audit, hosted matrix, and canonical +merge remain pending; details are in `gate3-increment-5-report.md`. The latest conditional run is green for pinned-model download and evaluation, loopback network, operator UI, and evidence-freshness policy at the earlier @@ -52,9 +51,9 @@ property, mutation, determinism, and real-environment evidence without weakening the mandatory CPU baseline. The operator has now placed ordinary feature work back behind a testing-depth -gate. Gate 3 is adopted and items 1–3 are complete; item 4 is implemented but -not yet canonical; items 5–7 and the quantitative exit criteria must pass -before unrelated feature work resumes. +gate. Gate 3 is adopted and items 1–4 are canonical; item 5 is locally complete +but not yet canonical; items 6–7 and the remaining quantitative exit criteria +must pass before unrelated feature work resumes. Correctness, security, data-loss, and test-infrastructure repairs remain permitted when narrowly scoped and paired with reproducing tests. @@ -293,12 +292,12 @@ resistance, environment evidence, or deterministic replay. | Measure | Current | Gate 3 minimum | |---|---:|---:| -| Repository statements | 76.76% canonical | 80.00% | -| Repository branches | 63.49% canonical | 65.00% | -| Mature CPU statements | 93.20% canonical | 94.00% | -| Mature CPU branches | 82.86% canonical | 84.00% | +| Repository statements | 78.69% item-5 candidate | 80.00% | +| Repository branches | 66.05% item-5 candidate | 65.00% | +| Mature CPU statements | 93.26% item-5 candidate | 94.00% | +| Mature CPU branches | 83.05% item-5 candidate | 84.00% | | Changed executable lines | 95.00% floor | 95.00% floor | -| Selected mutation score | 88.37% canonical on expanded persistence scope | at least 85% on expanded scope | +| Selected mutation score | 89.46% item-5 candidate on expanded service scope | at least 85% on expanded scope | | Unexpected warnings | 0 | 0 | | Active flaky quarantines | 0 | 0 | | Software conditional evidence age | at most 8 days | at most 8 days | @@ -358,9 +357,9 @@ small pure seam and measure it rather than mocking away the behavior under test. | `abliterate.py` | 50.91% / 38.03% candidate | at least 52% / 45% with mutation/math contracts | | `lora_ablation.py` | 9.57% / 0% | at least 70% / 55% for CPU-testable decisions | | `tourney.py` | 12.01% / 0% | at least 55% / 40% for lifecycle/state transitions | -| `bestiary_sync.py` | 0% / 0% | at least 80% / 65% using a loopback/fake transport | -| `models_client.py` | 27.12% / 3.57% | at least 80% / 65% for resolution/error contracts | -| `watchtower.py` | 24.63% / 0% | at least 70% / 55% with injected clock/client seams | +| `bestiary_sync.py` | 96.55% / 100% item-5 candidate | at least 80% / 65% using a loopback/fake transport | +| `models_client.py` | 92.86% / 92.00% item-5 candidate | at least 80% / 65% for resolution/error contracts | +| `watchtower.py` | 96.97% / 92.22% item-5 candidate | at least 70% / 55% with injected clock/client seams | | `interactive.py` | 10.49% / 0% | at least 70% / 55% with scripted I/O contracts | | `local_ui.py` | 48.15% / 15.22% | at least 70% / 55% for launch decisions and failures | | `ui_watchtower.py` | 0% / 0% | cover construction/callback contracts; real UI stays conditional | @@ -386,7 +385,9 @@ single repository-wide coverage PR. concurrency, atomicity, retry, cleanup-ownership, and Windows portability contracts. Canonical evidence is recorded in `gate3-increment-4-report.md`. -5. Add BESTIARY/model-client/watchtower transport and scheduler state contracts. +5. **Implemented locally at `f008787`.** Add BESTIARY/model-client/watchtower + transport and scheduler state contracts. Exact evidence is recorded in + `gate3-increment-5-report.md`; audit and canonical merge remain pending. 6. Add tournament, interactive, and UI decision-seam contracts. 7. Extend the installed tiny-model vertical slice and quantized-storage semantics, including the exact Float-to-integer restoration regression.