diff --git a/.aiwg/testing/gate3-increment-3-report.md b/.aiwg/testing/gate3-increment-3-report.md index d40cdfe..683800f 100644 --- a/.aiwg/testing/gate3-increment-3-report.md +++ b/.aiwg/testing/gate3-increment-3-report.md @@ -5,8 +5,10 @@ Exact base: `2c2b38b501f94af46ffac3a772f7bf9475154dc5` Exact implementation/evidence head: `e2e120c211adf3ec9a7909a401f5e7a34f17984f` Delivery head: the containing documentation commit, resolved with `git rev-parse HEAD` -Canonical merge: pending -Status: PASS locally; hosted exact-head audit and merge pending +Canonical merge: `c1b34503ddd3cb797e5d70671c47afcabfe73832` (PR #103, +rebase merged) +Status: PASS; exact-head audit, rebase merge, and canonical post-merge CI +complete ## Results @@ -88,6 +90,9 @@ decision logic and its wrappers only; CUDA, bitsandbytes hardware execution, MPS, MLX, network, download, remote, and operator-UI mappings are unchanged, so no new environment support claim is made. -Delivery requires the containing documentation commit to be pushed, audited at -its final SHA, pass all hosted checks with no unresolved review thread, and be -rebase merged before item 4 begins. +PR #103 passed its exact-head audit and all required hosted checks with no +unresolved review thread, then rebase merged at +`c1b34503ddd3cb797e5d70671c47afcabfe73832`. All seven jobs in canonical +post-merge run +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31920984688 passed +before item 4 began. diff --git a/.aiwg/testing/gate3-increment-4-report.md b/.aiwg/testing/gate3-increment-4-report.md new file mode 100644 index 0000000..e8e8453 --- /dev/null +++ b/.aiwg/testing/gate3-increment-4-report.md @@ -0,0 +1,124 @@ +# Gate 3 increment 4 checkpoint reliability report + +Date: 2026-08-15 +Exact base: `c1b34503ddd3cb797e5d70671c47afcabfe73832` +Exact core implementation/mutation head: +`9da022e6f3bc73d17fd31d4da70b42e3ca052ba2` +Delivery head: the containing documentation and Windows-CI commit, resolved with +`git rev-parse HEAD` +Canonical merge: pending +Status: PASS locally; hosted Windows contract, exact-head audit, and merge pending + +## Results + +| Gate | Result | +|---|---| +| Focused checkpoint/policy contracts | 176 passed in 6.08s at the exact core head; the delivery-head portability-policy selection also passed 176 tests in 6.36s | +| Mandatory CPU matrix | Exact core head — Python 3.10: 1,878 passed in 88.21s; Python 3.11: 1,878 passed in 80.63s. Exact delivery head — Python 3.12: 1,877 passed, 2 expected xfails in 45.49s. Nine conditional tests were deselected on each interpreter | +| Repository coverage | 76.60% statements / 63.25% branches before; 76.77% / 63.55% after | +| Mature CPU scope | 93.12% statements / 82.60% branches before; 93.19% / 82.85% after | +| Changed executable lines | 138/138 (100%) against the exact base on Python 3.12 | +| Touched production modules | 2/2 pass exact-base line and branch no-regression | +| Persistence seam | `persistence_contracts.py`: 100% statements and 100% branches (166 statements, 58 branches) | +| Abliteration runtime | 50.82% statements / 38.03% branches before; 50.91% / 38.03% after | +| Selective mutation | 1,614/1,827 killed (88.34%); 208 survived; 5 timed out; 0 no-test, suspicious, skipped, interrupted, or segfault results | +| Changed persistence mutation | 229/229 killed; 0 survived or timed out | +| Mutation runtime | 920.27s; 1,121,616 KiB maximum parent RSS; required-target guard passed | +| Repeatability | 493 tests passed in each of three file orders/hash seeds; 33.832s total; no flake candidates or consistent failures | +| Package | sdist and wheel built; metadata, contents, isolated import, version, module entry point, and console entry point passed on Python 3.12 | +| Supply chain | Pinned `uv 0.12.4` lock, all-extras import, Gitleaks, Python 3.10–3.12 vulnerability audits, wheel-bound CycloneDX SBOM, and license policy passed | +| Windows portability | A focused Python 3.12 hosted job now blocks on checkpoint atomicity, persistence, and pipeline contracts and retains JUnit evidence; first exact-head result pending | + +The mandatory Python lanes remain below the 240-second policy budget. The final +Python 3.12 test time measured 10.08s faster than the exact-base 55.57s while +adding 47 passing tests. The slowest Python 3.12 test remains the existing +offline model save/reload slice at 5.364s; no new checkpoint test exceeds an +owned duration budget. + +## Contracts added + +1. A staged checkpoint must contain the exact prepared metadata, non-empty JSON + model and tokenizer configurations, and either regular direct weights or a + complete safe sharded-weight map. Missing, empty, corrupt, truncated, + link-backed, or traversal-bearing artifacts fail before promotion. +2. Every regular staged artifact is flushed, nested directory entries are + flushed bottom-up where the platform supports directory fsync, and the + destination parent is flushed after backup, promotion, rollback, and durable + cleanup transitions. +3. A persistent sibling lock serializes cooperative threads and processes that + commit to the same destination. Concurrent complete writers may replace one + another, but readers never observe a partially promoted checkpoint. +4. Writer failure and `BaseException` cancellation are injected before commit + and at tree sync, backup replacement, backup-parent sync, promotion, + promotion-parent sync, and rollback. Existing destinations are restored; + newly created destinations are removed; ambiguous double failures fail + closed with an explicit recovery path. +5. Cleanup errors never mask the primary write failure. Only transaction-owned + staging and backup paths are removed, and a successfully promoted checkpoint + remains valid when obsolete-backup cleanup reports a durable warning. +6. Pipeline offload cleanup no longer clears ownership after a failed removal. + The owned path is retained for a deterministic retry, while successful + cleanup clears both the path and ownership flag. +7. The production rebirth path validates the complete staged checkpoint before + atomic promotion. Persistence is now a required mutation target and the + checkpoint atomicity/pipeline suites are part of mutation selection and the + three-order repeat gate. +8. The platform-specific lock and directory-sync branches are no longer backed + by an unverified CI comment: a focused `windows-latest` lane executes the + checkpoint contracts with the locked Python 3.12 environment and retained + JUnit evidence. + +## Defects and mutants detected + +The development mutation campaign initially left 29 persistence survivors. +They identified weak JSON type/path assertions, incomplete cleanup and +diagnostic call contracts, missing cancellation boundaries, and insufficient +parent-sync/rollback observability. The repaired exact-target campaign killed +all generated persistence mutants. The final complete campaign preserved that +229/229 result while exceeding the immutable aggregate 85% floor. + +The full fault-injection suite also detects partial directory trees, unsafe or +missing shards, symlinked artifacts, file-fsync failure, concurrent commit +interleavings, incomplete rollback, cleanup-warning suppression, stale staging +ownership, and offload-directory ownership loss. + +## Verification and evidence + +- Exact delivery-head Python 3.12 coverage/JUnit/trend evidence: + `/tmp/obliteratus-gate3-item4-final-JlA1k3/` +- Exact-base Python 3.12 coverage/JUnit evidence: + `/tmp/obliteratus-gate3-item4-base-2uZEXr/` +- Python 3.10 and 3.11 coverage/JUnit evidence: + `/tmp/obliteratus-gate3-item4-py310-exact-ldLxx6/` and + `/tmp/obliteratus-gate3-item4-py311-exact-H4wFBU/` +- Repeat evidence: + `/tmp/obliteratus-gate3-item4-repeat-H23MdN/` +- Exact core-head mutation evidence: + `/tmp/obliteratus-gate3-item4-mutation-PVKKOL/` +- Python 3.10/3.11 and quality-depth normalized trends: + `/tmp/obliteratus-gate3-item4-trends-lrrEjF/` +- Package evidence: + `/tmp/obliteratus-gate3-item4-package-CTZeC6/` +- Pinned-tool supply-chain evidence: + `/tmp/obliteratus-gate3-item4-supply-7aqGSc/` + +The canonical Ruff F gate, checksum-verified `actionlint`, `uv lock --check`, +conditional-policy validation, risk-map validation, changed-line gate, +touched-module gate, immutable coverage policy, normalized duration policy, +repeat gate, mutation score, installed-distribution checks, and supply-chain +decisions all pass. The ordinary worktree remained clean; all work and evidence +were isolated in the dedicated item-4 worktree and temporary evidence paths. + +## Scope and release decision + +This increment implements Gate 3 bounded delivery item 4. It does not reopen +ordinary feature work: items 5–7, conditional-evidence reconciliation, and the +final canonical Gate 3 publication remain open. The new hosted Windows lane is +a checkpoint portability contract, not a claim that accelerator, MPS, MLX, +network, download, remote, or operator-UI execution is supported by this +increment. + +Delivery requires the containing commit to be pushed, pass the first hosted +Windows checkpoint run and all existing required checks, be audited at its +exact final SHA with no unresolved review thread, and be rebase merged before +item 5 begins. diff --git a/.aiwg/testing/master-test-plan.md b/.aiwg/testing/master-test-plan.md index ce250de..807a075 100644 --- a/.aiwg/testing/master-test-plan.md +++ b/.aiwg/testing/master-test-plan.md @@ -12,20 +12,30 @@ Gate 3 increment 1 canonical commit: Gate 3 increment 2 canonical commit: `2c2b38b501f94af46ffac3a772f7bf9475154dc5` (PR #102, rebase merged) -The mandatory offline CPU selection passes on Python 3.10–3.12 with 1,644 -tests, 9 conditionally deselected tests, no skips, and no warnings. The slowest -hosted pytest selection is Python 3.12 at 158.989 seconds; its complete CI job -is 5 minutes 41 seconds because it also regenerates exact-base coverage. -Branch-enabled Python 3.12 coverage is 75.68% statements and 61.72% branches. -The documented mature CPU-testable scope is 92.71% statements and 81.55% -branches. The repository enforces 95% changed-line coverage, per-touched-module -line and branch no-regression, a 75% selective mutation floor, zero unexpected -warnings, and owned suite/test/marker/repeat duration budgets. The bounded -campaign kills 809 of 886 mutants (91.31%), and 391 high-consequence tests pass -in each of three file orders and hash seeds with no flake candidates. Package, -installed wheel/sdist, vulnerability, secret, license, and SBOM checks are -green. All seven jobs in the exact canonical-commit run passed: -https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085. +Gate 3 increment 3 canonical commit: +`c1b34503ddd3cb797e5d70671c47afcabfe73832` (PR #103, rebase merged) + +The latest canonical mandatory offline CPU selection passes on Python 3.10–3.12 +with 1,830–1,832 passing tests, 9 conditionally deselected tests, 2 expected +Python 3.12 xfails, no unexpected skips, and no warnings. Branch-enabled Python +3.12 coverage is 76.60% statements and 63.25% branches; the documented mature +CPU-testable scope is 93.12% statements and 82.60% branches. The repository +enforces 95% changed-line coverage, per-touched-module line and branch +no-regression, an 85% selective mutation floor, zero unexpected warnings, and +owned suite/test/marker/repeat duration budgets. The canonical increment-3 +campaign killed 1,424 of 1,637 mutants (86.99%), and 431 high-consequence tests +passed in each of three file orders and hash seeds. Package, installed +wheel/sdist, vulnerability, secret, license, and SBOM checks are green. All +seven jobs in the canonical post-merge run passed: +https://github.com/elder-plinius/OBLITERATUS/actions/runs/31920984688. + +Gate 3 increment 4 is implemented and locally verified at core evidence head +`9da022e6f3bc73d17fd31d4da70b42e3ca052ba2`, with delivery evidence recorded in +`gate3-increment-4-report.md`. Candidate coverage is 76.77% statements and +63.55% branches; mature CPU coverage is 93.19% statements and 82.85% branches; +the complete campaign kills 1,614 of 1,827 mutants (88.34%), including all +229 persistence mutants. Exact-head PR audit, the first hosted Windows +checkpoint contract, and canonical merge remain pending. The latest conditional run is green for pinned-model download and evaluation, loopback network, operator UI, and evidence-freshness policy at the earlier @@ -42,8 +52,9 @@ property, mutation, determinism, and real-environment evidence without weakening the mandatory CPU baseline. The operator has now placed ordinary feature work back behind a testing-depth -gate. Gate 3 is adopted and items 1–2 are complete, but items 3–7 and the -quantitative exit criteria must pass before unrelated feature work resumes. +gate. Gate 3 is adopted and items 1–3 are complete; item 4 is implemented but +not yet canonical; items 5–7 and the quantitative exit criteria must pass +before unrelated feature work resumes. Correctness, security, data-loss, and test-infrastructure repairs remain permitted when narrowly scoped and paired with reproducing tests. @@ -282,12 +293,12 @@ resistance, environment evidence, or deterministic replay. | Measure | Current | Gate 3 minimum | |---|---:|---:| -| Repository statements | 76.26% | 80.00% | -| Repository branches | 62.67% | 65.00% | -| Mature CPU statements | 92.87% | 94.00% | -| Mature CPU branches | 82.24% | 84.00% | +| Repository statements | 76.77% candidate | 80.00% | +| Repository branches | 63.55% candidate | 65.00% | +| Mature CPU statements | 93.19% candidate | 94.00% | +| Mature CPU branches | 82.85% candidate | 84.00% | | Changed executable lines | 95.00% floor | 95.00% floor | -| Selected mutation score | 86.15% on expanded numerical scope | at least 85% on expanded scope | +| Selected mutation score | 88.34% candidate on expanded persistence scope | at least 85% on expanded scope | | Unexpected warnings | 0 | 0 | | Active flaky quarantines | 0 | 0 | | Software conditional evidence age | at most 8 days | at most 8 days | @@ -344,7 +355,7 @@ small pure seam and measure it rather than mocking away the behavior under test. | Surface | Current statements / branches | Gate 3 intent | |---|---:|---| -| `abliterate.py` | 49.63% / 37.31% | at least 52% / 45% with mutation/math contracts | +| `abliterate.py` | 50.91% / 38.03% candidate | at least 52% / 45% with mutation/math contracts | | `lora_ablation.py` | 9.57% / 0% | at least 70% / 55% for CPU-testable decisions | | `tourney.py` | 12.01% / 0% | at least 55% / 40% for lifecycle/state transitions | | `bestiary_sync.py` | 0% / 0% | at least 80% / 65% using a loopback/fake transport | @@ -368,11 +379,13 @@ single repository-wide coverage PR. 2. **Completed at `2c2b38b` (PR #102).** Add numerical/reference-oracle and metamorphic contracts; expand mutation to the corresponding pure math. Canonical evidence is recorded in `gate3-increment-2-report.md`. -3. **Implemented in Gate 3 increment 3; exact-head PR audit and canonical merge - pending.** Add loader, architecture, dtype, quantization, and shared-weight - decision contracts with mutation evidence. Candidate evidence is recorded in - `gate3-increment-3-report.md`. -4. Add checkpoint failure injection, concurrency, atomicity, and retry contracts. +3. **Completed at `c1b3450` (PR #103).** Add loader, architecture, dtype, + quantization, and shared-weight decision contracts with mutation evidence. + Canonical evidence is recorded in `gate3-increment-3-report.md`. +4. **Implemented in Gate 3 increment 4; exact-head PR audit and canonical merge + pending.** Add checkpoint failure injection, concurrency, atomicity, retry, + cleanup-ownership, and Windows portability contracts. Candidate evidence is + recorded in `gate3-increment-4-report.md`. 5. Add BESTIARY/model-client/watchtower transport and scheduler state contracts. 6. Add tournament, interactive, and UI decision-seam contracts. 7. Extend the installed tiny-model vertical slice and quantized-storage diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2ba76a6..e25e958 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -393,6 +393,61 @@ jobs: if-no-files-found: error retention-days: 90 + checkpoint-windows: + name: Checkpoint contracts (Windows) + runs-on: windows-latest + timeout-minutes: 15 + env: + CUDA_VISIBLE_DEVICES: "" + HF_DATASETS_OFFLINE: "1" + HF_HUB_DISABLE_TELEMETRY: "1" + HF_HUB_OFFLINE: "1" + TOKENIZERS_PARALLELISM: "false" + TRANSFORMERS_OFFLINE: "1" + + steps: + - name: Check out repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + cache-dependency-path: | + pyproject.toml + uv.lock + + - name: Install locked test environment + shell: pwsh + run: | + python -m pip install "uv==$env:UV_VERSION" + $testEnv = Join-Path $env:RUNNER_TEMP "obliteratus-checkpoint-windows" + $env:UV_PROJECT_ENVIRONMENT = $testEnv + uv sync --locked --no-default-groups --extra dev --no-editable + "CHECKPOINT_TEST_ENV=$testEnv" >> $env:GITHUB_ENV + + - name: Run Windows checkpoint contracts + shell: pwsh + run: | + $python = Join-Path $env:CHECKPOINT_TEST_ENV "Scripts/python.exe" + & $python -m pytest ` + tests/test_checkpoint_atomicity.py ` + tests/test_persistence_contracts.py ` + tests/test_persistence_pipeline.py ` + --no-cov ` + --junitxml="$env:RUNNER_TEMP/checkpoint-windows.xml" ` + -q + + - name: Upload Windows checkpoint evidence + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: checkpoint-evidence-windows-py3.12 + path: ${{ runner.temp }}/checkpoint-windows.xml + if-no-files-found: error + retention-days: 90 + quality-depth: name: Quality depth runs-on: ubuntu-latest diff --git a/tests/test_ci_policy.py b/tests/test_ci_policy.py index d2d39c0..4d1d3d8 100644 --- a/tests/test_ci_policy.py +++ b/tests/test_ci_policy.py @@ -110,3 +110,21 @@ def test_ci_retains_normalized_test_and_quality_trends_for_ninety_days(): assert "test-trend-py${{ matrix.python-version }}.json" in workflow assert "quality-trend-py3.12.json" in workflow assert workflow.count("retention-days: 90") >= 2 + + +def test_ci_runs_checkpoint_portability_contracts_on_windows(): + workflow = WORKFLOW.read_text(encoding="utf-8") + checkpoint_job = workflow.split(" checkpoint-windows:\n", maxsplit=1)[1].split( + " quality-depth:\n", + maxsplit=1, + )[0] + + assert "runs-on: windows-latest" in checkpoint_job + assert 'python-version: "3.12"' in checkpoint_job + assert "uv sync --locked --no-default-groups --extra dev --no-editable" in checkpoint_job + assert "tests/test_checkpoint_atomicity.py" in checkpoint_job + assert "tests/test_persistence_contracts.py" in checkpoint_job + assert "tests/test_persistence_pipeline.py" in checkpoint_job + assert "--no-cov" in checkpoint_job + assert "checkpoint-evidence-windows-py3.12" in checkpoint_job + assert "if-no-files-found: error" in checkpoint_job