ci: verify checkpoint portability on Windows

This commit is contained in:
Joseph Magly
2026-08-16 00:28:15 -04:00
parent c74538a2be
commit 2d803e6a1c
5 changed files with 247 additions and 32 deletions
+10 -5
View File
@@ -5,8 +5,10 @@ Exact base: `2c2b38b501f94af46ffac3a772f7bf9475154dc5`
Exact implementation/evidence head: `e2e120c211adf3ec9a7909a401f5e7a34f17984f`
Delivery head: the containing documentation commit, resolved with
`git rev-parse HEAD`
Canonical merge: pending
Status: PASS locally; hosted exact-head audit and merge pending
Canonical merge: `c1b34503ddd3cb797e5d70671c47afcabfe73832` (PR #103,
rebase merged)
Status: PASS; exact-head audit, rebase merge, and canonical post-merge CI
complete
## Results
@@ -88,6 +90,9 @@ decision logic and its wrappers only; CUDA, bitsandbytes hardware execution,
MPS, MLX, network, download, remote, and operator-UI mappings are unchanged, so
no new environment support claim is made.
Delivery requires the containing documentation commit to be pushed, audited at
its final SHA, pass all hosted checks with no unresolved review thread, and be
rebase merged before item 4 begins.
PR #103 passed its exact-head audit and all required hosted checks with no
unresolved review thread, then rebase merged at
`c1b34503ddd3cb797e5d70671c47afcabfe73832`. All seven jobs in canonical
post-merge run
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31920984688 passed
before item 4 began.
+124
View File
@@ -0,0 +1,124 @@
# Gate 3 increment 4 checkpoint reliability report
Date: 2026-08-15
Exact base: `c1b34503ddd3cb797e5d70671c47afcabfe73832`
Exact core implementation/mutation head:
`9da022e6f3bc73d17fd31d4da70b42e3ca052ba2`
Delivery head: the containing documentation and Windows-CI commit, resolved with
`git rev-parse HEAD`
Canonical merge: pending
Status: PASS locally; hosted Windows contract, exact-head audit, and merge pending
## Results
| Gate | Result |
|---|---|
| Focused checkpoint/policy contracts | 176 passed in 6.08s at the exact core head; the delivery-head portability-policy selection also passed 176 tests in 6.36s |
| Mandatory CPU matrix | Exact core head — Python 3.10: 1,878 passed in 88.21s; Python 3.11: 1,878 passed in 80.63s. Exact delivery head — Python 3.12: 1,877 passed, 2 expected xfails in 45.49s. Nine conditional tests were deselected on each interpreter |
| Repository coverage | 76.60% statements / 63.25% branches before; 76.77% / 63.55% after |
| Mature CPU scope | 93.12% statements / 82.60% branches before; 93.19% / 82.85% after |
| Changed executable lines | 138/138 (100%) against the exact base on Python 3.12 |
| Touched production modules | 2/2 pass exact-base line and branch no-regression |
| Persistence seam | `persistence_contracts.py`: 100% statements and 100% branches (166 statements, 58 branches) |
| Abliteration runtime | 50.82% statements / 38.03% branches before; 50.91% / 38.03% after |
| Selective mutation | 1,614/1,827 killed (88.34%); 208 survived; 5 timed out; 0 no-test, suspicious, skipped, interrupted, or segfault results |
| Changed persistence mutation | 229/229 killed; 0 survived or timed out |
| Mutation runtime | 920.27s; 1,121,616 KiB maximum parent RSS; required-target guard passed |
| Repeatability | 493 tests passed in each of three file orders/hash seeds; 33.832s total; no flake candidates or consistent failures |
| Package | sdist and wheel built; metadata, contents, isolated import, version, module entry point, and console entry point passed on Python 3.12 |
| Supply chain | Pinned `uv 0.12.4` lock, all-extras import, Gitleaks, Python 3.103.12 vulnerability audits, wheel-bound CycloneDX SBOM, and license policy passed |
| Windows portability | A focused Python 3.12 hosted job now blocks on checkpoint atomicity, persistence, and pipeline contracts and retains JUnit evidence; first exact-head result pending |
The mandatory Python lanes remain below the 240-second policy budget. The final
Python 3.12 test time measured 10.08s faster than the exact-base 55.57s while
adding 47 passing tests. The slowest Python 3.12 test remains the existing
offline model save/reload slice at 5.364s; no new checkpoint test exceeds an
owned duration budget.
## Contracts added
1. A staged checkpoint must contain the exact prepared metadata, non-empty JSON
model and tokenizer configurations, and either regular direct weights or a
complete safe sharded-weight map. Missing, empty, corrupt, truncated,
link-backed, or traversal-bearing artifacts fail before promotion.
2. Every regular staged artifact is flushed, nested directory entries are
flushed bottom-up where the platform supports directory fsync, and the
destination parent is flushed after backup, promotion, rollback, and durable
cleanup transitions.
3. A persistent sibling lock serializes cooperative threads and processes that
commit to the same destination. Concurrent complete writers may replace one
another, but readers never observe a partially promoted checkpoint.
4. Writer failure and `BaseException` cancellation are injected before commit
and at tree sync, backup replacement, backup-parent sync, promotion,
promotion-parent sync, and rollback. Existing destinations are restored;
newly created destinations are removed; ambiguous double failures fail
closed with an explicit recovery path.
5. Cleanup errors never mask the primary write failure. Only transaction-owned
staging and backup paths are removed, and a successfully promoted checkpoint
remains valid when obsolete-backup cleanup reports a durable warning.
6. Pipeline offload cleanup no longer clears ownership after a failed removal.
The owned path is retained for a deterministic retry, while successful
cleanup clears both the path and ownership flag.
7. The production rebirth path validates the complete staged checkpoint before
atomic promotion. Persistence is now a required mutation target and the
checkpoint atomicity/pipeline suites are part of mutation selection and the
three-order repeat gate.
8. The platform-specific lock and directory-sync branches are no longer backed
by an unverified CI comment: a focused `windows-latest` lane executes the
checkpoint contracts with the locked Python 3.12 environment and retained
JUnit evidence.
## Defects and mutants detected
The development mutation campaign initially left 29 persistence survivors.
They identified weak JSON type/path assertions, incomplete cleanup and
diagnostic call contracts, missing cancellation boundaries, and insufficient
parent-sync/rollback observability. The repaired exact-target campaign killed
all generated persistence mutants. The final complete campaign preserved that
229/229 result while exceeding the immutable aggregate 85% floor.
The full fault-injection suite also detects partial directory trees, unsafe or
missing shards, symlinked artifacts, file-fsync failure, concurrent commit
interleavings, incomplete rollback, cleanup-warning suppression, stale staging
ownership, and offload-directory ownership loss.
## Verification and evidence
- Exact delivery-head Python 3.12 coverage/JUnit/trend evidence:
`/tmp/obliteratus-gate3-item4-final-JlA1k3/`
- Exact-base Python 3.12 coverage/JUnit evidence:
`/tmp/obliteratus-gate3-item4-base-2uZEXr/`
- Python 3.10 and 3.11 coverage/JUnit evidence:
`/tmp/obliteratus-gate3-item4-py310-exact-ldLxx6/` and
`/tmp/obliteratus-gate3-item4-py311-exact-H4wFBU/`
- Repeat evidence:
`/tmp/obliteratus-gate3-item4-repeat-H23MdN/`
- Exact core-head mutation evidence:
`/tmp/obliteratus-gate3-item4-mutation-PVKKOL/`
- Python 3.10/3.11 and quality-depth normalized trends:
`/tmp/obliteratus-gate3-item4-trends-lrrEjF/`
- Package evidence:
`/tmp/obliteratus-gate3-item4-package-CTZeC6/`
- Pinned-tool supply-chain evidence:
`/tmp/obliteratus-gate3-item4-supply-7aqGSc/`
The canonical Ruff F gate, checksum-verified `actionlint`, `uv lock --check`,
conditional-policy validation, risk-map validation, changed-line gate,
touched-module gate, immutable coverage policy, normalized duration policy,
repeat gate, mutation score, installed-distribution checks, and supply-chain
decisions all pass. The ordinary worktree remained clean; all work and evidence
were isolated in the dedicated item-4 worktree and temporary evidence paths.
## Scope and release decision
This increment implements Gate 3 bounded delivery item 4. It does not reopen
ordinary feature work: items 57, conditional-evidence reconciliation, and the
final canonical Gate 3 publication remain open. The new hosted Windows lane is
a checkpoint portability contract, not a claim that accelerator, MPS, MLX,
network, download, remote, or operator-UI execution is supported by this
increment.
Delivery requires the containing commit to be pushed, pass the first hosted
Windows checkpoint run and all existing required checks, be audited at its
exact final SHA with no unresolved review thread, and be rebase merged before
item 5 begins.
+40 -27
View File
@@ -12,20 +12,30 @@ Gate 3 increment 1 canonical commit:
Gate 3 increment 2 canonical commit:
`2c2b38b501f94af46ffac3a772f7bf9475154dc5` (PR #102, rebase merged)
The mandatory offline CPU selection passes on Python 3.103.12 with 1,644
tests, 9 conditionally deselected tests, no skips, and no warnings. The slowest
hosted pytest selection is Python 3.12 at 158.989 seconds; its complete CI job
is 5 minutes 41 seconds because it also regenerates exact-base coverage.
Branch-enabled Python 3.12 coverage is 75.68% statements and 61.72% branches.
The documented mature CPU-testable scope is 92.71% statements and 81.55%
branches. The repository enforces 95% changed-line coverage, per-touched-module
line and branch no-regression, a 75% selective mutation floor, zero unexpected
warnings, and owned suite/test/marker/repeat duration budgets. The bounded
campaign kills 809 of 886 mutants (91.31%), and 391 high-consequence tests pass
in each of three file orders and hash seeds with no flake candidates. Package,
installed wheel/sdist, vulnerability, secret, license, and SBOM checks are
green. All seven jobs in the exact canonical-commit run passed:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31874768085.
Gate 3 increment 3 canonical commit:
`c1b34503ddd3cb797e5d70671c47afcabfe73832` (PR #103, rebase merged)
The latest canonical mandatory offline CPU selection passes on Python 3.103.12
with 1,8301,832 passing tests, 9 conditionally deselected tests, 2 expected
Python 3.12 xfails, no unexpected skips, and no warnings. Branch-enabled Python
3.12 coverage is 76.60% statements and 63.25% branches; the documented mature
CPU-testable scope is 93.12% statements and 82.60% branches. The repository
enforces 95% changed-line coverage, per-touched-module line and branch
no-regression, an 85% selective mutation floor, zero unexpected warnings, and
owned suite/test/marker/repeat duration budgets. The canonical increment-3
campaign killed 1,424 of 1,637 mutants (86.99%), and 431 high-consequence tests
passed in each of three file orders and hash seeds. Package, installed
wheel/sdist, vulnerability, secret, license, and SBOM checks are green. All
seven jobs in the canonical post-merge run passed:
https://github.com/elder-plinius/OBLITERATUS/actions/runs/31920984688.
Gate 3 increment 4 is implemented and locally verified at core evidence head
`9da022e6f3bc73d17fd31d4da70b42e3ca052ba2`, with delivery evidence recorded in
`gate3-increment-4-report.md`. Candidate coverage is 76.77% statements and
63.55% branches; mature CPU coverage is 93.19% statements and 82.85% branches;
the complete campaign kills 1,614 of 1,827 mutants (88.34%), including all
229 persistence mutants. Exact-head PR audit, the first hosted Windows
checkpoint contract, and canonical merge remain pending.
The latest conditional run is green for pinned-model download and evaluation,
loopback network, operator UI, and evidence-freshness policy at the earlier
@@ -42,8 +52,9 @@ property, mutation, determinism, and real-environment evidence without weakening
the mandatory CPU baseline.
The operator has now placed ordinary feature work back behind a testing-depth
gate. Gate 3 is adopted and items 12 are complete, but items 37 and the
quantitative exit criteria must pass before unrelated feature work resumes.
gate. Gate 3 is adopted and items 13 are complete; item 4 is implemented but
not yet canonical; items 57 and the quantitative exit criteria must pass
before unrelated feature work resumes.
Correctness, security, data-loss, and test-infrastructure repairs remain
permitted when narrowly scoped and paired with reproducing tests.
@@ -282,12 +293,12 @@ resistance, environment evidence, or deterministic replay.
| Measure | Current | Gate 3 minimum |
|---|---:|---:|
| Repository statements | 76.26% | 80.00% |
| Repository branches | 62.67% | 65.00% |
| Mature CPU statements | 92.87% | 94.00% |
| Mature CPU branches | 82.24% | 84.00% |
| Repository statements | 76.77% candidate | 80.00% |
| Repository branches | 63.55% candidate | 65.00% |
| Mature CPU statements | 93.19% candidate | 94.00% |
| Mature CPU branches | 82.85% candidate | 84.00% |
| Changed executable lines | 95.00% floor | 95.00% floor |
| Selected mutation score | 86.15% on expanded numerical scope | at least 85% on expanded scope |
| Selected mutation score | 88.34% candidate on expanded persistence scope | at least 85% on expanded scope |
| Unexpected warnings | 0 | 0 |
| Active flaky quarantines | 0 | 0 |
| Software conditional evidence age | at most 8 days | at most 8 days |
@@ -344,7 +355,7 @@ small pure seam and measure it rather than mocking away the behavior under test.
| Surface | Current statements / branches | Gate 3 intent |
|---|---:|---|
| `abliterate.py` | 49.63% / 37.31% | at least 52% / 45% with mutation/math contracts |
| `abliterate.py` | 50.91% / 38.03% candidate | at least 52% / 45% with mutation/math contracts |
| `lora_ablation.py` | 9.57% / 0% | at least 70% / 55% for CPU-testable decisions |
| `tourney.py` | 12.01% / 0% | at least 55% / 40% for lifecycle/state transitions |
| `bestiary_sync.py` | 0% / 0% | at least 80% / 65% using a loopback/fake transport |
@@ -368,11 +379,13 @@ single repository-wide coverage PR.
2. **Completed at `2c2b38b` (PR #102).** Add numerical/reference-oracle and
metamorphic contracts; expand mutation to the corresponding pure math.
Canonical evidence is recorded in `gate3-increment-2-report.md`.
3. **Implemented in Gate 3 increment 3; exact-head PR audit and canonical merge
pending.** Add loader, architecture, dtype, quantization, and shared-weight
decision contracts with mutation evidence. Candidate evidence is recorded in
`gate3-increment-3-report.md`.
4. Add checkpoint failure injection, concurrency, atomicity, and retry contracts.
3. **Completed at `c1b3450` (PR #103).** Add loader, architecture, dtype,
quantization, and shared-weight decision contracts with mutation evidence.
Canonical evidence is recorded in `gate3-increment-3-report.md`.
4. **Implemented in Gate 3 increment 4; exact-head PR audit and canonical merge
pending.** Add checkpoint failure injection, concurrency, atomicity, retry,
cleanup-ownership, and Windows portability contracts. Candidate evidence is
recorded in `gate3-increment-4-report.md`.
5. Add BESTIARY/model-client/watchtower transport and scheduler state contracts.
6. Add tournament, interactive, and UI decision-seam contracts.
7. Extend the installed tiny-model vertical slice and quantized-storage
+55
View File
@@ -393,6 +393,61 @@ jobs:
if-no-files-found: error
retention-days: 90
checkpoint-windows:
name: Checkpoint contracts (Windows)
runs-on: windows-latest
timeout-minutes: 15
env:
CUDA_VISIBLE_DEVICES: ""
HF_DATASETS_OFFLINE: "1"
HF_HUB_DISABLE_TELEMETRY: "1"
HF_HUB_OFFLINE: "1"
TOKENIZERS_PARALLELISM: "false"
TRANSFORMERS_OFFLINE: "1"
steps:
- name: Check out repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
cache: pip
cache-dependency-path: |
pyproject.toml
uv.lock
- name: Install locked test environment
shell: pwsh
run: |
python -m pip install "uv==$env:UV_VERSION"
$testEnv = Join-Path $env:RUNNER_TEMP "obliteratus-checkpoint-windows"
$env:UV_PROJECT_ENVIRONMENT = $testEnv
uv sync --locked --no-default-groups --extra dev --no-editable
"CHECKPOINT_TEST_ENV=$testEnv" >> $env:GITHUB_ENV
- name: Run Windows checkpoint contracts
shell: pwsh
run: |
$python = Join-Path $env:CHECKPOINT_TEST_ENV "Scripts/python.exe"
& $python -m pytest `
tests/test_checkpoint_atomicity.py `
tests/test_persistence_contracts.py `
tests/test_persistence_pipeline.py `
--no-cov `
--junitxml="$env:RUNNER_TEMP/checkpoint-windows.xml" `
-q
- name: Upload Windows checkpoint evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: checkpoint-evidence-windows-py3.12
path: ${{ runner.temp }}/checkpoint-windows.xml
if-no-files-found: error
retention-days: 90
quality-depth:
name: Quality depth
runs-on: ubuntu-latest
+18
View File
@@ -110,3 +110,21 @@ def test_ci_retains_normalized_test_and_quality_trends_for_ninety_days():
assert "test-trend-py${{ matrix.python-version }}.json" in workflow
assert "quality-trend-py3.12.json" in workflow
assert workflow.count("retention-days: 90") >= 2
def test_ci_runs_checkpoint_portability_contracts_on_windows():
workflow = WORKFLOW.read_text(encoding="utf-8")
checkpoint_job = workflow.split(" checkpoint-windows:\n", maxsplit=1)[1].split(
" quality-depth:\n",
maxsplit=1,
)[0]
assert "runs-on: windows-latest" in checkpoint_job
assert 'python-version: "3.12"' in checkpoint_job
assert "uv sync --locked --no-default-groups --extra dev --no-editable" in checkpoint_job
assert "tests/test_checkpoint_atomicity.py" in checkpoint_job
assert "tests/test_persistence_contracts.py" in checkpoint_job
assert "tests/test_persistence_pipeline.py" in checkpoint_job
assert "--no-cov" in checkpoint_job
assert "checkpoint-evidence-windows-py3.12" in checkpoint_job
assert "if-no-files-found: error" in checkpoint_job