Conditional test operations

The mandatory pull-request workflow remains CPU-only, offline, and credential-free. Environment-bound contracts run through Conditional tests on manual dispatch, weekly at 06:17 UTC Sunday, and for published releases. OBLITERATUS maintainers own the workflow, policy, runner labels, credentials, and evidence review.

The canonical machine-readable policy is ci/conditional-test-policy.json. It maps every environment-bound mature-coverage exclusion to a runnable gate, defines the pinned tiny model, records prerequisites and expected cost, retains evidence for 30 days, and treats evidence older than eight days as stale. Every workflow run publishes a summary showing selected, successful, failed, and not-selected/no-fresh-evidence gates.

For a release, maintainers run the applicable hosted software gates manually against the exact signed tag before publication. Publishing the release triggers the same workflow again, and that release-event run is retained as the final freshness record. Unselected hardware or credential-bound jobs remain explicit non-evidence. See RELEASE_PROCESS.md for their place in the release sequence.

Unavailable hardware or credential-bound gates may use a committed environment waiver for at most 30 days. Each waiver names one gate, a canonical tracking issue, the reason, its opening and expiry dates, and the support claim it blocks. Invalid, duplicate, future-dated, overlong, or expired waivers fail the policy job. The current waivers are tracked by issue #110 and expire on 2026-09-15; they are not evidence that any waived backend works. They block support, compatibility, correctness, and performance claims for the waived environment.

The current accelerator waivers cover missing GitHub runner registration, not missing hardware. Titan has an RTX 4090 behind Gitea runners, and Mutsu is a 16 GB Apple M4 builder. Exact-head operator probes may be attached to issue #110, but they do not make the scheduled GitHub lanes runnable or establish a broad backend support claim.

Hosted gates

The model gate downloads only hf-internal-testing/tiny-random-gpt2@71034c5d8bde858ff824298bdedc65515b97d2b9 with trust_remote_code=False. Its cache key includes the model revision, Python version, and runner OS. The gate performs a forward pass, reopens the cache with Hub and Transformers offline modes enabled, asserts an uncached model fails offline, and runs the evaluator on two tiny samples. Its timeout is 25 minutes and its expected download is below 100 MB.

The network-service gate uses a disposable loopback HTTP server and no credentials. The operator-UI gate installs the locked spaces extra and constructs the Gradio application without launching a listener. These gates cost less than ten hosted runner-minutes each under normal conditions.

Run the same probes locally with:

uv sync --locked --extra dev
uv run --extra dev python scripts/run_conditional_gate.py model-download-runtime
uv run --extra dev python scripts/run_conditional_gate.py external-evaluation
uv run --extra dev python scripts/run_conditional_gate.py network-services

uv sync --locked --all-extras
uv run --all-extras python scripts/run_conditional_gate.py operator-ui

CUDA and bitsandbytes

Attach a dedicated runner with the labels self-hosted, linux, x64, and cuda. Set the repository variable ENABLE_CUDA_GATE=true for scheduled/release evidence, or select CUDA during manual dispatch. The gate verifies CUDA discovery, automatic selection, dtype selection, tensor placement, matrix multiplication, bitsandbytes availability, NF4 quantization, dequantization, shape, placement, and finite output. The expected cost is below 20 self-hosted runner-minutes. Mandatory Linux CI deliberately locks CPU-only Torch. The selected CUDA job reads that exact locked base version, replaces only Torch with the same-version official cu130 build, asserts a CUDA build was installed, and runs uv pip check before executing the probes.

For an operator run on the labeled machine:

uv sync --locked --extra dev --extra quantization
CUDA_TORCH_VERSION="$(.venv/bin/python -c \
  'import torch; print(torch.__version__.split("+", 1)[0])')"
UV_TORCH_BACKEND=cu130 uv pip install --python .venv/bin/python \
  --reinstall-package torch "torch==$CUDA_TORCH_VERSION"
uv pip check --python .venv/bin/python
.venv/bin/python scripts/run_conditional_gate.py cuda-runtime
.venv/bin/python scripts/run_conditional_gate.py bitsandbytes-runtime

Use the virtual environment interpreter directly after replacing Torch. A subsequent uv run or uv sync without the CUDA override may restore the portable CPU wheel from the lock. The agentic developer installer automates this ordering and performs a real CUDA tensor probe; see installer/setup.dev.manifest.yaml.

Quality-verifier coherence semantics

The built-in VERIFY stage does not treat length and vocabulary diversity as coherence. Each fixed factual prompt has deterministic semantic anchors, and the scorer rejects strong repetition and recognizable corpus contamination such as scraped Q&A profile/answer fragments. This is intentionally local and deterministic: release verification does not call another model or a network judge. Perplexity and extended capability checks remain separate measurements.

On Linux workstations that also install cuDNN system-wide, the dynamic loader can mix a host sublibrary with PyTorch's bundled cuDNN and raise CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH. The local launcher sets OBLITERATUS_DISABLE_CUDNN=1, keeping causal convolutions and attention on PyTorch's other CUDA kernels rather than moving the model to the CPU. Installer verification runs actual causal conv1d and SDPA operations, not only a generic CUDA allocation, so this failure is caught before a model pipeline begins.

Jetson CUDA support is tracked separately from this generic x64 CUDA lane. The mandatory Linux ARM64 preflight uses GitHub's hosted ubuntu-24.04-arm runner to prove locked CPU packaging, imports, CLI startup, and Jetson tooling contracts. It is not GPU evidence. Jetson depends on a JetPack/L4T-matched CUDA, cuDNN, and PyTorch runtime.

Physical testing uses only a trusted manual dispatch on the labels self-hosted, linux, ARM64, and jetson. The job preserves NVIDIA's vendor PyTorch, runs jetson-runtime, and uploads the sanitized conditional-jetson-<run-attempt> artifact for 30 days. It never runs for a pull request, schedule, or release. Contributor bootstrap, runner isolation, reporting commands, and acceptance criteria are documented in the NVIDIA Jetson support plan.

Apple MPS and MLX

MPS uses a runner labeled self-hosted, macOS, ARM64, and mps; enable its schedule with ENABLE_MPS_GATE=true. MLX uses the same first three labels plus mlx and ENABLE_MLX_GATE=true. The MPS probe checks discovery, selection, dtype and float64 fallback, placement, and a real matrix operation. The MLX probe uses the locked mlx==0.32.0 and mlx-lm==0.31.3 packages and verifies imports, array placement, evaluation, and matrix multiplication. Each should cost less than 20 self-hosted runner-minutes.

If the repository has no attached Apple runner, collect equivalent operator evidence on Apple Silicon and attach the JSON and JUnit files to the tracking issue:

uv sync --locked --extra dev
uv run --extra dev python scripts/run_conditional_gate.py mps-runtime

uv sync --locked --extra dev --group mlx
uv run --extra dev --group mlx python scripts/run_conditional_gate.py mlx-runtime

Remote provider

Remote evidence is opt-in. Configure a non-root, command-limited test account and:

variables OBLITERATUS_REMOTE_HOST, OBLITERATUS_REMOTE_USER, and optionally OBLITERATUS_REMOTE_PORT;
secrets OBLITERATUS_REMOTE_KEY and OBLITERATUS_REMOTE_KNOWN_HOSTS.

The known-hosts entry must be pinned after verifying the provider fingerprint through an independent channel. The workflow writes credentials to mode-0600 temporary files, never prints their contents, uses batch mode and strict host-key checking, and only runs echo ok plus python3 -c 'print(6 * 7)'. Missing prerequisites produce an actionable failure when the gate was selected; a direct local invocation may use --allow-missing to record explicit not_run evidence. Expected cost is below five hosted runner-minutes plus any provider charge.

Result semantics

scripts/run_conditional_gate.py requires at least one executed test and rejects any failure, error, or skip. A selected workflow job therefore cannot become green through an availability skip or unconditional success conversion. The final summary also fails if any selected job is not successful. Unselected jobs are explicitly reported as not_selected_no_fresh_evidence; they are not evidence of backend support. An unselected hardware or credential-bound gate with a valid waiver is reported as waived_no_support_claim, including its tracker, expiry, and blocked claim. Selecting and successfully running that gate produces success instead of relying on the waiver.

9.0 KiB Raw Blame History