Conditional test operations

The mandatory pull-request workflow remains CPU-only, offline, and credential-free. Environment-bound contracts run through Conditional tests on manual dispatch, weekly at 06:17 UTC Sunday, and for published releases. OBLITERATUS maintainers own the workflow, policy, runner labels, credentials, and evidence review.

The canonical machine-readable policy is ci/conditional-test-policy.json. It maps every environment-bound mature-coverage exclusion to a runnable gate, defines the pinned tiny model, records prerequisites and expected cost, retains evidence for 30 days, and treats evidence older than eight days as stale. Every workflow run publishes a summary showing selected, successful, failed, and not-selected/no-fresh-evidence gates.

Unavailable hardware or credential-bound gates may use a committed environment waiver for at most 30 days. Each waiver names one gate, a canonical tracking issue, the reason, its opening and expiry dates, and the support claim it blocks. Invalid, duplicate, future-dated, overlong, or expired waivers fail the policy job. The current waivers are tracked by issue #110 and expire on 2026-09-15; they are not evidence that any waived backend works.

Hosted gates

The model gate downloads only hf-internal-testing/tiny-random-gpt2@71034c5d8bde858ff824298bdedc65515b97d2b9 with trust_remote_code=False. Its cache key includes the model revision, Python version, and runner OS. The gate performs a forward pass, reopens the cache with Hub and Transformers offline modes enabled, asserts an uncached model fails offline, and runs the evaluator on two tiny samples. Its timeout is 25 minutes and its expected download is below 100 MB.

The network-service gate uses a disposable loopback HTTP server and no credentials. The operator-UI gate installs the locked spaces extra and constructs the Gradio application without launching a listener. These gates cost less than ten hosted runner-minutes each under normal conditions.

Run the same probes locally with:

uv sync --locked --extra dev
uv run --extra dev python scripts/run_conditional_gate.py model-download-runtime
uv run --extra dev python scripts/run_conditional_gate.py external-evaluation
uv run --extra dev python scripts/run_conditional_gate.py network-services

uv sync --locked --all-extras
uv run --all-extras python scripts/run_conditional_gate.py operator-ui

CUDA and bitsandbytes

Attach a dedicated runner with the labels self-hosted, linux, x64, and cuda. Set the repository variable ENABLE_CUDA_GATE=true for scheduled/release evidence, or select CUDA during manual dispatch. The gate verifies CUDA discovery, automatic selection, dtype selection, tensor placement, matrix multiplication, bitsandbytes availability, NF4 quantization, dequantization, shape, placement, and finite output. The expected cost is below 20 self-hosted runner-minutes.

For an operator run on the labeled machine:

uv sync --locked --extra dev
uv run --extra dev python scripts/run_conditional_gate.py cuda-runtime
uv run --extra dev python scripts/run_conditional_gate.py bitsandbytes-runtime

Apple MPS and MLX

MPS uses a runner labeled self-hosted, macOS, ARM64, and mps; enable its schedule with ENABLE_MPS_GATE=true. MLX uses the same first three labels plus mlx and ENABLE_MLX_GATE=true. The MPS probe checks discovery, selection, dtype and float64 fallback, placement, and a real matrix operation. The MLX probe uses the locked mlx==0.32.0 and mlx-lm==0.31.3 packages and verifies imports, array placement, evaluation, and matrix multiplication. Each should cost less than 20 self-hosted runner-minutes.

If the repository has no attached Apple runner, collect equivalent operator evidence on Apple Silicon and attach the JSON and JUnit files to the tracking issue:

uv sync --locked --extra dev
uv run --extra dev python scripts/run_conditional_gate.py mps-runtime

uv sync --locked --extra dev --group mlx
uv run --extra dev --group mlx python scripts/run_conditional_gate.py mlx-runtime

Remote provider

Remote evidence is opt-in. Configure a non-root, command-limited test account and:

variables OBLITERATUS_REMOTE_HOST, OBLITERATUS_REMOTE_USER, and optionally OBLITERATUS_REMOTE_PORT;
secrets OBLITERATUS_REMOTE_KEY and OBLITERATUS_REMOTE_KNOWN_HOSTS.

The known-hosts entry must be pinned after verifying the provider fingerprint through an independent channel. The workflow writes credentials to mode-0600 temporary files, never prints their contents, uses batch mode and strict host-key checking, and only runs echo ok plus python3 -c 'print(6 * 7)'. Missing prerequisites produce an actionable failure when the gate was selected; a direct local invocation may use --allow-missing to record explicit not_run evidence. Expected cost is below five hosted runner-minutes plus any provider charge.

Result semantics

scripts/run_conditional_gate.py requires at least one executed test and rejects any failure, error, or skip. A selected workflow job therefore cannot become green through an availability skip or unconditional success conversion. The final summary also fails if any selected job is not successful. Unselected jobs are explicitly reported as not_selected_no_fresh_evidence; they are not evidence of backend support. An unselected hardware or credential-bound gate with a valid waiver is reported as waived_no_support_claim, including its tracker, expiry, and blocked claim. Selecting and successfully running that gate produces success instead of relying on the waiver.

5.6 KiB Raw Blame History