OBLITERATUS/docs/conditional-testing.md

# Conditional test operations

The mandatory pull-request workflow remains CPU-only, offline, and credential-free.
Environment-bound contracts run through `Conditional tests` on manual dispatch,
weekly at 06:17 UTC Sunday, and for published releases. OBLITERATUS maintainers own
the workflow, policy, runner labels, credentials, and evidence review.

The canonical machine-readable policy is `ci/conditional-test-policy.json`. It maps
every environment-bound mature-coverage exclusion to a runnable gate, defines the
pinned tiny model, records prerequisites and expected cost, retains evidence for 30
days, and treats evidence older than eight days as stale. Every workflow run publishes
a summary showing selected, successful, failed, and not-selected/no-fresh-evidence
gates.

Unavailable hardware or credential-bound gates may use a committed environment
waiver for at most 30 days. Each waiver names one gate, a canonical tracking issue,
the reason, its opening and expiry dates, and the support claim it blocks. Invalid,
duplicate, future-dated, overlong, or expired waivers fail the policy job. The current
waivers are tracked by [issue #110](https://github.com/elder-plinius/OBLITERATUS/issues/110)
and expire on 2026-09-15; they are not evidence that any waived backend works.
They block support, compatibility, correctness, and performance claims for the
waived environment.

The current accelerator waivers cover missing GitHub runner registration, not
missing hardware. Titan has an RTX 4090 behind Gitea runners, and Mutsu is a 16 GB
Apple M4 builder. Exact-head operator probes may be attached to issue #110, but
they do not make the scheduled GitHub lanes runnable or establish a broad backend
support claim.

## Hosted gates

The model gate downloads only
`hf-internal-testing/tiny-random-gpt2@71034c5d8bde858ff824298bdedc65515b97d2b9`
with `trust_remote_code=False`. Its cache key includes the model revision, Python
version, and runner OS. The gate performs a forward pass, reopens the cache with Hub
and Transformers offline modes enabled, asserts an uncached model fails offline, and
runs the evaluator on two tiny samples. Its timeout is 25 minutes and its expected
download is below 100 MB.

The network-service gate uses a disposable loopback HTTP server and no credentials.
The operator-UI gate installs the locked `spaces` extra and constructs the Gradio
application without launching a listener. These gates cost less than ten hosted
runner-minutes each under normal conditions.

Run the same probes locally with:

```bash
uv sync --locked --extra dev
uv run --extra dev python scripts/run_conditional_gate.py model-download-runtime
uv run --extra dev python scripts/run_conditional_gate.py external-evaluation
uv run --extra dev python scripts/run_conditional_gate.py network-services

uv sync --locked --all-extras
uv run --all-extras python scripts/run_conditional_gate.py operator-ui
```

## CUDA and bitsandbytes

Attach a dedicated runner with the labels `self-hosted`, `linux`, `x64`, and `cuda`.
Set the repository variable `ENABLE_CUDA_GATE=true` for scheduled/release evidence,
or select CUDA during manual dispatch. The gate verifies CUDA discovery, automatic
selection, dtype selection, tensor placement, matrix multiplication, bitsandbytes
availability, NF4 quantization, dequantization, shape, placement, and finite output.
The expected cost is below 20 self-hosted runner-minutes. Mandatory Linux CI
deliberately locks CPU-only Torch. The selected CUDA job reads that exact locked
base version, replaces only Torch with the same-version official `cu130` build,
asserts a CUDA build was installed, and runs `uv pip check` before executing the
probes.

For an operator run on the labeled machine:

```bash
uv sync --locked --extra dev
CUDA_TORCH_VERSION="$(.venv/bin/python -c \
  'import torch; print(torch.__version__.split("+", 1)[0])')"
UV_TORCH_BACKEND=cu130 uv pip install --python .venv/bin/python \
  --reinstall-package torch "torch==$CUDA_TORCH_VERSION"
uv pip check --python .venv/bin/python
uv run --extra dev python scripts/run_conditional_gate.py cuda-runtime
uv run --extra dev python scripts/run_conditional_gate.py bitsandbytes-runtime
```

## Apple MPS and MLX

MPS uses a runner labeled `self-hosted`, `macOS`, `ARM64`, and `mps`; enable its
schedule with `ENABLE_MPS_GATE=true`. MLX uses the same first three labels plus `mlx`
and `ENABLE_MLX_GATE=true`. The MPS probe checks discovery, selection, dtype and
float64 fallback, placement, and a real matrix operation. The MLX probe uses the
locked `mlx==0.32.0` and `mlx-lm==0.31.3` packages and verifies imports, array
placement, evaluation, and matrix multiplication. Each should cost less than 20
self-hosted runner-minutes.

If the repository has no attached Apple runner, collect equivalent operator evidence
on Apple Silicon and attach the JSON and JUnit files to the tracking issue:

```bash
uv sync --locked --extra dev
uv run --extra dev python scripts/run_conditional_gate.py mps-runtime

uv sync --locked --extra dev --group mlx
uv run --extra dev --group mlx python scripts/run_conditional_gate.py mlx-runtime
```

## Remote provider

Remote evidence is opt-in. Configure a non-root, command-limited test account and:

- variables `OBLITERATUS_REMOTE_HOST`, `OBLITERATUS_REMOTE_USER`, and optionally
  `OBLITERATUS_REMOTE_PORT`;
- secrets `OBLITERATUS_REMOTE_KEY` and `OBLITERATUS_REMOTE_KNOWN_HOSTS`.

The known-hosts entry must be pinned after verifying the provider fingerprint through
an independent channel. The workflow writes credentials to mode-0600 temporary files,
never prints their contents, uses batch mode and strict host-key checking, and only
runs `echo ok` plus `python3 -c 'print(6 * 7)'`. Missing prerequisites produce an
actionable failure when the gate was selected; a direct local invocation may use
`--allow-missing` to record explicit `not_run` evidence. Expected cost is below five
hosted runner-minutes plus any provider charge.

## Result semantics

`scripts/run_conditional_gate.py` requires at least one executed test and rejects any
failure, error, or skip. A selected workflow job therefore cannot become green through
an availability skip or unconditional success conversion. The final summary also
fails if any selected job is not successful. Unselected jobs are explicitly reported
as `not_selected_no_fresh_evidence`; they are not evidence of backend support. An
unselected hardware or credential-bound gate with a valid waiver is reported as
`waived_no_support_claim`, including its tracker, expiry, and blocked claim. Selecting
and successfully running that gate produces `success` instead of relying on the
waiver.