mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-08-17 16:37:30 +02:00
103 lines
4.8 KiB
Markdown
103 lines
4.8 KiB
Markdown
# Conditional test operations
|
|
|
|
The mandatory pull-request workflow remains CPU-only, offline, and credential-free.
|
|
Environment-bound contracts run through `Conditional tests` on manual dispatch,
|
|
weekly at 06:17 UTC Sunday, and for published releases. OBLITERATUS maintainers own
|
|
the workflow, policy, runner labels, credentials, and evidence review.
|
|
|
|
The canonical machine-readable policy is `ci/conditional-test-policy.json`. It maps
|
|
every environment-bound mature-coverage exclusion to a runnable gate, defines the
|
|
pinned tiny model, records prerequisites and expected cost, retains evidence for 30
|
|
days, and treats evidence older than eight days as stale. Every workflow run publishes
|
|
a summary showing selected, successful, failed, and not-selected/no-fresh-evidence
|
|
gates.
|
|
|
|
## Hosted gates
|
|
|
|
The model gate downloads only
|
|
`hf-internal-testing/tiny-random-gpt2@71034c5d8bde858ff824298bdedc65515b97d2b9`
|
|
with `trust_remote_code=False`. Its cache key includes the model revision, Python
|
|
version, and runner OS. The gate performs a forward pass, reopens the cache with Hub
|
|
and Transformers offline modes enabled, asserts an uncached model fails offline, and
|
|
runs the evaluator on two tiny samples. Its timeout is 25 minutes and its expected
|
|
download is below 100 MB.
|
|
|
|
The network-service gate uses a disposable loopback HTTP server and no credentials.
|
|
The operator-UI gate installs the locked `spaces` extra and constructs the Gradio
|
|
application without launching a listener. These gates cost less than ten hosted
|
|
runner-minutes each under normal conditions.
|
|
|
|
Run the same probes locally with:
|
|
|
|
```bash
|
|
uv sync --locked --extra dev
|
|
uv run --extra dev python scripts/run_conditional_gate.py model-download-runtime
|
|
uv run --extra dev python scripts/run_conditional_gate.py external-evaluation
|
|
uv run --extra dev python scripts/run_conditional_gate.py network-services
|
|
|
|
uv sync --locked --all-extras
|
|
uv run --all-extras python scripts/run_conditional_gate.py operator-ui
|
|
```
|
|
|
|
## CUDA and bitsandbytes
|
|
|
|
Attach a dedicated runner with the labels `self-hosted`, `linux`, `x64`, and `cuda`.
|
|
Set the repository variable `ENABLE_CUDA_GATE=true` for scheduled/release evidence,
|
|
or select CUDA during manual dispatch. The gate verifies CUDA discovery, automatic
|
|
selection, dtype selection, tensor placement, matrix multiplication, bitsandbytes
|
|
availability, NF4 quantization, dequantization, shape, placement, and finite output.
|
|
The expected cost is below 20 self-hosted runner-minutes.
|
|
|
|
For an operator run on the labeled machine:
|
|
|
|
```bash
|
|
uv sync --locked --extra dev
|
|
uv run --extra dev python scripts/run_conditional_gate.py cuda-runtime
|
|
uv run --extra dev python scripts/run_conditional_gate.py bitsandbytes-runtime
|
|
```
|
|
|
|
## Apple MPS and MLX
|
|
|
|
MPS uses a runner labeled `self-hosted`, `macOS`, `ARM64`, and `mps`; enable its
|
|
schedule with `ENABLE_MPS_GATE=true`. MLX uses the same first three labels plus `mlx`
|
|
and `ENABLE_MLX_GATE=true`. The MPS probe checks discovery, selection, dtype and
|
|
float64 fallback, placement, and a real matrix operation. The MLX probe uses the
|
|
locked `mlx==0.32.0` and `mlx-lm==0.31.3` packages and verifies imports, array
|
|
placement, evaluation, and matrix multiplication. Each should cost less than 20
|
|
self-hosted runner-minutes.
|
|
|
|
If the repository has no attached Apple runner, collect equivalent operator evidence
|
|
on Apple Silicon and attach the JSON and JUnit files to the tracking issue:
|
|
|
|
```bash
|
|
uv sync --locked --extra dev
|
|
uv run --extra dev python scripts/run_conditional_gate.py mps-runtime
|
|
|
|
uv sync --locked --extra dev --group mlx
|
|
uv run --extra dev --group mlx python scripts/run_conditional_gate.py mlx-runtime
|
|
```
|
|
|
|
## Remote provider
|
|
|
|
Remote evidence is opt-in. Configure a non-root, command-limited test account and:
|
|
|
|
- variables `OBLITERATUS_REMOTE_HOST`, `OBLITERATUS_REMOTE_USER`, and optionally
|
|
`OBLITERATUS_REMOTE_PORT`;
|
|
- secrets `OBLITERATUS_REMOTE_KEY` and `OBLITERATUS_REMOTE_KNOWN_HOSTS`.
|
|
|
|
The known-hosts entry must be pinned after verifying the provider fingerprint through
|
|
an independent channel. The workflow writes credentials to mode-0600 temporary files,
|
|
never prints their contents, uses batch mode and strict host-key checking, and only
|
|
runs `echo ok` plus `python3 -c 'print(6 * 7)'`. Missing prerequisites produce an
|
|
actionable failure when the gate was selected; a direct local invocation may use
|
|
`--allow-missing` to record explicit `not_run` evidence. Expected cost is below five
|
|
hosted runner-minutes plus any provider charge.
|
|
|
|
## Result semantics
|
|
|
|
`scripts/run_conditional_gate.py` requires at least one executed test and rejects any
|
|
failure, error, or skip. A selected workflow job therefore cannot become green through
|
|
an availability skip or unconditional success conversion. The final summary also
|
|
fails if any selected job is not successful. Unselected jobs are explicitly reported
|
|
as `not_selected_no_fresh_evidence`; they are not evidence of backend support.
|