mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-08-17 16:37:30 +02:00
ci: add conditional environment test gates
This commit is contained in:
Joseph Magly
@@ -0,0 +1,102 @@
|
||||
# Conditional test operations
|
||||
|
||||
The mandatory pull-request workflow remains CPU-only, offline, and credential-free.
|
||||
Environment-bound contracts run through `Conditional tests` on manual dispatch,
|
||||
weekly at 06:17 UTC Sunday, and for published releases. OBLITERATUS maintainers own
|
||||
the workflow, policy, runner labels, credentials, and evidence review.
|
||||
|
||||
The canonical machine-readable policy is `ci/conditional-test-policy.json`. It maps
|
||||
every environment-bound mature-coverage exclusion to a runnable gate, defines the
|
||||
pinned tiny model, records prerequisites and expected cost, retains evidence for 30
|
||||
days, and treats evidence older than eight days as stale. Every workflow run publishes
|
||||
a summary showing selected, successful, failed, and not-selected/no-fresh-evidence
|
||||
gates.
|
||||
|
||||
## Hosted gates
|
||||
|
||||
The model gate downloads only
|
||||
`hf-internal-testing/tiny-random-gpt2@71034c5d8bde858ff824298bdedc65515b97d2b9`
|
||||
with `trust_remote_code=False`. Its cache key includes the model revision, Python
|
||||
version, and runner OS. The gate performs a forward pass, reopens the cache with Hub
|
||||
and Transformers offline modes enabled, asserts an uncached model fails offline, and
|
||||
runs the evaluator on two tiny samples. Its timeout is 25 minutes and its expected
|
||||
download is below 100 MB.
|
||||
|
||||
The network-service gate uses a disposable loopback HTTP server and no credentials.
|
||||
The operator-UI gate installs the locked `spaces` extra and constructs the Gradio
|
||||
application without launching a listener. These gates cost less than ten hosted
|
||||
runner-minutes each under normal conditions.
|
||||
|
||||
Run the same probes locally with:
|
||||
|
||||
```bash
|
||||
uv sync --locked --extra dev
|
||||
uv run --extra dev python scripts/run_conditional_gate.py model-download-runtime
|
||||
uv run --extra dev python scripts/run_conditional_gate.py external-evaluation
|
||||
uv run --extra dev python scripts/run_conditional_gate.py network-services
|
||||
|
||||
uv sync --locked --all-extras
|
||||
uv run --all-extras python scripts/run_conditional_gate.py operator-ui
|
||||
```
|
||||
|
||||
## CUDA and bitsandbytes
|
||||
|
||||
Attach a dedicated runner with the labels `self-hosted`, `linux`, `x64`, and `cuda`.
|
||||
Set the repository variable `ENABLE_CUDA_GATE=true` for scheduled/release evidence,
|
||||
or select CUDA during manual dispatch. The gate verifies CUDA discovery, automatic
|
||||
selection, dtype selection, tensor placement, matrix multiplication, bitsandbytes
|
||||
availability, NF4 quantization, dequantization, shape, placement, and finite output.
|
||||
The expected cost is below 20 self-hosted runner-minutes.
|
||||
|
||||
For an operator run on the labeled machine:
|
||||
|
||||
```bash
|
||||
uv sync --locked --extra dev
|
||||
uv run --extra dev python scripts/run_conditional_gate.py cuda-runtime
|
||||
uv run --extra dev python scripts/run_conditional_gate.py bitsandbytes-runtime
|
||||
```
|
||||
|
||||
## Apple MPS and MLX
|
||||
|
||||
MPS uses a runner labeled `self-hosted`, `macOS`, `ARM64`, and `mps`; enable its
|
||||
schedule with `ENABLE_MPS_GATE=true`. MLX uses the same first three labels plus `mlx`
|
||||
and `ENABLE_MLX_GATE=true`. The MPS probe checks discovery, selection, dtype and
|
||||
float64 fallback, placement, and a real matrix operation. The MLX probe uses the
|
||||
locked `mlx==0.32.0` and `mlx-lm==0.31.3` packages and verifies imports, array
|
||||
placement, evaluation, and matrix multiplication. Each should cost less than 20
|
||||
self-hosted runner-minutes.
|
||||
|
||||
If the repository has no attached Apple runner, collect equivalent operator evidence
|
||||
on Apple Silicon and attach the JSON and JUnit files to the tracking issue:
|
||||
|
||||
```bash
|
||||
uv sync --locked --extra dev
|
||||
uv run --extra dev python scripts/run_conditional_gate.py mps-runtime
|
||||
|
||||
uv sync --locked --extra dev --group mlx
|
||||
uv run --extra dev --group mlx python scripts/run_conditional_gate.py mlx-runtime
|
||||
```
|
||||
|
||||
## Remote provider
|
||||
|
||||
Remote evidence is opt-in. Configure a non-root, command-limited test account and:
|
||||
|
||||
- variables `OBLITERATUS_REMOTE_HOST`, `OBLITERATUS_REMOTE_USER`, and optionally
|
||||
`OBLITERATUS_REMOTE_PORT`;
|
||||
- secrets `OBLITERATUS_REMOTE_KEY` and `OBLITERATUS_REMOTE_KNOWN_HOSTS`.
|
||||
|
||||
The known-hosts entry must be pinned after verifying the provider fingerprint through
|
||||
an independent channel. The workflow writes credentials to mode-0600 temporary files,
|
||||
never prints their contents, uses batch mode and strict host-key checking, and only
|
||||
runs `echo ok` plus `python3 -c 'print(6 * 7)'`. Missing prerequisites produce an
|
||||
actionable failure when the gate was selected; a direct local invocation may use
|
||||
`--allow-missing` to record explicit `not_run` evidence. Expected cost is below five
|
||||
hosted runner-minutes plus any provider charge.
|
||||
|
||||
## Result semantics
|
||||
|
||||
`scripts/run_conditional_gate.py` requires at least one executed test and rejects any
|
||||
failure, error, or skip. A selected workflow job therefore cannot become green through
|
||||
an availability skip or unconditional success conversion. The final summary also
|
||||
fails if any selected job is not successful. Unselected jobs are explicitly reported
|
||||
as `not_selected_no_fresh_evidence`; they are not evidence of backend support.
|
||||
Reference in New Issue
Block a user