mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-08-30 06:30:37 +02:00
166 lines
8.6 KiB
Markdown
166 lines
8.6 KiB
Markdown
# Conditional test operations
|
|
|
|
The mandatory pull-request workflow remains CPU-only, offline, and credential-free.
|
|
Environment-bound contracts run through `Conditional tests` on manual dispatch,
|
|
weekly at 06:17 UTC Sunday, and for published releases. OBLITERATUS maintainers own
|
|
the workflow, policy, runner labels, credentials, and evidence review.
|
|
|
|
The canonical machine-readable policy is `ci/conditional-test-policy.json`. It maps
|
|
every environment-bound mature-coverage exclusion to a runnable gate, defines the
|
|
pinned tiny model, records prerequisites and expected cost, retains evidence for 30
|
|
days, and treats evidence older than eight days as stale. Every workflow run publishes
|
|
a summary showing selected, successful, failed, and not-selected/no-fresh-evidence
|
|
gates.
|
|
|
|
Unavailable hardware or credential-bound gates may use a committed environment
|
|
waiver for at most 30 days. Each waiver names one gate, a canonical tracking issue,
|
|
the reason, its opening and expiry dates, and the support claim it blocks. Invalid,
|
|
duplicate, future-dated, overlong, or expired waivers fail the policy job. The current
|
|
waivers are tracked by [issue #110](https://github.com/elder-plinius/OBLITERATUS/issues/110)
|
|
and expire on 2026-09-15; they are not evidence that any waived backend works.
|
|
They block support, compatibility, correctness, and performance claims for the
|
|
waived environment.
|
|
|
|
The current accelerator waivers cover missing GitHub runner registration, not
|
|
missing hardware. Titan has an RTX 4090 behind Gitea runners, and Mutsu is a 16 GB
|
|
Apple M4 builder. Exact-head operator probes may be attached to issue #110, but
|
|
they do not make the scheduled GitHub lanes runnable or establish a broad backend
|
|
support claim.
|
|
|
|
## Hosted gates
|
|
|
|
The model gate downloads only
|
|
`hf-internal-testing/tiny-random-gpt2@71034c5d8bde858ff824298bdedc65515b97d2b9`
|
|
with `trust_remote_code=False`. Its cache key includes the model revision, Python
|
|
version, and runner OS. The gate performs a forward pass, reopens the cache with Hub
|
|
and Transformers offline modes enabled, asserts an uncached model fails offline, and
|
|
runs the evaluator on two tiny samples. Its timeout is 25 minutes and its expected
|
|
download is below 100 MB.
|
|
|
|
The network-service gate uses a disposable loopback HTTP server and no credentials.
|
|
The operator-UI gate installs the locked `spaces` extra and constructs the Gradio
|
|
application without launching a listener. These gates cost less than ten hosted
|
|
runner-minutes each under normal conditions.
|
|
|
|
Run the same probes locally with:
|
|
|
|
```bash
|
|
uv sync --locked --extra dev
|
|
uv run --extra dev python scripts/run_conditional_gate.py model-download-runtime
|
|
uv run --extra dev python scripts/run_conditional_gate.py external-evaluation
|
|
uv run --extra dev python scripts/run_conditional_gate.py network-services
|
|
|
|
uv sync --locked --all-extras
|
|
uv run --all-extras python scripts/run_conditional_gate.py operator-ui
|
|
```
|
|
|
|
## CUDA and bitsandbytes
|
|
|
|
Attach a dedicated runner with the labels `self-hosted`, `linux`, `x64`, and `cuda`.
|
|
Set the repository variable `ENABLE_CUDA_GATE=true` for scheduled/release evidence,
|
|
or select CUDA during manual dispatch. The gate verifies CUDA discovery, automatic
|
|
selection, dtype selection, tensor placement, matrix multiplication, bitsandbytes
|
|
availability, NF4 quantization, dequantization, shape, placement, and finite output.
|
|
The expected cost is below 20 self-hosted runner-minutes. Mandatory Linux CI
|
|
deliberately locks CPU-only Torch. The selected CUDA job reads that exact locked
|
|
base version, replaces only Torch with the same-version official `cu130` build,
|
|
asserts a CUDA build was installed, and runs `uv pip check` before executing the
|
|
probes.
|
|
|
|
For an operator run on the labeled machine:
|
|
|
|
```bash
|
|
uv sync --locked --extra dev --extra quantization
|
|
CUDA_TORCH_VERSION="$(.venv/bin/python -c \
|
|
'import torch; print(torch.__version__.split("+", 1)[0])')"
|
|
UV_TORCH_BACKEND=cu130 uv pip install --python .venv/bin/python \
|
|
--reinstall-package torch "torch==$CUDA_TORCH_VERSION"
|
|
uv pip check --python .venv/bin/python
|
|
.venv/bin/python scripts/run_conditional_gate.py cuda-runtime
|
|
.venv/bin/python scripts/run_conditional_gate.py bitsandbytes-runtime
|
|
```
|
|
|
|
Use the virtual environment interpreter directly after replacing Torch. A subsequent
|
|
`uv run` or `uv sync` without the CUDA override may restore the portable CPU wheel
|
|
from the lock. The agentic developer installer automates this ordering and performs
|
|
a real CUDA tensor probe; see `installer/setup.dev.manifest.yaml`.
|
|
|
|
## Quality-verifier coherence semantics
|
|
|
|
The built-in VERIFY stage does not treat length and vocabulary diversity as
|
|
coherence. Each fixed factual prompt has deterministic semantic anchors, and the
|
|
scorer rejects strong repetition and recognizable corpus contamination such as
|
|
scraped Q&A profile/answer fragments. This is intentionally local and deterministic:
|
|
release verification does not call another model or a network judge. Perplexity and
|
|
extended capability checks remain separate measurements.
|
|
|
|
On Linux workstations that also install cuDNN system-wide, the dynamic loader can
|
|
mix a host sublibrary with PyTorch's bundled cuDNN and raise
|
|
`CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH`. The local launcher sets
|
|
`OBLITERATUS_DISABLE_CUDNN=1`, keeping causal convolutions and attention on
|
|
PyTorch's other CUDA kernels rather than moving the model to the CPU. Installer
|
|
verification runs actual causal `conv1d` and SDPA operations, not only a generic
|
|
CUDA allocation, so this failure is caught before a model pipeline begins.
|
|
|
|
Jetson CUDA support is tracked separately from this generic x64 CUDA lane. The
|
|
mandatory `Linux ARM64 preflight` uses GitHub's hosted `ubuntu-24.04-arm` runner
|
|
to prove locked CPU packaging, imports, CLI startup, and Jetson tooling contracts.
|
|
It is not GPU evidence. Jetson depends on a JetPack/L4T-matched CUDA, cuDNN, and
|
|
PyTorch runtime.
|
|
|
|
Physical testing uses only a trusted manual dispatch on the labels `self-hosted`,
|
|
`linux`, `ARM64`, and `jetson`. The job preserves NVIDIA's vendor PyTorch, runs
|
|
`jetson-runtime`, and uploads the sanitized `conditional-jetson-<run-attempt>`
|
|
artifact for 30 days. It never runs for a pull request, schedule, or release.
|
|
Contributor bootstrap, runner isolation, reporting commands, and acceptance
|
|
criteria are documented in the [NVIDIA Jetson support plan](platforms/jetson.md).
|
|
|
|
## Apple MPS and MLX
|
|
|
|
MPS uses a runner labeled `self-hosted`, `macOS`, `ARM64`, and `mps`; enable its
|
|
schedule with `ENABLE_MPS_GATE=true`. MLX uses the same first three labels plus `mlx`
|
|
and `ENABLE_MLX_GATE=true`. The MPS probe checks discovery, selection, dtype and
|
|
float64 fallback, placement, and a real matrix operation. The MLX probe uses the
|
|
locked `mlx==0.32.0` and `mlx-lm==0.31.3` packages and verifies imports, array
|
|
placement, evaluation, and matrix multiplication. Each should cost less than 20
|
|
self-hosted runner-minutes.
|
|
|
|
If the repository has no attached Apple runner, collect equivalent operator evidence
|
|
on Apple Silicon and attach the JSON and JUnit files to the tracking issue:
|
|
|
|
```bash
|
|
uv sync --locked --extra dev
|
|
uv run --extra dev python scripts/run_conditional_gate.py mps-runtime
|
|
|
|
uv sync --locked --extra dev --group mlx
|
|
uv run --extra dev --group mlx python scripts/run_conditional_gate.py mlx-runtime
|
|
```
|
|
|
|
## Remote provider
|
|
|
|
Remote evidence is opt-in. Configure a non-root, command-limited test account and:
|
|
|
|
- variables `OBLITERATUS_REMOTE_HOST`, `OBLITERATUS_REMOTE_USER`, and optionally
|
|
`OBLITERATUS_REMOTE_PORT`;
|
|
- secrets `OBLITERATUS_REMOTE_KEY` and `OBLITERATUS_REMOTE_KNOWN_HOSTS`.
|
|
|
|
The known-hosts entry must be pinned after verifying the provider fingerprint through
|
|
an independent channel. The workflow writes credentials to mode-0600 temporary files,
|
|
never prints their contents, uses batch mode and strict host-key checking, and only
|
|
runs `echo ok` plus `python3 -c 'print(6 * 7)'`. Missing prerequisites produce an
|
|
actionable failure when the gate was selected; a direct local invocation may use
|
|
`--allow-missing` to record explicit `not_run` evidence. Expected cost is below five
|
|
hosted runner-minutes plus any provider charge.
|
|
|
|
## Result semantics
|
|
|
|
`scripts/run_conditional_gate.py` requires at least one executed test and rejects any
|
|
failure, error, or skip. A selected workflow job therefore cannot become green through
|
|
an availability skip or unconditional success conversion. The final summary also
|
|
fails if any selected job is not successful. Unselected jobs are explicitly reported
|
|
as `not_selected_no_fresh_evidence`; they are not evidence of backend support. An
|
|
unselected hardware or credential-bound gate with a valid waiver is reported as
|
|
`waived_no_support_claim`, including its tracker, expiry, and blocked claim. Selecting
|
|
and successfully running that gate produces `success` instead of relying on the
|
|
waiver.
|