NVIDIA Jetson support plan

Issue: https://github.com/elder-plinius/OBLITERATUS/issues/31

Internal testing assessment: .aiwg/testing/jetson-support-assessment.md

Decision

OBLITERATUS should treat Jetson AGX support as a distinct platform target, not as a generic Linux ARM build. Generic linux/aarch64 packaging can prove that the Python package resolves and imports on ARM, but it does not prove CUDA on Jetson because Jetson devices depend on JetPack/L4T-matched CUDA, cuDNN, and PyTorch builds.

The first supported target should be Jetson AGX Orin 64 GB on JetPack 6.x, run through either NVIDIA's Jetson PyTorch instructions or a JetPack-compatible container. Xavier-era JetPack 5 can be documented as best-effort until a runner exists.

Evidence

NVIDIA documents Jetson PyTorch as JetPack-specific pip wheels with GPU acceleration and cuDNN support, installed on top of the matching JetPack version:

The NVIDIA NGC l4t-pytorch container is explicitly a Jetson/JetPack image and requires matching the container tag to the installed JetPack/L4T release. The jetson-containers project provides a practical composition path for PyTorch and Transformers on Jetson, including compatible autotag selection:

https://github.com/dusty-nv/jetson-containers

PyTorch upstream support is moving: PyTorch 2.11 release notes say PyPI wheels now include CUDA 13.0 for Linux aarch64, while CUDA 12.6 and 12.8 wheels remain available from download.pytorch.org. That improves generic ARM CUDA, but it does not remove the need to match Jetson's installed JetPack/CUDA stack:

https://github.com/pytorch/pytorch/releases

bitsandbytes currently publishes Linux aarch64 CUDA builds and lists targets for CUDA 11.8 through 13.x. AGX Orin's Ampere GPU should satisfy the compute capability requirement for 4-bit and 8-bit quantization, but the runtime still has to be validated on the Jetson image actually used by contributors:

https://huggingface.co/docs/bitsandbytes/en/installation

Current repository gaps

pyproject.toml maps Linux and Windows Torch resolution to the CPU-only PyTorch index. That is correct for default CI, but it means the locked default environment will not discover Jetson CUDA.
uv.lock includes a Linux aarch64 bitsandbytes wheel, but Torch is locked as 2.13.0+cpu on Linux. A Jetson runtime must intentionally override Torch after syncing the default lock, similar to the existing CUDA conditional lane.
.github/workflows/conditional-tests.yml has one CUDA lane labeled self-hosted, linux, x64, cuda and installs an official cu130 Torch build. It cannot run on Jetson because the runner labels and Torch source are x64 CUDA-specific.
docs/conditional-testing.md documents generic CUDA and bitsandbytes probes, not Jetson labels, JetPack requirements, or container startup.
Dockerfile is based on python:3.11-slim, installs requirements.txt, and is documented as local-only. It is not a Jetson image and should not be used for a support claim.
obliteratus/device.py can already detect CUDA through PyTorch, so no special Jetson detection is needed for the first milestone. The blocker is installing a CUDA-enabled Jetson PyTorch runtime and proving the existing device and bitsandbytes contracts there.
tests/conditional/test_cuda_runtime.py is a good starting probe, but it does not record Jetson-specific platform evidence such as JetPack/L4T release, CUDA version, Python version, architecture, Torch build, and bitsandbytes binary availability.

Implementation options

Option A: Native Jetson runner, minimal repo changes

Attach a self-hosted runner on the Jetson with labels such as self-hosted, linux, ARM64, jetson, and cuda. Add a new conditional gate jetson-cuda-runtime that:

syncs OBLITERATUS without replacing the system/vendor Jetson Torch build;
verifies platform.machine() is aarch64;
records /etc/nv_tegra_release, nvcc --version, torch.__version__, torch.version.cuda, torch.cuda.get_device_name(0), and bitsandbytes import/runtime status;
runs the existing CUDA and bitsandbytes probes.

Tradeoff: lowest abstraction and fastest to validate real hardware, but it requires maintaining the Jetson's host Python/runtime state carefully.

Option B: Jetson container lane

Base a Jetson-specific image on an NVIDIA L4T PyTorch image or build it with jetson-containers from pytorch plus transformers. Install OBLITERATUS into that image with dependency constraints that do not replace the container's Torch. Run the same jetson-cuda-runtime probe inside the container.

Tradeoff: best reproducibility and closest to the reporter's concern about patched/static libraries, but the image has to track JetPack/L4T tags and may need per-JetPack dependency pins.

Option C: Generic Linux ARM build plus separate Jetson UAT

Add a hosted or self-hosted Linux ARM packaging job that installs OBLITERATUS with CPU Torch on aarch64, then keep Jetson CUDA as manual UAT evidence until a stable runner exists.

Tradeoff: useful packaging signal for ARM contributors, but it must not be presented as Jetson GPU support because it does not exercise CUDA, cuDNN, device placement, quantization kernels, or JetPack compatibility.

Recommended path

Use Option B for the support claim and Option C as a cheap early-warning signal. The container path matches Jetson's dependency model, avoids polluting the default lock with Jetson-only Torch URLs, and gives contributors a repeatable recipe. A native runner can still be used to execute the container and collect evidence. Treat generic ARM64 as Tier 0/Tier 1 preflight only; require real-device Jetson smoke and release validation before publishing a support claim.

Acceptance criteria

docs/conditional-testing.md documents Jetson as a separate conditional gate with supported JetPack versions, runner labels, and local/container commands.
ci/conditional-test-policy.json includes a jetson-cuda-runtime gate with a tracking issue and evidence freshness rule.
.github/workflows/conditional-tests.yml has a Jetson-selected job that runs only on a labeled Jetson runner or through a Jetson-compatible container.
The gate records JetPack/L4T, architecture, Torch CUDA version, CUDA device name, memory, and bitsandbytes status in JSON evidence.
The gate runs without skips: python scripts/run_conditional_gate.py jetson-cuda-runtime.
README support language says Jetson AGX Orin is supported only for the specific JetPack/runtime combinations that have fresh green conditional evidence.

User guidance until support lands

Do not install from the default OBLITERATUS lock and expect Jetson CUDA to work. Start from a JetPack-compatible PyTorch runtime, then install OBLITERATUS without replacing Torch. If using a container, prefer an L4T PyTorch image or a jetson-containers build that matches the device's JetPack/L4T release.

7.0 KiB Raw Blame History