NVIDIA Jetson support plan

Issue: https://github.com/elder-plinius/OBLITERATUS/issues/31

Status

Native Jetson AGX support is not claimed yet. OBLITERATUS should treat Jetson as a dedicated conditional runtime lane, not as part of the default pull-request gate and not as a generic Linux ARM build.

The current OBLITERATUS CUDA path delegates discovery to PyTorch through torch.cuda.is_available(). If a Jetson AGX host reports no CUDA inside OBLITERATUS, the first thing to verify is the JetPack/L4T/PyTorch/container stack, because NVIDIA publishes Jetson-specific PyTorch builds intended for specified JetPack versions.

Decision

Support Jetson through a JetPack-pinned runtime contract:

Keep ordinary PR CI CPU-only, offline, and architecture-neutral.
Add a Jetson conditional gate once a Jetson runner is available.
Prefer an NVIDIA-supported Jetson PyTorch container or NVIDIA Jetson PyTorch wheel for the exact JetPack release under test.
Do not use the existing x64 CUDA gate as Jetson evidence.
Do not treat a generic linux/arm64 build as evidence that CUDA works on Jetson.

Why generic ARM is insufficient

Jetson support couples at least five moving pieces:

Jetson hardware family and compute capability.
JetPack version.
Jetson Linux/L4T version and Ubuntu base image.
CUDA, cuDNN, TensorRT, and related NVIDIA libraries.
PyTorch build or container version.

NVIDIA's Jetson PyTorch documentation says the PyTorch packages are installed on top of a specified JetPack version, and the compatibility table maps PyTorch versions to NVIDIA framework containers/wheels and JetPack versions. A generic ARM build can prove that Python code imports on aarch64; it cannot prove that CUDA, cuDNN, TensorRT, or PyTorch CUDA dispatch works on Jetson.

Initial support matrix

Start with the hardware reported in issue #31: Jetson AGX devices with 64 GB unified memory.

Recommended first support tier:

Tier	Hardware	JetPack	OS / CUDA baseline	Evidence requirement
Target	Jetson AGX Orin 64 GB	6.2	Jetson Linux 36.4.3 / CUDA 12.6	Native Jetson runner or NVIDIA Jetson container on Jetson hardware
Evaluate	Jetson AGX Thor	7.x	Jetson Linux 38/39 / Ubuntu 24.04 / CUDA 13.x family	Separate runner and issue before claiming support
Legacy	Jetson AGX Xavier	5.1.x	Jetson Linux 35.x / Ubuntu 20.04 / CUDA 11.x family	Defer unless a maintainer/user provides hardware and demand

Do not collapse these tiers into one "ARM64" support claim.

Installation shape

The generic local Dockerfile uses python:3.11-slim and is not the Jetson runtime image. A Jetson runtime should use one of these approaches:

Start from an NVIDIA Jetson-compatible PyTorch framework container for the selected JetPack version, then install OBLITERATUS without replacing the container's validated PyTorch stack.
On a flashed Jetson host, install the NVIDIA Jetson PyTorch wheel matching the installed JetPack release, then install OBLITERATUS in a virtual environment without allowing dependency resolution to replace torch.

The lock policy should make the Jetson torch source explicit. The existing Linux PR lock intentionally uses CPU-only PyTorch. A Jetson install path needs an override or separate constraints file that preserves NVIDIA's Jetson PyTorch runtime.

Conditional gate

Add a new gate instead of modifying the x64 CUDA gate:

Gate id: jetson-runtime
Runner labels: self-hosted, linux, ARM64, jetson
Optional labels by tier: orin, jetpack-6 or thor, jetpack-7
Trigger: manual dispatch and release/scheduled validation only
Evidence retention: same 30-day conditional-evidence policy as other hardware gates

The gate should verify:

platform.machine() is aarch64 or equivalent ARM64.
torch.cuda.is_available() is true.
torch.version.cuda is not None.
torch.cuda.get_device_name(0) identifies the Jetson GPU class.
OBLITERATUS resolves device=auto to cuda.
A small CUDA tensor operation completes with finite output.
The existing offloaded-surgery CUDA probe passes.
bitsandbytes NF4/4-bit quantization is either proven on that exact Jetson stack or documented as unsupported for the tier.
A tiny Hugging Face model run passes only when the model-download gate is explicitly selected and the runner has the required account/cache policy.

Acceptance criteria

Jetson support can be claimed for a tier only after all of the following are true:

The supported Jetson module and JetPack version are named in this document.
The install instructions pin the JetPack-compatible PyTorch container or wheel source.
The Jetson conditional gate produces non-skipped green evidence on the exact commit being claimed.
The release notes distinguish generic ARM importability from Jetson CUDA support.
The docs state memory expectations for 64 GB unified memory and recommend small models for validation before large ablation runs.

Operational notes

Jetson's unified memory is shared by the OS, CUDA, model weights, activations, and file cache. Treat "64 GB" as a capacity class, not guaranteed usable model memory. Use small models for smoke tests, then move larger GPU validation to dedicated CUDA hosts such as Titan when those resources are available.

Logging into a Hugging Face account is expected only for gated/private models, license-gated models, or rate-limit avoidance. It should not be required for the offline CPU PR gate or for the Jetson CUDA hardware probe. Any model-download validation must remain an explicit conditional gate.

Sources

REF-JETSON-PYTORCH-INSTALL: NVIDIA, Installing PyTorch for Jetson Platform.
REF-JETSON-PYTORCH-RELEASES: NVIDIA, PyTorch for Jetson Platform release notes.
REF-JETPACK-62: NVIDIA, JetPack 6.2 release notes.
REF-JETPACK-7-DOWNLOADS: NVIDIA, JetPack SDK downloads and notes.
REF-UV-PYTORCH: Astral, Using uv with PyTorch.
REF-BITSANDBYTES-INSTALL: Hugging Face, bitsandbytes installation guide.

6.0 KiB Raw Blame History