docs: specify Jetson support path

2026-08-29 22:20:36 +02:00 · 2026-08-21 19:35:21 -04:00
parent b340cc025e
commit 8814dbe359
15 changed files with 352 additions and 1 deletions
@@ -0,0 +1,71 @@
+# Jetson support architecture note
+
+Date: 2026-08-21
+Issue: https://github.com/elder-plinius/OBLITERATUS/issues/31
+
+## Summary
+
+OBLITERATUS should support NVIDIA Jetson AGX as a JetPack-specific conditional
+runtime target. A generic ARM build is useful as import/build coverage, but it
+is not sufficient for Jetson CUDA support because the working runtime depends on
+JetPack, L4T, CUDA, cuDNN, TensorRT, and NVIDIA's Jetson-compatible PyTorch
+build or container.
+
+## Current repo state
+
+- `obliteratus/device.py` uses `torch.cuda.is_available()` for CUDA discovery.
+- `docs/conditional-testing.md` defines hardware gates outside the mandatory
+  PR workflow.
+- `.github/workflows/conditional-tests.yml` has an x64 CUDA gate using
+  `UV_TORCH_BACKEND=cu130`.
+- `pyproject.toml` locks Linux PR resolution to CPU-only PyTorch through the
+  `pytorch-cpu` index.
+- `Dockerfile` is a generic local `python:3.11-slim` image, not a Jetson image.
+
+These choices are correct for contributor PR turnaround, but they do not create
+a Jetson-compatible runtime.
+
+## Required changes before claiming support
+
+1. Add a `jetson-runtime` conditional gate and policy entry.
+2. Add a Jetson install path that preserves NVIDIA's Jetson PyTorch stack.
+3. Add a Jetson container recipe or documented base-image override for the
+   selected JetPack tier.
+4. Add a tiny hardware probe that records JetPack/L4T, CUDA, PyTorch, device
+   name, and OBLITERATUS `device=auto` behavior.
+5. Decide whether `bitsandbytes` is supported, tier-limited, or disabled on the
+   selected Jetson stack based on hardware evidence.
+
+## Design constraints
+
+- Keep PR CI CPU-only and offline.
+- Keep x64 CUDA and Jetson CUDA evidence separate.
+- Do not let `uv sync` replace NVIDIA's Jetson PyTorch wheel/container runtime
+  with the generic CPU lock.
+- Treat JetPack 6.x Orin and JetPack 7.x Thor/Orin as separate evidence tiers.
+- Avoid publishing support claims without non-skipped conditional evidence.
+
+## Recommended first implementation
+
+Implement Jetson AGX Orin 64 GB on JetPack 6.2 first. Add a self-hosted runner
+with labels `self-hosted`, `linux`, `ARM64`, `jetson`, `orin`, and
+`jetpack-6`. The first gate should run only CUDA discovery, a small matrix
+operation, OBLITERATUS device selection, and the existing offloaded-surgery
+probe. Add `bitsandbytes` only after the same runner proves NF4 quantization
+works with the selected PyTorch/CUDA stack.
+
+## Evidence basis
+
+- `REF-JETSON-PYTORCH-INSTALL` records NVIDIA's Jetson-specific PyTorch install
+  path and states the packages are intended for specified JetPack versions.
+- `REF-JETSON-PYTORCH-RELEASES` maps PyTorch releases to NVIDIA framework
+  containers/wheels and JetPack versions.
+- `REF-JETPACK-62` records JetPack 6.2 as Jetson Linux 36.4.3 and CUDA
+  12.6.
+- `REF-JETPACK-7-DOWNLOADS` records the current JetPack 7.2.1 stack as Jetson
+  Linux 39.2.1, Ubuntu 24.04, and CUDA 13.2.1.
+- `REF-UV-PYTORCH` explains that PyTorch uses separate accelerator indexes and
+  local-version builds such as CPU and CUDA variants.
+- `REF-BITSANDBYTES-INSTALL` documents Linux `aarch64` CUDA support targets,
+  but OBLITERATUS should still require project-specific Jetson evidence before
+  claiming quantization support.
@@ -0,0 +1,13 @@
+ref_id: REF-BITSANDBYTES-INSTALL
+grade: LOW
+baseline: LOW
+source_type: library_documentation
+upgrades: []
+downgrades:
+  - Library support matrix is not Jetson-specific OBLITERATUS runtime evidence.
+allowed_language:
+  - "bitsandbytes documents Linux aarch64 CUDA support..."
+  - "Quantization should still be proven on the selected Jetson stack..."
+forbidden_language:
+  - "bitsandbytes is confirmed for OBLITERATUS on Jetson..."
+  - "Jetson quantization support can be claimed without hardware evidence..."
@@ -0,0 +1,14 @@
+ref_id: REF-JETPACK-62
+grade: MODERATE
+baseline: LOW
+source_type: vendor_documentation
+upgrades:
+  - Primary NVIDIA release notes for the recommended JetPack 6.2 target tier.
+downgrades:
+  - Release notes establish platform stack contents, not OBLITERATUS runtime support.
+allowed_language:
+  - "NVIDIA lists JetPack 6.2 as..."
+  - "JetPack 6.2 is an appropriate explicit platform tier..."
+forbidden_language:
+  - "JetPack 6.2 support is certified..."
+  - "Generic ARM evidence proves this stack..."
@@ -0,0 +1,13 @@
+ref_id: REF-JETPACK-7-DOWNLOADS
+grade: MODERATE
+baseline: LOW
+source_type: vendor_documentation
+upgrades:
+  - Current NVIDIA release/download page for JetPack 7 stack contents.
+downgrades:
+  - Download page is not project runtime evidence.
+allowed_language:
+  - "NVIDIA currently lists..."
+  - "JetPack 7 should be a separate evidence tier..."
+forbidden_language:
+  - "JetPack 7 works with OBLITERATUS..."
@@ -0,0 +1,14 @@
+ref_id: REF-JETSON-PYTORCH-INSTALL
+grade: MODERATE
+baseline: LOW
+source_type: vendor_documentation
+upgrades:
+  - Primary vendor documentation for Jetson PyTorch installation.
+downgrades:
+  - Vendor documentation can change without project-controlled reproducibility.
+allowed_language:
+  - "NVIDIA documents..."
+  - "NVIDIA's install guide states..."
+forbidden_language:
+  - "OBLITERATUS supports Jetson..."
+  - "This guarantees compatibility..."
@@ -0,0 +1,14 @@
+ref_id: REF-JETSON-PYTORCH-RELEASES
+grade: MODERATE
+baseline: LOW
+source_type: vendor_documentation
+upgrades:
+  - Primary vendor compatibility table for Jetson PyTorch releases.
+downgrades:
+  - Release tables are necessary but not sufficient runtime evidence.
+allowed_language:
+  - "NVIDIA maps..."
+  - "The compatibility table lists..."
+forbidden_language:
+  - "All listed combinations work for OBLITERATUS..."
+  - "No hardware testing is needed..."
@@ -0,0 +1,13 @@
+ref_id: REF-UV-PYTORCH
+grade: MODERATE
+baseline: LOW
+source_type: tool_documentation
+upgrades:
+  - Primary tool documentation for uv PyTorch resolution behavior.
+downgrades:
+  - Packaging behavior still needs validation in this repository's lock model.
+allowed_language:
+  - "uv documents..."
+  - "PyTorch accelerator variants need explicit resolver handling..."
+forbidden_language:
+  - "uv automatically solves Jetson packaging..."
@@ -0,0 +1,10 @@
+id: REF-BITSANDBYTES-INSTALL
+title: bitsandbytes Installation Guide
+source_type: library_documentation
+publisher: Hugging Face
+url: https://huggingface.co/docs/bitsandbytes/installation
+accessed_at: "2026-08-21T23:31:01Z"
+relevant_claims:
+  - bitsandbytes supports NVIDIA CUDA GPUs with compute capability 6.0 or newer.
+  - Linux aarch64 CUDA builds are documented for CUDA Toolkit 11.8 through 13.2 with specific SM targets.
+  - LLM.int8 requires Turing-class or newer hardware, while NF4/FP4 quantization requires Pascal-class or newer hardware.
@@ -0,0 +1,10 @@
+id: REF-JETPACK-62
+title: JetPack 6.2 Release Notes
+source_type: vendor_documentation
+publisher: NVIDIA
+url: https://docs.nvidia.com/jetson/archives/jetpack-archived/jetpack-62/release-notes/index.html
+accessed_at: "2026-08-21T23:31:01Z"
+relevant_claims:
+  - JetPack 6.2 includes Jetson Linux 36.4.3.
+  - JetPack 6.2 includes a compute stack with CUDA 12.6, TensorRT 10.3, cuDNN 9.3, VPI 3.2, DLA 3.1, and DLFW 24.0.
+  - JetPack 6.2 targets Jetson Orin modules and includes updated Super Mode behavior for Orin Nano and Orin NX modules.
@@ -0,0 +1,10 @@
+id: REF-JETPACK-7-DOWNLOADS
+title: NVIDIA JetPack SDK Downloads and Notes
+source_type: vendor_documentation
+publisher: NVIDIA
+url: https://developer.nvidia.com/embedded/jetpack/downloads
+accessed_at: "2026-08-21T23:31:01Z"
+relevant_claims:
+  - The current JetPack 7.2.1 release is paired with Jetson Linux 39.2.1.
+  - JetPack 7.2.1 lists CUDA 13.2.1, TensorRT 10.16.2, and cuDNN 9.20.0.
+  - JetPack 7 uses an Ubuntu 24.04 L4T base and aligns Jetson software with SBSA.
@@ -0,0 +1,11 @@
+id: REF-JETSON-PYTORCH-INSTALL
+title: Installing PyTorch for Jetson Platform
+source_type: vendor_documentation
+publisher: NVIDIA
+url: https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform/index.html
+accessed_at: "2026-08-21T23:31:01Z"
+relevant_claims:
+  - NVIDIA provides Jetson PyTorch pip wheels with GPU acceleration and cuDNN support.
+  - The packages are intended to be installed on top of a specified JetPack version.
+  - Installation prerequisites include JetPack on the Jetson device and system packages.
+  - PyTorch installation verification starts by importing torch on the Jetson platform.
@@ -0,0 +1,11 @@
+id: REF-JETSON-PYTORCH-RELEASES
+title: PyTorch for Jetson Platform Release Notes
+source_type: vendor_documentation
+publisher: NVIDIA
+url: https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform-release-notes/pytorch-jetson-rel.html
+accessed_at: "2026-08-21T23:31:01Z"
+relevant_claims:
+  - NVIDIA's compatibility table maps PyTorch versions to NVIDIA framework containers or wheels and JetPack versions.
+  - JetPack 6.2 entries map to NVIDIA framework containers 25.02 through 25.06.
+  - JetPack 7.x entries map to NVIDIA framework containers 25.08 and later.
+  - NVIDIA notes that standalone iGPU containers are no longer produced starting with the 26.03 release.
@@ -0,0 +1,11 @@
+id: REF-UV-PYTORCH
+title: Using uv with PyTorch
+source_type: tool_documentation
+publisher: Astral
+url: https://docs.astral.sh/uv/guides/integration/pytorch/
+accessed_at: "2026-08-21T23:31:01Z"
+relevant_claims:
+  - uv can manage PyTorch dependencies while controlling accelerator selection.
+  - PyTorch wheels use dedicated indexes outside PyPI for many builds.
+  - PyTorch encodes accelerator builds in local version specifiers such as +cpu and +cu130.
+  - Different PyTorch accelerator builds are published on different indexes.
@@ -84,7 +84,7 @@ Jetson CUDA support is tracked separately from this generic x64 CUDA lane. A
 generic Linux ARM build can prove package portability, but it does not prove
 Jetson GPU support because Jetson depends on a JetPack/L4T-matched CUDA, cuDNN,
 and PyTorch runtime. The support plan, recommended container path, and acceptance
-criteria are documented in [NVIDIA Jetson support plan](jetson-support.md).
+criteria are documented in [NVIDIA Jetson support plan](platforms/jetson.md).

 ## Apple MPS and MLX

@@ -0,0 +1,136 @@
+# NVIDIA Jetson support plan
+
+Issue: https://github.com/elder-plinius/OBLITERATUS/issues/31
+
+## Status
+
+Native Jetson AGX support is not claimed yet. OBLITERATUS should treat Jetson
+as a dedicated conditional runtime lane, not as part of the default pull-request
+gate and not as a generic Linux ARM build.
+
+The current OBLITERATUS CUDA path delegates discovery to PyTorch through
+`torch.cuda.is_available()`. If a Jetson AGX host reports no CUDA inside
+OBLITERATUS, the first thing to verify is the JetPack/L4T/PyTorch/container
+stack, because NVIDIA publishes Jetson-specific PyTorch builds intended for
+specified JetPack versions.
+
+## Decision
+
+Support Jetson through a JetPack-pinned runtime contract:
+
+- Keep ordinary PR CI CPU-only, offline, and architecture-neutral.
+- Add a Jetson conditional gate once a Jetson runner is available.
+- Prefer an NVIDIA-supported Jetson PyTorch container or NVIDIA Jetson PyTorch
+  wheel for the exact JetPack release under test.
+- Do not use the existing x64 CUDA gate as Jetson evidence.
+- Do not treat a generic `linux/arm64` build as evidence that CUDA works on
+  Jetson.
+
+## Why generic ARM is insufficient
+
+Jetson support couples at least five moving pieces:
+
+- Jetson hardware family and compute capability.
+- JetPack version.
+- Jetson Linux/L4T version and Ubuntu base image.
+- CUDA, cuDNN, TensorRT, and related NVIDIA libraries.
+- PyTorch build or container version.
+
+NVIDIA's Jetson PyTorch documentation says the PyTorch packages are installed
+on top of a specified JetPack version, and the compatibility table maps PyTorch
+versions to NVIDIA framework containers/wheels and JetPack versions. A generic
+ARM build can prove that Python code imports on `aarch64`; it cannot prove that
+CUDA, cuDNN, TensorRT, or PyTorch CUDA dispatch works on Jetson.
+
+## Initial support matrix
+
+Start with the hardware reported in issue #31: Jetson AGX devices with 64 GB
+unified memory.
+
+Recommended first support tier:
+
+| Tier | Hardware | JetPack | OS / CUDA baseline | Evidence requirement |
+| --- | --- | --- | --- | --- |
+| Target | Jetson AGX Orin 64 GB | 6.2 | Jetson Linux 36.4.3 / CUDA 12.6 | Native Jetson runner or NVIDIA Jetson container on Jetson hardware |
+| Evaluate | Jetson AGX Thor | 7.x | Jetson Linux 38/39 / Ubuntu 24.04 / CUDA 13.x family | Separate runner and issue before claiming support |
+| Legacy | Jetson AGX Xavier | 5.1.x | Jetson Linux 35.x / Ubuntu 20.04 / CUDA 11.x family | Defer unless a maintainer/user provides hardware and demand |
+
+Do not collapse these tiers into one "ARM64" support claim.
+
+## Installation shape
+
+The generic local Dockerfile uses `python:3.11-slim` and is not the Jetson
+runtime image. A Jetson runtime should use one of these approaches:
+
+1. Start from an NVIDIA Jetson-compatible PyTorch framework container for the
+   selected JetPack version, then install OBLITERATUS without replacing the
+   container's validated PyTorch stack.
+2. On a flashed Jetson host, install the NVIDIA Jetson PyTorch wheel matching
+   the installed JetPack release, then install OBLITERATUS in a virtual
+   environment without allowing dependency resolution to replace `torch`.
+
+The lock policy should make the Jetson torch source explicit. The existing
+Linux PR lock intentionally uses CPU-only PyTorch. A Jetson install path needs
+an override or separate constraints file that preserves NVIDIA's Jetson PyTorch
+runtime.
+
+## Conditional gate
+
+Add a new gate instead of modifying the x64 CUDA gate:
+
+- Gate id: `jetson-runtime`
+- Runner labels: `self-hosted`, `linux`, `ARM64`, `jetson`
+- Optional labels by tier: `orin`, `jetpack-6` or `thor`, `jetpack-7`
+- Trigger: manual dispatch and release/scheduled validation only
+- Evidence retention: same 30-day conditional-evidence policy as other hardware
+  gates
+
+The gate should verify:
+
+- `platform.machine()` is `aarch64` or equivalent ARM64.
+- `torch.cuda.is_available()` is true.
+- `torch.version.cuda` is not `None`.
+- `torch.cuda.get_device_name(0)` identifies the Jetson GPU class.
+- OBLITERATUS resolves `device=auto` to `cuda`.
+- A small CUDA tensor operation completes with finite output.
+- The existing offloaded-surgery CUDA probe passes.
+- `bitsandbytes` NF4/4-bit quantization is either proven on that exact Jetson
+  stack or documented as unsupported for the tier.
+- A tiny Hugging Face model run passes only when the model-download gate is
+  explicitly selected and the runner has the required account/cache policy.
+
+## Acceptance criteria
+
+Jetson support can be claimed for a tier only after all of the following are
+true:
+
+- The supported Jetson module and JetPack version are named in this document.
+- The install instructions pin the JetPack-compatible PyTorch container or
+  wheel source.
+- The Jetson conditional gate produces non-skipped green evidence on the exact
+  commit being claimed.
+- The release notes distinguish generic ARM importability from Jetson CUDA
+  support.
+- The docs state memory expectations for 64 GB unified memory and recommend
+  small models for validation before large ablation runs.
+
+## Operational notes
+
+Jetson's unified memory is shared by the OS, CUDA, model weights, activations,
+and file cache. Treat "64 GB" as a capacity class, not guaranteed usable model
+memory. Use small models for smoke tests, then move larger GPU validation to
+dedicated CUDA hosts such as Titan when those resources are available.
+
+Logging into a Hugging Face account is expected only for gated/private models,
+license-gated models, or rate-limit avoidance. It should not be required for the
+offline CPU PR gate or for the Jetson CUDA hardware probe. Any model-download
+validation must remain an explicit conditional gate.
+
+## Sources
+
+- `REF-JETSON-PYTORCH-INSTALL`: NVIDIA, Installing PyTorch for Jetson Platform.
+- `REF-JETSON-PYTORCH-RELEASES`: NVIDIA, PyTorch for Jetson Platform release notes.
+- `REF-JETPACK-62`: NVIDIA, JetPack 6.2 release notes.
+- `REF-JETPACK-7-DOWNLOADS`: NVIDIA, JetPack SDK downloads and notes.
+- `REF-UV-PYTORCH`: Astral, Using uv with PyTorch.
+- `REF-BITSANDBYTES-INSTALL`: Hugging Face, bitsandbytes installation guide.