fix: harden quantized checkpoint integration

This commit is contained in:
Joseph Magly
2026-08-16 12:57:17 -04:00
parent b6a1727cb6
commit b25b3f3b66
13 changed files with 1379 additions and 228 deletions
+23 -12
View File
@@ -1821,10 +1821,13 @@ protocol; anything here that surprised us is marked **(finding)**.
detection must accept both `_scale`/`_scale_2` suffixes and the dotted
`.weight_scale` family **(finding: first state-dict test run failed on
exactly this)**.
4. **compressed-tensors stores reciprocal scales.** Dequant divides rather
than multiplies. Round-tripping reciprocals through FP8-E4M3 storage is
measurably coarser (cosine ~0.986 vs ~0.999 for direct storage on random
weights) — expected, not a bug.
4. **compressed-tensors inverts only the global NVFP4 scale.** Its documented
converter maps ModelOpt `weight` to `weight_packed`, leaves `weight_scale`
unchanged, and maps `weight_scale_2` to the reciprocal
`weight_global_scale`. Dequantization therefore multiplies by the FP8 block
scale and divides by the global scale. Treating both scales as reciprocal
produced a self-consistent synthetic test but did not match the published
checkpoint contract **(maintainer audit finding)**.
5. **torch 2.13.0+cpu**: `torch.float4_e2m1fn_x2` exists but the runtime
upcast probe fails on this CPU build, so the manual nibble-LUT path is
what actually runs there; the probe design (rather than a version check)
@@ -1833,9 +1836,11 @@ protocol; anything here that surprised us is marked **(finding)**.
so the FP8/NVFP4 loud-failure guard in `_dequantize_weight` must run
*before* the bitsandbytes branch or the error message is misleading
**(finding: unit test caught the wrong branch firing)**.
7. NVFP4 round-trip error on random Gaussian weights: relative Frobenius
error ~0.10 direct / ~0.18 reciprocal, cosine >0.98 — this is the
intrinsic precision of the format, not implementation error.
7. NVFP4 round-trip error on the deterministic random-Gaussian fixtures is
below 0.15 relative Frobenius error with cosine above 0.995 for both
ModelOpt and compressed-tensors global-scale storage. Seeds, construction,
and thresholds are recorded in
`tests/fixtures/quant_dequant_provenance.json`.
8. **Real checkpoints are mixed-precision.** `nvidia/Nemotron-3-Nano-Omni-
30B-A3B-Reasoning-NVFP4` ships `quant_method: modelopt` with
`quant_algo: MIXED_PRECISION`: FP8 (8-bit float, per-tensor
@@ -1857,7 +1862,12 @@ Smoke-test results on real checkpoints: see Appendix E continuation below
---
## Appendix E2: Nemotron-3-Nano-Omni smoke runs (2026-07-24, 1x A100-80GB)
## Appendix E2: Contributor-reported Nemotron-3-Nano-Omni smoke runs (2026-07-24)
The following A100 results were supplied with the contribution. They do not
include immutable checkpoint revisions or retained machine-verifiable logs and
were not independently reproduced during maintainer integration. They are
historical engineering notes, not release evidence or a support claim.
Real-checkpoint validation of the FP8/NVFP4 dequantization path. Both
targets loaded with **zero quantized tensors left, zero NaNs, and
@@ -1944,10 +1954,11 @@ Notes:
- The compressed-tensors per-channel FP8 dense alternate
(`RedHatAI/Llama-3.1-8B-Instruct-FP8-dynamic`) was not run: the exact
code path (`dequantize_fp8_per_channel`, direct scales) was already
exercised by the Nemotron FP8 checkpoint, and Llama-3.1 is a gated
repo. CT-NVFP4 (reciprocal scales) is covered by synthetic unit tests
only — no public checkpoint was needed after the Nemotron layout
parsed cleanly.
exercised by the contributor's Nemotron FP8 run, and Llama-3.1 is a gated
repo. CT-NVFP4 is covered by deterministic synthetic contract tests only;
those tests follow the published `weight_packed`/`weight_scale`/
`weight_global_scale` mapping. A pinned public-checkpoint run remains
necessary before making a runtime support claim.
---