We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.
Figures & tables
Figure 1: Accuracy and throughput comparisons of Nemotron 3 Super with GPT-OSS-120B and Qwen3.5-122B. Nemotron 3 Super achieves comparable accuracies across popular benchmarks but provides the highest inference throughput; on 8k input sequence lengths and 64k output sequence lengths, Nemotron 3 Super provides up to 2.2 × and 7.5 × higher throughput than GPT-OSS-120B and Qwen3.5-122B, respectively. We measured throughput on B200 GPUs with vLLM and TRT-LLM and use the best out of the two frameworks for each model. For GPT-OSS-120B we use MXFP4 weights, MXFP8 activations, and FP8 KV-Cache; for Qwen3.5-122B we use BF16. We used the OpenHands harness to evaluate SWE-Bench.
Figure 2: Nemotron 3 Super layer pattern. Similar to Nemotron 3 Nano, we use a hybrid Mamba-Attention architecture, but Nemotron 3 Super is the first model to scale sparsely using LatentMoE layers rather than standard MoE layers.
Configuration
Nemotron 3 Super 120B-A12B Base
Total Layers
88
Model Dimension
4096
Q-Heads ( nq )
32
KV-Heads ( nkv )
2
Head Dimension
128
Mamba State Dimension
128
Table 1: Nemotron 3 Super Architectural Dimensions. The model employs a hybrid Mamba-2 and MoE design with strategic global attention layers to optimize the balance between sequence modeling performance and inference throughput.
Figure 3: Standard MoE vs. LatentMoE. In LatentMoE, tokens are projected from the hidden dimension d into a smaller latent dimension ℓ for routing and expert computation, reducing routed parameter loads and all-to-all traffic by a factor d/ℓ . These savings are used to increase both the total number of experts and the top- K active experts per token by the same factor, improving model accuracy at approximately constant inference cost.
Figure 5
Layer Type
Format
Rationale
All Linear Layers Unless Otherwise Noted
NVFP4
Final 15% of Network
BF16
Promote training stability at scale
Latent Projections
BF16
Strategically kept in BF16 as step-time impact is negligible
MTP Layers
BF16
Preserves multi-token prediction capabilities
QKV & Attention Projections
BF16
Maintain fidelity of few attention layers
Mamba Output Projection
MXFP8
Mitigates high incidence of underflows observed when quantizing this layer to NVFP4 at smaller scales
Table 3: Precision by Layer Type
Figure 6: Channel Magnitude Patterns in Weights of Expert Layers in Nemotron 3 Super. Patterns emerge as training progresses. Top: Early layer routed expert FC1 weight matrix at 0.5T tok and 23T tok. Bottom: Early layer routed expert FC2 weight matrix at 500B tok and 23T tok. Low-norm output channels of FC1 align with low-norm input channels of FC2.
Figure 7: Number of Zero-Valued Weight Gradient Elements on Nemotron 3 Nano. Left: Released Nemotron Nano 3 model ( NVIDIA, 2025e ) trained to 25T tokens in BF16. Right: Ablation study trained to 1T tokens in BF16 and with our NVFP4 recipe. The NVFP4 model at 1T tokens reaches a similar zero-valued weight gradient count as the BF16 model at 25T tokens. Switching from NVFP4 to BF16 at 0.5T tokens causes zero-valued weight gradients to return to baseline levels. The high prevalence of small-magnitude gradients (<1e-12) in BF16 suggests NVFP4 quantization underflows already-small values to zero.
Figure 8: Origin of Zero Element Weight Gradients in Nemotron 3 Nano, shown for routed expert layers increasing in layer depth. Averaged across all routed experts in a layer index. Top: Tensors sampled at 500B tokens. Percent of zero-valued weight gradients are higher for FC1 than FC2 and attributed almost entirely to underflows in dgrad of FC2. Bottom: Tensors sampled at 750B tokens. Percent of zero-valued weight gradients are equivalently high for FC1 and FC2. Zero-valued weight gradients in FC1 are attributed to underflows in dgrad of FC2. Zero element weight gradients in FC2 are attributed mainly to underflows in fprop of FC1, with a minor contribution of underflows in wgrad of FC2.
Figure 9: Improvement in downstream task evaluation accuracy after switching the network precision to MXFP8. Values greater than zero indicate improvement in accuracy over the NVFP4 model. None of the downstream task evaluation metrics showed sustained improvement after training in MXFP8.
Figure 10: Data mixtures for each phase of pre-training.
Figure 11: Average accuracy across 12 benchmarks for trained checkpoints versus the best offline checkpoint merge during pretraining. During the stable LR phase, offline merging yields a consistent 2–4 point improvement. During the LR decay phase (shaded), the gap narrows as trained checkpoints benefit from actual learning rate annealing.
Task
Metric
N-3-Super
Ling-flash
GLM-4.5
120B-A12B-Base
base-2.0
Air-Base
General Knowledge
MMLU
5-shot, acc
86.01
81.00
81.00
MMLU-Pro
5-shot, CoT EM
75.65
62.10
58.20
AGIEval-En
3/5-shot, CoT EM
77.92
61.70
62.40
GPQA-Diamond
5-shot, CoT EM
60.00
36.00
23.20
Table 4: Comparison of Ling-flash-base-2.0 , GLM-4.5-Air-Base , and Nemotron Super 120B-A12B Base . Best results are marked in bold.
Figure 12: Overview of the post-training pipeline for Nemotron 3 Super.
Figure 13: Agentic Command Line Interface Dataset Construction & Training Pipeline
Figure 14: Overview of the synthetic data generation pipeline for specialized conversational tool-use SFT data used for Nemotron 3 Super.
Figure 15: Overview of the pipeline for general-purpose tool-calling data used in Nemotron 3 Super.
Figure 16: SFT data blend for Nemotron 3 Super.
Benchmark
N-3-Super
Qwen3.5-122B-A10B
GPT-OSS-120B
General Knowledge
MMLU-Pro
83.73
86.70
81.00
Reasoning
AIME25 (no tools)
90.21
90.36
92.50
HMMT Feb25 (no tools)
93.67
91.40
90.00
HMMT Feb25 (with tools)
94.73
89.55
-
Table 5: Evaluation suite for Nemotron 3 Super. We compare against Qwen-3.5-122B-A10B and GPT-OSS-120B .
Configuration
FP8 Checkpoint
BF16 Baseline
Embedding
BF16
BF16
Attention GEMM (QKV and Out Projection)
BF16
BF16
KV Cache + Attention BMM1
FP8
FP8
Attention BMM2
BF16
BF16
MoE GEMM (Sparse Experts and Shared Experts)
FP8
BF16
MoE Latent Projection GEMM
BF16
BF16
Table 6: Precision settings for the FP8 checkpoint compared with the BF16 baseline.
Configuration
AutoQuantize Searched?
NVFP4 Checkpoint
BF16 Baseline
Embedding
No
BF16
BF16
Attention QKV Projection GEMM
Yes
BF16
BF16
Attention Output Projection GEMM
Yes
FP8 / BF16
BF16
KV Cache + Attention BMM1
No
FP8
FP8
Attention BMM2
No
BF16
BF16
Sparse Expert (Routed) GEMM
Yes
NVFP4
BF16
Table 7: Precision settings for the backbone NVFP4 checkpoint compared with the BF16 baseline.
Benchmark
N-3-Super
N-3-Super FP8
N-3-Super NVFP4
General Knowledge
MMLU-Pro
83.73
83.63
83.33
Reasoning
HMMT Feb25 (with tools)
94.73
94.38
95.36
GPQA (no tools)
79.23
79.36
79.42
LiveCodeBench (v6 2024-08 ↔ 2025-05)
78.69
78.44
78.44
Table 8: Evaluation suite for Nemotron 3 Super. We compare our FP8 and NVFP4 optimized models with BF16 model.
Accuracy
Completion Tokens
Verbosity Increase
Weight and
livecodebench
scicode
livecodebench
scicode
livecodebench
scicode
Activation
SSM cache
(pass@1
(pass@1
(pass@1
(pass@1
(pass@1
(pass@1
Precision
recipe
avg-of-8)
avg-of-8)
avg-of-8)
avg-of-8)
avg-of-8)
avg-of-8)
W16A16
FP32
72.91
40.90
21769
3680
0.00%
0.00%
W16A16
FP16
73.24
42.01
29812
3760
36.95%
2.19%
W16A16
FP16+SR (Philox 5)
72.00
41.94
21392
3580
− 1.73%
− 2.72%
Table 9: Impact of SSM cache recipe on code benchmarks and verbosity.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 17: Per-benchmark accuracy for trained checkpoints versus the best offline checkpoint merge across the full 25T-token training run. The 12 benchmarks span general knowledge (MMLU-Pro, MMLU), code generation (HumanEval, HumanEval+, MBPP, MBPP+), mathematical reasoning (GSM8K, MATH-500), and commonsense understanding (RACE, ARC-Challenge, HellaSwag, WinoGrande). The shaded region indicates the LR decay phase.
Algorithm
Details
MMLU-Pro
GPQA
LiveCodeBench
AA-LCR
BF16
—
83.49
79.92
72.907
53.00
Default NVFP4 PTQ (Baseline algorithm)
Static per-tensor scales are computed using max-value calibration; per-block scales are computed dynamically from block maximum values.
82.99
79.29
70.18
55.50
Weight per-block scales minimizing MSE
Weight per-block scales are swept to minimize per-block MSE.
83.31
79.92
71.37
56.75
Weight per-block scales to minimize output MSE
Weight per-block scales are swept independently to minimize GEMM output MSE.
83.05
78.98
71.00
57.06
GPTQ
GPTQ ( Frantar et al., 2023 ) is used for weight quantization.
We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.
We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super at matched user throughput constraints. In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests. Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality. We evaluate Puzzle-75B-A9B on a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the model retains strong downstream accuracy relative to the parent model across a wide range of tasks. These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability. Our model is publicly available on Hugging Face.
Akhiad Bercovich, Talor Abramovich, Daniel Afrimi +67
We introduce TurboMamba, a language-model architecture for faster inference, faster training, and better token prediction. A global Mamba-3 model runs once per completed token block. A small Transformer uses this global context and earlier tokens within the block to predict the next token. Three lightweight additions improve local predictions: direct access to the previous block's last token, a small correction to next-token scores, and a learned mixture with recent-token frequencies. We compare twelve small-scale implementations on public FineWeb-Edu, with comparable parameter counts and shared training-budget caps. The baselines include Transformer, Mamba-3, Jamba, and Zamba. The Jamba and Zamba versions use Mamba-3 in place of their original Mamba modules. TurboMamba achieves the lowest mean test prediction loss (negative log-likelihood, NLL) among all twelve models. Its mean NLL is 9.4% lower than that of Mamba-3, 7.3% lower than that of the Jamba version, and 18.3% lower than that of the Zamba version. End-to-end training throughput is 2.60x that of Mamba-3, 2.81x that of the Jamba version, and 4.82x that of the Zamba version. We measure end-to-end batch latency for a batch of 32 sequences, each with a 256-token prompt and 128 generated tokens. On this workload, TurboMamba achieves a 3.12x speedup over Mamba-3, a 3.53x speedup over the Jamba version, and a 6.21x speedup over the Zamba version.