cs.LGApr 14, 2026
SaveNemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Abstract
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.
Figures & tables
Figure 1: Accuracy and throughput comparisons of Nemotron 3 Super with GPT-OSS-120B and Qwen3.5-122B. Nemotron 3 Super achieves comparable accuracies across popular benchmarks but provides the highest inference throughput; on 8k input sequence lengths and 64k output sequence lengths, Nemotron 3 Super provides up to 2.2 and 7.5 higher throughput than GPT-OSS-120B and Qwen3.5-122B, respectively. We measured throughput on B200 GPUs with vLLM and TRT-LLM and use the best out of the two frameworks for each model. For GPT-OSS-120B we use MXFP4 weights, MXFP8 activations, and FP8 KV-Cache; for Qwen3.5-122B we use BF16. We used the OpenHands harness to evaluate SWE-Bench.
Figure 2: Nemotron 3 Super layer pattern. Similar to Nemotron 3 Nano, we use a hybrid Mamba-Attention architecture, but Nemotron 3 Super is the first model to scale sparsely using LatentMoE layers rather than standard MoE layers.
| Configuration | Nemotron 3 Super 120B-A12B Base |
| Total Layers | 88 |
| Model Dimension | 4096 |
| Q-Heads ( ) | 32 |
| KV-Heads ( ) | 2 |
| Head Dimension | 128 |
| Mamba State Dimension | 128 |
Table 1: Nemotron 3 Super Architectural Dimensions. The model employs a hybrid Mamba-2 and MoE design with strategic global attention layers to optimize the balance between sequence modeling performance and inference throughput.
Figure 3: Standard MoE vs. LatentMoE. In LatentMoE, tokens are projected from the hidden dimension into a smaller latent dimension for routing and expert computation, reducing routed parameter loads and all-to-all traffic by a factor . These savings are used to increase both the total number of experts and the top- active experts per token by the same factor, improving model accuracy at approximately constant inference cost.
| Layer Type | Format | Rationale |
| All Linear Layers Unless Otherwise Noted | NVFP4 | |
| Final 15% of Network | BF16 | Promote training stability at scale |
| Latent Projections | BF16 | Strategically kept in BF16 as step-time impact is negligible |
| MTP Layers | BF16 | Preserves multi-token prediction capabilities |
| QKV & Attention Projections | BF16 | Maintain fidelity of few attention layers |
| Mamba Output Projection | MXFP8 | Mitigates high incidence of underflows observed when quantizing this layer to NVFP4 at smaller scales |
Table 3: Precision by Layer Type
Figure 6: Channel Magnitude Patterns in Weights of Expert Layers in Nemotron 3 Super. Patterns emerge as training progresses. Top: Early layer routed expert FC1 weight matrix at 0.5T tok and 23T tok. Bottom: Early layer routed expert FC2 weight matrix at 500B tok and 23T tok. Low-norm output channels of FC1 align with low-norm input channels of FC2.
Figure 7: Number of Zero-Valued Weight Gradient Elements on Nemotron 3 Nano. Left: Released Nemotron Nano 3 model ( NVIDIA, 2025e ) trained to 25T tokens in BF16. Right: Ablation study trained to 1T tokens in BF16 and with our NVFP4 recipe. The NVFP4 model at 1T tokens reaches a similar zero-valued weight gradient count as the BF16 model at 25T tokens. Switching from NVFP4 to BF16 at 0.5T tokens causes zero-valued weight gradients to return to baseline levels. The high prevalence of small-magnitude gradients (<1e-12) in BF16 suggests NVFP4 quantization underflows already-small values to zero.
Figure 8: Origin of Zero Element Weight Gradients in Nemotron 3 Nano, shown for routed expert layers increasing in layer depth. Averaged across all routed experts in a layer index. Top: Tensors sampled at 500B tokens. Percent of zero-valued weight gradients are higher for FC1 than FC2 and attributed almost entirely to underflows in dgrad of FC2. Bottom: Tensors sampled at 750B tokens. Percent of zero-valued weight gradients are equivalently high for FC1 and FC2. Zero-valued weight gradients in FC1 are attributed to underflows in dgrad of FC2. Zero element weight gradients in FC2 are attributed mainly to underflows in fprop of FC1, with a minor contribution of underflows in wgrad of FC2.
Figure 9: Improvement in downstream task evaluation accuracy after switching the network precision to MXFP8. Values greater than zero indicate improvement in accuracy over the NVFP4 model. None of the downstream task evaluation metrics showed sustained improvement after training in MXFP8.
Figure 10: Data mixtures for each phase of pre-training.
Figure 11: Average accuracy across 12 benchmarks for trained checkpoints versus the best offline checkpoint merge during pretraining. During the stable LR phase, offline merging yields a consistent 2–4 point improvement. During the LR decay phase (shaded), the gap narrows as trained checkpoints benefit from actual learning rate annealing.
| Task | Metric | N-3-Super | Ling-flash | GLM-4.5 |
| 120B-A12B-Base | base-2.0 | Air-Base | ||
| General Knowledge | ||||
| MMLU | 5-shot, acc | 86.01 | 81.00 | 81.00 |
| MMLU-Pro | 5-shot, CoT EM | 75.65 | 62.10 | 58.20 |
| AGIEval-En | 3/5-shot, CoT EM | 77.92 | 61.70 | 62.40 |
| GPQA-Diamond | 5-shot, CoT EM | 60.00 | 36.00 | 23.20 |
Table 4: Comparison of Ling-flash-base-2.0 , GLM-4.5-Air-Base , and Nemotron Super 120B-A12B Base . Best results are marked in bold.
Figure 12: Overview of the post-training pipeline for Nemotron 3 Super.
Figure 13: Agentic Command Line Interface Dataset Construction & Training Pipeline
Figure 14: Overview of the synthetic data generation pipeline for specialized conversational tool-use SFT data used for Nemotron 3 Super.
Figure 15: Overview of the pipeline for general-purpose tool-calling data used in Nemotron 3 Super.
Figure 16: SFT data blend for Nemotron 3 Super.
| Benchmark | N-3-Super | Qwen3.5-122B-A10B | GPT-OSS-120B |
| General Knowledge | |||
| MMLU-Pro | 83.73 | 86.70 | 81.00 |
| Reasoning | |||
| AIME25 (no tools) | 90.21 | 90.36 | 92.50 |
| HMMT Feb25 (no tools) | 93.67 | 91.40 | 90.00 |
| HMMT Feb25 (with tools) | 94.73 | 89.55 | - |
Table 5: Evaluation suite for Nemotron 3 Super. We compare against Qwen-3.5-122B-A10B and GPT-OSS-120B .
| Configuration | FP8 Checkpoint | BF16 Baseline |
| Embedding | BF16 | BF16 |
| Attention GEMM (QKV and Out Projection) | BF16 | BF16 |
| KV Cache + Attention BMM1 | FP8 | FP8 |
| Attention BMM2 | BF16 | BF16 |
| MoE GEMM (Sparse Experts and Shared Experts) | FP8 | BF16 |
| MoE Latent Projection GEMM | BF16 | BF16 |
Table 6: Precision settings for the FP8 checkpoint compared with the BF16 baseline.
| Configuration | AutoQuantize Searched? | NVFP4 Checkpoint | BF16 Baseline |
| Embedding | No | BF16 | BF16 |
| Attention QKV Projection GEMM | Yes | BF16 | BF16 |
| Attention Output Projection GEMM | Yes | FP8 / BF16 | BF16 |
| KV Cache + Attention BMM1 | No | FP8 | FP8 |
| Attention BMM2 | No | BF16 | BF16 |
| Sparse Expert (Routed) GEMM | Yes | NVFP4 | BF16 |
Table 7: Precision settings for the backbone NVFP4 checkpoint compared with the BF16 baseline.
| Benchmark | N-3-Super | N-3-Super FP8 | N-3-Super NVFP4 |
| General Knowledge | |||
| MMLU-Pro | 83.73 | 83.63 | 83.33 |
| Reasoning | |||
| HMMT Feb25 (with tools) | 94.73 | 94.38 | 95.36 |
| GPQA (no tools) | 79.23 | 79.36 | 79.42 |
| LiveCodeBench (v6 2024-08 2025-05) | 78.69 | 78.44 | 78.44 |
Table 8: Evaluation suite for Nemotron 3 Super. We compare our FP8 and NVFP4 optimized models with BF16 model.
| Accuracy | Completion Tokens | Verbosity Increase | |||||
| Weight and | livecodebench | scicode | livecodebench | scicode | livecodebench | scicode | |
| Activation | SSM cache | (pass@1 | (pass@1 | (pass@1 | (pass@1 | (pass@1 | (pass@1 |
| Precision | recipe | avg-of-8) | avg-of-8) | avg-of-8) | avg-of-8) | avg-of-8) | avg-of-8) |
| W16A16 | FP32 | 72.91 | 40.90 | 21769 | 3680 | 0.00% | 0.00% |
| W16A16 | FP16 | 73.24 | 42.01 | 29812 | 3760 | 36.95% | 2.19% |
| W16A16 | FP16+SR (Philox 5) | 72.00 | 41.94 | 21392 | 3580 | 1.73% | 2.72% |
Table 9: Impact of SSM cache recipe on code benchmarks and verbosity.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 17: Per-benchmark accuracy for trained checkpoints versus the best offline checkpoint merge across the full 25T-token training run. The 12 benchmarks span general knowledge (MMLU-Pro, MMLU), code generation (HumanEval, HumanEval+, MBPP, MBPP+), mathematical reasoning (GSM8K, MATH-500), and commonsense understanding (RACE, ARC-Challenge, HellaSwag, WinoGrande). The shaded region indicates the LR decay phase.
| Algorithm | Details | MMLU-Pro | GPQA | LiveCodeBench | AA-LCR |
| BF16 | — | 83.49 | 79.92 | 72.907 | 53.00 |
| Default NVFP4 PTQ (Baseline algorithm) | Static per-tensor scales are computed using max-value calibration; per-block scales are computed dynamically from block maximum values. | 82.99 | 79.29 | 70.18 | 55.50 |
| Weight per-block scales minimizing MSE | Weight per-block scales are swept to minimize per-block MSE. | 83.31 | 79.92 | 71.37 | 56.75 |
| Weight per-block scales to minimize output MSE | Weight per-block scales are swept independently to minimize GEMM output MSE. | 83.05 | 78.98 | 71.00 | 57.06 |
| GPTQ | GPTQ ( Frantar et al., 2023 ) is used for weight quantization. | 83.11 | 80.05 | 69.79 | 57.87 |
Table 10: PTQ Algorithm Ablation