We introduce Nemotron 3.5 Super and Nemotron 3.5 Lightning, open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads. Super is a 120B-parameter, 12B-active multimodal model, while Lightning is a smaller 30B-parameter, 3B-active text model; both designed for efficient deployment. Both models combine large-scale distillation, long-context training, and native speculative decoding, while Super further incorporates Indic continued pretraining and agentic software-engineering mid-training. We develop a comprehensive post-training pipeline spanning supervised fine-tuning and reinforcement learning (RLVR and RLHF), complemented in Super by specialized teacher training and multi- teacher on-policy distillation (MOPD). This transforms Super from a text-only base model into a multimodal agent capable of reasoning over text, images, video, and interactive tool environments. We further explore test-time scaling through parallel agent execution and self-verification. Across training and deployment, we jointly optimize capability and efficiency. Indic continued pretraining improves Indian-language benchmarks by up to 9.82 percentage points, while agentic mid-training strengthens software-engineering capabilities and robustness across agent harnesses. The post-trained Super model achieves 51.78% on SWE-bench Pro, surpassing Nemotron 3 Ultra despite using approximately 4.6x fewer active parameters. Post-training MTP boosting improves speculative- decoding acceptance length by up to 21.8% under probabilistic draft sampling, while NVFP4 quantization achieves 99.62% median accuracy recovery relative to BF16. Inference optimizations enable up to 4,856 output tokens/s/GPU on GB200 while sustaining at least 50 tokens/s/user.
Figures & tables
Figure 1: Benchmark performance of Nemotron 3.5 Super (120B-A12B) compared with six leading open models across agentic tasks, code generation, mathematical and scientific reasoning, multimodal understanding, long-context reasoning, and chat preference. Higher scores indicate better performance. Full evaluation results and methodology are provided in Section 3.8 and Table 10.
Priority
Native
Romanized
Additional
Native
language
Input
Output
Input
Output
language
Input
Output
Bengali
114.0
138.9
12.2
19.0
Assamese
16.3
31.1
Gujarati
115.5
196.4
12.0
17.2
Bodo
8.6
18.7
Hindi
118.1
149.3
12.8
18.4
Dogri
23.4
39.9
Kannada
103.5
180.5
11.3
19.1
Kashmiri
11.4
33.1
Malayalam
108.9
183.5
11.4
19.3
Konkani
16.7
28.9
Table 1: Reported input and output token counts by language and script, in billions (B), using a Gemma-4 tokenizer. Input counts may overlap across configurations; they are not unique source volume or total pipeline processing. Subtotals sum displayed rounded values. The output-only corpus total remains 2,825.6B (approximately 2.83T), not the number of tokens used to train Super. The additional ten languages have native-script data only.
Figure 2: Four-stage Indic synthetic-data pipeline implemented using NeMo Data Designer for the twelve priority languages. Selected English source documents, primarily from Nemotron-CC, contain fewer than 4,096 Gemma-4 tokens and are translated in full, without splitting. Translations are retained after passing the initial or post-rewrite quality check. Romanization starts from native-script text and uses the same model with script-specific prompts and quality checks. Sampled human review helps improve prompts and checks; it does not cover every document.
Checkpoint
MILU (10)
Indic MMLU-Pro (12)
Indic ARC-C (12)
Indic ARC-E (12)
Indic GSM8K strict (11)
Before Indic CPT (distilled)
72.48
51.83
84.17
90.73
72.67
After Indic CPT
77.90
56.81
92.83
96.77
82.49
Change (percentage points)
+5.42
+4.98
+8.66
+6.04
+9.82
Table 2: Super performance on Indic benchmarks before and after Indic CPT. Scores are percentages; parentheses show the number of languages. Changes are calculated from unrounded scores.
Figure 3: Indic task performance and language quality for Nemotron 3 Super (blue-gray) and Nemotron 3.5 Super (green). Left: six task benchmarks. Right: preliminary IndiVibe conditional mean preference scores across four language-quality dimensions, not direct head-to-head win rates. Labels show changes in percentage points (chrF points for WMT24++). Benchmark, scoring, and checkpoint details will appear in the companion Indic report.
Checkpoint
Indic share
Four-benchmark mean
Base reference
-
45.5
Mixture ablation
10%
63.1
Mixture ablation
20%
64.6
Mixture ablation
30%
65.6
Table 3: Lightning 3.5 research ablations. Each ablation uses a 200B-token budget. The reported mean covers MILU, MMLU-Pro, ARC-Challenge, and GSM8K. The base checkpoint is shown for reference; the fixed-budget comparison covers the three ablation runs.
Data source
Description
Share
Issue–PR
GitHub issues with repository context and the merged patch
41%
Repositories
Whole repositories, with files that import each other placed together
20%
Agentic traces
Trajectories from many agent scaffolds, tool responses loss-masked
10%
Commit diffs
Multi-file commits and per-file edit histories
3%
Build and CI logs
Raw build and continuous-integration logs
1%
Pretraining replay
Phase-2 pretraining blend
20%
Table 4: Mid-training blend for Nemotron 3.5 Super, as shares of training tokens.
Base
Mid-trained
SWE-bench Multilingual
65.9
65.6
SWE-bench Pro
52.1
51.7
DeepSWE
6.4
10.4
BrowseComp (subset)
64.2
65.0
SWE-bench Verified, by harness
OpenHands
71.3
71.9
Table 5: Effect of mid-training on Nemotron 3.5 Super, measured after the same experimental 33B-token (2K-step) SFT on the base and the mid-trained checkpoint (pass@1, %). SWE-bench Multilingual and Pro use OpenHands, and DeepSWE is averaged over 4 runs. The OpenHands results come from a separate evaluation from the other harnesses.
Task
Metric
Nemotron 3.5 Super
Nemotron 3.5 Lightning
General Knowledge
MMLU
exact match
84.14
78.63
MMLU-Pro
exact match, TigerLab extract
76.25
67.50
AGIEval-En
CoT exact match, custom extract
75.65
68.62
SuperGPQA
CoT
47.65
33.21
GPQA Diamond
5-shot, CoT exact match
66.16
33.33
Table 6: Base model evaluation results for Nemotron 3.5 Super 120B-A12B Base and Lightning 3.5 30B-A3B Base . For sampled code and mathematical reasoning benchmarks, pass@1 is estimated from 32 generations per prompt where indicated. A dash indicates that a corresponding result is not available.
Figure 4: Overview of the Super-V3.5-VL post-training pipeline. Teacher models are developed from a shared trunk through supervised fine-tuning (SFT) and reinforcement learning (RL) stages, specializing in different capabilities. These teacher models are subsequently merged into a unified student model through MoPD (Multi-Teacher On-Policy Distillation).
Figure 5: Synthetic vision-data generation through diverse input selection, question generation, grounded reasoning, and quality filtering.
Task
Baseline
Ours
Δ
Single-image spatial reasoning
ERQA
49.2
57.5
+8.3
Omni3D
48.6
53.4
+4.8
OmniSpatial
52.7
58.6
+5.9
SPBench
60.1
63.8
+3.7
Multi-view spatial reasoning
Table 7: Effect of agentic vision training on spatial reasoning and video understanding using the SpatialClaw harness.
Model
temp=0
temp=1
Nemotron 3.5 Super (SFT checkpoint)
4.589
4.048
+ 4k MTP-only training iterations
5.080
4.730
Nemotron 3.5 Super (Final)
4.287
3.787
+ MTP head from the SFT-stage run
4.528
4.436
+ 25k MTP Boosting iterations
4.718
4.612
Table 8: Effect of MTP Boosting on average acceptance length (AL), at draft length 7. Top : MTP training applied directly to an SFT checkpoint. Bottom : Nemotron 3.5 Super evaluated with the native MTP, the resulting MTP head from the SFT-stage run, and after 25k of boosting iterations. temp=0 is greedy decoding; temp=1 uses probabilistic draft sampling. For each training phase we generated data using the corresponding backbone.
Category
Nemotron 3.5 Super
Gemma-4 31B
Qwen3.5 122B-A10B
Qwen3.8-Flash-Next
Native
Boosted MTP
low
medium
xhigh
Coding
4.966 (4.343)
5.563 (5.441)
5.792 ( 5.459 )
5.630 (5.221)
5.506 (5.310)
5.355 (5.124)
3.975 (3.714)
Humanities
3.703 (3.245)
3.975 (3.817)
4.488 ( 4.039 )
4.584 (3.956)
4.122 (3.763)
4.073 (3.628)
3.873 (3.005)
Math
4.533 (4.093)
4.841 (4.825)
5.676 ( 5.321 )
5.149 (4.826)
5.207 (4.982)
5.169 (4.815)
3.942 (3.672)
Multilingual
4.945 (4.266)
5.548 (5.358)
5.703 ( 5.435 )
4.995 (4.613)
5.056 (4.911)
4.942 (4.678)
4.218 (3.926)
QA
4.083 (3.624)
4.526 ( 4.378 )
4.366 (4.092)
4.513 (4.111)
3.988 (3.913)
4.096 (3.748)
3.626 (3.322)
Table 9: MTP average acceptance lengths on the SPEED-Bench qualitative split using draft length 7. Main values use greedy decoding with temp=0 ; parenthetical values use temp=1 with probabilistic draft sampling. Reasoning is turned on for all models. For Nemotron 3.5 Super, Native is the pre-boosting MTP; for Qwen3.8-Flash-Next the sub-columns give the reasoning effort. Bold marks the best value per row, computed separately for temp=0 and temp=1 .
Figure 6: Joint MTP acceptance rate at each draft position on the SPEED-Bench qualitative split (draft length 7).
Benchmark
N3.5-Super 120B-A12B
N3-Ultra 550B-A55B
Qwen3.5 122B-A10B
Qwen3.8 Flash-Next
DS-V4 Flash-0731
Inkling Small
GLM-5.3 Flash
Agentic
SWE-bench Verified
71.10
73.93
71.87
72.60
77.80
75.27
79.10
SWE-bench Multilingual
67.72
68.78
62.00
79.00
72.00
71.44
78.30
SWE-bench Pro
51.78
49.66
45.60
60.37
56.91
56.54
59.70
Tau2 Retail
69.24
73.68
55.37
71.60
73.79
71.82
78.18
Tau2 Airline
76.50
79.75
63.50
80.75
83.50
62.75
81.75
Table 10: Evaluation suite for Nemotron 3.5 Super. We compare against six open models. All SWE evaluations use OpenCode as a default harness. Terminal Bench 2.1 uses Terminus 2 and a 3 hours task timeout.
Text Arena
N3.5-Super 120B-A12B NVFP4
N3-Ultra 550B-A55B NVFP4
Qwen3.5 122B-A10B
DS-V4 Flash
Inkling Small
GLM-5.3 Flash
Gemma-4 26B-A4B
Overall
With Style Control
1420 ± 12 (2,328 votes)
1426 ± 7 (11,570 votes)
1416 ± 4 (29,737 votes)
1436 ± 4 (51,988 votes)
1404 ± 5 (31,831 votes)
1474 ± 5 (32,013 votes)
1438 ± 7 (6,095 votes)
With Factuality
1441 ± 20 (2,328 votes)
1386 ± 12 (11,570 votes)
1431 ± 9 (29,737 votes)
1433 ± 6 (51,988 votes)
1429 ± 7 (31,831 votes)
1438 ± 7 (32,013 votes)
1450 ± 20 (6,095 votes)
No Adjustment
1413 ± 12 (2,328 votes)
1445 ± 7 (11,570 votes)
1417 ± 4 (29,737 votes)
1432 ± 4 (51,988 votes)
1412 ± 5 (31,831 votes)
1471 ± 5 (32,013 votes)
1434 ± 7 (6,095 votes)
English
With Style Control
1446 ± 20 (884 votes)
1440 ± 9 (5,381 votes)
1430 ± 6 (13,158 votes)
1445 ± 5 (23,133 votes)
1417 ± 6 (13,047 votes)
1479 ± 7 (11,852 votes)
1450 ± 12 (2,249 votes)
Table 11: Text Arena Elo scores. We report number of votes and CI from Arena. Note, N3.5-Super 120B-A12B NVFP4 are preliminary results, and Arena does not report public scores for Qwen3.8 Flash-Next.
Figure 7: Accuracy–compute trade-off on SWE-bench Pro. We evaluate each model, record the resulting agent trajectories, and use the captured traces to benchmark task throughput (tasks per hour). The vertical axis reports Pass@1 accuracy averaged over three evaluation runs. The horizontal axis reports the aggregate GPU time, in GPU-hours, required to complete 2,193 tasks (three repeats of the public set). The compute footprint is four NVIDIA B200 GPUs. All models use NVFP4 precision except DeepSeek V4 Flash 0731, which uses MXFP4.
Figure 8: Nemotron 3.5 Super performance across six agent harnesses on SWE-bench Pro and Terminal-Bench 2.1, using unmodified harness system prompts and standard configurations. For SWE-bench Pro, Git history is removed and a user instruction prohibits cheating or seeking online resources.
Figure 9: Test-time scaling with the final post-trained checkpoint. Blue bars show empirical pass@1, averaged over the candidate rollouts; green segments show the improvement from LAV selection, with total bar height indicating selected accuracy. Candidate-pool sizes are shown below each benchmark. BrowseComp is evaluated on a 400-problem subset used for internal evaluation tracking.
Reference
Predefined recipes
Layer / axis
BF16
FP8
Conservative
Aggressive
Very aggressive
Vision path
C-RADIO encoder
BF16
BF16
BF16
BF16
BF16
VL connector
BF16
BF16
BF16
BF16
BF16
Latent MoE
MoE + shared GEMM
BF16
FP8
NVFP4 a
NVFP4
NVFP4
Table 12: Per-module precision across predefined recipe tiers. The Aggressive column is the released recipe. NVFP4 denotes e2m1-based 4-bit quantization; FP8 denotes e4m3; W4A16 denotes NVFP4 weights with 16-bit activations; +SR denotes stochastic rounding. Reference columns denote precision policies, not complete benchmark serving configurations; the very-aggressive column was explored but not released. Whole-model BPE includes quantization scales and BF16-stored router weights: Aggressive is measured from the released checkpoint; the other columns are policy-based estimates.
Benchmark
BF16
MSE
Max
MSE 4-over-6
MMLU-ProX
76.92
73.92 ( −3.00 )
72.77 ( −4.15 )
74.77 ( −2.15 )
GPQA-D w/o tools
89.52
90.03 ( +0.51 )
89.90 ( +0.38 )
89.52 ( ±0.00 )
APEX Shortlist
88.56
86.97 ( −1.60 )
87.37 ( −1.20 )
88.70 ( +0.13 )
SciCode Subtask
47.52
48.37 ( +0.86 )
46.81 ( −0.70 )
46.79 ( −0.72 )
SciCode Problem
15.77
16.54 ( +0.77 )
15.77 ( ±0.00 )
16.35 ( +0.58 )
AA-LCR
69.00
68.06 ( −0.94 )
66.50 ( −2.50 )
67.00 ( −2.00 )
Table 13: Accuracy for the aggressive tier calibrated using three different algorithms: MSE, Max, and MSE 4-over-6. Deltas are relative to the BF16 baseline.
BF16
PTQ
QAD
Iter. 200
Iter. 400
Iter. 600
Benchmark / checkpoint
100/0
75/25
100/0
75/25
100/0
75/25
MMLU-ProX
74.49
71.39
70.81
71.50
70.49
71.04
70.79
71.65
GPQA-D w/o tools
90.28
90.09
89.58
90.09
91.10
89.77
90.21
89.39
APEX Shortlist
89.36
86.04
89.49
88.30
88.30
89.10
88.03
88.96
SciCode subtask
48.25
48.43
48.75
47.84
48.31
47.30
48.58
47.38
Table 14: Accuracy results for the BF16 baseline, the 4.8-BPE MSE PTQ checkpoint, and QAD checkpoints trained with either 100% SFT or a 75/25 SFT/RL mixture. Higher is better.
BF16
PTQ
QAD
Iter. 200
Iter. 400
Iter. 600
Benchmark / checkpoint
100/0
75/25
100/0
75/25
100/0
75/25
MMLU-ProX
3,381
3,395
3,286
3,480
3,389
3,477
3,411
3,492
GPQA-D w/o tools
11,991
12,157
12,548
13,262
12,975
12,972
12,477
12,350
APEX Shortlist
61,817
64,980
64,208
65,010
64,711
63,650
66,093
64,762
SciCode
6,594
6,504
6,513
6,872
7,040
6,959
6,912
6,001
Table 15: Average completion-token counts for the BF16 baseline, the 4.8-BPE MSE PTQ checkpoint, and the QAD checkpoints. Lower values indicate less verbose generations. The final row reports the change in total completion tokens (summed over all benchmarks) relative to the BF16 baseline.
BF16
4.8 BPE
4.82 BPE
4.9 BPE
Benchmark / metric
Base
PTQ
Q200
Q300
PTQ
Q200
Q300
PTQ
Q200
Q300
MMLU-ProX
76.92
73.82
74.03
73.75
73.66
74.49
75.07
75.11
73.26
74.28
GPQA-D w/o tools
89.52
90.34
90.03
90.21
89.84
89.71
90.03
89.90
89.84
89.84
APEX Shortlist
88.56
89.36
86.84
88.96
88.16
87.10
90.03
88.43
88.70
88.96
SciCode subtask
47.52
48.22
46.88
47.31
47.33
49.08
48.58
48.17
46.69
47.44
SciCode problem
15.77
16.92
14.42
16.15
15.38
16.73
16.92
16.54
15.00
15.00
Table 16: Accuracy of BF16 baseline and the PTQ and QAD checkpoints obtained from the 4.8, 4.82, and 4.9 BPE recipes. Q200 and Q300 denote QAD checkpoints after 200 and 300 iterations, respectively. Higher is better.
BF16
4.8 BPE
4.82 BPE
4.9 BPE
Benchmark / metric
Base
PTQ
Q200
Q300
PTQ
Q200
Q300
PTQ
Q200
Q300
MMLU-ProX
3,487
3,498
3,516
3,508
3,522
3,441
3,520
3,509
3,510
3,537
GPQA-D w/o tools
14,424
14,553
15,284
14,802
14,774
14,709
14,388
14,943
14,867
14,765
APEX Shortlist
68,401
66,810
67,748
67,934
66,830
66,687
66,721
66,866
67,717
64,926
SciCode
6,459
6,960
7,404
6,904
6,535
6,782
6,822
6,972
7,078
7,205
AA-LCR
4,444
4,242
4,367
4,100
4,280
4,971
4,153
4,158
4,319
4,664
Table 17: Average completion-token counts for the PTQ and QAD checkpoints obtained from the 4.8, 4.82, and 4.9 BPE recipes. Lower token counts indicate less verbose generations. The final row reports the change in total completion tokens (summed over all benchmarks) relative to the BF16 baseline.
Metric / benchmark
NVFP4 KV cache
FP8 KV cache
KV cache payload BPE
4.5
8.0
Median accuracy recovery (%)
99.36
99.52
MMLU-ProX
74.47
73.76
Baby Vision
28.61
29.51
CharXiv (RQ)
80.30
79.88
MathVision
77.84
77.79
Table 18: Uniform NVFP4 KV cache versus FP8 KV cache configurations. Both use aggressive NVFP4 recipe with QAD at 200 iterations. Median accuracy recovery is relative to BF16 checkpoint with FP8 KV cache.
Δ accuracy (pp) against the BF16 reference
Benchmark
Rollouts
q/k/v BF16
q/k/v MXFP8
q/k/v FP8
GraphWalks ( ≤ 1M) C,S
950×1
−0.52 (t=-0.70)
+0.15 (t=+0.21)
−0.83 (t=-1.03)
MMLongBench-Doc C,S
1091×3
−0.76 (t=-1.42)
−0.90 (t=-1.81)
−0.34 (t=-0.61)
AA-LCR S
100×16
+1.12 (t=+0.89)
−0.12 (t=-0.10)
+1.56 (t=+1.19)
MRCR-128k P
1470×3
−2.81 (t=-3.90)
−2.65 (t=-3.61)
−3.63 (t=-4.97)
Table 19: Long-context capability: distinction (C, content), disambiguation (P, position), aggregation (S, summary); all quantized models serve an FP8 KV cache with W4A4 experts, and differ only in q/k/v projection precision. t=mean(Δ)/SE(Δ) , where Δ is the per-task difference of task means and the standard error is taken over tasks , not rollouts — so the four rows are directly comparable.
Target model
MTP drafter
Acceptance length
Projected speedup
Backbone
LM head
Experts
LM head
DL=5
DL=7
DL=5
BF16
BF16
BF16
BF16
4.115
4.612
-
NVFP4
BF16
BF16
BF16
4.109
4.592
1
NVFP4
BF16
NVFP4
BF16
4.066
4.518
1.043
NVFP4
BF16
NVFP4
FP8
4.069
4.514
1.114
NVFP4
BF16
NVFP4
W4A16
4.031
4.469
1.136
Table 20: Acceptance length (AL) for different target-model and MTP-drafter precisions, and projected speedups over the baseline. Results are measured on the SPEED-Bench qualitative split at temperature 1. Speedup is projected relative to the NVFP4 backbone using unquantized MTP and lm_head . The shaded row denotes the selected configuration.
Layer group
Quantization-config name
Layer type
General backbone
mlp.*
Gate/up/down projections
fc
Fully connected projection
lm_head
Language-model output head
DSpark-specific
markov_head.markov_w2
Markov output projection
LiLiCorr-specific
lilicorr.context_proj
Context/hidden-state/token projections
lilicorr.pass_hidden_proj
Table 21: Quantized layers specification in DSpark and LiLiCorr draft models.
Benchmark
BF16 tokens per trajectory
NVFP4 tokens per trajectory
BF16 acc.
NVFP4 acc.
Δ (pp)
Recovery (%)
Agentic
SWE-Bench Verified
24,569
23,414
52.20
52.20
0.00
100.00
SWE-Bench Multilingual
31,071
33,924
37.13
37.00
-0.13
99.65
Tau2 Telecom
7,076
7,075
59.65
60.96
+1.32
102.21
Tau3 Banking
19,107
17,998
9.90
9.48
-0.41
95.83
Terminal-Bench v2.1
76,219
69,523
24.44
22.05
-2.39
90.22
Table 22: Lightning 3.5 BF16 versus W4A16 NVFP4 using scores from the saved evaluation view. Token columns retain the original average tokens per trajectory.
Benchmark
BF16 tokens per trajectory
NVFP4 tokens per trajectory
BF16 acc.
NVFP4 acc.
Δ (pp)
Recovery (%)
Agentic
SWE-Bench Verified
22,353
21,166
71.10
71.07
-0.03
99.95
SWE-Bench Multilingual
29,710
30,610
67.72
67.00
-0.72
98.93
SWE-Bench Pro
25,630
25,036
51.78
51.30
-0.48
99.08
Tau2 Telecom
5,784
5,659
73.19
73.19
0.00
100.00
Tau2 Retail
4,028
3,983
69.24
69.30
+0.05
100.08
Table 23: Super 3.5 BF16 versus released NVFP4 across the full benchmark suite. Token columns report average tokens per trajectory; Δ and recovery are calculated from accuracy.
Figure 10: Serving performance using vLLM on the 20-turn agentic proxy workload, with 2K input tokens and 1K output tokens per turn and ReplaySSM disabled. The graphs compare NVFP4 and BF16 maximum throughput at >=50 tokens/s/user and minimum latency.
Figure 11: Serving performance using vllm for the 20-turn agentic proxy workload, with 2K input tokens and 1K output tokens per turn. The graph shows the max-throughput of each model at tps/user >= 50 .
Deployment
Conc.
TPS/GPU
TPS/user
TTFT (s)
Aggregated
616
2,969
96.6
0.49
Disaggregated
528
3,276
155.9
1.12
Gain
+10.3%
+61%
Table 25: Best usable point for Nemotron 3.5 Super VL NVFP4 on 11 NVIDIA VR200 GPUs with Dynamo and vLLM. The workload replays SWE-bench Pro agentic coding sessions. A usable point has a median output speed of at least 50 tokens/s/user and a median TTFT below 5 s. Disaggregated uses 7 prefill and 4 decode GPUs. TPS/user and TTFT are medians. MTP uses a fixed acceptance length of 4.06 and draft length 5. TPS denotes tokens/s.
Dataset
Splits
Prompts
Nemotron-Competitive-Programming-v1
InfiniByte
1,174,694
Nemotron-SFT-Agentic-v2
interactive agent, search, tool calling
991,900
Nemotron-SFT-Instruction-Following-Chat-v2
reasoning on
929,237
Nemotron-Post-Training-Dataset-v2
chat, code, math, STEM, multilingual †
800,000
Nemotron-RL-Super-Training-Blends
RLHF, RLVR, SWE
479,303
Nemotron-SFT-OpenCode-v1
all six subsets
460,254
Table 26: Prompt sources of both training stages. After filtering, the first stage retains 5.67 M conversations and 22.6 B tokens, of which 14.9 B fall within its 4,096 -token training window. The second stage retains 11.01 M of 18.2 M generated samples and 35.6 B tokens, of which 34.9 B fall within its 8,192 -token training window. † 200K chat, all 175K code, 200K math, and 100K STEM prompts, plus 25K prompts in each of German, Spanish, French, Italian, and Japanese.
Setting
Original ( Rusanovsky et al., 2026 )
Nemotron 3.5 Super
Block size (drafted slots)
16 ( 15 )
8 ( 7 )
Candidates per slot K
8
8
Head layers, width, heads, FFN
2 , 1024 , 8 , 2048
2 , 1024 , 8 , 2048
Cosine scale σ
8
8
Body key-value heads
8
8
Body convolution (kernel, group)
none
2 , 16
Table 27: Training configuration of LiLiCorr for Nemotron 3.5 Super, compared with the original work.
BF16 drafter
NVFP4 drafter
Category
temp=0
temp=1
temp=0
temp=1
coding
5.70±0.01
5.31±0.08
5.65±0.03
5.29±0.04
rag
5.67±0.01
5.29±0.03
5.69±0.05
5.33±0.01
multilingual
5.56±0.04
5.13±0.12
5.53±0.02
5.04±0.02
summarization
5.38±0.01
4.96±0.07
5.37±0.01
4.96±0.02
math
4.94±0.05
4.54±0.14
4.93±0.02
4.56±0.09
Table 28: Average acceptance length (AL) of LiLiCorr on the SPEED-Bench qualitative split, at draft length 7 , served in vLLM at concurrency 64 with 8,192 -token sliding-window attention, verified by the NVFP4 Nemotron 3.5 Super. temp=0 is greedy decoding, and temp=1 uses probabilistic draft sampling with top- p of 0.95 ; both are reported as the mean and standard deviation over three seeds.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Book level translation scores on the PG19 test set containing 100 books. Models are instructed to translate each book into German, Spanish, French, Italian, Japanese, and Chinese, forming 600 total translations. We then score each translation using the SEGALE ( Wang et al., 2025a ) pipeline.
Side
Language
Records
Tokens
Source
English
4,335
258,667,792
Target
Japanese
785
64,757,505
Target
Chinese (Simplified)
786
55,994,705
Target
Italian
757
51,381,730
Target
Spanish (Mexico)
766
49,770,267
Target
German
728
48,193,156
Appendix
Table 29: Language and token distribution for PG19 long translation data.
Side
Language
Records
Tokens
Source
English (Latin)
1,031
451,571,611
Source
Spanish (Latin)
173
74,851,307
Source
French (Latin)
112
46,515,356
Source
Mandarin Chinese (Han)
85
31,270,147
Source
Japanese
37
16,259,086
Source total
1,438
620,467,507
Appendix
Table 30: Language and token distribution for FinePDFs-Edu long translation data.
Benchmark
Modality
Rollouts
ISL (k)
OSL (k) / rollout
Trunc. %
Baby Vision
vision
388×3
0.5
42.3 → 11.2 ( 0.27 × )
39.0 → 1.9
CharXiv (RQ)
vision
1,000×3
3.1
11.8 → 3.3 ( 0.28 × )
9.2 → 0.6
MMLongBench-Doc
vision
1,091×3
75.0
10.8 → 3.0 ( 0.28 × )
9.9 → 0.4
OCR Reasoning
vision
1,069×3
1.5–1.6 a
16.8 → 5.2 ( 0.31 × )
13.7 → 0.6
OSWorld
vision/agentic
361×3
—
13.6 → 5.1 ( 0.38 × )
1.2 → 0.0
Tau3 Banking
text/agentic
97×8
—
35.2 → 13.7 ( 0.39 × )
20.9 → 0.3
Appendix
Table 31: Output length per rollout and truncation, pre-MOPD versus MOPD, both in BF16. OSL is the total over a rollout’s policy-model calls, excluding the user simulator. Post-training RL removed the non-terminating tail, not baseline verbosity.
pre-MOPD Runaway?
L59–L83 (12) α/β under
BF16
FP4
pre-MOPD BF16
pre-MOPD FP4
MOPD BF16
MOPD FP4
No
No
0.100/0.137
0.097/0.146
0.100/0.142
0.097/0.140
No
Yes
0.091/0.144
0.061 / 0.162
0.092/0.149
0.092/0.147
Yes
No
0.055 / 0.161
0.093/0.139
0.081 /0.164
0.081 /0.161
Yes
Yes
0.059 / 0.151
0.063 / 0.155
0.077 /0.160
0.078 /0.155
Appendix
Table 32: Routed ( α ) and shared ( β ) expert contributions in 12 MoE layers of L59–L83, split by whether each tau3-banking prompt produced a runaway loop under pre-MOPD BF16 rollout versus pre-MOPD FP4 rollout. MOPD rollouts reported 1 runaway loop failure in BF16, and 3 failures in FP4. All MoE layers before L59 agree to within 0.031 regardless of pre-MOPD versus MOPD, and BF16 versus FP4 quantization.
pre-MOPD
MOPD
P(stop) at the stop vs. elsewhere (median)
0.994 vs. 2.0×10−9
0.996 vs. 6.5×10−11
Terminations under FP4 experts
23 / 23 rollouts
24 / 24 rollouts
FP4 experts shift the decision
17/23 away ( p=0.035 )
20/24 toward ( p=0.002 )
Peak P(stop) , runaway rollouts (median)
0.043
0.729
</think> at the peak
2 / 10 rollouts a
6 / 10 rollouts
Appendix
Table 33: The tau3-banking’s stop-versus-continue decision, P(stop) , at lm_head output. Here lm_head stays BF16 to isolate the opposite intervention to Table 39 . The first three rows are measured on the rollouts where </think> was emitted. The last two rows are measured on the rollouts causing a runaway loop in pre-MOPD. Binomial two-sided direction test: p=21−n∑i=kn(in) , k out of n rollouts.
Quantity
Measured
Reading
Keys in representable window
4,730–7,981
population competing
Distinct e4m3 P levels
35–80
levels available
Keys sharing a level
60–120
the dilution
Ties at the top e4m3 level
1–3
the peak stays separated
Attn. score mass preserved
0.994–1.001
mass intact, resolution lost
Spearman ρ (context len., keys/level)
+0.73 – +0.94
more keys per level when longer
Appendix
Table 34: tau3-banking’s score-ladder dilution under an FP8 KV cache, per head, across 12 contexts of 10,532–63,641 prefix tokens. The softmax output P is cast to e4m3 for the PV GEMM; the logits entering the softmax are not.
accuracy
recovery %
context
n
BF16
q/k/v BF16
q/k/v MXFP8
q/k/v FP8
0–8k
290
0.768
97.9
97.5
95.3
8–32k
578
0.552
94.6
95.1
95.5
32–64k
295
0.559
95.0
95.3
91.4
64–131k
307
0.596
93.8
94.3
92.1
Appendix
Table 35: MRCR-128k accuracy recovery against the BF16 reference, by served context, over 1,470 tasks at three repetitions. Context bins use n_tokens , which undercounts the served prompt by about 3% . The o projection is FP8 in every recovery column.
BF16 accuracy by served context
pooled
candidates
0–8k
8–32k
32–64k
64–131k
BF16
recovery %
2
0.883
0.749
0.793
0.823
0.800
94.9
4
0.806
0.585
0.522
0.694
0.637
91.7
8
0.616
0.320
0.353
0.290
0.378
96.0
Appendix
Table 36: MRCR-128k BF16 accuracy by number of near-identical candidates × served context, over 1,470 tasks at three repetitions. The two axes are independent by construction: each context band carries a 33% mix of the three candidate counts, and the candidate strata are length-matched at 36.4 – 37.8 k mean prompt tokens. Recovery is q/k/v/o FP8 against the BF16 reference.
Attention {q,k,v,o} precision
Score
Δ vs BF16 baseline
All four at NVFP4
0.5713
−2.25 pp
All four BF16
0.5900
−0.38 pp
All four FP8
0.5962
+0.25 pp
o FP8, q/k/v BF16
0.6031
+0.94 pp
q/k/v FP8, o BF16
0.6050
+1.12 pp
Appendix
Table 37: Ablation of attention projection precision on AA-LCR. This benchmark does not exercise disambiguation.
Benchmark
FP8 (W8A8)
NVFP4 (W4A16)
ratio
OSWorld
8.3k
10.6k
1.27×
SciCode
12.0k
13.9k
1.16×
Tau3 Banking
14.3k
15.5k
1.08×
AA-Omniscience
6.4k
6.7k
1.06×
AA-LCR
5.7k
5.9k
1.02×
Tau2-AA
5.2k
5.2k
1.01×
Appendix
Table 38: output generation length per rollout for a quantization recipe pair differing in Mamba output projection only: W8A8 versus W4A16. Counts are policy-model output, excluding the user simulator on the agentic benchmarks.
FP8 (W8A16)
NVFP4 (W4A16)
Ratio
Weight error
0.0265
0.0952
3.6 ×
Top-1 agreement with BF16
98.6 %
95.8 %
—
KL mean / p99 (bits)
0.0013 / 0.0080
0.0121 / 0.0797
9.3 × / 10 ×
Stop/continue flip rate
0.781 %
2.450 %
3.14 ×
Appendix
Table 39: Unembedding sensitivity, per-tensor FP8 versus NVFP4, measured on an identical hidden-state input with fake quantization.
Component
Setting
Scored text
user turn (SFT) or prompt (RL), dataset-level boilerplate removed
Verbatim overlap
word 8- and 13-grams; shared n-grams / n-grams of the benchmark question
Embedding model
Qwen3-Embedding-0.6B
Chunking
512-byte windows with 256-byte stride (sample and question)
Semantic overlap
maximum cosine similarity over chunk pairs
Candidate selection
per question: top 20 by combined score + top 2 per signal
Appendix
Table 40: Configuration of the decontamination runs.
Label
Contaminating
Definition given to the judge
direct_copy
yes
reproduces the benchmark question (and/or its answer) verbatim or with only minor edits
paraphrase
yes
restates the same specific problem as the benchmark question, with wording or irrelevant details changed
answer_leakage
yes
solving the training problem yields the benchmark’s answer, e.g. the benchmark is a special case, sub-problem, or key step of the training problem
method_leakage
yes
the benchmark is solved by the same method as the training problem, differing only in surface details or numeric values
style_overlap
no
shares only the topic, domain, or format: general knowledge, not the specific problem
no_contamination
no
unrelated; any similarity is coincidental
Appendix
Table 41: Judge labels. A sample is removed if any of its pairs receives a contaminating label with severity ≥3 .
Terminal-Bench 2.1
GPQA Diamond
Signal (alone)
Recall (%)
False pos.
Recall (%)
False pos.
Verbatim overlap ≥0.5
26.3
0 / 731
25.7
0 / 12,032
Semantic overlap ≥0.85
61.2
0 / 731
47.4
0 / 12,032
LLM judge (severity ≥3 )
87.5
0 / 731
79.1
1 / 12,032
Appendix
Table 42: Synthetic evaluation: recall on transformed benchmark questions and false positives on same-domain negatives, for each signal alone.
Augmentation
Terminal-Bench 2.1 (%)
GPQA Diamond (%)
Verbatim copy
100.0
100.0
Case flipping
100.0
100.0
Burial among other questions
100.0
100.0
Local word reordering
99.7
99.4
LLM paraphrase
97.2
93.2
LLM translation
94.7
79.4
Appendix
Table 43: Synthetic evaluation: recall by augmentation, averaged over strengths, for the union of the three signals (a positive counts as recovered only if it is matched to its own source question).
We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.
We introduce Nemotron-Cascade 2, an open 30B MoE model with 3B activated parameters that delivers best-in-class reasoning and strong agentic capabilities. Despite its compact size, its mathematical and coding reasoning performance approaches that of frontier open models. It is the second open-weight LLM, after DeepSeekV3.2-Speciale-671B-A37B, to achieve Gold Medal-level performance in the 2025 International Mathematical Olympiad (IMO), the International Olympiad in Informatics (IOI), and the ICPC World Finals, demonstrating remarkably high intelligence density with 20x fewer parameters. In contrast to Nemotron-Cascade 1, the key technical advancements are as follows. After SFT on a meticulously curated dataset, we substantially expand Cascade RL to cover a much broader spectrum of reasoning and agentic domains. Furthermore, we introduce multi-domain on-policy distillation from the strongest intermediate teacher models for each domain throughout the Cascade RL process, allowing us to efficiently recover benchmark regressions and sustain strong performance gains along the way. We release the collection of model checkpoint and training data.