Reasoning models often become more accurate when they can generate longer answers at test time. This behavior is known as test-time scaling. However, RL with verifiable rewards (RLVR) usually trains them with a fixed rollout cap. We find that this cap changes model behavior, not only training cost. After training, models use few additional tokens beyond the training budget. They outperform the base models near that budget, but their accuracy stops improving and falls below the base model at larger budgets. The rollout cap thus becomes a learned constraint, that limits model's test-time scaling. We call this effect budget internalization. We observe it across Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B and two reasoning training tasks: mathematical reasoning and diverse, procedurally generated problems. This effect also transfers from math training to science and coding tasks. Training-objective level ablations on Qwen3.5-4B show that most changes recover little test-time scaling. Stronger KL regularization and lower training-data truncation preserve more scaling, but stronger KL reduces in-budget accuracy. Forcing an answer on truncated training rollouts largely removes budget internalization, but also removes most of the in-budget gain. SFT on artificially truncated traces reproduces budget internalization, whereas SFT on naturally short traces does not, which shows that the effect is not specific to RL. This result links budget internalization to length-censored post-training rather than to an RL-specific objective term. Thus, a rollout cap does not only control training cost. It can trade low-budget gains for lower performance when additional inference compute is available.
Figures & tables
Figure 1: Budget internalization after fixed-budget math RLVR. Trained points and labels are means over three validation-selected checkpoints; shading shows the seed minimum–maximum. Blue curves are base-model evaluations. Appendix I reports selected and final results for each seed.
Checkpoint
Mean length
Qwen3.5-4B base
27.32
4k-trained
4.97 [ 4.71, 5.25 ]
8k-trained
6.22 [ 5.64, 6.95 ]
Gemma4-E2B base
6.85
1k-trained
1.05 [ 0.99, 1.11 ]
2k-trained
1.90 [ 1.57, 2.48 ]
Table 1: Mean AIME output length at each model’s largest budget in Figure 1 (Qwen: 64k; Gemma and Nemotron: 16k). Trained entries are seed mean [min, max]; bases use one evaluation.
Figure 2: AIME accuracy after fixed-budget Reasoning Gym training across three model families, with dashed lines for trained checkpoints and colors matching Figure 1 . Appendix Table 9 reports the exact values.
Figure 3: Accuracy on GPQA-Diamond and LiveCodeBench v6 after fixed-budget math (solid budget colors) and Reasoning Gym (brown dashed diamonds) training. Appendix Tables 10 and 11 report exact values; Appendix Figure 21 reports DeepScaleR output lengths.
Task
2k
4k
8k
16k
32k
64k
AIME
100.00
100.00
95.00
66.67
33.33
6.67
GPQA-Diamond
100.00
95.79
67.00
16.67
0.51
0.00
LiveCodeBench v6
99.81
97.14
81.52
70.86
51.81
45.14
Table 2: Qwen3.5-4B base-model truncation rate (%) across evaluation budgets. Appendix E reports the corresponding curves and exact values for every model family and fixed-budget checkpoint.
Figure 4: Qwen3.5-4B accuracy under ordinary base-model decoding, base-model answer forcing, the 4k-trained checkpoint, and forcing the trained checkpoint upward. Appendix Figure 18 and Table 7 report truncation rates and output lengths.
Ablation
Variant
2k
4k
8k (train)
16k
32k
64k
Base model
0.0 ±0.0
0.3 ±0.6
9.2 ±6.2
34.2 ±10.8
65.4 ±10.1
78.8 ±8.5
GRPO
DeepScaleR
1.1 ±1.7
22.7 ±9.8
50.4 ±10.4
56.5 ±10.0
56.5 ±10.0
56.5 ±10.0
Data
GSM8K
0.0 ±0.0
2.8 ±2.7
20.7 ±9.2
52.7 ±11.2
70.6 ±9.4
72.0 ±9.2
Nemotron-RL-Math-v2
3.4 ±2.5
19.9 ±8.0
43.3 ±10.1
54.0 ±9.4
54.6 ±9.5
54.6 ±9.5
Truncation
Loss masking
1.0 ±1.0
16.1 ±7.6
38.3 ±10.7
56.8 ±9.9
59.7 ±9.4
59.7 ±9.4
DAPO overlength penalty
6.4 ±4.2
27.2 ±9.9
44.2 ±10.5
49.7 ±10.2
50.0 ±10.2
50.0 ±10.2
Table 3: AIME avg@16 accuracy (%) of Qwen3.5-4B after GRPO ablations at an 8k training budget and after rejection fine-tuning. Gray ± values are 95% confidence intervals across the 60 problem-level avg@16 scores. Green and red values improve or degrade relative to the base model; bold and underlined values are best and second best in each column.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Training caps
Peak LR
Minimum LR
Optimizer
Weight decay
Training backend
Qwen3.5-4B
4k, 8k
5×10−7
5×10−8
Adam
0.01
Megatron
Gemma4-E2B
1k, 2k
3×10−6
3×10−7
FusedAdam
0.1
FSDP2
Nemotron-3-Nano-4B
1k, 2k
2×10−6
2×10−7
FusedAdam
0.1
FSDP2
Appendix
Table 4: Hyperparameters for the headline fixed-budget GRPO runs. Settings not shown in the table are shared across model families as described above.
Model
Temperature
Top- k
Qwen3.5-4B
1.0
20
Gemma4-E2B
1.0
64
Nemotron-3-Nano-4B
1.0
Model-card default
Appendix
Table 5: Evaluation sampling hyperparameters from the corresponding Hugging Face model cards ( Qwen Team, 2026a ; Gemma Team, 2026 ; NVIDIA, 2026 ) . For Qwen, we use the recommended reasoning parameters across all benchmarks.
Figure 5: AIME avg@16 response curves for the Qwen3.5-4B 8k training ablations. The plot includes the base model, the standard 8k GRPO run, and all eight completed ablations. The two KL runs use β=0.001 and β=0.01 . The vertical line marks the 8k training budget. Table 3 reports the exact values.
Dataset
Pass Rate at 64K
Truncation at 8K
Median Output Tokens
GSM8K
92%
24%
3,744
DeepScaleR
77%
79%
19,056
Nemotron-RL-Math-v2
31%
99%
60,879
Appendix
Table 6: Difficulty and length statistics for the math training datasets, measured with Qwen3.5-4B in thinking mode. Truncation at 8K is the share of responses that do not finish within 8K tokens.
Figure 6: Budget-cue ablation under a shared 64k generation cap. Left: no cue versus a 4k cue for the base model and 4k cue-conditioned checkpoint. Right: no cue versus a 16k cue for the base model and 16k cue-conditioned checkpoint. The stated budget changes, but the available generation budget does not.
Figure 7: Base-model placement ablation at a shared 32k generation cap. Columns show 4k- and 16k-token budget cues; rows show combined AIME avg@3 and mean output tokens. The no-cue control is shared across cue placements.
Figure 8: Matched-placement ablation for separately trained 4k cue-conditioned checkpoints at a shared 32k generation cap. Within each placement, the base model and the checkpoint trained with that placement are evaluated with no cue (blue) or a 4k budget cue (green). Rows show combined AIME avg@3 and mean output tokens.
Figure 18: Termination diagnostics and cumulative generation length for ordinary base-model decoding, base-model answer forcing, the 4k-trained checkpoint, and forcing the trained checkpoint upward with Wait . For answer forcing, truncation includes continuations shortened at the stated answer window and saved final calls that exhaust their limit; length counts both stages. Answer forcing receives an additional answer window at each nominal reasoning setting; Table 7 gives the comparison at matched total caps.
Task
Quantity
4k
8k
16k
32k
64k
AIME
Total budget (tokens)
4,216
8,312
16,504
32,888
65,656
Accuracy (%)
0.56 / 9.44
5.56 / 21.67
34.44 / 45.56
65.00 / 69.44
– / 79.44
Mean base tokens (k)
4.22
8.18
14.70
22.86
–
Mean BF tokens (k)
4.18
8.09
14.27
21.76
25.87
GPQA-Diamond
Total budget (tokens)
4,112
8,208
16,400
32,784
65,552
Accuracy (%)
4.04 / 60.27
31.31 / 66.84
69.53 / 75.76
78.11 / 77.27
78.28 / 79.46
Appendix
Table 7: Qwen3.5-4B base decoding and answer forcing at matched total generated-token budgets. Columns identify the nominal answer-forcing reasoning setting; the exact total cap is shown for each task. Accuracy entries list ordinary base decoding / base answer forcing in percent; separate rows report mean generated tokens in thousands. Answer-forcing lengths include both the reasoning prefix and the answer continuation. A dash marks the unavailable AIME base comparison at a total budget of 65,656 tokens: the saved base run was capped at 65,536 and still contained unfinished responses. LCB base means are ranges bounding two API-error attempts with missing usage between zero and the total token budget each.
Checkpoint
Metric
Evaluation budget
2k
4k
8k
16k
32k
64k
Qwen3.5-4B
Base
Accuracy
0.00
1.11
7.78
38.89
65.56
78.33
Length
2.00
3.99
7.82
14.17
21.77
27.32
4k-trained
Accuracy
17.22
35.93
44.26
47.59
48.15
48.15
Length
1.90
3.15
4.37
4.93
4.97
4.97
Appendix
Table 8: Exact means underlying Figure 1 . Trained values average three validation-selected training seeds; base values use one evaluation. Accuracy is in percent and mean output length in thousands of tokens. Per-seed results for both checkpoint choices appear in Appendix I .
Table 11: Exact GPQA-Diamond and LiveCodeBench v6 accuracy values for the Reasoning Gym checkpoints in Figure 3 , reported in percent.
Figure 20: Truncation rates across evaluation budgets on AIME, GPQA-Diamond, and LiveCodeBench v6. Fixed-budget training substantially reduces truncation, particularly near and above the training budget.
Figure 21: Mean output-length response curves on GPQA-Diamond and LiveCodeBench v6 after fixed-budget mathematical-reasoning training.
Figure 22: Pass@3 response curves for Qwen3.5-4B on AIME, GPQA-Diamond, and LiveCodeBench v6.
Figure 23: Pass@3 response curves for Gemma4-E2B on AIME, GPQA-Diamond, and LiveCodeBench v6.
Figure 24: Pass@3 response curves for Nemotron-3-Nano-4B on AIME, GPQA-Diamond, and LiveCodeBench v6.
Figure 25: Gemma4-E2B accuracy versus total generated-token allowance, including the additional answer window for answer forcing. Each panel compares ordinary base decoding, base answer forcing, and the 2k-trained checkpoint; ordinary curves include the matched total caps reported in Table 13 .
Task
Quantity
1k
2k
4k
8k
AIME
Total budget (tokens)
1,144
2,168
4,216
8,312
Accuracy (%)
0.00 / 3.33
2.78 / 17.78
17.22 / 30.56
35.00 / 37.22
Mean base tokens (k)
1.14
2.16
4.03
6.29
Mean BF tokens (k)
1.04
2.06
3.95
6.19
GPQA-Diamond
Total budget (tokens)
1,040
2,064
4,112
8,208
Accuracy (%)
9.09 / 22.56
30.98 / 28.28
40.57 / 37.54
40.57 / 42.93
Appendix
Table 13: Gemma4-E2B ordinary base decoding and base answer forcing at matched total generated-token budgets. Columns give the nominal reasoning setting; accuracy entries list base / answer forcing (%), and mean lengths count all generated output in thousands of tokens. The answer allowance is 120 tokens for AIME, 16 for GPQA, and 1,024 for LiveCodeBench; saved answer continuations are shortened and rescored as described in Appendix C.1 .
Figure 26: Nemotron-3-Nano-4B accuracy versus total generated-token allowance, including the additional answer window for answer forcing. Each panel compares ordinary base decoding, base answer forcing, and the 2k-trained checkpoint; ordinary curves include the matched total caps reported in Table 14 .
Task
Quantity
2k
4k
8k
16k
AIME
Total budget (tokens)
2,168
4,216
8,312
16,504
Accuracy (%)
26.11 / 30.56
42.22 / 39.44
55.00 / 55.00
60.00 / 62.22
Mean base tokens (k)
1.95
3.16
4.50
5.52
Mean BF tokens (k)
1.94
3.22
4.52
5.83
GPQA-Diamond
Total budget (tokens)
2,064
4,112
8,208
16,400
Accuracy (%)
14.48 / 43.10
27.95 / 49.83
37.71 / 49.16
44.11 / 51.85
Appendix
Table 14: Nemotron-3-Nano-4B ordinary base decoding and base answer forcing at matched total generated-token budgets, using the same accounting and accuracy convention as Table 13 .
Condition
Seed
Step
Accuracy (%)
Mean response length (k tokens)
512
1k
2k
4k
8k
16k
32k
64k
512
1k
2k
4k
8k
16k
32k
64k
Qwen 4k
42
290
0.00
0.00
11.67
33.89
42.78
46.11
46.67
46.67
0.51
1.00
1.95
3.27
4.58
5.18
5.25
5.25
43
290
0.00
0.00
18.89
36.11
43.33
45.56
46.11
46.11
0.51
1.00
1.90
3.13
4.20
4.68
4.71
4.71
44
300
0.00
3.89
21.11
37.78
46.67
51.11
51.67
51.67
0.51
0.99
1.86
3.06
4.34
4.93
4.95
4.95
Qwen 8k
42
290
0.00
0.00
2.22
27.78
48.89
50.56
50.56
50.56
0.51
1.00
1.99
3.61
4.95
5.10
5.28
5.64
43
160
0.00
0.00
0.56
19.44
47.78
55.00
55.00
55.00
0.51
1.00
2.00
3.84
6.29
6.95
6.95
6.95
Appendix
Table 15: Validation-selected checkpoints for every training seed. Accuracy is AIME avg@3 in percent; mean response length is in thousands of tokens at each evaluation budget. Lengths are capped at the stated budget and averaged over all 180 responses. Step is the optimizer checkpoint step.
Condition
Seed
Step
Accuracy (%)
Mean response length (k tokens)
512
1k
2k
4k
8k
16k
32k
64k
512
1k
2k
4k
8k
16k
32k
64k
Qwen 4k
42
300
0.00
0.00
16.67
34.44
43.89
47.78
47.78
47.78
0.51
1.00
1.93
3.25
4.67
5.34
5.44
5.44
43
300
0.00
0.56
22.78
41.67
47.78
50.00
50.56
50.56
0.51
1.00
1.91
3.08
4.26
4.72
4.76
4.76
44
300
0.00
3.89
21.11
37.78
46.67
51.11
51.67
51.67
0.51
0.99
1.86
3.06
4.34
4.93
4.95
4.95
Qwen 8k
42
300
0.00
0.00
2.78
27.78
55.56
55.56
55.56
55.56
0.51
1.00
1.99
3.61
4.95
5.02
5.11
5.28
43
300
0.00
0.00
0.00
18.33
50.00
53.89
53.89
53.89
0.51
1.00
2.00
3.83
6.11
6.47
6.47
6.47
Appendix
Table 16: Final step-300 checkpoints for every training seed. Accuracy and mean response length at each evaluation budget use the same definitions and units as Table 15 .