Budget Internalization: Fixed-Budget RLVR Trades Test-Time Scaling for In-Budget Accuracy
Organizations: JetBrains Research · Czech Technical University in Prague
Abstract
Reasoning models often become more accurate when they can generate longer answers at test time. This behavior is known as test-time scaling. However, RL with verifiable rewards (RLVR) usually trains them with a fixed rollout cap. We find that this cap changes model behavior, not only training cost. After training, models use few additional tokens beyond the training budget. They outperform the base models near that budget, but their accuracy stops improving and falls below the base model at larger budgets. The rollout cap thus becomes a learned constraint, that limits model's test-time scaling. We call this effect budget internalization. We observe it across Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B and two reasoning training tasks: mathematical reasoning and diverse, procedurally generated problems. This effect also transfers from math training to science and coding tasks. Training-objective level ablations on Qwen3.5-4B show that most changes recover little test-time scaling. Stronger KL regularization and lower training-data truncation preserve more scaling, but stronger KL reduces in-budget accuracy. Forcing an answer on truncated training rollouts largely removes budget internalization, but also removes most of the in-budget gain. SFT on artificially truncated traces reproduces budget internalization, whereas SFT on naturally short traces does not, which shows that the effect is not specific to RL. This result links budget internalization to length-censored post-training rather than to an RL-specific objective term. Thus, a rollout cap does not only control training cost. It can trade low-budget gains for lower performance when additional inference compute is available.
Figures & tables
| Checkpoint | Mean length |
|---|---|
| Qwen3.5-4B base | 27.32 |
| 4k-trained | 4.97 [ 4.71, 5.25 ] |
| 8k-trained | 6.22 [ 5.64, 6.95 ] |
| Gemma4-E2B base | 6.85 |
| 1k-trained | 1.05 [ 0.99, 1.11 ] |
| 2k-trained | 1.90 [ 1.57, 2.48 ] |
| Task | 2k | 4k | 8k | 16k | 32k | 64k |
|---|---|---|---|---|---|---|
| AIME | 100.00 | 100.00 | 95.00 | 66.67 | 33.33 | 6.67 |
| GPQA-Diamond | 100.00 | 95.79 | 67.00 | 16.67 | 0.51 | 0.00 |
| LiveCodeBench v6 | 99.81 | 97.14 | 81.52 | 70.86 | 51.81 | 45.14 |
| Ablation | Variant | 2k | 4k | 8k (train) | 16k | 32k | 64k |
|---|---|---|---|---|---|---|---|
| Base model | 0.0 | 0.3 | 9.2 | 34.2 | 65.4 | 78.8 | |
| GRPO | DeepScaleR | 1.1 | 22.7 | 50.4 | 56.5 | 56.5 | 56.5 |
| Data | GSM8K | 0.0 | 2.8 | 20.7 | 52.7 | 70.6 | 72.0 |
| Nemotron-RL-Math-v2 | 3.4 | 19.9 | 43.3 | 54.0 | 54.6 | 54.6 | |
| Truncation | Loss masking | 1.0 | 16.1 | 38.3 | 56.8 | 59.7 | 59.7 |
| DAPO overlength penalty | 6.4 | 27.2 | 44.2 | 49.7 | 50.0 | 50.0 |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Training caps | Peak LR | Minimum LR | Optimizer | Weight decay | Training backend |
|---|---|---|---|---|---|---|
| Qwen3.5-4B | 4k, 8k | Adam | 0.01 | Megatron | ||
| Gemma4-E2B | 1k, 2k | FusedAdam | 0.1 | FSDP2 | ||
| Nemotron-3-Nano-4B | 1k, 2k | FusedAdam | 0.1 | FSDP2 |
| Model | Temperature | Top- |
|---|---|---|
| Qwen3.5-4B | 1.0 | 20 |
| Gemma4-E2B | 1.0 | 64 |
| Nemotron-3-Nano-4B | 1.0 | Model-card default |
| Dataset | Pass Rate at 64K | Truncation at 8K | Median Output Tokens |
|---|---|---|---|
| GSM8K | 92% | 24% | 3,744 |
| DeepScaleR | 77% | 79% | 19,056 |
| Nemotron-RL-Math-v2 | 31% | 99% | 60,879 |
| Task | Quantity | 4k | 8k | 16k | 32k | 64k |
|---|---|---|---|---|---|---|
| AIME | Total budget (tokens) | 4,216 | 8,312 | 16,504 | 32,888 | 65,656 |
| Accuracy (%) | 0.56 / 9.44 | 5.56 / 21.67 | 34.44 / 45.56 | 65.00 / 69.44 | – / 79.44 | |
| Mean base tokens (k) | 4.22 | 8.18 | 14.70 | 22.86 | – | |
| Mean BF tokens (k) | 4.18 | 8.09 | 14.27 | 21.76 | 25.87 | |
| GPQA-Diamond | Total budget (tokens) | 4,112 | 8,208 | 16,400 | 32,784 | 65,552 |
| Accuracy (%) | 4.04 / 60.27 | 31.31 / 66.84 | 69.53 / 75.76 | 78.11 / 77.27 | 78.28 / 79.46 |
| Checkpoint | Metric | Evaluation budget | |||||
|---|---|---|---|---|---|---|---|
| 2k | 4k | 8k | 16k | 32k | 64k | ||
| Qwen3.5-4B | |||||||
| Base | Accuracy | 0.00 | 1.11 | 7.78 | 38.89 | 65.56 | 78.33 |
| Length | 2.00 | 3.99 | 7.82 | 14.17 | 21.77 | 27.32 | |
| 4k-trained | Accuracy | 17.22 | 35.93 | 44.26 | 47.59 | 48.15 | 48.15 |
| Length | 1.90 | 3.15 | 4.37 | 4.93 | 4.97 | 4.97 | |
| Model | Checkpoint | Evaluation budget | |||||
|---|---|---|---|---|---|---|---|
| 2k | 4k | 8k | 16k | 32k | 64k | ||
| Qwen3.5-4B | Base | 0.00 | 0.00 | 5.00 | 33.33 | 64.44 | 75.56 |
| RG 4k-trained | 11.11 | 31.11 | 43.89 | 50.56 | 50.56 | 50.56 | |
| Task | Checkpoint | Evaluation budget | |||||
|---|---|---|---|---|---|---|---|
| 2k | 4k | 8k | 16k | 32k | 64k | ||
| Qwen3.5-4B | |||||||
| GPQA | Base | 0.00 | 3.70 | 30.30 | 69.02 | 78.11 | 78.28 |
| 4k-trained | 32.66 | 67.17 | 69.19 | 69.19 | 69.19 | 69.19 | |
| 8k-trained | 4.88 | 41.92 | 73.06 | 76.09 | 76.09 | 76.09 | |
| LCB-v6 | Base | 0.00 | 2.67 | 17.52 | 27.81 | 41.90 | 44.38 |
| Model | Task | Evaluation budget | |||||
|---|---|---|---|---|---|---|---|
| 2k | 4k | 8k | 16k | 32k | 64k | ||
| Qwen3.5-4B, RG 4k | GPQA | 26.77 | 59.09 | 71.21 | 71.55 | 71.55 | 71.55 |
| LCB-v6 | 23.62 | 29.71 | 33.52 | 39.43 | 40.57 | 40.57 | |
| Checkpoint | 2k | 4k | 8k | 16k | 32k | 64k |
|---|---|---|---|---|---|---|
| AIME | ||||||
| Base | 100.00 | 100.00 | 95.00 | 66.67 | 33.33 | 6.67 |
| 4k-trained | 85.56 | 22.22 | 1.11 | 1.11 | 1.11 | 1.11 |
| 8k-trained | 98.33 | 75.56 | 33.33 | 0.00 | 0.00 | 0.00 |
| GPQA-Diamond | ||||||
| Base | 100.00 | 95.79 | 67.00 | 16.67 | 0.51 | 0.00 |
| Task | Quantity | 1k | 2k | 4k | 8k |
|---|---|---|---|---|---|
| AIME | Total budget (tokens) | 1,144 | 2,168 | 4,216 | 8,312 |
| Accuracy (%) | 0.00 / 3.33 | 2.78 / 17.78 | 17.22 / 30.56 | 35.00 / 37.22 | |
| Mean base tokens (k) | 1.14 | 2.16 | 4.03 | 6.29 | |
| Mean BF tokens (k) | 1.04 | 2.06 | 3.95 | 6.19 | |
| GPQA-Diamond | Total budget (tokens) | 1,040 | 2,064 | 4,112 | 8,208 |
| Accuracy (%) | 9.09 / 22.56 | 30.98 / 28.28 | 40.57 / 37.54 | 40.57 / 42.93 |
| Task | Quantity | 2k | 4k | 8k | 16k |
|---|---|---|---|---|---|
| AIME | Total budget (tokens) | 2,168 | 4,216 | 8,312 | 16,504 |
| Accuracy (%) | 26.11 / 30.56 | 42.22 / 39.44 | 55.00 / 55.00 | 60.00 / 62.22 | |
| Mean base tokens (k) | 1.95 | 3.16 | 4.50 | 5.52 | |
| Mean BF tokens (k) | 1.94 | 3.22 | 4.52 | 5.83 | |
| GPQA-Diamond | Total budget (tokens) | 2,064 | 4,112 | 8,208 | 16,400 |
| Accuracy (%) | 14.48 / 43.10 | 27.95 / 49.83 | 37.71 / 49.16 | 44.11 / 51.85 |
| Condition | Seed | Step | Accuracy (%) | Mean response length (k tokens) | ||||||||||||||
| 512 | 1k | 2k | 4k | 8k | 16k | 32k | 64k | 512 | 1k | 2k | 4k | 8k | 16k | 32k | 64k | |||
| Qwen 4k | 42 | 290 | 0.00 | 0.00 | 11.67 | 33.89 | 42.78 | 46.11 | 46.67 | 46.67 | 0.51 | 1.00 | 1.95 | 3.27 | 4.58 | 5.18 | 5.25 | 5.25 |
| 43 | 290 | 0.00 | 0.00 | 18.89 | 36.11 | 43.33 | 45.56 | 46.11 | 46.11 | 0.51 | 1.00 | 1.90 | 3.13 | 4.20 | 4.68 | 4.71 | 4.71 | |
| 44 | 300 | 0.00 | 3.89 | 21.11 | 37.78 | 46.67 | 51.11 | 51.67 | 51.67 | 0.51 | 0.99 | 1.86 | 3.06 | 4.34 | 4.93 | 4.95 | 4.95 | |
| Qwen 8k | 42 | 290 | 0.00 | 0.00 | 2.22 | 27.78 | 48.89 | 50.56 | 50.56 | 50.56 | 0.51 | 1.00 | 1.99 | 3.61 | 4.95 | 5.10 | 5.28 | 5.64 |
| 43 | 160 | 0.00 | 0.00 | 0.56 | 19.44 | 47.78 | 55.00 | 55.00 | 55.00 | 0.51 | 1.00 | 2.00 | 3.84 | 6.29 | 6.95 | 6.95 | 6.95 | |
| Condition | Seed | Step | Accuracy (%) | Mean response length (k tokens) | ||||||||||||||
| 512 | 1k | 2k | 4k | 8k | 16k | 32k | 64k | 512 | 1k | 2k | 4k | 8k | 16k | 32k | 64k | |||
| Qwen 4k | 42 | 300 | 0.00 | 0.00 | 16.67 | 34.44 | 43.89 | 47.78 | 47.78 | 47.78 | 0.51 | 1.00 | 1.93 | 3.25 | 4.67 | 5.34 | 5.44 | 5.44 |
| 43 | 300 | 0.00 | 0.56 | 22.78 | 41.67 | 47.78 | 50.00 | 50.56 | 50.56 | 0.51 | 1.00 | 1.91 | 3.08 | 4.26 | 4.72 | 4.76 | 4.76 | |
| 44 | 300 | 0.00 | 3.89 | 21.11 | 37.78 | 46.67 | 51.11 | 51.67 | 51.67 | 0.51 | 0.99 | 1.86 | 3.06 | 4.34 | 4.93 | 4.95 | 4.95 | |
| Qwen 8k | 42 | 300 | 0.00 | 0.00 | 2.78 | 27.78 | 55.56 | 55.56 | 55.56 | 55.56 | 0.51 | 1.00 | 1.99 | 3.61 | 4.95 | 5.02 | 5.11 | 5.28 |
| 43 | 300 | 0.00 | 0.00 | 0.00 | 18.33 | 50.00 | 53.89 | 53.89 | 53.89 | 0.51 | 1.00 | 2.00 | 3.83 | 6.11 | 6.47 | 6.47 | 6.47 | |
Test-Time Scaling
Test-time scaling studies how extra inference-time computation can improve model answers, decisions, and generated outputs.
See all 244 papers