cs.LGNov 19, 2026

Budget Internalization: Fixed-Budget RLVR Trades Test-Time Scaling for In-Budget Accuracy

Authors: Ilia Zavidnyi, Alexander Kovrigin, Arina Puchkova, Kirill Gelvan, Sergey Titov

Organizations: JetBrains Research · Czech Technical University in Prague

Abstract

Reasoning models often become more accurate when they can generate longer answers at test time. This behavior is known as test-time scaling. However, RL with verifiable rewards (RLVR) usually trains them with a fixed rollout cap. We find that this cap changes model behavior, not only training cost. After training, models use few additional tokens beyond the training budget. They outperform the base models near that budget, but their accuracy stops improving and falls below the base model at larger budgets. The rollout cap thus becomes a learned constraint, that limits model's test-time scaling. We call this effect budget internalization. We observe it across Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B and two reasoning training tasks: mathematical reasoning and diverse, procedurally generated problems. This effect also transfers from math training to science and coding tasks. Training-objective level ablations on Qwen3.5-4B show that most changes recover little test-time scaling. Stronger KL regularization and lower training-data truncation preserve more scaling, but stronger KL reduces in-budget accuracy. Forcing an answer on truncated training rollouts largely removes budget internalization, but also removes most of the in-budget gain. SFT on artificially truncated traces reproduces budget internalization, whereas SFT on naturally short traces does not, which shows that the effect is not specific to RL. This result links budget internalization to length-censored post-training rather than to an RL-specific objective term. Thus, a rollout cap does not only control training cost. It can trade low-budget gains for lower performance when additional inference compute is available.

Figures & tables

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. From Base Rollouts to RL Reasoning: A Budgeted Search Perspective

    Sep 1, 2026Wenhe Sun, Cunxiang Wang, Zijun Yao +1Inference-Time SearchRL for Language Model Reasoning

  2. DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards

    May 8, 2026Haoyu Hu, Xuandong Zhao, Xuhai "Orson'' Xu +1Reinforcement LearningGroup Relative Policy Optimization