Within the landscape of inference-time scaling methods for foundation models, a width-based approach to scaling--which involves the insertion of <pause> tokens in the input stream to delay model responses--offers a unique advantage by increasing model expressivity while remaining highly parallelizable at both training and inference. The existing literature on training models to utilize <pause> tokens relies on the standard cross-entropy objective in which the model output is read out and evaluated only at the final step of a pause sequence. This approach provides no mechanism for the model to regulate its own processing or to signal readiness to respond, treating the additional compute steps as a barrier to skip over rather than a resource to be used adaptively. We propose a supervised loss, Catch Your Breath (CYB), framed as a sequential-decision problem, that trains a model to dynamically and autonomously scale the number of compute steps used for each input token. The model indicates the need for additional compute steps by emitting a special <don't know> output, delaying its response via a pause. The model can abstain multiple times to obtain longer delays. Our experiments demonstrate that when <pause> tokens are included in the input stream, pretraining or fine-tuning with CYB significantly outperforms standard cross-entropy, reducing perplexity and enhancing downstream accuracy with no additional computational or memory cost.
Figures & tables
Figure 1: (a) Sequence processing in a transformer with <don’t know> outputs that trigger <pause> inputs. (b) The CYB loss yields variable per-token computing steps. When the <don’t know> probability is high, additional computing steps are provided. Shading indicates the expected number of computing steps to predict the corresponding token. Spacing reflects token boundaries.
Figure 2: Catch Your Breath: A sequential-decision framework to obtain a single output token. The blue boxes represent model abstention decisions, the purple boxes model choices of output tokens. The orange boxes are choices made by the environment about whether the model is granted a delay in responding. The green and red boxes represent end-state rewards. The unconditional stop-probability at step i , ωi , has been rewritten as a hazard probability ω~i=ωi/∑j=iMωj .
Figure 3: C4 evaluation set perplexity (lower is better) for pretrained Gemma1-2B ( left ) and Gemma3-4B ( right ) models (black bar) and various models fine-tuned on the C4 train set. Model variants include a baseline condition, where no modification is made to training objective or token sequence (brown bar), Think Before You Speak (TBYS) with 1, 2, and 3 pause tokens after each input token (green bars), and the CYB with 1, 2, and 3 pause tokens.
Figure 4
Figure 6: C4 evaluation set perplexity for various fine-tuned Gemma1-2B models based on a given data regime. 1 × 1 and 2 × 1 refer to 1 and 2 passes through the fine-tuning set, respectively. 1 × 2 and 1 × 3 refer to a single pass through 2 or 3 times the dataset size of 1 × 1. The dashed lines indicate the perplexity of each model in the 1 × 3 training regime, except for Baseline 4 × seq where we only report performance in the 1 × 1 regime.
Figure 7: Latency distribution for four hyperparameter settings of CYB. Blue histograms show the distribution over evaluation tokens of the expected latency (the number of pause steps utilized). Purple bar graphs show the latency distribution averaged across tokens.
Method
HellaSwag
MMLU
GSM8K
(10-shot)
(5-shot)
(8-shot)
Baseline
24.68
49.16
47.60
TBYS
24.74
52.70
53.16 [SEM 0.23]
CYB
27.94
54.00
53.91 [SEM 0.30]
Table 1: Downstream evaluation top-1 accuracy (%). HellaSwag and MMLU are evaluated on C4 fine-tuned checkpoints. GSM8K results are means and standard errors over 10 replications in which we additionally fine tune on the GSM8K training set.
Low Median
High Median
Low Variance
isn, wasn, didn, doesn, don, according, able, However, plenty, etc, Inc, &, non, addition
g, e, to, entry, S, won, co, New, Y, V, F, Z, World, of, Air
materials, images, @, site, pictures, photos, sites, code, College, wood, page, file, professionals, is, homes
Table 2: Statistics of pause counts for selected tokens following CYB training
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Gaze fixations of a human reader. Indices indicate order of gaze within a sentence along with fixation duration (in msec). Reprinted from Just and Carpenter ( 1980 , Figure 1).
Term
Meaning
M
maximum number of steps granted by world for any token
S
r.v. denoting # steps at which model generates result (either by world or model <don’t know> ), 1≤S≤M
ωi
prob. that world terminates processing at step i
di
prob. of model abstention ( <don’t know> ) at step i
si
prob. of outputting result (self- or world-induced) at step i ; shorthand for Pr(S=i)
ti
prob. that the model selects ground-truth token at step i
Appendix
Table B.1 : Notation Used in the Main Article
Figure C.2 : C4 evaluation set perplexity for various hyperparameter settings of CYB and two variants, CYB-VA and CYB-DP, on Gemma1-2B fine-tuning experiment. The black dashed line indicates the perplexity of CYB. Hyperparameters ρ refer to the distribution of stopping times and are indicated by ratios for steps 1-4 (i.e., the initial input token and the following 3 pause tokens). Hyperparameter α is a penalty term associated with CYB-DP. See Table B.1 for notation.
Figure C.3 : Distribution of stopping times for CYB-VA. Blue histograms show the distribution over evaluation tokens of the expected number of pause steps. Purple bar graphs show the distribution over the number of pause steps across tokens.
Figure C.4 : Distribution of stopping times for CYB-DP with α=0.1 and α=0.01 . Blue histograms show the distribution over evaluation tokens of the expected number of pause steps. Purple bar graphs show the distribution over the number of pause steps across tokens.
Figure E.5 : Density plots indicating that <don’t know> probability is positively correlated with improvement in accuracy, for steps 1, 2, and 3.
Method
Perplexity
Rel. Improvement (%)
Pretrained Gemma3-4B
11.765
-13.57%
Baseline
10.670
-3.00%
Baseline 4 × seq
10.694
-3.24%
TBYS 1 pause
10.512
-1.48%
TBYS 2 pauses
10.422
-0.61%
TBYS 3 pauses
10.359
—
Appendix
Table F.2 : Results for fine-tuned (on C4) Gemma3-4B . Perplexity on eval set of C4. For relative improvement, we use the formula ( 5 ). In bold we highlight the best result – the lowest perplexity and the highest relative improvement.
Method
Perplexity
Rel. Improvement (%)
Pretrained Gemma1-2B
13.018
-8.64%
Baseline
12.173
-1.58%
Baseline 4 × seq
11.953
+0.25%
TBYS 1 pause
12.098
-0.96%
TBYS 2 pauses
12.024
-0.34%
TBYS 3 pauses
11.983
—
Appendix
Table F.3: Results for fine-tuned (on C4) Gemma1-2B . Perplexity on eval set of C4. For relative improvement, we use the formula ( 5 ). In bold we highlight the best result – the lowest perplexity and the highest relative improvement.
Method
Perplexity
Rel. Improvement (%)
Pretrained Gemma1-2B
13.018
-4.14%
Baseline
12.893
-3.14%
TBYS 3 pauses
12.500
—
CYB 3 pauses
12.054
+3.57%
Appendix
Table F.4 : Results for trained from scratch (on C4) Gemma1-2B . Perplexity on eval set of C4. For relative improvement, we use the formula ( 5 ). In bold we highlight the best result – the lowest perplexity and the highest relative improvement.
Figure F.6 : Model trained with CYB loss produces variable pause durations on tokens.
Figure F.7 : Model trained with CYB loss produces variable pause durations on tokens.
Test-time scaling via recurrent/iterative Transformers enables large language models to spend more computation at inference, but most pretrained recurrent LMs run a fixed number of iterations, wasting compute on easy tokens and lacking token-wise adaptivity. Following the core idea of Adaptive Computation Time(ACT) and Early Exit(EE), we propose AdaPonderLM, a self-supervised recurrent language model that learns token-wise early exiting during pretraining without manually tuned per-token/per-layer pruning ratios. AdaPonderLM uses iteration-specific MLP gates with a monotonic halting mask to decide when each token stops recurring, and introduces a KV reuse mechanism that reuses cached key/value states for halted tokens, ensuring train--test consistency and practical acceleration. Across Pythia backbones from 70M to 410M (pretraining) and up to 2.8B (continued pretraining), AdaPonderLM reduces inference compute at about 10% while maintaining comparable language modeling perplexity and competitive downstream accuracy. Our analysis shows the learned gates allocate more computation to high-NLL (hard) tokens, exhibiting adaptive computation time behavior in a fully self-supervised setting. Meanwhile, under iso-FLOPs, the learned halting policy consistently outperforms fixed pruning, showing AdaPonderLM allocates compute to the right tokens rather than just reducing average depth.
Shixiang Song, He Li, Zitong Wang +6
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Sun Yat-sen University +2
Test-time scaling has shown that allocating more additional computation at inference can improve generation quality, motivating a natural follow-up question: where should this computation be spent? Building on this insight, we introduce PonderLM-3, a pretraining framework for token-wise adaptive pondering that learns to selectively allocate additional computation under purely self-supervised objectives, built on top of the PonderLM-2 backbone. This makes additional inference computation an allocatable per-token resource, so tokens receive more computation only when it is beneficial, rather than paying a uniform extra cost. To make this allocation learnable while maintaining train-inference consistency, PonderLM-3 injects a differentiable attention mask during pretraining and pairs it with a matching hard pruning rule at inference. PonderLM-3 defines a stronger Pareto frontier: compared with existing recursive or adaptive baselines, it achieves lower pretraining perplexity at equal inference FLOPs. On downstream benchmarks, PonderLM-3 attains comparable performance to fixed-step PonderLM-2 under the same maximum number of additional computation steps, while using fewer inference FLOPs in practice. Overall, PonderLM-3 provides an end-to-end differentiable and train-inference consistent framework for token-wise adaptive computation, enabling additional inference compute to be allocated where it is most useful rather than paid uniformly by every token.
He Li, Feichen Song, Boyi Zeng +4
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai AI Laboratory · Shanghai Jiao Tong University +1
Pause-token methods improve LLM reasoning by inserting special tokens into sequences. Prior work explains these gains through computational expressivity. However, there is relatively little investigation into the training dynamics of pause tokens. We explore how pause tokens reshape the training dynamics of fine-tuning. Two controlled pilots expose distinct asymmetries. On a synthetic continual-learning task, masked pauses overwrite a previously-learned distribution roughly 4x less at matched final adaptation (H1, mode retention); on a synthetic math-reasoning probe, the boundary-adjacent token comes to encode substantially more downstream-step information (H2, non-myopic compression). We formalize a training rule consistent with both - Masked Boundary Pause (MBP), pause tokens placed at reasoning-step boundaries with their loss masked. Across 1B-8B Qwen and Llama models, MBP consistently improves reasoning, achieving gains of up to 6 points on math and 2.5 points on code, while preserving general language understanding abilities. We further demonstrate that this mode-preserving strategy extend gains to GRPO. These results recast pause tokens as a training-dynamics intervention on the retention-adaptation trade-off, rather than merely an inference-time computation device.
Jaehyeon Kim, Suhwan Kim, Nakyung Lee +4
HodooAI Lab, Seoul, South Korea · Seoul National University, Seoul, South Korea