Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
Organizations: Google DeepMind
Abstract
Within the landscape of inference-time scaling methods for foundation models, a width-based approach to scaling--which involves the insertion of <pause> tokens in the input stream to delay model responses--offers a unique advantage by increasing model expressivity while remaining highly parallelizable at both training and inference. The existing literature on training models to utilize <pause> tokens relies on the standard cross-entropy objective in which the model output is read out and evaluated only at the final step of a pause sequence. This approach provides no mechanism for the model to regulate its own processing or to signal readiness to respond, treating the additional compute steps as a barrier to skip over rather than a resource to be used adaptively. We propose a supervised loss, Catch Your Breath (CYB), framed as a sequential-decision problem, that trains a model to dynamically and autonomously scale the number of compute steps used for each input token. The model indicates the need for additional compute steps by emitting a special <don't know> output, delaying its response via a pause. The model can abstain multiple times to obtain longer delays. Our experiments demonstrate that when <pause> tokens are included in the input stream, pretraining or fine-tuning with CYB significantly outperforms standard cross-entropy, reducing perplexity and enhancing downstream accuracy with no additional computational or memory cost.
Figures & tables
| Method | HellaSwag | MMLU | GSM8K |
| (10-shot) | (5-shot) | (8-shot) | |
| Baseline | 24.68 | 49.16 | 47.60 |
| TBYS | 24.74 | 52.70 | 53.16 [SEM 0.23] |
| CYB | 27.94 | 54.00 | 53.91 [SEM 0.30] |
| Low Median | High Median | |
| Low Variance | isn, wasn, didn, doesn, don, according, able, However, plenty, etc, Inc, &, non, addition | devices, players, students, projects, applications, challenges, environment, systems, stories, games, groups, events, patients, families, teams |
| High Variance | g, e, to, entry, S, won, co, New, Y, V, F, Z, World, of, Air | materials, images, @, site, pictures, photos, sites, code, College, wood, page, file, professionals, is, homes |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Term | Meaning |
| maximum number of steps granted by world for any token | |
| r.v. denoting # steps at which model generates result (either by world or model <don’t know> ), | |
| prob. that world terminates processing at step | |
| prob. of model abstention ( <don’t know> ) at step | |
| prob. of outputting result (self- or world-induced) at step ; shorthand for | |
| prob. that the model selects ground-truth token at step |
| Method | Perplexity | Rel. Improvement (%) |
| Pretrained Gemma3-4B | 11.765 | -13.57% |
| Baseline | 10.670 | -3.00% |
| Baseline 4 seq | 10.694 | -3.24% |
| TBYS 1 pause | 10.512 | -1.48% |
| TBYS 2 pauses | 10.422 | -0.61% |
| TBYS 3 pauses | 10.359 | — |
| Method | Perplexity | Rel. Improvement (%) |
| Pretrained Gemma1-2B | 13.018 | -8.64% |
| Baseline | 12.173 | -1.58% |
| Baseline 4 seq | 11.953 | +0.25% |
| TBYS 1 pause | 12.098 | -0.96% |
| TBYS 2 pauses | 12.024 | -0.34% |
| TBYS 3 pauses | 11.983 | — |
| Method | Perplexity | Rel. Improvement (%) |
| Pretrained Gemma1-2B | 13.018 | -4.14% |
| Baseline | 12.893 | -3.14% |
| TBYS 3 pauses | 12.500 | — |
| CYB 3 pauses | 12.054 | +3.57% |
Adaptive Computation
Adaptive computation allocates model depth, tokens, attention, or sensing effort according to input difficulty and resource constraints.
See all 115 papers