There Is No Free Loop: When Does Looping a Trained Model Pay for Its Inference Compute?
Organizations: Microsoft Applied Sciences Group
Abstract
Looping re-applies a block of layers so that each token passes through the block more than once. Adding a loop to a trained model is attractive because it leaves the parameter count, and hence the disk size, the memory footprint and the training cost unchanged. Unfortunately, every generated token pays for the loop. Re-applying m of N layers makes each token about (N+m)/N times as expensive. Loops that a model learns during large-scale pretraining justify this price, but whether loops added after pretraining justify this cost remains unclear. We compare a looped model with two simple baselines. We first compare it with the same model trained identically without the loop (equal parameters). Secondly, we compare it with the best unlooped use of the same inference floating-point operations (equal FLOPs). Across Qwen3 and Qwen3.5 models from 0.8B to 9B parameters, only the looped 0.8B model beats both baselines. At 4B and 9B, the looped model does not beat the first baseline, i.e. the same model trained without the loop. This is not because the benchmarks are too easy, since on the same benchmarks the unlooped 9B model still scores up to 13 points higher than the 4B model, showing that there is ample room for improvement. Introducing loops in an already pretrained model therefore only pays for small models with a fixed parameter budget.
Figures & tables
| Work | Model | Tokens |
| Huginn [ 18 ] | 3.5B, from scratch | 800B |
| Ouro [ 40 ] | 1.4B / 2.6B, from scratch | 7.7T |
| Retrofitted recurrence [ 39 ] | 1B, converted | 50B |
| Closing the Loop [ 6 ] | 1.7B, converted | 60–84B |
| TaH2 [ 5 ] | 1.7–8B, post-training | 3.4–16B |
| Ours | 0.8–9B, post-training | 0.5B |
| Model | Loop learned by | Layer passes (cost) | Seeds | Benchmarks | Twin looped |
|---|---|---|---|---|---|
| Qwen3-8B ‡ | none (training-free) | 36 45 (1.25 ) | – | GSM8K | 89.4 91.4 ( ) |
| MATH | 76.0 76.6 ( ) | ||||
| Qwen3.5-0.8B | short full FT | 24 30 (1.25 ) | 4 (4/4) | GSM8K, MATH | 44.2 48.3 ( ) |
| Qwen3.5-0.8B | 5.2B CPT, then short FT | 24 30 (1.25 ) | 2 (2/2) | GSM8K, MATH | 31.6 37.6 ( ) △ |
| Qwen3.5-4B | short full FT | 32 40 (1.25 ) | 2 (1/2) | GSM8K, MATH | 82.7 82.3 ( ) |
| Qwen3.5-9B | short full FT | 32 40 (1.25 ) | 2 (0/2) | GSM8K, MATH | 86.5 85.9 ( ) |
| Model | Loop learned by | Unlooped comparator | Comparator cost | Seeds | Benchmarks | Unlooped looped |
|---|---|---|---|---|---|---|
| Qwen3.5-0.8B | short full FT | best cheaper § | loop | 4 (4/4) | GSM8K, MATH | 45.1 48.0 ( ) |
| both + BF | best cheaper, BF § | loop | 4 (4/4) | GSM8K, MATH | 54.8 57.3 ( ) | |
| both 3 FT + BF | best of 12, BF § | loop | 2 (2/2) | GSM8K, MATH | 60.2 64.1 ( ) △ | |
| Qwen3.5-0.8B | 5.2B CPT, then short FT | best cheaper § | loop | 2 (2/2) | GSM8K, MATH | 32.8 37.2 ( ) △ |
| both + BF | best cheaper, BF § | loop | 2 (2/2) | GSM8K, MATH | 40.4 46.4 ( ) △ | |
| Qwen3.5-4B | short full FT | best cheaper § | loop | 2 (1/2) | GSM8K, MATH | 83.4 82.5 ( ) |
| Setting | GSM8K | MATH-500 | Macro | Finished (%) | Cost ( ) |
|---|---|---|---|---|---|
| Unlooped | 54.7 | 33.0 | 43.9 | 72 / 44 | 1.00 |
| Looped | 61.2 | 35.1 | 48.2 | 76 / 44 | 1.20 |
| Unlooped + repetition stop | 59.7 | 40.4 | 50.0 | 86 / 67 | 0.80 |
| Unlooped + budget forcing | 63.8 | 44.6 | 54.2 | 97 / 73 | 0.96 |
| Looped + budget forcing | 68.5 | 44.6 | 56.5 | 97 / 68 | 1.18 |
| Unlooped, 1.25 FT + budget forcing | 64.1 | 46.2 | 55.2 | 97 / 71 | 0.96 |
| Benchmark | 4B | 9B | Size gap | Loop at 4B | Loop at 9B |
|---|---|---|---|---|---|
| GSM8K | 90.0 | 93.7 | 3.7 | 0.0 | 1.1 |
| MATH-500 | 81.5 | 86.0 | 4.5 | 0.5 | 0.8 |
| AIME 2024 | 49.3 | 61.7 | 12.4 | 0.4 | 2.1 |
| AIME 2025 | 41.1 | 53.8 | 12.7 | 0.5 | 4.6 |
| GPQA-Diamond | 36.5 | 42.9 | 6.4 | 0.3 | 2.0 |
| Macro | 59.7 | 67.6 | 7.9 | 0.0 | 1.3 |
| Model | Tokens (B) | Train FLOPs | Inference | Loss | HellaSwag | ARC-E | ARC-C | LAMBADA |
|---|---|---|---|---|---|---|---|---|
| Dense, pretrained without loop | 1.50 | 1.00 | 1.00 | 3.308 | 28.3 | 42.9 | 22.9 | 18.9 |
| Dense, FLOP-matched to looped pretraining | 1.83 | 1.22 | 1.00 | 3.250 | 28.5 | 43.5 | 24.3 | 20.4 |
| Looped during pretraining | 1.50 | 1.22 | 1.22 | 3.289 | 28.4 | 43.5 | 23.8 | 19.2 |
| Dense, 16 layers | 1.50 | 1.22 | 1.22 | 3.268 | 28.4 | 43.6 | 24.5 | 19.2 |
| Dense + best searched loop (no training) | 1.50 | 1.00 | 1.28 | 3.306 | 28.2 | 43.0 | 22.8 | 19.1 |
| Dense, 1.2 tokens, no loop | 1.80 | 1.20 | 1.00 | 3.278 | 28.3 | 42.8 | 23.7 | 19.4 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| , | layers of the unlooped model, layers that the loop re-runs |
| inference cost of the looped model relative to its twin (Eq. 1 ) | |
| , | looped model, its unlooped twin |
| , | accuracy and inference FLOPs of an evaluation |
| unlooped evaluations available to the equal-FLOP test | |
| , | gain at equal parameters and at equal FLOPs (Eqs. 2 – 3 ) |
| Model | Benchmark | Twin looped | 95% CI | Equivalent | |
|---|---|---|---|---|---|
| 4B (4 seeds) | MATH-500 | 81.5 81.0 | [ 1.9, 0.8] | 1.00 | yes |
| 4B (4 seeds) | AIME 2024 | 49.3 49.7 | [ 1.7, 2.7] | 1.00 | no |
| 4B (4 seeds) | AIME 2025 | 41.1 41.6 | [ 3.0, 4.2] | 1.00 | no |
| 4B (4 seeds) | GPQA-D | 36.5 36.2 | [ 3.4, 2.9] | 1.00 | no |
| 4B (4 seeds) | GSM8K | 90.0 90.0 | [ 0.7, 0.8] | 1.00 | yes |
| 4B (4 seeds) | macro | 59.7 59.7 | [ 1.1, 1.1] | – | yes |
| Timing | Setting | Seed | MATH-500 | AIME24 | AIME25 | GPQA-D | GSM8K | Macro | |
|---|---|---|---|---|---|---|---|---|---|
| before | A1: ARD continued pretraining SFT † | 0 | 1 | 78.4 | 47.9 | 35.0 | 36.4 | 89.6 | 57.5 |
| before | A1: ARD continued pretraining SFT † | 0 | 2 | 80.6 | 50.8 | 39.2 | 37.4 | 89.5 | 59.5 |
| before | A1: ARD continued pretraining SFT | 1 | 1 | 77.6 | 44.6 | 37.9 | 27.8 | 87.9 | 55.1 |
| before | A1: ARD continued pretraining SFT | 1 | 2 | 81.4 | 45.8 | 40.8 | 31.3 | 89.8 | 57.8 |
| before | A1c: control † | 0 | 1 | 80.6 | 49.2 | 40.4 | 35.9 | 89.7 | 59.1 |
| before | A1c: control | 1 | 1 | 79.4 | 49.6 | 42.1 | 35.9 | 89.4 | 59.3 |
| Model | Latency (ms) | ( ) | Throughput (tokens/s) | ( ) | Batch | Memory (GB) |
|---|---|---|---|---|---|---|
| 0.8B | 20.6 25.3 | 1.23 | 6,842 5,286 | 0.77 | 693 710 | 1.5 1.5 |
| 4B | 29.2 36.8 | 1.26 | 2,552 2,026 | 0.79 | 324 298 | 8.1 8.1 |
| 9B | 29.4 36.0 | 1.22 | 2,400 1,923 | 0.80 | 245 247 | 16.9 17.0 |
| AIME 2024 | GSM8K (thinking) | |||
| Method | acc. | time | acc. | time |
| Qwen3-1.7B | ||||
| greedy | 16.7 | 1.00 | 81.0 | 1.00 |
| self-consistency, 2 samples | 13.3 | 2.01 | 83.0 | 1.34 |
| self-consistency, 4 samples | 26.7 | 3.23 | 85.0 | 1.90 |
| looped, PPL T | 13.3 | 2.43 | 84.0 | 1.75 |
LLM Inference
Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.
See all 321 papers