cs.LGOct 30, 2026

There Is No Free Loop: When Does Looping a Trained Model Pay for Its Inference Compute?

Authors: Rhea Sanjay Sukthanker, Pashmina Cameron

Organizations: Microsoft Applied Sciences Group

Abstract

Looping re-applies a block of layers so that each token passes through the block more than once. Adding a loop to a trained model is attractive because it leaves the parameter count, and hence the disk size, the memory footprint and the training cost unchanged. Unfortunately, every generated token pays for the loop. Re-applying m of N layers makes each token about (N+m)/N times as expensive. Loops that a model learns during large-scale pretraining justify this price, but whether loops added after pretraining justify this cost remains unclear. We compare a looped model with two simple baselines. We first compare it with the same model trained identically without the loop (equal parameters). Secondly, we compare it with the best unlooped use of the same inference floating-point operations (equal FLOPs). Across Qwen3 and Qwen3.5 models from 0.8B to 9B parameters, only the looped 0.8B model beats both baselines. At 4B and 9B, the looped model does not beat the first baseline, i.e. the same model trained without the loop. This is not because the benchmarks are too easy, since on the same benchmarks the unlooped 9B model still scores up to 13 points higher than the 4B model, showing that there is ample room for improvement. Introducing loops in an already pretrained model therefore only pays for small models with a fixed parameter budget.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Closing the Loop: Practical Training Recipes for Looped Language Models

    Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model TrainingLLM Training

  2. How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference

    Oct 29, 2026Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan +5Recurrent TransformersEfficient Transformer Inference

  3. UpLoop: Upcycling Large Language Models with Gated Loop

    Oct 18, 2026Tieyuan Chen, Huabin Liu, Yingtao Zhang +4Recurrent Language ModelsRecurrent Transformers