Parallel Looped Transformers (PLTs) use otherwise idle decoding compute by overlapping recurrent iterations. However, vanilla PLTs scale poorly with depth: they exhibit training-loss spikes, and their quality gains shrink as more loops are added. The benefit of additional loops depends on the information available to each iteration, so we examine whether the existing computational dependencies limit effective depth scaling. We find that only the preceding in-trajectory state is passed across loop boundaries, even though completed cross-trajectory representations are already available. Based on this finding, we propose InfuseLoop, a cross-trajectory latent injection mechanism. It aggregates causally available hidden states from later prediction trajectories through lightweight attention residuals. This design preserves token-parallel training and concurrent loop execution, and requires only O(Rd) additional hidden-state storage. Experiments show that InfuseLoop eliminates training-loss spikes and improves validation-loss scaling up to eight loops. At eight loops, InfuseLoop reduces perplexity by 3.4--17.1% relative to vanilla PLTs and by 10.5--37.3% relative to the non-looped baseline. It also improves average downstream accuracy by 0.84 and 1.99 percentage points over these two baselines. InfuseLoop incurs only 0.3% decoding time over vanilla PLTs and 23.3% over the non-looped baseline, while keeping the same logical KV-cache footprint as vanilla PLTs.
Figures & tables
Figure 1: Overview of InfuseLoop. Each loop combines the latest token embedding with the current state and causally available future states through attention residual aggregation. The enlarged view details the attention-based aggregation of current latent state and future states.
Figure 2: Execution schedules and information flow. (a) Looped Transformers execute successive loops serially before generating the next token. (b) InfuseLoop overlaps computation across prediction positions and loop iterations, and introduces future-state injection (red) on top of vanilla PLT.
Figure 3: State availability and source selection for ht(4) in a six-loop model ( R=6 ). ht(4) is computed at cycle c=τ(t,4) , and the outlined states are the sources selected from cycle c−1 .
Input sequence X=(x1,…,xL) ,
next-token targets Y ,
number of loops R , backbone Fθ
Algorithm 1 Token-parallel training of InfuseLoop
Current token xc ,
previous-cycle outputs p1,…,pR ,
number of loops R , backbone Fθ
Algorithm 2 One decoding cycle of InfuseLoop
Figure 4: Validation loss during pretraining with (a) R=4 and (b) R=8 loops.
Loops
Model
Val. loss ↓
Perplexity ↓
Downstream tasks ↑
Wiki2
LAMBADA
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WG
OBQA
Avg.
R=1
Single
2.9941
39.72
67.02
24.58
36.44
49.92
28.58
63.38
51.70
33.00
41.09
R=4
LT
2.9022
34.78
41.07
24.35
40.17
54.34
30.55
66.43
54.70
33.80
43.48
PLT
2.9252
36.19
62.96
24.71
39.24
51.68
28.58
65.78
51.70
32.60
42.04
InfuseLoop
2.9112
35.70
45.08
24.49
39.49
53.75
31.40
65.89
52.96
33.00
43.00
R=8
LT
2.8775
33.78
35.04
23.99
41.88
55.77
31.91
65.51
54.38
32.20
43.66
Table 1: Validation loss, perplexity, and downstream accuracy (%) after training on 6B tokens.
Context length
Model
512
2,048
8,192
Single
114.69
114.69
114.69
InfuseLoop-4
157.72 (+37.5%)
125.45 (+9.4%)
117.38 (+2.3%)
InfuseLoop-8
215.10 (+87.5%)
139.79 (+21.9%)
120.96 (+5.5%)
Table 2: Logical cache footprint per token (KB) with window size w=64 . Parentheses show the overhead relative to Single.
Figure 5: Decoding cost and workload sensitivity. (a) GPU kernel time with a 2,048-token initial context, normalized to the total kernel time of Single at the same batch size. (b, c) Throughput relative to Single across batch sizes and context lengths.
Figure 6: Depth scaling and performance–latency Pareto frontiers. Top: validation loss; bottom: average downstream accuracy. Interpolated R=2 points and projected R>8 extensions are visual guides only; Pareto membership is computed from measured configurations.
R=4
R=8
Configuration
Loss ↓
Avg. ↑
Loss ↓
Avg. ↑
InfuseLoop
3.2482
38.26
3.2436
38.35
Source selection
Base only
3.2660
38.04
3.2799
37.59
FullState
3.3975
37.43
3.4244
37.21
In-trajectory
3.2735
38.19
3.2839
37.87
Table 3: Component ablations. All configurations are trained on 1B tokens.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
Backbone configuration
Qwen3-0.6B
Transformer layers ( K )
28
Hidden dimension ( d )
1,024
FFN intermediate dimension
3,072
Query heads
16
KV heads
8
Appendix
Table 4: Backbone configuration and parameter counts.
Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream task performance at a fixed backbone parameter count. However, each loop depends on the preceding loop, so increasing recurrent depth also increases decoding latency. Existing approaches include Parallel Loop Transformers, which reduce latency by overlapping loop computation across token positions, but yield rapidly diminishing marginal accuracy gains at larger depths. To address this challenge, we introduce LoopFold, which folds computation at multiple recurrent depths into parallel batches for the same token prefix. A lightweight predictor estimates inputs to later loops so they can start before earlier loops finish. A corrector then refines the outputs of these later loops using newly available results from earlier loops. All components are trained jointly using only the final next-token prediction loss. In experiments with 1.4B models, deeper LoopFold configurations achieve progressively higher average zero-shot accuracy across eight tasks, rising from 53.41% to 56.38%. Vanilla looped models and PLT show limited gains over the same depth range. Compared with a nine-loop vanilla model, LoopFold uses five sequential stages and additional parallel computation to reduce decoding latency by 46%.
Feichen Song, Boyi Zeng, Yiqin Hao +7
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
Looping in transformer models reuses weights by repeatedly applying the same transformer, or parts of it, making the model deeper without increasing its parameter count. Whether looping is useful depends on which constraint dominates: model size, training compute, or autoregressive inference efficiency. We study these trade-offs jointly in dense decoder-only transformers trained from scratch, varying which parts of the model are looped, how many loops are applied, and how information is passed between loops. Our experiments span models with up to 1B parameters, trained on up to 155B tokens with up to 1.07×1021 FLOPs and 32 loops. We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained. Adding loops is not uniformly beneficial: gains eventually saturate, reverse, or become unstable. This motivates asking whether forcing every loop to use the same weights limits the benefit of deeper looping. We address this by adding small loop-specific low-rank adapters that improve quality, with larger gains as more loops are added. Looped transformers also suffer a decoding slowdown proportional to the number of loops. Parallel Loop Transformers address this by passing hidden states between loops across adjacent token positions, allowing loop stages to run concurrently. However, in our experiments the original formulation does not consistently retain the quality gains of ordinary looping. We find that preserving the current-token embedding and separately normalizing it and the previous-token hidden state before combining them recovers strong performance across model scales and training budgets. Together, these results show that effective looping requires matching the design to the resource constraint and inference setting.
Looped Transformers have attracted substantial attention as a parameter-efficient approach to increasing computational depth through repeated application of shared Transformer blocks. However, their practical advantages over conventional Transformers remain under debate: each additional loop incurs another Transformer pass and requires caching another set of KV states, causing inference FLOPs and KV-cache memory to grow continuously with loop depth. This overhead becomes particularly severe at large loop counts and long context, preventing the parameter efficiency of Looped Transformers from translating into practical inference efficiency. In this paper, we find that much of the additional computation and storage introduced by looping is redundant. As recurrence proceeds, state changes become increasingly concentrated on a small subset of tokens; attention-output differences are dominated by a sparse and stable subset of key columns; and KV residuals between adjacent loops become progressively more amenable to low-bit quantization. Building on these observations, we introduce FlashLoop, a training-free inference framework that reduces cross-loop redundancy through token-sparse updates, sparse attention, and KV-residual quantization. Across several Looped Transformers models, FlashLoop delivers lossless accuracy while achieving up to 1.64× end-to-end speedup and up to 6× KV-cache memory reduction, substantially improving the practicality of scaling Looped Transformers to greater computational depths and longer context.
Wanqi Yang, Shiwei Liu
ELLIS Institute Tübingen Max Planck Institute for Intelligent Systems Tübingen AI Center