LoopFold: Parallelizing Recurrent Depth in Looped Transformers

Feichen Song; Boyi Zeng; Yiqin Hao; Shixiang Song; He Li; Yizhou Yang; Zitong Wang; Yifan Liu

cs.CLOct 26, 2026

LoopFold: Parallelizing Recurrent Depth in Looped Transformers

Authors: Feichen Song, Boyi Zeng, Yiqin Hao, Shixiang Song, He Li, Yizhou Yang, Zitong Wang, Yifan Liu,

Organizations: Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute

Abstract

Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream task performance at a fixed backbone parameter count. However, each loop depends on the preceding loop, so increasing recurrent depth also increases decoding latency. Existing approaches include Parallel Loop Transformers, which reduce latency by overlapping loop computation across token positions, but yield rapidly diminishing marginal accuracy gains at larger depths. To address this challenge, we introduce LoopFold, which folds computation at multiple recurrent depths into parallel batches for the same token prefix. A lightweight predictor estimates inputs to later loops so they can start before earlier loops finish. A corrector then refines the outputs of these later loops using newly available results from earlier loops. All components are trained jointly using only the final next-token prediction loss. In experiments with 1.4B models, deeper LoopFold configurations achieve progressively higher average zero-shot accuracy across eight tasks, rising from 53.41% to 56.38%. Vanilla looped models and PLT show limited gains over the same depth range. Compared with a nine-loop vanilla model, LoopFold uses five sequential stages and additional parallel computation to reduce decoding latency by 46%.

Figures & tables

Figure 1: LoopFold with schedule

1+3+3

. Three stages and width 3 reach target depth index

D=5

using seven backbone evaluations. (a) Predict deeper inputs from

H_{1}-H_{0}

. (b) Evaluate three states together and correct using earlier-loop updates. (c) Repeat the stage and predict the next token.

Model	Target depth index $D$	Backbone evaluations	Sequential stages $S$	Width $n$
Vanilla looped model	$D$	$D$	$D$	1
LoopFold	$n+S-1$	$1+n(S-1)$	$S$	$n$

Table 1: Backbone computation per token (any model size).

D

: target depth index;

S

: sequential stages;

n

: parallel width after the first loop. Evaluations count all parallel lanes.

Figure 2: Parallel width and Pile perplexity in LoopFold (1.4B, 30B training tokens). Each curve fixes sequential stages

S

and varies width

n

. Schedule

1+n+\cdots+n

reaches target depth index

D=n+S-1

Figure 3: Pile PPL versus training compute and decoding latency (1.4B, 30B tokens). (a) LoopFold:

n=2

S=3

–

6

, ending with

1+2+2+2+2+2

; vanilla looped: 3–11 loops. (b) LoopFold uses

1+3+3

1+4+4+4

, and

1+5+5+5+5

D=5,7,9

. Timing: batch 1, 1,024-token prompts/continuations, greedy decoding, same GPU, BF16, tensor parallelism disabled.

Model	Training (ZFLOPs)	LAMBADA OAI	ARC-E	ARC-C	Wino	PIQA	Hella	SciQ	RACE	Avg-8
Vanilla Looped	1.20	59.65	52.90	28.24	57.14	69.86	50.55	78.90	32.25	53.69
PLT		57.27	50.42	26.96	54.70	69.26	48.47	79.10	33.11	52.41
Hidden Decoding		54.41	48.23	26.62	52.09	68.99	43.66	75.70	30.33	50.00
LoopFold ( $n=2,S=3$ )		57.60	53.16	28.16	55.33	69.59	49.57	79.40	33.11	53.24
Vanilla Looped	1.68	60.14	53.32	28.75	57.77	70.35	51.44	77.20	34.64	54.20
PLT		55.89	51.85	27.56	54.70	68.99	47.70	77.80	33.01	52.19

Table 2: Zero-shot accuracy at matched training backbone compute (1.4B, 30B Pile tokens). Vanilla looped models and PLT use 5/7/9 loops; Hidden Decoding uses 5/7/9 streams per token. LoopFold schedules

1+2+2

1+2+2+2

, and

1+2+2+2+2

correspond to

D=4,5,6

. Accuracy is in percent; Avg-8 is the unweighted mean across eight tasks.

Model	Latency (ms/token)	LAMBADA OAI	ARC-E	ARC-C	Wino	PIQA	Hella	SciQ	RACE	Avg-8
PLT-5	24.93	57.27	50.42	26.96	54.70	69.26	48.47	79.10	33.11	52.41
PLT-7	25.35	55.89	51.85	27.56	54.70	68.99	47.70	77.80	33.01	52.19
PLT-9	25.16	56.84	51.05	27.22	55.41	70.29	47.72	76.30	31.96	52.10
Hidden Decoding-5	23.63	54.41	48.23	26.62	52.09	68.99	43.66	75.70	30.33	50.00
Hidden Decoding-7	23.95	52.92	48.32	25.60	52.49	68.88	43.44	74.70	30.62	49.62
Hidden Decoding-9	23.27	53.19	49.20	26.54	51.78	68.23	43.65	73.40	30.62	49.57

Table 3: Zero-shot accuracy and decoding latency (1.4B, 30B Pile tokens). LoopFold uses

1+3+3

1+4+4+4

, and

1+5+5+5+5

D=5,7,9

, paired with 3/4/5-loop vanilla looped models. PLT/Hidden Decoding suffixes count loops/streams per token. Timing: batch 1, 1,024-token prompts/continuations, greedy decoding, same GPU, BF16, tensor parallelism disabled. Accuracy is in percent; Avg-8 is the unweighted mean across eight tasks.

Figure 4: Mean zero-shot accuracy across target depth indices (1.4B, 30B Pile tokens). At

D=5,7,9

, LoopFold uses

1+3+3

1+4+4+4

, and

1+5+5+5+5

in 3/4/5 stages; vanilla looped models and PLT use

D

loops. Accuracy is the unweighted mean (%) across eight tasks.

Model	$B=1$	$B=2$	$B=4$	$B=8$	$B=16$
Vanilla Looped $D=5$	8.14	16.15	31.67	65.31	130.18
LoopFold $D=5$	13.44	26.24	52.26	105.35	209.12
Speed ratio to vanilla looped model	$1.65\times$	$1.62\times$	$1.65\times$	$1.61\times$	$1.61\times$

Table 4: Decoding throughput at

D=5

(1.4B, tokens/s). LoopFold uses

1+3+3

, with seven backbone evaluations in three stages, versus five loops in the vanilla looped model.

B

is batch size. Timing: 1,024-token prompts/continuations, greedy decoding, same GPU, BF16, tensor parallelism disabled. Speed ratio: LoopFold/vanilla looped throughput at equal

B

; blanks indicate unavailable measurements.

Variant	Later-stage KV policy	KV storage	Latency (ms/token)	Pile PPL $\downarrow$
No KV sharing	independent cache	$7\times$	99.89	7.62
First-KV reuse	reuse all positions	$5\times$	103.21	7.68
Hard overwrite-64	overwrite most recent 64	$5\times$	104.94	7.63

Table 5: KV sharing (1.4B, 30B Pile tokens,

1+2+2+2

D=5

). First-KV reuse retains each depth’s first cache; Hard overwrite-64 also refreshes the latest 64 positions. Storage is relative to one full-backbone cache. Variants train with their inference cache policies. Timing: batch 1, 1,024-token prompts/continuations, greedy decoding, same GPU, BF16, tensor parallelism disabled.

Variant	Initial states	Output correction	Pile PPL $\downarrow$	$\Delta$ PPL
LoopFold	$J_{\mathrm{pred}}$	$J_{\mathrm{corr}}$	9.13	–
No prediction	repeated $H_{1}$	$J_{\mathrm{corr}}$	9.29	+0.16
No correction	$J_{\mathrm{pred}}$	none	9.47	+0.34

Table 6: Component ablations (410M, 25B Pile tokens,

1+4+4+4

D=7

J_{\mathrm{pred}}

and

J_{\mathrm{corr}}

denote prediction and correction. No prediction repeats

H_{1}

; No correction uses raw backbone outputs. Variants train separately;

\Delta

PPL is the increase over the complete model.

Variant	$J_{\mathrm{pred}}$	$J_{\mathrm{corr}}$	Pile PPL $\downarrow$
LoopFold	Learned	Learned	9.13
Identity prediction	$I$	Learned	9.21
Identity correction	Learned	$I$	9.20

Table 7: Learned versus identity maps (410M, 25B Pile tokens,

1+4+4+4

D=7

). Each identity variant fixes prediction

J_{\mathrm{pred}}

or correction

J_{\mathrm{corr}}

I

throughout training, while learning the backbone and the other map.

Variant	Metric	$H_{2}$ (S1)	$H_{3}$ (S1 $\to$ S2)	$H_{4}$ (S1 $\to$ S2 $\to$ S3)
LoopFold	Cosine $\uparrow$	0.922	0.774 $\to$ 0.811	0.636 $\to$ 0.653 $\to$ 0.823
LoopFold	Relative $L_{2}$ $\downarrow$	0.394	0.642 $\to$ 0.588	0.809 $\to$ 0.792 $\to$ 0.573
No prediction	Cosine $\uparrow$	0.971	0.874 $\to$ 0.924	0.643 $\to$ 0.701 $\to$ 0.805
No prediction	Relative $L_{2}$ $\downarrow$	0.239	0.494 $\to$ 0.389	0.777 $\to$ 0.726 $\to$ 0.594
No correction	Cosine $\uparrow$	0.506	0.517 $\to$ 0.870	0.234 $\to$ 0.224 $\to$ 0.414
No correction	Relative $L_{2}$ $\downarrow$	0.866	0.857 $\to$ 0.516	0.977 $\to$ 1.027 $\to$ 0.920

Table 8: Provisional-state evolution (410M, 25B Pile tokens,

1+4+4+4

D=7

). Reference states come from each variant’s own sequential backbone on the same input. S1 denotes initialization; arrows mark subsequent stages. Cosine uses full state tensors; relative

L_{2}

divides the Frobenius error by the reference state’s norm.

Recurrent Transformers

Recurrent Transformers reuse layers or state across steps to extend context, reasoning depth, and sequence modeling with less parameter growth.

See all 179 papers

Decoding Looped Transformers Better for (Almost) Free
Oct 1, 2026Weihao Liu, Huangjie Zheng, Tianrong Chen +5Efficient Transformer Inference Language Model Decoding
Apple
StateLoop: Persistent State for Looped Transformers
Oct 26, 2026Feichen Song, Boyi Zeng, He Li +7Persistent Memory for Language Models Language Modeling
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
Sep 24, 2026Wanqi Yang, Shiwei LiuEfficient Transformer Inference Recurrent Transformers
ELLIS Institute Tübingen Max Planck Institute for Intelligent Systems Tübingen AI Center

Oct 1, 2026J/K move · Enter open · S save

Decoding Looped Transformers Better for (Almost) Free

Weihao Liu, Huangjie Zheng, Tianrong Chen, Rohit Dilip+4

Apple

Efficient Transformer Inference Language Model Decoding Recurrent Transformers Contrastive Decoding

Open paper AskPDF arXiv

Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training. We introduce LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting the final prediction with an earlier recurrent pass, operating either in logit space with one extra output pass (LoopCD-Logits) or in hidden-state space with zero output overhead (LoopCD-Hidden). Across four looped Transformer families, LoopCD delivers substantial, consistent gains at full recurrent depth: LoopCD-Logits raises Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. Crucially, these performance gains enable halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%. By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute.

Abstract

Recurrent Transformers

Explore similar work

Decoding Looped Transformers Better for (Almost) Free

StateLoop: Persistent State for Looped Transformers

FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates