ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals

Yuekun Yao; Yupei Du; Dawei Zhu; Huan Sun

cs.CLNov 3, 2026

ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals

Authors: Yuekun Yao, Yupei Du, Dawei Zhu, Huan Sun

Organizations: The Ohio State University · Saarland University

Abstract

Looped transformers increase effective depth by repeatedly applying a shared set of transformer layers, offering a parameter-efficient way to scale computation. However, standard residual connections aggregate information across recurrent depth into a single stream, providing limited control over which intermediate representations from preceding layers should be reused. In this paper, we introduce ARLoop, a looped transformer architecture based on attention residuals, where the input of each layer is selectively aggregated through attention over model depth. Distinct from standard transformers, we find that looped transformers do not require access to the full residual history: sliding-window depth attention matches or outperforms full-history attention at larger recurrent depths. Across controlled experiments with multiple parameter scales (140M and 370M) and recurrent depths (R = 1 to 5), ARLoop consistently outperforms prior residual designs for looped transformers. More importantly, we further scale ARLoop to 0.6B and 1.2B parameters under a realistic LLM pretraining recipe, demonstrating that the gains transfer to realistic pretraining, with particularly strong improvements on math and coding tasks. Our 1.2B model, pretrained on only 0.5T tokens with three recurrences, outperforms Huginn-3.5B (the only fully-open looped LLM as of our release date) across all evaluated tasks and is competitive with similar-sized non-looped models such as Qwen3.5-2B despite being trained on substantially less data.

Figures & tables

Figure 1: Performance of ARLoop on general reasoning, knowledge, mathematics, and coding benchmarks. ARLoop-1.2B outperforms both Huginn-3.5B and Qwen3.5-2B on all eight benchmarks, while ARLoop-0.6B outperforms its parameter-matched PreNormLoop baseline on seven of eight benchmarks.

Figure 2: An example illustrating attention residuals in looped transformers. (a) Each loop contains one transformer layer (i.e. a self-attention and an FFN), with parameters shared across loops. We group every

S=2

sublayers into a residual block, yielding

b_{1}=z_{1}+z_{2}

and

b_{2}=z_{3}+z_{4}

. (b) Full-history depth attention ( Kimi Team, 2026 ) accesses all available residual sources. (c) Sliding-window depth attention retains the embedding and a window of recent residual sources.

Model	140M			370M
	$R=1$	$R=3$	$R=5$	$R=1$	$R=3$
Sandwich-Norm ( Geiping et al., 2025 )	18.98	17.19	16.59	13.92	12.97
DeepLoop ( Li et al., 2026 )	18.72	17.05	16.52	13.69	12.78
Parcae w/o input injection ( Prairie et al., 2026 )	18.37	16.60	16.01	13.55	12.31
Parcae w. input injection ( Prairie et al., 2026 )	18.37	16.55	15.96	13.55	12.30
PreNormLoop	18.37	16.91	16.34	13.55	12.48

Table 1: Validation perplexity (

\downarrow

) for models at the 140M and 370M scales.

Scale	$R$	PPL $\downarrow$			Downstream $\uparrow$
Scale	$R$	PreNormLoop	ARLoop full	ARLoop sw	PreNormLoop	ARLoop full	ARLoop sw
140M	2	17.48	17.27	17.30	42.22	41.96	42.27
	3	16.91	16.46	16.38	42.72	43.69	43.09
	4	16.59	16.07	16.03	43.27	44.09	43.73
370M	2	12.83	12.68	12.55	48.98	48.70	49.25
	3	12.48	12.27	12.17	49.29	50.23	50.67

Table 2: Validation perplexity and average zero-shot downstream accuracy (%). Bold marks the best result for each metric within each row.

Panel of Figure 3 — Figure 3: Left : Attention weights over residual sources (i.e. blocks) at each sublayer for the 140M ARLoop full model with $R=4$ , averaged over the validation set. Green lines indicate the sliding windows used in the corresponding ARLoop sw variant. Right : Pairwise cosine similarity between residual blocks for the same model.

Open-weight LLMs
Model	Looped	FLOPs	Data (T)	ARC-C	Hella.	Wino.	OBQA	GSM8K	MATH	MBPP	HumanEval
Qwen3.5-2B	None	432	36	54.3	67.5	66.4	37.8	60.6	35.8	49.5	32.9
Gemma 3-4B	None	96	4.0	58.4	77.7	72.3	43.0	38.1	25.4	60.1	34.2
Ouro-1.4B	4	259	7.7	59.9	74.4	70.5	41.4	78.2	70.5	68.3	52.4
Fully-open LLMs
OLMo 2-7B	None	180	4.1	64.6	82.0	76.8	45.6	68.1	20.9	32.0	12.2

Table 3: Comparison of open-weight and fully-open language models. Training FLOPs are estimated using the

6ND

approximation and reported in units of

10^{21}

. Data (T) denotes pretraining tokens in trillions. Looped denotes the default number of recurrences used by each model. Rows in light blue refer to looped language models, including our fully-open models.

Figure 4: Layer effects on 50 MATH examples for PreNormLoop-0.6B and ARLoop-0.6B . Shaded regions mark Loop1 (layers 0–17), Loop2 (18–35), and Loop3 (36–53).

Memory (GiB) $\downarrow$
Model	140M	370M	0.6B	1.2B
PreNorm	32.8	80.0	66.5	65.4
ARLoop full	33.0	80.1	66.6	65.4
ARLoop sw	32.9	80.1	66.6	65.4
Throughput (tokens/s) $\uparrow$
PreNorm	131.1k	45.8k	28.9k	15.0k

Table 4: Training memory and throughput at

R=4

on a single NVIDIA H100, with batch sizes of 8, 8, 4, and 2 for the 140M, 370M, 0.6B, and 1.2B models, respectively.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

		140M		370M
		PreNorm	ARLoop	PreNorm	ARLoop
Architecture	Physical Transformer layers $L$	6	6	12	12
	Hidden dimension $d_{\mathrm{model}}$	768	768	1,024	1,024
	MLP dimension $d_{\mathrm{intermediate}}$	3,072	3,072	4,096	4,096
	Attention heads (query / key–value)	6 / 6	6 / 6	8 / 8	8 / 8
	Head dimension	128	128	128	128

Appendix

Table 5: Architecture details for the controlled 140M and 370M experiments. Each Transformer layer contains an attention sublayer and an MLP sublayer. The AttnRes block and window settings apply to the corresponding residual variants; Table 2 lists the evaluated configurations. PreNorm uses additive pre-normalized residual updates, while ARLoop learns the input mixture at each effective sublayer.

Benchmark	Settings / reported metric	Framework
Language modeling
FineWeb-Edu (held out)	PPL
Zero-shot downstream evaluation
ARC-Challenge	Log-likelihood, 0-shot; accuracy	lm-eval-harness
ARC-Easy	Log-likelihood, 0-shot; accuracy	lm-eval-harness
HellaSwag	Log-likelihood, 0-shot; normalized accuracy	lm-eval-harness

Appendix

Table 6: Evaluation protocol for the controlled 140M and 370M experiments.

Model	140M, $R=3$ , w/o value res. (Val PPL $\downarrow$ )
PreNorm	18.19
Sandwich-Norm ( Geiping et al., 2025 )	Collapse
DeepLoop ( Li et al., 2026 )	18.44
Parcae + input injection ( Prairie et al., 2026 )	17.83
ARLoop full	17.52
ARLoop sw	18.20

Appendix

Table 7: Validation perplexity without value residuals. Bold marks the best available result.

140M
$R$	Residual design	ARC-C	ARC-E	HellaSwag	LAMBADA OpenAI	LAMBADA Standard	OBQA	PIQA	RACE	SciQ	WinoGrande	Avg.
1	PreNorm	21.84	56.36	34.49	25.77	19.13	32.20	63.49	28.33	82.40	50.04	41.41
1	ARLoop full	23.72	55.22	33.90	25.58	18.18	32.20	64.15	28.90	83.90	52.49	41.82
2	PreNorm	23.55	56.40	35.40	26.97	21.56	28.40	63.93	32.34	82.30	51.38	42.22
	ARLoop full	23.63	55.51	35.64	27.19	19.85	29.80	63.87	30.72	83.20	50.20	41.96
	ARLoop sw	22.61	57.53	35.77	26.99	21.35	31.80	63.11	30.53	81.80	51.22	42.27

Appendix

Table 8: Zero-shot downstream accuracy (%) for the 140M and 370M models in Table 2 . HellaSwag, OpenBookQA (OBQA), and PIQA use normalized accuracy; all other tasks use accuracy. Training and evaluation use the same recurrence

R

;

R=1

is non-recurrent. ARLoop full and ARLoop sw use residual blocks of 6 sublayers at 140M and 12 at 370M; ARLoop sw retains the input embedding and the three most recent blocks. Bold indicates the highest accuracy within each scale and recurrence group.

Figure 5: Additional full-history diagnostics: (a) 140M, $R=3$ ; (b) 370M, $R=3$ ; and (c) 370M, $R=4$ . Top: residual-source attention weights, with green outlines marking the embedding plus the three most recent blocks. Bottom: pairwise residual-block cosine similarity, with E denoting the embedding.

Architecture	Full history	Sliding window ( $W=3$ )	$\Delta$ PPL
Looped: $L=6$ , $R=4$	16.07	16.03	$-0.04$
Non-recurrent: $L=24$ , $R=1$	13.65	16.78	$+3.13$

Appendix

Table 9: Validation perplexity at matched effective depth. Both models use the same training and architecture setups, except that non-recurrence models use more unique parameters to match the effective depth. The sliding window retains the embedding plus three blocks.

Figure 6: Residual-block similarity for the non-recurrent 24-layer, $W=3$ model, measured on the same validation positions as the looped-model cosine diagnostics. E is the embedding; blocks 1–8 belong to one forward pass, not separate recurrences.

Model	Recurrence	Residual block size (sublayers)
		2	4	6
140M 6 layers	Non-recurrent	18.20	18.25	18.26
	Loop $=2$	17.19	17.18	17.27
	Loop $=3$	16.59	16.48	16.46
	Loop $=4$	16.13	16.02	16.07

Appendix

Table 10: Validation PPL for completed 140M full-history block-size experiments. The best result within each model and recurrence is bolded.

Model	Recurrence	Sliding-window blocks $W$
		2	3	4	5
140M ARLoop sw	4	16.17	16.03	16.07	16.14

Appendix

Table 11: Validation PPL versus sliding-window size for 140M ARLoop sw models with residual blocks of six sublayers.

W

counts retained residual blocks; the input embedding is always retained.

	PreNorm 0.6B	ARLoop 0.6B	ARLoop 1.2B
Physical Transformer layers $L$	18	18	24
Hidden dimension	1,536	1,536	2,048
SwiGLU intermediate dimension	4,608	4,608	5,632
Query / key–value heads	16 / 4	16 / 4	16 / 4
Head dimension	96	96	128
Residual block size (sublayers)	–	18	24

Appendix

Table 12: Architecture and optimization settings for large-scale pretraining. Residual block/window settings apply only to ARLoop . Each physical layer contains one attention and one MLP sublayer.

Figure 7: Three-stage pretraining recipe. The upper panels show the domain sampling shares at each stage. The lower panel shows the learning rate schedules: 1,000 warmup updates followed by cosine decay to 1% of the peak. Source-level allocations are listed in Table 14 .

Benchmark	Settings / reported metric	Framework
General knowledge and reasoning
ARC-Challenge	Log-likelihood, 25-shot; normalized accuracy	lm-eval-harness
HellaSwag	Log-likelihood, 10-shot; normalized accuracy	lm-eval-harness
WinoGrande	Log-likelihood, 5-shot; accuracy	lm-eval-harness
OpenBookQA	Log-likelihood, 0-shot; normalized accuracy	lm-eval-harness
Mathematical reasoning

Appendix

Table 13: Evaluation setup for large-scale pretraining. We evaluate pretrained base checkpoints without chat templates. CoT denotes chain-of-thought demonstrations. MATH uses the full 5,000-example test set. These settings also apply to the external checkpoints re-evaluated for Table 3 .

Domain	Dataset source	0.6B model (180B tokens)				1.2B model (500B tokens)
Domain	Dataset source	Stage 1	Stage 2	Stage 3	Total	Stage 1	Stage 2	Stage 3	Total
General	FineWeb-Edu ( Penedo et al., 2024 )	89.02	26.36	22.06	137.44	247.29	73.22	61.27	381.78
General	StackExchange	2.16	0.64	0.04	2.84	6.00	1.78	0.10	7.88
Math	FineMath 3+ ( Allal et al., 2025 )	1.08	0.36	0.07	1.51	3.00	1.00	0.20	4.20
	InfiWebMath 3+ ( Allal et al., 2025 )	1.08	0.36	0.07	1.51	3.00	1.00	0.20	4.20
	FineMath 4+ ( Allal et al., 2025 )	1.08	1.96	0.90	3.94	3.00	5.44	2.50	10.94

Appendix

Table 14: Source-level allocation (billion tokens for each cell) for large-scale pretraining.

Figure 8: Output influence (left) and layer-contribution influence (right) on 50 shared zero-shot MBPP examples. Curves compare the ARLoop-0.6B and PreNormLoop-0.6B checkpoints at

R=3

, using max aggregation and normalization by each curve’s total influence. Dashed lines separate the three recurrences.

Figure 9: Output influence (left) and layer-contribution influence (right) on 50 shared zero-shot MATH examples, with the same aggregation and normalization as Figure 8 . The right panel supplies the data for Figure 4 . Dashed lines separate the three recurrences.

Eval $R$	Training recurrence	140M		370M
Eval $R$	Training recurrence	Val PPL $\downarrow$	downstream avg. $\uparrow$	Val PPL $\downarrow$	downstream avg. $\uparrow$
4	ARLoop full ( $R=4$ )	16.07	44.09	12.01	50.93
	ARLOOP-Poisson-Seq	16.69	43.24	12.33	49.62
	ARLOOP-Poisson-Batch	16.52	43.30	12.36	50.53

Appendix

Table 15: Fixed versus dynamic recurrence training. Dynamic Poisson draws have mean 4 and are clamped to

[1,4]

. Bold marks the best result at each scale.

Figure 10: Validation perplexity versus evaluation recurrence for the per-sequence Poisson dynamic full-history ARLoop models in Table 15 . Each curve evaluates the same completed checkpoint at

R=1,\ldots,8

. The vertical dashed line at

R=4

marks the maximum training recurrence; the gray shaded region denotes evaluation beyond this depth. Horizontal dashed lines retain the fixed-

R=4

baseline reference levels (16.07 at 140M and 12.01 at 370M), not recurrence sweeps of the fixed models. Panels use separate vertical scales.

Recurrent Transformers

Recurrent Transformers reuse layers or state across steps to extend context, reasoning depth, and sequence modeling with less parameter growth.

See all 187 papers

How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference
Oct 29, 2026Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan +5Recurrent Transformers Efficient Transformer Inference
FAIR at Meta · MIT
DeepLoop: Depth Scaling for Looped Transformers
Jul 15, 2026Shuzhen Li, Yifan Zhang, Jiacheng Guo +2Residual Learning Recurrent Transformers
Princeton University · University of California, Los Angeles
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Parallel Training of Recurrent Networks Recurrent Transformers
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute

Oct 29, 2026J/K move · Enter open · S save

How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference

Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan, Shruti Bhosale+4

FAIR at Meta · MIT

Recurrent Transformers Efficient Transformer Inference Parallel Decoding

Open paper AskPDF arXiv

Looping in transformer models reuses weights by repeatedly applying the same transformer, or parts of it, making the model deeper without increasing its parameter count. Whether looping is useful depends on which constraint dominates: model size, training compute, or autoregressive inference efficiency. We study these trade-offs jointly in dense decoder-only transformers trained from scratch, varying which parts of the model are looped, how many loops are applied, and how information is passed between loops. Our experiments span models with up to 1B parameters, trained on up to 155B tokens with up to

1.07\times10^{21}

FLOPs and 32 loops. We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained. Adding loops is not uniformly beneficial: gains eventually saturate, reverse, or become unstable. This motivates asking whether forcing every loop to use the same weights limits the benefit of deeper looping. We address this by adding small loop-specific low-rank adapters that improve quality, with larger gains as more loops are added. Looped transformers also suffer a decoding slowdown proportional to the number of loops. Parallel Loop Transformers address this by passing hidden states between loops across adjacent token positions, allowing loop stages to run concurrently. However, in our experiments the original formulation does not consistently retain the quality gains of ordinary looping. We find that preserving the current-token embedding and separately normalizing it and the previous-token hidden state before combining them recovers strong performance across model scales and training budgets. Together, these results show that effective looping requires matching the design to the resource constraint and inference setting.

Abstract

Recurrent Transformers

Explore similar work

How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference

DeepLoop: Depth Scaling for Looped Transformers

LoopFold: Parallelizing Recurrent Depth in Looped Transformers