Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder

Cheng Liang; Tengfei Zhang; Wenjie DU; Jianyu Wu; Changlian Ma; Shuchen Pu; Ya Zhang; Yanfeng Wang

cs.CLOct 20, 2026

Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder

Authors: Cheng Liang, Tengfei Zhang, Wenjie DU, Jianyu Wu, Changlian Ma, Shuchen Pu, Ya Zhang, Yanfeng Wang,

Organizations: Shanghai Jiaotong University · University of Science and Technology of China · Nanyang Technological University · Nanjing University

Abstract

Looped Transformers increase effective depth by repeatedly applying shared layers, offering a way to scale computation without increasing parameter count. However, full-stack looping places context construction and iterative state refinement within the same recurrent computation, without explicitly distinguishing their roles. We introduce the Asymmetric Looped Transformer (AsymLoop), which separates single-pass context encoding from recurrent state refinement. A single-pass causal encoder produces layer-specific Global key-value (KV) caches that remain fixed across iterations, while a weight-tied recurrent decoder generates Local KV from its evolving state. Joint attention over both sources under a single softmax allows each update to integrate persistent contextual information with the current intermediate state. For training, we apply a heavy-tail-guided layerwise learning-rate method to the encoder and decoder. Experiments on FineWeb-Edu across four model scales, from 60M to 1B, show consistently lower validation perplexity and higher average downstream accuracy than Full Loop under matched effective depth and training token budgets. At 1B, AsymLoop improves average zero-shot accuracy across seven commonsense reasoning benchmarks from 49.58% to 50.85%. Further experiments extend these benefits to longer training budgets, an OLMo-style backbone, and sparse mixture-of-experts models. These results support asymmetric context encoding and state refinement as an effective design for recurrent language models.

Figures & tables

Figure 1: AsymLoop architecture and Global–Local attention. (a) Dense executes each layer once. (b) Full Loop repeats the entire stack with shared parameters. (c) AsymLoop runs a causal encoder once and a shared-parameter decoder for

K

iterations. The encoder output provides layer-specific Global KV that remains fixed across iterations. (d) Each decoder layer jointly attends to fixed Global KV and Local KV computed from its current state, using a single softmax with causal masking for both sources.

Figure 2: Validation and zero-shot performance across model sizes. Full Loop (

K=2

) and AsymLoop (

K=3

) have the same effective depth. (a, b) Validation cross-entropy versus model parameter count and training token budget, respectively. (c) Average zero-shot accuracy across seven tasks.

Zero-shot
Scale	Model	OBQA	WinoGrande	ARC-c	ARC-e	HellaSwag	SIQA	PIQA	Avg.
60M	Dense	17.80	50.83	17.41	45.33	27.39	34.54	59.03	36.05
	Full Loop	16.80	50.28	20.22	46.51	27.14	37.00	58.71	36.66
	AsymLoop	17.80	50.59	18.17	47.31	27.72	36.49	59.41	36.79
135M	Dense	18.80	48.54	21.42	52.90	29.06	37.41	61.53	38.52
	Full Loop	19.80	51.85	22.87	51.52	29.09	38.69	62.19	39.43

Table 1: Zero- and five-shot commonsense reasoning accuracy (

\uparrow

\%

). Full Loop (

K=2

) and AsymLoop (

K=3

) have the same effective depth. Avg. is the mean across seven tasks. Darker/lighter blue mark the best/second-best results at each model size and shot setting.

Zero-shot
Model	OBQA	WinoGrande	ARC-c	ARC-e	HellaSwag	SIQA	PIQA	Avg.
Dense	25.40	54.70	31.06	66.33	37.69	39.61	68.44	46.18
Full Loop ( $K=2$ )	27.20	53.43	31.91	66.20	39.04	41.40	68.55	46.82
AsymLoop ( $K=3$ , ALRS)	26.80	55.80	32.68	68.35	38.54	40.02	69.04	47.32
Five-shot
Dense	21.20	54.54	30.72	67.00	37.21	41.20	68.77	45.81

Table 2: Zero- and five-shot downstream accuracy (

\uparrow

\%

) of 350M models trained on 35B tokens. Avg. is the mean across seven tasks. Darker/lighter blue mark the best/second-best results within each shot setting.

Scale	Loop / Total Layers	$K$	Effective Depth	Architecture	Budget	PPL
60M	0 / 6	1	6	Dense	1.68B	23.72
	3 / 6	1	6	AsymLoop (Ours)	1.68B	21.79
	6 / 6	2	12	Full Loop	1.68B	23.03
	3 / 6	3	12	AsymLoop (Ours)	1.68B	21.45
	6 / 6	3	18	Full Loop	1.68B	22.77
	3 / 6	5	18	AsymLoop (Ours)	1.68B	21.30

Table 3: Effect of recurrent depth. Validation perplexity (

\downarrow

) of Dense, Full Loop, and AsymLoop across model sizes at matched effective depths. Darker/lighter blue mark lower/higher perplexity within each depth-matched pair.

(a) Architectural components
Model Variant	PPL $\downarrow$ ( $K=3$ )	PPL $\downarrow$ ( $K=5$ )
AsymLoop (ALRS)	23.751	23.580
AsymLoop (uniform LR)	24.727	24.619
w/o QKNorm	24.984	24.663
w/o Global KV	25.566	25.751
w/o Local KV	25.656	25.465

Table 4: Ablations at the 60M scale. Validation perplexity (

\downarrow

). (a) Architectural components and (b) learning-rate ablations on AsymLoop (

K=3,5

). (c) Uniform LR versus ALRS on Full Loop (

K=2

). Darker/lighter blue mark the best/second-best results within each panel of (a, b).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	60M	135M	350M	1B
Dense parameters	91.8M	134.3M	368.2M	1.34B
AsymLoop parameters	95.4M	141.4M	393.4M	1.44B
Layers $L$ ( $E+D$ )	6 (3+3)	12 (6+6)	24 (12+12)	24 (12+12)
$d_{\text{model}}$	768	768	1024	2048
$d_{\text{ff}}$	2048	2048	2736	5461
Heads	6	12	16	32

Appendix

Table 5: Model architectures and pre-training hyperparameters. Backbone configurations follow He et al. (2026) . Parameter counts include the untied input embeddings and output head.

Figure 3: Learning-rate dynamics under ALRS at 135M and 350M. Left: Full training trajectories. Middle: The first 20% of training. Right: Decoder-to-encoder mean learning-rate ratios. Vertical dotted lines mark the end of allocation updates.

Figure 4: Normalized stable rank under uniform learning rates and ALRS. Results for 135M and 350M AsymLoop models (

K=3

). For each matrix

W

, we compute

\|W\|_{F}^{2}/[\|W\|_{2}^{2}\min(\mathrm{shape}(W))]

and average equally within each module group.

Model Variant	CE $\downarrow$ ( $K=1$ )	CE $\downarrow$ ( $K=3$ )	CE $\downarrow$ ( $K=5$ )
AsymLoop	3.238735	3.207915	3.203515
w/o Global KV	3.260346	3.241257	3.248483
Global KV gain $G(K)$	0.021611	0.033343	0.044968
Gain beyond $K=1$	0.000000	0.011732	0.023357

Appendix

Table 6: Global KV and recurrent depth. Validation CE (

\downarrow

) with and without Global KV.

G(K)

is the CE reduction from Global KV; the final row reports the additional gain

G(K)-G(1)

Model Variant	PPL $\downarrow$
AsymLoop	24.727
w/o Global KV	25.566
w/ additive injection	25.116
Full Loop	25.053

Appendix

Table 7: Context-access variants and a Full Loop baseline. Validation perplexity (

\downarrow

). Full Loop uses

K=2

; all other variants use

K=3

. Darker/lighter blue mark the best/second-best results.

Reference	$K$	Effective depth	CE	PPL
Dense, Uniform	1	6	3.271820	26.359274
Dense, layerwise LR	1	6	3.209192	24.759083
Full Loop, Uniform	2	12	3.241455	25.570904
Full Loop, Uniform	3	18	3.229346	25.263120
AsymLoop w/o QKNorm, Uniform	3	12	3.218239	24.984094
AsymLoop w/o QKNorm, Uniform	5	18	3.205305	24.663015

Appendix

Table 8: Additional validation results. Cross-entropy and perplexity (

\downarrow

) under a common evaluation protocol. Darker/lighter blue mark the best/second-best results in each metric column.

	Forward GFLOPs/token		Additional cost (%)
Model size	Full Loop	AsymLoop	Global KV proj.	Global attn.	Total
60M	0.256967	0.292356	2.75	11.02	13.77
135M	0.464585	0.535364	3.05	12.19	15.23
350M	1.476657	1.677984	3.41	10.23	13.63
1B	5.365891	5.869208	3.75	5.63	9.38

Appendix

Table 9: Analytical forward FLOPs at matched effective depth. Full Loop uses

R=2

and AsymLoop uses

K=3

, with

N=1024

and

V=32128

. Costs are reported per input position; overhead is relative to Full Loop.

Figure 5: Training compute–performance trade-offs across model scales and recurrent depths. Average zero-shot accuracy over seven downstream tasks versus analytical training FLOPs. The horizontal axis uses a logarithmic scale in EFLOPs (

10^{18}

FLOPs). Panels pair AsymLoop with the baseline of matched effective depth: (a) Dense and AsymLoop

K=1

(

L

), (b) Full Loop

K=2

and AsymLoop

K=3

(

2L

), and (c) Full Loop

K=3

and AsymLoop

K=5

(

3L

). Purple, yellow, and orange denote AsymLoop, Full Loop, and Dense, respectively.

Model size	Model	FFN	Params	s/step	Peak mem.	Validation CE
60M	Full Loop $K{=}2$	2048	91.83M	0.232	20.2 GiB	3.0434
	+ FLOPs matching	2656	100.23M	0.230	21.6 GiB	3.0139
	AsymLoop $K{=}3$	2048	95.37M	0.216	21.7 GiB	2.9684
135M	Full Loop $K{=}2$	2048	134.30M	0.286	30.9 GiB	$2.81550\pm 0.00221$
	+ FLOPs matching	2656	151.11M	0.296	33.6 GiB	$2.78983\pm 0.00197$
	AsymLoop $K{=}3$	2048	141.38M	0.367	33.8 GiB	$2.74947\pm 0.00299$

Appendix

Table 10: Validation cross-entropy and training cost with a widened Full Loop baseline. The widened baseline matches AsymLoop’s analytical per-step training FLOPs. The 60M results are single runs; the 135M results are means

\pm

sample standard deviations over three training seeds. Step time and peak memory are measured on eight H100 GPUs. Darker/lighter blue marks the best/second-best cross-entropy at each model scale.

Model size	AsymLoop	Full Loop (wide)	FFN Width	Difference vs. AsymLoop
	Measured GFLOPs/step		Widened Full Loop
60M	1134.878	1108.907	2656	$-2.29\%$
135M	2118.157	2092.198	2656	$-1.23\%$

Appendix

Table 11: Measured matrix FLOPs per training step. Counts include forward, backward, and activation recomputation, with sequence length 1024 and batch size 1.

Figure 6: Downstream performance during pre-training at 1B. Zero-shot accuracy on seven benchmarks and their average. Purple and yellow denote AsymLoop and Full Loop, respectively.

Model	Depth	CE $\downarrow$	PPL $\downarrow$
Dense	8	3.508664	33.403615
Full Loop ( $K=2$ )	16	3.501389	33.161466
AsymLoop ( $K=3$ )	16	3.341073	28.249416

Appendix

Table 12: Validation performance on an OLMo-style backbone. Cross-entropy and perplexity (

\downarrow

). Depth denotes effective depth. Darker/lighter blue marks the best/second-best results.

Model	GSM8K Accuracy (%) $\uparrow$	Relative gain over Dense (%) $\uparrow$
Dense	8.94	–
Full Loop ( $K=2$ )	10.24	+14.54
AsymLoop ( $K=3$ )	11.75	+31.43

Appendix

Table 13: GSM8K accuracy of 1B models after three epochs of full-parameter supervised fine-tuning. The best result is highlighted in blue.

60M configuration: 1.6B training tokens
Model	Effective depth	CE $\downarrow$	PPL $\downarrow$
Dense + MoE	6	2.9856	19.798
Full Loop K2 + MoE	12	2.9363	18.845
AsymLoop K3 + MoE	12	2.8986	18.149
135M configuration: 3B training tokens
Dense + MoE	12	2.7489	15.625

Appendix

Table 14: Validation performance of sparse MoE models. Cross-entropy and perplexity (

\downarrow

) with eight experts and top-2 routing. Darker/lighter blue marks the best/second-best results at each model scale.

Recurrent Language Models

See all 31 papers

StateLoop: Persistent State for Looped Transformers
Oct 26, 2026Feichen Song, Boyi Zeng, He Li +7Persistent Memory for Language Models Language Modeling
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Recurrent Transformers Efficient Transformer Inference
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
Decoding Looped Transformers Better for (Almost) Free
Oct 1, 2026Weihao Liu, Huangjie Zheng, Tianrong Chen +5Efficient Transformer Inference Language Model Decoding
Apple

Oct 26, 2026J/K move · Enter open · S save

StateLoop: Persistent State for Looped Transformers

Feichen Song, Boyi Zeng, He Li, Shixiang Song+6

Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute

Persistent Memory for Language Models Language Modeling Recurrent Transformers

Open paper AskPDF arXiv

Looped Transformers offer a parameter-efficient approach to language modeling by repeatedly applying a shared backbone. Vanilla looped Transformers use each backbone output directly as the input to the next iteration. These outputs may contain information useful several iterations later, but retaining it across the loop depends entirely on the single hidden state. We introduce StateLoop, which maintains one additional persistent state per token to retain information across iterations. At each iteration, a learned gate controls how much of the previous state is retained and how much information from the backbone output is incorporated. The updated state is projected back and added to the backbone output for subsequent iterations and final prediction. This design reduces Pile perplexity and improves average accuracy across eight zero-shot tasks over Vanilla Looped. Its perplexity advantage over Vanilla Looped grows with trained loop depth, consistent with persistent state becoming increasingly useful as recurrent computation deepens.

Abstract

Recurrent Language Models

Explore similar work

StateLoop: Persistent State for Looped Transformers

LoopFold: Parallelizing Recurrent Depth in Looped Transformers

Decoding Looped Transformers Better for (Almost) Free