Parcae: Scaling Laws For Stable Looped Language Models

Hayden Prairie; Zachary Novack; Taylor Berg-Kirkpatrick; Daniel Y. Fu

cs.LGApr 14, 2026

Parcae: Scaling Laws For Stable Looped Language Models

Authors: Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, Daniel Y. Fu

Organizations: University of California, San Diego · Together AI

Abstract

Traditional fixed-depth architectures scale quality by increasing training FLOPs, typically through increased parameterization, at the expense of a higher memory footprint, or data. A potential alternative is looped architectures, which instead increase FLOPs by sending activations through a block of layers in a loop. While promising, existing recipes for training looped architectures can be unstable, suffering from residual explosion and loss spikes. We address these challenges by recasting looping as a nonlinear time-variant dynamical system over the residual stream. Via a linear approximation to this system, we find that instability occurs in existing looped architectures as a result of large spectral norms in their injection parameters. To address these instability issues, we propose Parcae, a novel stable, looped architecture that constrains the spectral norm of the injection parameters via discretization of a negative diagonal parameterization. As a result, Parcae achieves up to 6.3% lower validation perplexity over prior large-scale looped models. Using our stable looped architecture, we investigate the scaling properties of looping as a medium to improve quality by increasing FLOPs in training and test-time. For training, we derive predictable power laws to scale FLOPs while keeping parameter count fixed. Our initial scaling laws suggest that looping and data should be increased in tandem, given a fixed FLOP budget. At test-time, we find that Parcae can use looping to scale compute, following a predictable, saturating exponential decay. When scaled up to 1.3B parameters, we find that Parcae improves CORE and Core-Extended quality by 2.99 and 1.18 points when compared to strong Transformer baselines under a fixed parameter and data budget, achieving a relative quality of up to 87.5% a Transformer twice the size.

Figures & tables

Figure 1: Parcae and the Scaling Laws of Looping. ( Left ) Parcae constrains the spectral norm of

\overline{\bm{A}}

and normalizes the input injection, stabilizing the residual stream

h_{t}

across loops. ( Right ) We observe looping to be an orthogonal axis of scaling compute which follows a power law.

Figure 2: Training Instability of Looped Architectures. ( left ) Pre-Norm looped models diverge, while residual norm. and Parcae converge. ( right ) Instability stems from an exploding recurrent state norm

||h_{T}||_{2}

, the hidden embedding norm after

T

recurrences.

Method	$\overline{\bm{A}}$	$\overline{\bm{B}}$	$\rho(\overline{\bm{A}})$	LTI Stability
Addition	$I$	$I$	$\rho(\overline{\bm{A}})=1$	marginally-stable
Concatenation	$\mathbb{R}^{d_{h}\times d_{h}}$	$\mathbb{R}^{d_{h}\times d_{e}}$	$\rho(\overline{\bm{A}})\in\mathbb{R}$	unstable
Parcae (ours)	$\text{ZOH}(\texttt{Diag}(-\exp(\mathbb{R}^{d_{h}}))$	$\text{Euler}(\mathbb{R}^{d_{h}\times d_{e}})$	$\rho(\overline{\bm{A}})<1$	stable

Table 1: Comparison of Prior Update Rule Stability based on LTI Representation.

	Model	$\mathbf{T}$	Val.	WikiText	Hellaswag	ARC-c	ARC-e	PIQA	BoolQ	SciQ	Avg.
100M	RDM	16	14.23	63.27	27.16	17.66	42.38	59.14	51.35	72.50	45.03
100M	Parcae	16	13.59	60.33	27.18	18.09	43.10	59.30	61.83	71.50	46.83
350M	RDM	8	10.76	41.31	28.55	20.90	47.26	61.75	61.53	76.70	49.45
350M	Parcae	8	10.09	37.53	29.23	21.08	48.78	62.08	60.73	78.80	50.12

Table 3: Zero-Shot and Perplexity Results Trained on RDM Setup. Comparison of Parcae and RDM [ 27 ] on a variety of open source benchmarks and perplexity held-out validation set and Wikitext [ 50 ] . Best results are bolded .

Configuration	$T\!=\!1$	$T\!=\!4$	$T\!=\!8$	$T\!=\!1$	$T\!=\!4$	$T\!=\!8$	$T\!=\!1$	$T\!=\!4$	$T\!=\!8$
	Val Loss ( $\downarrow$ )			Core ( $\uparrow$ )			Core Ext ( $\uparrow$ )
RDM	Divergent Training			Divergent Training			Divergent Training
+ Constrained $\overline{\bm{A}}$	8.99	3.15	2.97	$-2.0_{\pm 0.1}$	$11.0_{\pm 0.1}$	$13.2_{\pm 0.2}$	$0.5_{\pm 0.1}$	$7.8_{\pm 0.0}$	$9.1_{\pm 0.5}$
+ Per-Seq. Sampling	3.38	3.01	2.98	$\mathbf{7.6_{\pm 0.2}}$	$13.4_{\pm 0.2}$	$14.0_{\pm 0.2}$	$\mathbf{5.9_{\pm 0.4}}$	$9.3_{\pm 0.2}$	$\mathbf{9.9_{\pm 0.2}}$
+ Prelude Norm	3.28	2.97	2.95	$7.5_{\pm 0.3}$	$\mathbf{13.5_{\pm 0.0}}$	$\mathbf{14.0_{\pm 0.2}}$	$5.8_{\pm 0.3}$	$\mathbf{9.4_{\pm 0.1}}$	$9.7_{\pm 0.3}$

Table 4: Stability Results Trained on Transformer Setup. To illustrate stability, we retrofit a baseline 140M Transformer into a RDM and then sequentially add our stability improvements.

	Model	$\mathbf{T}$	Val. PPL ( $\downarrow$ )	Lambada PPL ( $\downarrow$ )	Core ( $\uparrow$ )	Core-Extended ( $\uparrow$ )
140M	Transformer	–	21.48	127.39	13.00 ± 0.15	8.80 ± 0.21
140M	Parcae	8	19.06	80.64	14.04 ± 0.20	9.67 ± 0.28
370M	Transformer	–	15.79	40.77	17.46 ± 0.03	11.71 ± 0.22
370M	Parcae	8	14.49	32.74	20.00 ± 0.06	12.75 ± 0.31
770M	Transformer	–	13.08	22.37	22.42 ± 0.20	14.20 ± 0.63
770M	Parcae	8	12.49	19.71	25.07 ± 0.33	15.19 ± 0.43

Table 5: Comparing Parcae to Fixed-Depth Transformers. We pretrain Transformers and Parcae with a nanochat setup at several scales, evaluating on a held-out validation set, Lambada [ 58 ] , Core, and Core-Extended [ 45 ] . Best results are bolded.

Figure 4: Looping Scales Training Compute Optimally. ( Left ) Parametric isoLoss contours over $\mu_{\text{rec}}$ and data. The efficient frontier (blue line) traces the lowest FLOP budget required to achieve each loss level, showing that optimal training requires increased looping. ( Right ) Parabolic isoFLOP fits for 140M and 370M models reveal a clear optimum $\mu_{\text{rec}}$ at each FLOP budget, indicating that looping is an orthogonal scaling axis to data.

Figure 5: Optimal

\mu_{\text{rec}}

and Tokens Follows Predictable Power Laws. We fit a parabola to each isoFLOP budget for both 140M and 370M Parcae models, using its minima to approximate the optimal

\mu_{\text{rec}}

and token budget at each scale. We observe that optimal recurrence ( left plots ) and tokens ( right plots ) follow a predictable power law with similar coefficients at both scales.

Figure 7: Test-Time Scaling of Parcae . When evaluating Parcae models from Table 5 , we observe test-time looping follows a predictable saturating trend, consistent across model sizes.

Figure 8: Scaling Test-Time Compute follows a Predictable Power Laws. We plot the validation loss with different

\mu_{\text{rec}}

as a function of test-time recurrence

T

, and find the fitted exponential decay (solid curve for each

\mu_{\text{rec}}

) tightly captures the test-time performance of looping.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$d$	Embedding dimension of the model
$t$	Discrete temporal state axis of $\mathcal{R}$ on $\mathbb{N}$
$b$	Global batch size used during pretraining
$\mathcal{P}$	Initial prelude block of a recurrent architecture
$\mathcal{R}$	Middle recurrent block of a recurrent architecture
$\mathcal{C}$	Final coda block of a recurrent architecture

Appendix

Table 7: Glossary of notation and terminology. ( Top ) Frequently used dimensions for tensors. ( Middle ) Definition of Parcae blocks. ( Bottom ) Tensors and distributions are used to express recurrent depth models.

Figure 9: Training instability of recurrent depth models across different learning rates. We show both training losses and recurrent state norm to understand divergence and state explosion.

Figure 10: Training curves showing per-sequence sampling effectively eliminates loss spikes in training over per-micro-batch sampling.

Figure 11: Comparison of recurrent residual and state norm metrics (defined in Section A.1 ), which show that per-sequence sampling enables stronger fixed point behavior in training.

	Method	T=1	T=4	T=8	T=16
100M	Per-Batch	300.32	36.75	16.65	13.81
100M	Per-Sequence	70.47	17.15	14.08	13.59
350M	Per-Batch	167.61	12.80	10.40	10.24
350M	Per-Sequence	17.92	10.49	10.09	10.11

Appendix

Table 8: Per-Microbatch vs. Per-Sequence Comparison . We compare perplexity of Parcae models trained with per-microbatch sampling [ 27 ] and per-sequence sampling, using different recurrences (

T

) on a held-out validation set. Bolded results indicate best at each scale.

Figure 12: Comparison of our sampling method with [ 27 ] . It can be observed that the actual distribution of forward recurrence for [ 27 ] is a shifted Poisson distribution. The implications of this sampling strategy can be better visualized in Figure 13 .

Figure 13: A distributional mismatch can be observed from the recurrent sampling method of [ 27 ] . Specifically, if our desired pre-training distribution for

\mu_{\text{rec}}

is a Poisson distribution, the distribution total recurrence

T

of [ 27 ] is truncated based on

\mu_{\text{bwd}}

. However, our sampling method decouples the effects of

\mu_{\text{bwd}}

\Lambda

, allowing the recurrent distribution to be faithfully sampled from.

Figure 14: Training and validation curves of three 100 million parameter Parcae models pretrained on 10 billion tokens, comparing different truncated back-propagation methods (baseline is a model with no back-propagation truncation). Each model has identical architecture and hyperparameters, with

\mu_{\text{rec}}

and

\mu_{\text{bwd}}

both being set to eight, all using

\Lambda\sim\text{Poisson}

. It can be observed that even though each model has similar training loss and validation loss when using

T=8

, our implementation more faithfully follows the validation loss of full back-propagation. Specifically for

T=4

, our implementation significantly improves validation loss compared to [ 27 ] sampling method.

Figure 15: Validation curves of six different recurrent depth models, pretrained on 10 billion tokens, with a fixed architecture and hyperparameters. Each model is pretrained with a fixed

\mu_{\text{bwd}}

of 8 and varying

\mu_{\text{rec}}

[4,8,14,20,26,32]

. The key observation is that scaling up

\mu_{\text{rec}}

while keeping

\mu_{\text{bwd}}

fixed results in models that perform worse than if just pretrained on

\mu_{\text{rec}}

of eight.

	$\mu_{\text{rec}}=4$	$\mu_{\text{rec}}=8$	$\mu_{\text{rec}}=14$	$\mu_{\text{rec}}=20$	$\mu_{\text{rec}}=26$	$\mu_{\text{rec}}=32$
Val Loss	2.477	2.453	2.456	2.457	2.458	2.458
Val Perplexity	11.906	11.624	11.665	11.671	11.692	11.687

Appendix

Table 9: Validation loss and perplexity for looped models trained with different

\mu_{\text{rec}}

and a fixed

\mu_{\text{bwd}}=4

. We use

T=\mu_{\text{rec}}

. Surprisingly,

\mu_{\text{rec}}=8

performs the best.

Figure 16: Validation and training curves of looped models, pretrained on 8.5 billion tokens. Each model is trained with a fixed

\mu_{\text{rec}}=20

and

\mu_{bwd}\in[4,6,8,10,12]

. Observe that scaling up

\mu_{\text{bwd}}

improves validation performance at higher and lower recurrences monotonically for

T=1,16,64

	$\mu_{\text{bwd}}=4$	$\mu_{\text{bwd}}=6$	$\mu_{\text{bwd}}=8$	$\mu_{\text{bwd}}=10$	$\mu_{\text{bwd}}=12$
Val Loss	2.500	2.490	2.480	2.479	2.474
Val Perplexity	12.09	12.06	11.94	11.93	11.86

Appendix

Table 10: Validation loss and perplexity of looped models trained with variable

\mu_{\text{bwd}}

, but fixed

\mu_{\text{rec}}

Figure 17: Late Stage Instability of 1.3B Parcae models. We observe loss spikes and state explosion at the final stages of our large-scale run.

Figure 18: Spectral Norms of

\overline{\bm{A}},\overline{\bm{B}},\bm{C}

throughout training 1.3B Parcae. We find that the spectral norm of

\overline{\bm{A}}

and

\overline{\bm{B}}

remain stable throughout training, while the spectral norm of

\bm{C}

grows.

Figure 19: Comparison of

\bm{C}

Amplification with Spectral Norm. We observe that the actual expansion ratio of

\bm{C}

is small and decreasing slowly throughout training.

Figure 20: Empirical Average of Recurrent State Norm over $T$ iterations. For each checkpoint we have for our failed 1.3B Parcae model run, we evaluate the recurrent norm through $T=24$ recurrences at test time, on a held out validation set of fineweb-edu [ 60 ] . We find that after an initial explosion on the first recurrence, the state remains relatively stable.

Figure 21: Recurrent State Norm Progression After Each Transformer Block for $T=1$ . For each checkpoint we have for our failed 1.3B Parcae model run, we evaluate the recurrent norm after injection and each non-linear transformer block for only $T=1$ . We find that the non-linear parts of Parcae have little effect on explosion, which instead mainly stems from the initial injection of prelude output $e$ .

Figure 22: State Norm Progression Throughout each Transformer Layer in the Prelude Block. For each checkpoint we have for our failed 1.3B Parcae model run, we evaluate residual norm after each transformer block in the prelude $\mathcal{P}$ . We find that a single layer creates an explosion of the residual norm and leads to divergence.

Figure 23: Prelude Norm Stabilizes Recurrent Norm. We find that prelude norm helps stabilize recurrent state norm in Parcae models following the setup in Section 5.1 for Transformers.

Figure 24: Prelude Norm Improves Quality. We find that in our 140M and 370M Parcae models trained in the same setup as Section 5.1 for Transformers, normalizing the prelude output leads to better convergence.

Figure 25: Parametric Fit of Looping. Visualization of our parametric function $\widehat{\mathcal{L}}_{\text{train}}(\mu_{\text{rec}},D)$ , which displays the IsoLoss contours for both 140M Parcae ( left ) and 370M Parcae ( right ) models.

Model	$\boldsymbol{E}$	$\boldsymbol{A}$	$\boldsymbol{a}$	$\boldsymbol{B}$	$\boldsymbol{b}$	Huber ( $\times 10^{-4}$ )
Small (140M)	2.662	522733.307	0.771	25420.102	0.525	0.44
Medium (370M)	2.439	832134.346	0.775	6386.865	0.448	0.01

Appendix

Table 11: Optimal Scaling Coefficients for Parametric Fits.

	$\mathcal{L}_{\infty}{+}Ze^{-zT}$	$\mathcal{L}_{\infty}{+}Z(1{+}T)^{-z}$	$\mathcal{L}_{\infty}{+}ZT^{-z}$	$ZT^{-z}$
In-Distribution
140M	2.52	5.42	11.11	112.89
370M	1.88	5.26	10.77	104.95
Extrapolation ( $T>\mu_{\text{rec}}$ )
140M	3.18	21.41	43.99	397.90
370M	2.29	18.51	38.68	369.83

Appendix

Table 12: Functional form comparison for test-time scaling. We report average Huber loss (

\times 10^{-7}

) across all per-curve fits, both in-distribution (all

T

) and in extrapolation (fit

T\leq\mu_{\text{rec}}

, evaluate

T>\mu_{\text{rec}}

). Lower is better.

Model	Mean % Err	Max % Err
140M	0.16%	0.59%
370M	0.05%	0.22%

Appendix

Table 13: Mean and max absolute percent error between

\mathcal{L}_{\infty}

and

\mathcal{L}(T{=}\mu_{\text{rec}})

across all isoFLOP configurations.

	$Ze^{-z\mu^{-\gamma}T}$	$Ze^{-zT/\mu}$ ( $\gamma{=}1$ )	$Ze^{-zT}$ (no $\mu$ )
Train (isoFLOP)
140M	0.001116	0.001177	0.003253
370M	0.000229	0.000283	0.001438
Test (held-out, $\mu_{\text{rec}}{=}8$ )
140M	0.000207	0.000212	0.000266
370M	0.000133	0.000131	0.000189

Appendix

Table 14: Ablation of

\mu_{\text{rec}}

conditioning in the unified test-time law. We report total Huber loss on the isoFLOP training set and on held-out Table 5 models (

\mu_{\text{rec}}=8

, fixed data budget). Lower is better.

Figure 26: Out-of-Distribution Prediction of Unified Parametric Fit. We visualize the prediction of our unified parametric fit (orange) and an oracle fit using the empirical loss at

T=\mu_{\text{rec}}

for

\widehat{\mathcal{L}}_{\text{train}}

(blue) against empirical validation loss with increasing

T

for models trained in Section 5.1 .

Category	Task	Type	Shots	Core
Understanding	HellaSwag [ 88 ] (0-shot)	MC	0	✓
	HellaSwag [ 88 ] (10-shot)	MC	10	✓
	Lambada [ 58 ]	LM	0	✓
	Winograd WSC [ 9 ]	S	0	✓
	WinoGrande [ 66 ]	S	0	✓
	BIG-Bench Language ID [ 72 ]	MC	10	✓

Appendix

Table 15: Full list of downstream evaluation Tasks marked with ✓ are included in the Core [ 45 ] ; all tasks are included in Core-Extended [ 45 ] . Type indicates the scoring method: MC (multiple choice, lowest mean NLL), S (schema-based NLL), or LM (exact greedy match).

	FLOPs		Optimal $\mu_{\mathrm{rec}}^{*}$		Fixed-Depth ( $\mu_{\mathrm{rec}}=1$ )
	( $\times 10^{18}$ )	$\mu_{\mathrm{rec}}^{*}$	Core	Core Ext.	Core	Core Ext.
140M	$1$	2	$7.6\pm 0.3$	$5.7\pm 0.5$	$\mathbf{7.9\pm 0.2}$	$\mathbf{6.1\pm 0.1}$
	$2$	2	$9.0\pm 0.2$	$6.2\pm 0.1$	$\mathbf{10.5\pm 0.1}$	$\mathbf{6.4\pm 0.2}$
	$4$	4	$\mathbf{11.2\pm 0.0}$	$\mathbf{8.4\pm 0.2}$	$10.7\pm 0.1$	$8.1\pm 0.3$
	$8$	6	$10.5\pm 0.1$	$\mathbf{7.8\pm 0.2}$	$\mathbf{11.8\pm 0.2}$	$7.7\pm 0.2$
	$16$	8	$\mathbf{14.6\pm 0.1}$	$\mathbf{9.8\pm 0.4}$	$13.0\pm 0.2$	$8.8\pm 0.4$

Appendix

Table 16: Expanded Core Scores Comparison of Looping Optimal Frontier over Purely Scaling Data. Including variance bars now.

	Parcae-100M	Parcae-350M	RDM-100M	RDM-350M
Parameters	114,242,560	378,558,464	114,242,560	382,765,056
Layers in $\mathcal{P}$	1	1	1	1
Layers in $\mathcal{C}$	1	1	1	1
Layers in $\mathcal{R}$	1	2	1	2
$d_{\text{model}}$	1,024	2,048	1,024	2,048
$d_{\text{intermediate}}$	3,520	7,040	3,520	7,040

Appendix

Table 17: Model definitions of both Parcae and baseline residual-norm RDMs [ 27 ] .

		Small (140M)	Medium (370M)	Large (770M)	XLarge (1.3B)
Architecture	Layers (Transformer)	6	12	18	24
	Layers in $\mathcal{P}$ (Parcae)	2	4	6	8
	Layers in $\mathcal{R}$ (Parcae)	2	4	6	8
	Layers in $\mathcal{C}$ (Parcae)	2	4	6	8
	$d_{\text{model}}$	768	1,024	1,280	1,536
	$d_{\text{intermediate}}$	3,072	4,096	5,120	6,144

Appendix

Table 18: Model definitions of both Parcae and baseline Transformers.

	Small (140M)	Medium (370M)	Large (770M)	XLarge (1.3B)
Transformer Parameters	143,141,184	385,903,104	773,375,040	1,333,868,544
Parcae Parameters	144,323,136	388,003,328	776,655,680	1,338,591,744
Additional Parameters	1,181,952	2,100,224	3,280,640	4,723,200
Additional (%)	0.83%	0.54%	0.42%	0.35%

Appendix

Table 19: Comparison of Parcae and Transformer parameter count.

Optimizer	Parameters
AdamW [ 43 ]	Token embeddings ( wte )
	LM head ( lm_head )
	Normalization layers ( RMSNorm )
	Value embedding gates ( ve_gate )
	All 1D parameters
AdamW [ 43 ] (Parcae only)	Injection parameters ( $\bm{A}$ , $\Delta$ , $\bm{B}$ )

Appendix

Table 20: Optimizer parameter group assignment for Parcae and baseline Transformers.

	Small (140M)	Medium (370M)	Large (770M)	XLarge (1.3B)
Training Tokens	11.2B	29.6B	61.6B	104B
Batch Size (sequences)	256	256	256	256
Sequence Length	2,048	2,048	2,048	2,048
Precision	bf16-mixed
AdamW LR	$8\times 10^{-3}$	$8\times 10^{-3}$	$6\times 10^{-3}$	$6\times 10^{-3}$
AdamW $(\beta_{1},\beta_{2})$	$(0.8,0.95)$

Appendix

Table 21: Hyperparameter used from training Parcae and Transformer models in Section 5.1 for Transformers.

Tokenizer	Vocab Size	Bytes/Token $\uparrow$
		Train	Val
GPT-2 ( gpt2 )	50,257	4.67	4.63
GPT-4 ( cl100k )	100,277	4.81	4.76
Ours	32,768	4.72	4.65

Appendix

Table 22: Compression ratio (bytes per token) on FineWeb-Edu for tokenizer used in training.

Explore similar work

Jun 16, 2026

On the Residual Scaling of Looped Transformers: Stability and Transferability

Looped (weight-tied) Transformers apply a shared residual block

N

times (

h \leftarrow h + \varepsilon\,f(h)

, same

f

at each step), increasing effective depth without adding parameters. Prior depth-scaling analyses prescribe

\varepsilon = 1/\!\sqrt{L}

for depth-

L

residual networks. We show that this is insufficient for looped architectures: weight sharing makes residual updates correlated across iterations, requiring the stronger scaling

\varepsilon = 1/N

. For multi-layer blocks (

L

unique layers looped

N

times), we derive a factored parameterization

\varepsilon = \lambda/(N\!\sqrt{L})

that separates the two sources of growth:

1/N

controls the within-layer loop correlation, and

1/\!\sqrt{L}

controls the across-layer variance. A key consequence is that the optimal learning rate depends only on the number of unique layers

L

, not on the loop count

N

, enabling direct hyperparameter transfer from small to large

N

without retuning. Experiments on looped Transformers confirm that

1/N

scaling improves trainability and yields better loss than

1/\!\sqrt{N}

scaling across loop counts.

Shaowen Wang, Bingrui Li, Ge Zhang +3

Tsinghua University · ByteDance Seed · M-A-P

Residual Learning Hyperparameter Transfer Recurrent Transformers Neural Network Training Dynamics

Ask PDF

May 11, 2026

Simply Stabilizing the Loop via Fully Looped Transformer

Scaling model performance typically requires increasing model size. Looped Transformer offers a compelling alternative by iteratively reusing the same Transformer blocks, trading additional computation for improved performance without increasing parameter count or context length. Because the number of loop iterations can be adjusted at inference, it also provides a natural mechanism for balancing performance and test-time compute. However, Looped Transformer still suffers from training instability when the number of loop iterations increases. Our analysis reveals that this instability stems from two sources: gradient oscillation and residual explosion. To address these two problems, we propose the Fully Looped Transformer, which introduces two parameter-free modifications: (1) Fully Looped Architecture, which distributes inter-loop signals across all layers to mitigate residual explosion; (2) Attention Injection, which reuses the existing attention block to suppress gradient oscillation. These modifications stabilize training dynamics, enabling the Fully Looped Transformer to be trained stably up to 12 loop iterations, whereas other baseline looped models collapse in this regime. In milder settings where Looped Transformer does not collapse, Fully Looped Transformer still improves average downstream-task performance by up to 13.2%. Overall, our experiments demonstrate that Fully Looped Transformer improves training stability, enhances downstream performance, and provides preliminary adaptability under different test-time compute budgets by varying loop iterations at inference.

Rao Fu, Zixuan Yang, Jiankun Zhang +4

Hong Kong Baptist University · Jilin University

Test-Time Scaling Recurrent Transformers Neural Network Training Dynamics

Ask PDF

May 9, 2026

Sparse Layers are Critical to Scaling Looped Language Models

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

Ryan Lee, Jacob Biloki, Edward J. Hu +1

1USC Information Sciences Institute · 2Netflix · 3Independent Researcher

Mixture-of-Experts Language Models Recurrent Transformers Language Model Scaling LLM Inference Acceleration Language Modeling Recursive Language Models

Ask PDF