Looped Transformers increase effective depth by repeatedly applying a shared block, offering a parameter-efficient approach to language modeling. However, most existing works directly reuse each loop's output as the next's input, which can lead to hidden-state scale growth and perturbation amplification as recurrence deepens. We propose Rectified Looped Transformer (RLT), which redesigns feedback between loops using an interpolation schedule inspired by flow-based modeling. At each loop, RLT forms the next input by linearly interpolating between the current state and its prediction. Such interpolation distributes state changes across loops, encouraging a smoother trajectory from the initial input to the final output. Theoretical analysis proves that, under appropriate assumptions, RLT keeps state magnitudes uniformly bounded across loops and reduces the worst-case perturbation amplification. We evaluate RLT on language modeling corpora and downstream benchmarks. At matched training and inference depths, RLT consistently achieves lower perplexity than a standard looped Transformer, while exhibiting greater robustness to input perturbations and variance in inference depth. Consistent with the theoretical results, latent trajectory analysis shows that RLT maintains more stable hidden-state norms across loops, which also facilitates extrapolation-based decoding strategies. Notably, RLT achieves these gains without requiring additional trainable parameters or extra recurrent block evaluations, thus providing a simple yet more stable feedback mechanism for looped Transformers.
Figures & tables
Figure 1: Overview of Rectified Looped Transformer (RLT). (a) SLT and RLT share the same looped architecture and uniform all-step supervision. RLT replaces direct feedback with interpolation between the current input and the predicted endpoint. (b) Schematic latent dynamics under matched and mismatched loop counts, illustrating more uniform state advancement and more stable final predictions in RLT, while SLT may fall short or drift beyond the target region.
Variants
Structure
Perplexity ↓
Accuracy (%) ↑
Pile
FWE
OWT
COPA
HS
LB
OBQA
PIQA
RACE
SciQ
ARC
SIQA
WG
Avg Acc
K=8
SLT
3⊗8
11.08
27.58
25.39
64.00
36.50
33.00
29.50
62.50
28.57
66.50
30.22
36.92
55.91
44.36
RLT
3⊗8
10.82
26.74
24.61
62.00
36.50
34.00
28.00
60.87
29.52
70.50
31.31
38.46
44.09
43.53
K=4
SLT
3⊗4
11.76
29.76
27.30
64.00
35.50
26.00
27.50
59.78
26.67
66.50
32.38
37.44
48.03
42.38
Table 1: Language modeling and downstream performance at matched depth K=Ktrain=Ktest . Structure denotes shared layers ⊗ loops. Accuracy is reported in percentages; Avg Acc is the unweighted mean across ten benchmarks. Bold marks the better result within each depth group.
Figure 2: Cross-loop hidden-state scales relative to the first-loop output h1 (left) and the initial embedding input x0 (right), with Ktrain=Ktest=8 .
Figure 3: Final-output displacement after injecting Gaussian noise into the initial embedding input x0 . Distances are averaged over tokens and measured before RMS normalization.
Figure 4: Performance under recurrent perturbations applied to h before feedback (top) or x after feedback (bottom). ΔCE and ΔAcc are changes relative to each model’s clean baseline.
Figure 6Figure 7
Figure 9: 3D PCA projections of final-token hidden outputs h for three OpenWebText windows. SLT and RLT are on the top and bottom rows, respectively. Colors indicate the inference budget; stars and squares mark h1 and hK . The outlined green square marks the matched-depth endpoint.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Setting
Number of shared Transformer layers
3
Default training / inference loops (Ktrain/Ktest)
8 / 8
Hidden dimension d
2,048
Attention heads / head dimension
32 / 64
Feed-forward intermediate dimension
5,120
Feed-forward activation
GELU
Appendix
Table 3: Default model configuration.
Hyperparameter
Setting
Optimizer
AdamW
Peak learning rate
6×10−4
Minimum learning rate
6×10−5
Warmup steps
4,000
Optimizer steps
14,000
Adam (β1,β2)
(0.9,0.95)
Appendix
Table 4: Training hyperparameters shared by SLT and RLT.
Corpus
Available windows
Evaluation windows
Evaluation tokens
The Pile
364,548
36,455
37.33M
FineWeb-Edu
8,437
844
0.864M
OpenWebText
4,331
433
0.443M
Total
377,316
37,732
38.64M
Appendix
Table 5: Corpora used for perplexity evaluation.
Benchmark
Original size
Evaluation size
Metric
COPA
100
100
Acc
HellaSwag
10,042
200
Accnorm
LAMBADA
5,153
200
Acc
OpenBookQA
500
200
Accnorm
PIQA
1,838
184
Accnorm
RACE-high
1,045
105
Acc
Appendix
Table 6: Downstream evaluation sample sizes and metrics.
Configuration
Weight quantization
Activation format
W8A16
INT8, per output channel
BF16
W4A16
INT4, groups of 128
BF16
W8A8
INT8, per output channel
Dynamic per-token INT8
Appendix
Table 7: Quantization configurations for the shared Transformer blocks. Weight groups are formed along the input dimension within each output channel.
Figure 10: 3D PCA projections of final-token hidden outputs for three additional OpenWebText windows. Each row corresponds to one window; SLT and RLT occupy the left and right columns, respectively. Colors indicate the inference budget K=Ktest ; stars and squares mark h1 and hK . The outlined green square marks the matched-depth endpoint.
Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream task performance at a fixed backbone parameter count. However, each loop depends on the preceding loop, so increasing recurrent depth also increases decoding latency. Existing approaches include Parallel Loop Transformers, which reduce latency by overlapping loop computation across token positions, but yield rapidly diminishing marginal accuracy gains at larger depths. To address this challenge, we introduce LoopFold, which folds computation at multiple recurrent depths into parallel batches for the same token prefix. A lightweight predictor estimates inputs to later loops so they can start before earlier loops finish. A corrector then refines the outputs of these later loops using newly available results from earlier loops. All components are trained jointly using only the final next-token prediction loss. In experiments with 1.4B models, deeper LoopFold configurations achieve progressively higher average zero-shot accuracy across eight tasks, rising from 53.41% to 56.38%. Vanilla looped models and PLT show limited gains over the same depth range. Compared with a nine-loop vanilla model, LoopFold uses five sequential stages and additional parallel computation to reduce decoding latency by 46%.
Feichen Song, Boyi Zeng, Yiqin Hao +7
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
Scaling model performance typically requires increasing model size. Looped Transformer offers a compelling alternative by iteratively reusing the same Transformer blocks, trading additional computation for improved performance without increasing parameter count or context length. Because the number of loop iterations can be adjusted at inference, it also provides a natural mechanism for balancing performance and test-time compute. However, Looped Transformer still suffers from training instability when the number of loop iterations increases. Our analysis reveals that this instability stems from two sources: gradient oscillation and residual explosion. To address these two problems, we propose the Fully Looped Transformer, which introduces two parameter-free modifications: (1) Fully Looped Architecture, which distributes inter-loop signals across all layers to mitigate residual explosion; (2) Attention Injection, which reuses the existing attention block to suppress gradient oscillation. These modifications stabilize training dynamics, enabling the Fully Looped Transformer to be trained stably up to 12 loop iterations, whereas other baseline looped models collapse in this regime. In milder settings where Looped Transformer does not collapse, Fully Looped Transformer still improves average downstream-task performance by up to 13.2%. Overall, our experiments demonstrate that Fully Looped Transformer improves training stability, enhances downstream performance, and provides preliminary adaptability under different test-time compute budgets by varying loop iterations at inference.
Looped Transformers offer a parameter-efficient approach to language modeling by repeatedly applying a shared backbone. Vanilla looped Transformers use each backbone output directly as the input to the next iteration. These outputs may contain information useful several iterations later, but retaining it across the loop depends entirely on the single hidden state. We introduce StateLoop, which maintains one additional persistent state per token to retain information across iterations. At each iteration, a learned gate controls how much of the previous state is retained and how much information from the backbone output is incorporated. The updated state is projected back and added to the backbone output for subsequent iterations and final prediction. This design reduces Pile perplexity and improves average accuracy across eight zero-shot tasks over Vanilla Looped. Its perplexity advantage over Vanilla Looped grows with trained loop depth, consistent with persistent state becoming increasingly useful as recurrent computation deepens.
Feichen Song, Boyi Zeng, He Li +7
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute