Looped Transformers increase effective depth by repeatedly applying a shared block, offering a parameter-efficient approach to language modeling. However, most existing works directly reuse each loop's output as the next's input, which can lead to hidden-state scale growth and perturbation amplification as recurrence deepens. We propose Rectified Looped Transformer (RLT), which redesigns feedback between loops using an interpolation schedule inspired by flow-based modeling. At each loop, RLT forms the next input by linearly interpolating between the current state and its prediction. Such interpolation distributes state changes across loops, encouraging a smoother trajectory from the initial input to the final output. Theoretical analysis proves that, under appropriate assumptions, RLT keeps state magnitudes uniformly bounded across loops and reduces the worst-case perturbation amplification. We evaluate RLT on language modeling corpora and downstream benchmarks. At matched training and inference depths, RLT consistently achieves lower perplexity than a standard looped Transformer, while exhibiting greater robustness to input perturbations and variance in inference depth. Consistent with the theoretical results, latent trajectory analysis shows that RLT maintains more stable hidden-state norms across loops, which also facilitates extrapolation-based decoding strategies. Notably, RLT achieves these gains without requiring additional trainable parameters or extra recurrent block evaluations, thus providing a simple yet more stable feedback mechanism for looped Transformers.
Figures & tables
Figure 1: Overview of Rectified Looped Transformer (RLT). (a) SLT and RLT share the same looped architecture and uniform all-step supervision. RLT replaces direct feedback with interpolation between the current input and the predicted endpoint. (b) Schematic latent dynamics under matched and mismatched loop counts, illustrating more uniform state advancement and more stable final predictions in RLT, while SLT may fall short or drift beyond the target region.
Variants
Structure
Perplexity ↓
Accuracy (%) ↑
Pile
FWE
OWT
COPA
HS
LB
OBQA
PIQA
RACE
SciQ
ARC
SIQA
WG
Avg Acc
K=8
SLT
3⊗8
11.08
27.58
25.39
64.00
36.50
33.00
29.50
62.50
28.57
66.50
30.22
36.92
55.91
44.36
RLT
3⊗8
10.82
26.74
24.61
62.00
36.50
34.00
28.00
60.87
29.52
70.50
31.31
38.46
44.09
43.53
K=4
SLT
3⊗4
11.76
29.76
27.30
64.00
35.50
26.00
27.50
59.78
26.67
66.50
32.38
37.44
48.03
42.38
Table 1: Language modeling and downstream performance at matched depth K=Ktrain=Ktest . Structure denotes shared layers ⊗ loops. Accuracy is reported in percentages; Avg Acc is the unweighted mean across ten benchmarks. Bold marks the better result within each depth group.
Figure 2: Cross-loop hidden-state scales relative to the first-loop output h1 (left) and the initial embedding input x0 (right), with Ktrain=Ktest=8 .
Figure 3: Final-output displacement after injecting Gaussian noise into the initial embedding input x0 . Distances are averaged over tokens and measured before RMS normalization.
Figure 4: Performance under recurrent perturbations applied to h before feedback (top) or x after feedback (bottom). ΔCE and ΔAcc are changes relative to each model’s clean baseline.
Figure 6Figure 7
Figure 9: 3D PCA projections of final-token hidden outputs h for three OpenWebText windows. SLT and RLT are on the top and bottom rows, respectively. Colors indicate the inference budget; stars and squares mark h1 and hK . The outlined green square marks the matched-depth endpoint.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Setting
Number of shared Transformer layers
3
Default training / inference loops (Ktrain/Ktest)
8 / 8
Hidden dimension d
2,048
Attention heads / head dimension
32 / 64
Feed-forward intermediate dimension
5,120
Feed-forward activation
GELU
Appendix
Table 3: Default model configuration.
Hyperparameter
Setting
Optimizer
AdamW
Peak learning rate
6×10−4
Minimum learning rate
6×10−5
Warmup steps
4,000
Optimizer steps
14,000
Adam (β1,β2)
(0.9,0.95)
Appendix
Table 4: Training hyperparameters shared by SLT and RLT.
Corpus
Available windows
Evaluation windows
Evaluation tokens
The Pile
364,548
36,455
37.33M
FineWeb-Edu
8,437
844
0.864M
OpenWebText
4,331
433
0.443M
Total
377,316
37,732
38.64M
Appendix
Table 5: Corpora used for perplexity evaluation.
Benchmark
Original size
Evaluation size
Metric
COPA
100
100
Acc
HellaSwag
10,042
200
Accnorm
LAMBADA
5,153
200
Acc
OpenBookQA
500
200
Accnorm
PIQA
1,838
184
Accnorm
RACE-high
1,045
105
Acc
Appendix
Table 6: Downstream evaluation sample sizes and metrics.
Configuration
Weight quantization
Activation format
W8A16
INT8, per output channel
BF16
W4A16
INT4, groups of 128
BF16
W8A8
INT8, per output channel
Dynamic per-token INT8
Appendix
Table 7: Quantization configurations for the shared Transformer blocks. Weight groups are formed along the input dimension within each output channel.
Figure 10: 3D PCA projections of final-token hidden outputs for three additional OpenWebText windows. Each row corresponds to one window; SLT and RLT occupy the left and right columns, respectively. Colors indicate the inference budget K=Ktest ; stars and squares mark h1 and hK . The outlined green square marks the matched-depth endpoint.