Parallel Looped Transformers (PLTs) use otherwise idle decoding compute by overlapping recurrent iterations. However, vanilla PLTs scale poorly with depth: they exhibit training-loss spikes, and their quality gains shrink as more loops are added. The benefit of additional loops depends on the information available to each iteration, so we examine whether the existing computational dependencies limit effective depth scaling. We find that only the preceding in-trajectory state is passed across loop boundaries, even though completed cross-trajectory representations are already available. Based on this finding, we propose InfuseLoop, a cross-trajectory latent injection mechanism. It aggregates causally available hidden states from later prediction trajectories through lightweight attention residuals. This design preserves token-parallel training and concurrent loop execution, and requires only O(Rd) additional hidden-state storage. Experiments show that InfuseLoop eliminates training-loss spikes and improves validation-loss scaling up to eight loops. At eight loops, InfuseLoop reduces perplexity by 3.4--17.1% relative to vanilla PLTs and by 10.5--37.3% relative to the non-looped baseline. It also improves average downstream accuracy by 0.84 and 1.99 percentage points over these two baselines. InfuseLoop incurs only 0.3% decoding time over vanilla PLTs and 23.3% over the non-looped baseline, while keeping the same logical KV-cache footprint as vanilla PLTs.
Figures & tables
Figure 1: Overview of InfuseLoop. Each loop combines the latest token embedding with the current state and causally available future states through attention residual aggregation. The enlarged view details the attention-based aggregation of current latent state and future states.
Figure 2: Execution schedules and information flow. (a) Looped Transformers execute successive loops serially before generating the next token. (b) InfuseLoop overlaps computation across prediction positions and loop iterations, and introduces future-state injection (red) on top of vanilla PLT.
Figure 3: State availability and source selection for ht(4) in a six-loop model ( R=6 ). ht(4) is computed at cycle c=τ(t,4) , and the outlined states are the sources selected from cycle c−1 .
Input sequence X=(x1,…,xL) ,
next-token targets Y ,
number of loops R , backbone Fθ
Algorithm 1 Token-parallel training of InfuseLoop
Current token xc ,
previous-cycle outputs p1,…,pR ,
number of loops R , backbone Fθ
Algorithm 2 One decoding cycle of InfuseLoop
Figure 4: Validation loss during pretraining with (a) R=4 and (b) R=8 loops.
Loops
Model
Val. loss ↓
Perplexity ↓
Downstream tasks ↑
Wiki2
LAMBADA
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WG
OBQA
Avg.
R=1
Single
2.9941
39.72
67.02
24.58
36.44
49.92
28.58
63.38
51.70
33.00
41.09
R=4
LT
2.9022
34.78
41.07
24.35
40.17
54.34
30.55
66.43
54.70
33.80
43.48
PLT
2.9252
36.19
62.96
24.71
39.24
51.68
28.58
65.78
51.70
32.60
42.04
InfuseLoop
2.9112
35.70
45.08
24.49
39.49
53.75
31.40
65.89
52.96
33.00
43.00
R=8
LT
2.8775
33.78
35.04
23.99
41.88
55.77
31.91
65.51
54.38
32.20
43.66
Table 1: Validation loss, perplexity, and downstream accuracy (%) after training on 6B tokens.
Context length
Model
512
2,048
8,192
Single
114.69
114.69
114.69
InfuseLoop-4
157.72 (+37.5%)
125.45 (+9.4%)
117.38 (+2.3%)
InfuseLoop-8
215.10 (+87.5%)
139.79 (+21.9%)
120.96 (+5.5%)
Table 2: Logical cache footprint per token (KB) with window size w=64 . Parentheses show the overhead relative to Single.
Figure 5: Decoding cost and workload sensitivity. (a) GPU kernel time with a 2,048-token initial context, normalized to the total kernel time of Single at the same batch size. (b, c) Throughput relative to Single across batch sizes and context lengths.
Figure 6: Depth scaling and performance–latency Pareto frontiers. Top: validation loss; bottom: average downstream accuracy. Interpolated R=2 points and projected R>8 extensions are visual guides only; Pareto membership is computed from measured configurations.
R=4
R=8
Configuration
Loss ↓
Avg. ↑
Loss ↓
Avg. ↑
InfuseLoop
3.2482
38.26
3.2436
38.35
Source selection
Base only
3.2660
38.04
3.2799
37.59
FullState
3.3975
37.43
3.4244
37.21
In-trajectory
3.2735
38.19
3.2839
37.87
Table 3: Component ablations. All configurations are trained on 1B tokens.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
Backbone configuration
Qwen3-0.6B
Transformer layers ( K )
28
Hidden dimension ( d )
1,024
FFN intermediate dimension
3,072
Query heads
16
KV heads
8
Appendix
Table 4: Backbone configuration and parameter counts.