Looped language models increase computational depth by repeatedly applying shared layers, offering a way to improve model quality without adding parameters. However, sequential dependencies between loop iterations increase autoregressive decoding latency. We introduce Dual-Slot, an architecture that reorganizes serial looped computation into parallel latent and prediction slots. Rather than waiting for the current token's latent computation to finish, the prediction slot combines the previous token's deep latent state with the current representation, while the latent slot produces a state for the next decoding step. This shifts the latent dependency across adjacent tokens, replacing two sequential module invocations with a single paired invocation while preserving standard autoregressive generation. To preserve token-level parallelism during training, we use a small number of Jacobi iterations to refine latent states across the sequence in parallel. Experiments on Qwen-style dense and mixture-of-experts (MoE) models show consistent improvements in language modeling and average downstream accuracy. On the 1B dense model, Dual-Slot reduces Pile validation perplexity from 7.25 to 7.00 and improves average five-shot accuracy across nine task configurations by 2.26 percentage points. After supervised fine-tuning, it improves zero-shot GSM8K accuracy by 5.83 and 3.49 percentage points for the 1B dense and A0.6B MoE models, respectively. Across the evaluated batch sizes, Dual-Slot retains 93.1-96.5% of vanilla decoding throughput for the dense model and 98.1-99.6% for the MoE model.
Figures & tables
Figure 1: Dual-Slot training and inference. 1) Parallel Training (left). The prefix processes the full sequence once, and an initial warm pass through the core produces the initial latent slots. Jacobi updates then refine the slots at all positions in parallel, with next-token loss computed only from the final prediction states. 2) Autoregressive Decoding (right). At decoding step t , the shared Loop Core processes xt and xt+ht−1 in one invocation. The prediction-slot output yt predicts token t+1 , while the latent-slot output ht is carried to the next step.
Figure 2: Language modeling performance of the 1B dense and A0.6B MoE models. From left to right: perplexity on Pile validation, WikiText-2 (word-level), LAMBADA-OpenAI, and LAMBADA-Standard. Lower is better. Dual-Slot improves over Vanilla across all four benchmarks for both model families.
Figure 3: Scaling behavior of Dual-Slot. (a) Pile validation loss versus model size. (b) Loss versus training tokens for the 1B dense and A0.6B MoE models. (c) Average zero-shot accuracy over nine downstream task configurations versus model size. Model size denotes the nominal dense parameter count or the MoE active parameter count. Lower is better in (a) and (b); higher is better in (c).
Model
Method
LAMBADA
PIQA
Wino Grande
SciQ
Hella Swag
ARC
RACE high
Avg.
OpenAI
Standard
Easy
Challenge
Zero-shot
1B Dense
Vanilla
62.64
52.42
71.93
60.38
89.30
42.09
60.82
27.56
35.31
55.83
Dual-Slot
64.95
56.69
72.58
59.75
90.20
43.74
61.95
27.47
36.56
57.10
A0.6B MoE
Vanilla
64.43
52.61
72.85
59.04
88.60
44.12
63.05
26.79
34.83
56.26
Dual-Slot
64.70
55.62
73.23
58.33
89.00
45.33
65.07
31.06
34.35
57.41
Table 1: Downstream performance of the 1B dense and A0.6B MoE models. Zero-shot and five-shot accuracy (%). Avg. is the unweighted mean over the nine task configurations. Bold indicates the best score within each model and shot setting, including ties.
Model
Method
Zero-shot
Eight-shot
EM (%) ↑
Gain (pp.)
EM (%) ↑
Gain (pp.)
1B Dense
Vanilla
13.12
–
10.84
–
1B Dense
Dual-Slot
18.95
+5.83
14.94
+4.10
A0.6B MoE
Vanilla
11.60
–
7.96
–
A0.6B MoE
Dual-Slot
15.09
+3.49
11.83
+3.87
Table 2: Mathematical reasoning after supervised fine-tuning. GSM8K test final-answer EM (%) under zero-shot and eight-shot prompting for the 1B dense and A0.6B MoE models. Dual-Slot consistently improves accuracy across both model families and evaluation settings.
Figure 4: Comparison with Parallel Loop Transformer (PLT2). (a) Final Pile validation loss across dense model scales. (b) Pile validation loss over training tokens for the 1B models. (c) Average zero-shot raw accuracy over nine downstream task configurations across model scales. Dual-Slot consistently achieves lower validation loss and higher downstream accuracy than PLT2 across the evaluated scales.
Method
Pile PPL
LAMBADA
PIQA
Wino Grande
SciQ
Hella Swag
ARC
RACE high
Avg.
OpenAI
Standard
Easy
Challenge
150M
Vanilla
11.53
37.34
24.26
62.24
51.07
75.80
29.32
45.33
19.37
29.38
41.57
PLT2
11.19
38.44
25.00
64.25
50.20
74.80
29.28
46.04
19.11
28.52
41.74
Dual-Slot
11.10
38.11
27.05
63.93
51.85
75.70
29.89
46.63
18.77
28.90
42.32
300M
Table 3: Detailed comparison with Parallel Loop Transformer. Pile validation perplexity ( ↓ ) and zero-shot raw accuracy (%, ↑ ) across four dense model scales. Avg. denotes the unweighted mean over the nine downstream task configurations. Bold indicates the best result within each model scale.
1B Dense
A0.6B MoE
BS
Vanilla
Dual-Slot
DS/V
BS
Vanilla
Dual-Slot
DS/V
1
80.51
74.92
0.9306×
1
17.25
16.92
0.9809×
4
307.53
292.29
0.9504×
4
68.29
67.97
0.9953×
32
2378.11
2294.87
0.9650×
32
549.12
546.74
0.9957×
128
9527.89
9056.98
0.9506×
128
2191.15
2162.76
0.9870×
Table 4: Autoregressive decoding throughput. Throughput (tokens/s) on a single GPU using BF16, SDPA, and the standard KV cache. DS/V denotes the throughput ratio between Dual-Slot and Vanilla.
Figure 5: Convergence of Jacobi refinement. From left to right: evaluation loss and inter-iteration hidden-state RMSE for the 1B dense model, followed by the corresponding measurements for the A0.6B MoE model. Evaluation loss decreases rapidly up to the training depths ( K=2,3 ) and changes comparatively little with further updates, while the RMSE between consecutive hidden states diminishes under continued refinement.
Figure 6: Fixed versus random Jacobi iterations. Validation loss (left) and inter-iteration hidden-state RMSE (right) for models trained with fixed K=2 , fixed K=3 , or random K∈{2,3} . With random K , loss changes only slightly beyond the training iteration counts, and hidden states stabilize faster within the evaluated range.
Method
Val. loss ↓
Loss gap
Raw Add w/o gate
2.0485
0.0000
RMS matching w/o gate
2.0497
+0.0012
Raw Add w/ gate
2.0474
−0.0011
RMS matching w/ gate
2.0490
+0.0006
Table 11
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Parameters
Layers
dmodel
dff
Q heads
KV heads
Dual-Slot core
Prefix/Core/Suffix
150M
149,541,120
12
768
3,328
12
4
L4–L7
4/4/4
300M
303,203,200
15
1,024
4,608
16
4
L5–L9
5/5/5
600M
604,627,328
21
1,280
5,632
20
5
L7–L13
7/7/7
1B
1,011,626,496
24
1,536
7,168
24
6
L8–L15
8/8/8
Appendix
Table 7: Dense model configurations. Vanilla and Dual-Slot have identical physical architectures and parameter counts. Dual-Slot core layers use zero-based indices.
Model
Active params.
Total params.
Layers
dmodel
dffexpert
Q/KV heads
Dual-Slot core
Prefix/Core/Suffix
A0.15B
150,130,944
794,218,752
12
768
416
12/4
L4–L7
4/4/4
A0.3B
304,186,240
1,790,542,720
15
1,024
576
16/4
L5–L9
5/5/5
A0.6B
606,347,648
3,785,499,008
21
1,280
704
20/5
L7–L13
7/7/7
Appendix
Table 8: MoE model configurations. Active parameters follow the conventional count for one slot with Top-8 routing. Each model has 64 routed experts, and core layers use zero-based indices.
Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid Latent Attention (HLA), which keeps exact keys and values within a sliding window of W recent tokens and stores each older token as a compact latent that the query of each loop reads directly, without reconstructing keys and values. We uptrain HLA on Ouro looped models (T=4) with 1.4B and 2.6B parameters, keeping the pretrained weights frozen and training only the added parameters to reproduce the original attention. The cache shrinks by 10.7x per token, fitting 4.0-8.8x as many concurrent sequences per GPU, and decoding throughput improves by 2.5x at 1K-token contexts and by up to 7.4x at 16K. HLA retains over 97% of the original accuracy on math, knowledge and reasoning benchmarks, and 96-100% on long-context retrieval up to 16K tokens. After supervised fine-tuning, it performs on par with the fine-tuned original model on competition-level math.
Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We propose a novel dual-path block that can flexibly scale compute, the number of sequential operations applied to a hidden state, and capacity, the parameters available at a single step. For this we expose both axes as parallel pathways within a single layer: a deep sublayer re-applied K times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once. Independent per-token gates combine both axes and allow detailed per-token routing analyses. We show that across two FLOP budgets, our dual-path model surpasses iso-FLOP matched models on language modeling and downstream evaluations, while using fewer parameters than the baseline at matched FLOPs. The learned gates are directly interpretable and show systematic per-token allocation with function words and lexical content trend wide, while punctuation, symbols, and arithmetic tokens trend deep.
Markus Frey, Behzad Shomali, Joachim Koehler +1
Lamarr Institute · Fraunhofer IAIS · University of Bonn
Autoregressive language models generate text one token at a time, yet natural language is inherently structured in multi-token units, including phrases, n-grams, and collocations that carry meaning jointly. This one-token bottleneck limits both the expressiveness of the model during pre-training and its throughput at inference time. Existing remedies such as speculative decoding or diffusion-based language models either leave the underlying bottleneck intact or sacrifice the causal structure essential to language modeling. We propose BitLM, a language model that represents each token as a fixed-length binary code and employs a lightweight diffusion head to denoise multiple tokens in parallel within each block. Crucially, BitLM preserves left-to-right causal attention across blocks while making joint lexical decisions within each block, combining the reliability of autoregressive modeling with the parallelism of iterative refinement. By replacing the large-vocabulary softmax with bitwise denoising, BitLM reframes token generation as iterative commitment in a compact binary space, enabling more efficient pre-training and substantially faster inference without altering the causal foundation that makes language models effective. Our results demonstrate that the one-token-at-a-time paradigm is not a fundamental requirement but an interface choice, and that changing it can yield a stronger and faster language model. We hope BitLM points toward a promising direction for next-generation language model architectures.
Shaobin Zhuang, Yuang Ai, Jiaming Han +7
Shanghai Jiao Tong University · MMLab, The Chinese University of Hong Kong · Institute of Automation, Chinese Academy of Sciences +1