Dual-Slot: From Serial Loops to Parallel Slots in Language Models

Yiqin Hao; Zitong Wang; Boyi Zeng; He Li; Shixiang Song; Feichen Song; Yifan Liu; Ran Tao

cs.CLOct 18, 2026

Dual-Slot: From Serial Loops to Parallel Slots in Language Models

Authors: Yiqin Hao, Zitong Wang, Boyi Zeng, He Li, Shixiang Song, Feichen Song, Yifan Liu, Ran Tao,

Organizations: LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai AI Laboratory

Abstract

Looped language models increase computational depth by repeatedly applying shared layers, offering a way to improve model quality without adding parameters. However, sequential dependencies between loop iterations increase autoregressive decoding latency. We introduce Dual-Slot, an architecture that reorganizes serial looped computation into parallel latent and prediction slots. Rather than waiting for the current token's latent computation to finish, the prediction slot combines the previous token's deep latent state with the current representation, while the latent slot produces a state for the next decoding step. This shifts the latent dependency across adjacent tokens, replacing two sequential module invocations with a single paired invocation while preserving standard autoregressive generation. To preserve token-level parallelism during training, we use a small number of Jacobi iterations to refine latent states across the sequence in parallel. Experiments on Qwen-style dense and mixture-of-experts (MoE) models show consistent improvements in language modeling and average downstream accuracy. On the 1B dense model, Dual-Slot reduces Pile validation perplexity from 7.25 to 7.00 and improves average five-shot accuracy across nine task configurations by 2.26 percentage points. After supervised fine-tuning, it improves zero-shot GSM8K accuracy by 5.83 and 3.49 percentage points for the 1B dense and A0.6B MoE models, respectively. Across the evaluated batch sizes, Dual-Slot retains 93.1-96.5% of vanilla decoding throughput for the dense model and 98.1-99.6% for the MoE model.

Figures & tables

Figure 1: Dual-Slot training and inference. 1) Parallel Training (left). The prefix processes the full sequence once, and an initial warm pass through the core produces the initial latent slots. Jacobi updates then refine the slots at all positions in parallel, with next-token loss computed only from the final prediction states. 2) Autoregressive Decoding (right). At decoding step

t

, the shared Loop Core processes

x_{t}

and

x_{t}+h_{t-1}

in one invocation. The prediction-slot output

y_{t}

predicts token

t+1

, while the latent-slot output

h_{t}

is carried to the next step.

Figure 2: Language modeling performance of the 1B dense and A0.6B MoE models. From left to right: perplexity on Pile validation, WikiText-2 (word-level), LAMBADA-OpenAI, and LAMBADA-Standard. Lower is better. Dual-Slot improves over Vanilla across all four benchmarks for both model families.

Figure 3: Scaling behavior of Dual-Slot. (a) Pile validation loss versus model size. (b) Loss versus training tokens for the 1B dense and A0.6B MoE models. (c) Average zero-shot accuracy over nine downstream task configurations versus model size. Model size denotes the nominal dense parameter count or the MoE active parameter count. Lower is better in (a) and (b); higher is better in (c).

Zero-shot
Model	Method	LAMBADA		PIQA	Wino Grande	SciQ	Hella Swag	ARC		RACE high	Avg.
Model	Method	OpenAI	Standard					Easy	Challenge
1B Dense	Vanilla	62.64	52.42	71.93	60.38	89.30	42.09	60.82	27.56	35.31	55.83
1B Dense	Dual-Slot	64.95	56.69	72.58	59.75	90.20	43.74	61.95	27.47	36.56	57.10
A0.6B MoE	Vanilla	64.43	52.61	72.85	59.04	88.60	44.12	63.05	26.79	34.83	56.26
A0.6B MoE	Dual-Slot	64.70	55.62	73.23	58.33	89.00	45.33	65.07	31.06	34.35	57.41

Table 1: Downstream performance of the 1B dense and A0.6B MoE models. Zero-shot and five-shot accuracy (%). Avg. is the unweighted mean over the nine task configurations. Bold indicates the best score within each model and shot setting, including ties.

Model	Method	Zero-shot		Eight-shot
Model	Method	EM (%) $\uparrow$	Gain (pp.)	EM (%) $\uparrow$	Gain (pp.)
1B Dense	Vanilla	13.12	–	10.84	–
1B Dense	Dual-Slot	18.95	$+5.83$	14.94	$+4.10$
A0.6B MoE	Vanilla	11.60	–	7.96	–
A0.6B MoE	Dual-Slot	15.09	$+3.49$	11.83	$+3.87$

Table 2: Mathematical reasoning after supervised fine-tuning. GSM8K test final-answer EM (%) under zero-shot and eight-shot prompting for the 1B dense and A0.6B MoE models. Dual-Slot consistently improves accuracy across both model families and evaluation settings.

Figure 4: Comparison with Parallel Loop Transformer (PLT2). (a) Final Pile validation loss across dense model scales. (b) Pile validation loss over training tokens for the 1B models. (c) Average zero-shot raw accuracy over nine downstream task configurations across model scales. Dual-Slot consistently achieves lower validation loss and higher downstream accuracy than PLT2 across the evaluated scales.

150M
Method	Pile PPL	LAMBADA		PIQA	Wino Grande	SciQ	Hella Swag	ARC		RACE high	Avg.
Method	Pile PPL	OpenAI	Standard	PIQA	Wino Grande	SciQ	Hella Swag	Easy	Challenge	RACE high	Avg.
Vanilla	11.53	37.34	24.26	62.24	51.07	75.80	29.32	45.33	19.37	29.38	41.57
PLT2	11.19	38.44	25.00	64.25	50.20	74.80	29.28	46.04	19.11	28.52	41.74
Dual-Slot	11.10	38.11	27.05	63.93	51.85	75.70	29.89	46.63	18.77	28.90	42.32
300M

Table 3: Detailed comparison with Parallel Loop Transformer. Pile validation perplexity (

\downarrow

) and zero-shot raw accuracy (%,

\uparrow

) across four dense model scales. Avg. denotes the unweighted mean over the nine downstream task configurations. Bold indicates the best result within each model scale.

1B Dense				A0.6B MoE
BS	Vanilla	Dual-Slot	DS/V	BS	Vanilla	Dual-Slot	DS/V
1	80.51	74.92	$0.9306\times$	1	17.25	16.92	$0.9809\times$
4	307.53	292.29	$0.9504\times$	4	68.29	67.97	$0.9953\times$
32	2378.11	2294.87	$0.9650\times$	32	549.12	546.74	$0.9957\times$
128	9527.89	9056.98	$0.9506\times$	128	2191.15	2162.76	$0.9870\times$

Table 4: Autoregressive decoding throughput. Throughput (tokens/s) on a single GPU using BF16, SDPA, and the standard KV cache. DS/V denotes the throughput ratio between Dual-Slot and Vanilla.

Figure 5: Convergence of Jacobi refinement. From left to right: evaluation loss and inter-iteration hidden-state RMSE for the 1B dense model, followed by the corresponding measurements for the A0.6B MoE model. Evaluation loss decreases rapidly up to the training depths (

K=2,3

) and changes comparatively little with further updates, while the RMSE between consecutive hidden states diminishes under continued refinement.

Figure 6: Fixed versus random Jacobi iterations. Validation loss (left) and inter-iteration hidden-state RMSE (right) for models trained with fixed

K=2

, fixed

K=3

, or random

K\in\{2,3\}

. With random

K

, loss changes only slightly beyond the training iteration counts, and hidden states stabilize faster within the evaluated range.

Method	Val. loss $\downarrow$	Loss gap
Raw Add w/o gate	2.0485	$0.0000$
RMS matching w/o gate	2.0497	$+0.0012$
Raw Add w/ gate	2.0474	$-0.0011$
RMS matching w/ gate	2.0490	$+0.0006$

Table 11

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Parameters	Layers	$d_{\mathrm{model}}$	$d_{\mathrm{ff}}$	Q heads	KV heads	Dual-Slot core	Prefix/Core/Suffix
150M	149,541,120	12	768	3,328	12	4	L4–L7	4/4/4
300M	303,203,200	15	1,024	4,608	16	4	L5–L9	5/5/5
600M	604,627,328	21	1,280	5,632	20	5	L7–L13	7/7/7
1B	1,011,626,496	24	1,536	7,168	24	6	L8–L15	8/8/8

Appendix

Table 7: Dense model configurations. Vanilla and Dual-Slot have identical physical architectures and parameter counts. Dual-Slot core layers use zero-based indices.

Model	Active params.	Total params.	Layers	$d_{\mathrm{model}}$	$d_{\mathrm{ff}}^{\mathrm{expert}}$	Q/KV heads	Dual-Slot core	Prefix/Core/Suffix
A0.15B	150,130,944	794,218,752	12	768	416	12/4	L4–L7	4/4/4
A0.3B	304,186,240	1,790,542,720	15	1,024	576	16/4	L5–L9	5/5/5
A0.6B	606,347,648	3,785,499,008	21	1,280	704	20/5	L7–L13	7/7/7

Appendix

Table 8: MoE model configurations. Active parameters follow the conventional count for one slot with Top-8 routing. Each model has 64 routed experts, and core layers use zero-based indices.

Autoregressive Language Modeling

Autoregressive language modeling predicts sequences one step at a time, using prior tokens to generate text, speech, code, and other data.

See all 133 papers

Hybrid Latent Attention for Looped Language Models
Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model Inference LLM Inference Acceleration
Virginia Tech · Independent Researcher
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
May 28, 2026Markus Frey, Behzad Shomali, Joachim Koehler +1Recurrent Transformers Language Model Scaling
Lamarr Institute · Fraunhofer IAIS · University of Bonn
BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
May 12, 2026Shaobin Zhuang, Yuang Ai, Jiaming Han +7Multi-Token Prediction Autoregressive Language Modeling
Shanghai Jiao Tong University · MMLab, The Chinese University of Hong Kong · Institute of Automation, Chinese Academy of Sciences +1

Oct 6, 2026J/K move · Enter open · S save

Hybrid Latent Attention for Looped Language Models

Yuhan Chen, Siyuan Zhang, Nan Wang, Feiyang Kang+1

Virginia Tech · Independent Researcher

Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Compression Hybrid Attention Recursive Language Models

Open paper AskPDF arXiv

Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid Latent Attention (HLA), which keeps exact keys and values within a sliding window of W recent tokens and stores each older token as a compact latent that the query of each loop reads directly, without reconstructing keys and values. We uptrain HLA on Ouro looped models (T=4) with 1.4B and 2.6B parameters, keeping the pretrained weights frozen and training only the added parameters to reproduce the original attention. The cache shrinks by 10.7x per token, fitting 4.0-8.8x as many concurrent sequences per GPU, and decoding throughput improves by 2.5x at 1K-token contexts and by up to 7.4x at 16K. HLA retains over 97% of the original accuracy on math, knowledge and reasoning benchmarks, and 96-100% on long-context retrieval up to 16K tokens. After supervised fine-tuning, it performs on par with the fine-tuned original model on competition-level math.

Abstract

Autoregressive Language Modeling

Explore similar work

Hybrid Latent Attention for Looped Language Models

A Dual-Path Architecture for Scaling Compute and Capacity in LLMs

BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion