Beyond the Sequentiality Bottleneck: A Hybrid Diffusion-Autoregressive Framework for Efficient Multimodal Reasoning

Wei Cai; Jian Zhao; Yuchen Yuan; Xuelong Li

cs.AIOct 20, 2026

Beyond the Sequentiality Bottleneck: A Hybrid Diffusion-Autoregressive Framework for Efficient Multimodal Reasoning

Authors: Wei Cai, Jian Zhao, Yuchen Yuan, Xuelong Li

Organizations: Peking University · Institute of Artificial Intelligence (TeleAI), China Telecom

Abstract

Autoregressive models (ARMs) face a "Sequentiality Bottleneck" in reasoning, while diffusion language models (DLMs) offer parallelism but lack semantic coherence. To addresses both limitations, we propose Collaborative Reasoning Asymmetric Learning (CoRAL), a hybrid framework via an Asymmetric Planner-Executor (APE) architecture: a DLM Planner generates K diverse reasoning trajectories in parallel, while a large ARM Executor selects the optimal plan and synthesizes the final answer. To train the Planner effectively, we introduce Reward-Guided Denoising for CoT (RDC), which integrates outcome-based rewards into the diffusion process to steer it toward goal-directed reasoning. Experiments on vision-language benchmarks show that CoRAL significantly outperforms prior diffusion-based methods and achieves competitive performance against strong ARMs, achieving both superior efficiency and quality.

Figures & tables

Figure 1: Comparison of thinking in DLM, ARM, and DLM $\rightarrow$ ARM, DLMs: Parallel thinking, ARM: Sequential thinking, DLM $\rightarrow$ ARM (CoRAL): The DLM first plans in parallel, the ARM evaluates and executes

Figure 2: An overview of our CoRAL framework.

Model	Trip-Planning		CapMAS			CapArena	DetailCaps-4870	GPQA
Model	Acc	ThroughPut	CLAIR	Coverage	Factuality	CapArena-Auto	CAPTURE	Acc	ThroughPut
AR model
LLaVA-1.5-7B	0.05	0.24	62.20	34.50	52.60	-93.00	51.05	0.32	0.15
InternVL3-8B	0.10	0.23	78.36	52.67	78.75	-29.83	57.80	0.33	0.29
VILA-1.5-7B	0.07	0.19	68.47	42.52	58.69	-49.75	55.30	0.32	0.19
GLM-4v-9B	0.11	0.25	80.51	56.37	80.28	-19.71	57.50	0.34	0.32

Table 1: Comparison of different models on various benchmarks

Figure 3: Performance comparison across different inference steps on the benchmark datasets. “Mask-pred training” refers to training conducted on the same dataset with the traditional masked prediction objective

Figure 4: (Left) The accuracy-cost trade-off for various models. (Center) Performance of the DLM and ARM when serving as the Planner ( $M_{p}$ ) and Executor ( $M_{e}$ ), respectively. (Right) The accuracy-runtime trade-off for various models.

			Trip-Planning	CLAIR	Coverage	CAPTURE
	ARM	DLM	RDC	Acc	Avg	Avg	Avg
Qwen2.5-VL-7B (ARM-Only)	✓			0.12	80.48	57.32	60.61
LLaDA-V (DLM-Only)		✓		0.21	65.54	49.23	59.63
CoRAL	✓	✓	✓	0.31	86.75	58.36	63.88
$w/o$ RDC	✓	✓		0.25	82.36	57.83	62.27
$w/o$ ARM		✓	✓	0.26	74.35	52.21	61.75

Table 2: Ablations of each component of our CoRAL

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setup	Planner	Executor	Interaction	Trip-Planning	DetailCaps
	( $M_{P}$ )	( $M_{E}$ )		(Acc $\uparrow$ )	(CAPTURE $\uparrow$ )
ARM Only	—	Qwen2.5-VL	—	0.12	60.61
DLM Only	—	LLaDA-V	—	0.21	59.63
ARM $\to$ ARM	Qwen2.5-VL	Qwen2.5-VL	Text	0.18	61.15
DLM $\to$ DLM	LLaDA-V	LLaDA-V	Text	0.23	60.20
ARM $\to$ DLM	Qwen2.5-VL	LLaDA-V	Text	0.20	59.80

Appendix

Table 3: Comparison of Different Architectures

Figure 5: An template of DLM Thinking Planner.

Figure 7: An template of DLM Thinking Planner.

Hybrid Language Models

See all 15 papers

Latent-DARM: Bridging Discrete Diffusion And Autoregressive Models For Reasoning
Mar 10, 2026Lina Berrayana, Ahmed Heakl, Abdullah Sohail +3Multi-Agent Reasoning Multi-Agent LLMs
EPFL · MBZUAI · ETH Zürich +1
Think First, Diffuse Fast: Improving Diffusion Language Model Reasoning via Autoregressive Plan Conditioning
Feb 20, 2026Earl J St SauverDiffusion Language Models LLM Planning
Independent Researcher
Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models
Apr 7, 2026Keuntae Kim, Mingyu Kang, Yong Suk ChoiVLM Reasoning Diffusion Language Models
Department of Computer Science, Hanyang University · Department of Artificial Intelligence, Hanyang University

Mar 10, 2026J/K move · Enter open · S save

Latent-DARM: Bridging Discrete Diffusion And Autoregressive Models For Reasoning

Lina Berrayana, Ahmed Heakl, Abdullah Sohail, Thomas Hofmann+2

EPFL · MBZUAI · ETH Zürich+1

Multi-Agent Reasoning Multi-Agent LLMs Diffusion Language Models

Open paper AskPDF arXiv

Most multi-agent systems rely exclusively on autoregressive language models (ARMs) that are based on sequential generation. Although effective for fluent text, ARMs limit global reasoning and plan revision. On the other hand, Discrete Diffusion Language Models (DDLMs) enable non-sequential, globally revisable generation and have shown strong planning capabilities, but their limited text fluency hinders direct collaboration with ARMs. We introduce Latent-DARM, a latent-space communication framework bridging DDLM (planners) and ARM (executors), maximizing collaborative benefits. Across mathematical, scientific, and commonsense reasoning benchmarks, Latent-DARM outperforms text-based interfaces on average, improving accuracy from 27.0% to 36.0% on DART-5 and from 0.0% to 14.0% on AIME2024. Latent-DARM approaches the results of state-of-the-art reasoning models while using less than 2.2% of its token budget. This work advances multi-agent collaboration among agents with heterogeneous models.

Abstract

Hybrid Language Models

Explore similar work

Latent-DARM: Bridging Discrete Diffusion And Autoregressive Models For Reasoning

Think First, Diffuse Fast: Improving Diffusion Language Model Reasoning via Autoregressive Plan Conditioning

Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models