Diffusion large language models (dLLMs) are emerging as promising alternatives to autoregressive (AR) LLMs. Recently, this paradigm has been extended to multimodal tasks, leading to the development of diffusion multimodal large language models (dMLLMs). These models are expected to retain the reasoning capabilities of LLMs while enabling faster inference through parallel generation. However, when combined with Chain-of-Thought (CoT) reasoning, dMLLMs exhibit two critical issues. First, we observe that dMLLMs often generate the final answer token at a very early timestep. This trend indicates that the model determines the answer before sufficient reasoning, leading to degraded reasoning performance. Second, during the initial timesteps, dMLLMs show minimal dependency on visual prompts, exhibiting a fundamentally different pattern of visual information utilization compared to AR vision-language models. In summary, these findings indicate that dMLLMs tend to generate premature final answers without sufficiently grounding on visual inputs. To address these limitations, we propose Position and Step Penalty (PSP) and Visual Reasoning Guidance (VRG). PSP penalizes tokens in later positions during early timesteps, delaying premature answer generation and encouraging progressive reasoning across timesteps. VRG, inspired by classifier-free guidance, amplifies visual grounding signals to enhance the model's alignment with visual evidence. Extensive experiments across various dMLLMs demonstrate that our method achieves up to 7.5% higher accuracy while delivering more than 3x speedup compared to reasoning with four times more diffusion steps.
Figures & tables
Figure 1: Example responses of LaViDa (generation length = 64, diffusion steps = 32). State at answer generation refers to the output state at the moment when the final answer (A, B, C) is generated, while Final Response denotes the model’s final response generated by each method.
Figure 2: Overview of our method. (a) shows the base inference process, which exhibits early answer generation and less visual grounding. (b) illustrates our proposed inference process that introduces PSP and VRG to mitigate these two issues.
Figure 3: Results of the final answer generation step on the M3CoT validation set using LaViDa. The default remasking strategy is Low-confidence.
Figure 4: Comparison of PDM measurements on the M3CoT validation set between the autoregressive-based model LLaVA-1.5 and the diffusion-based model LaViDa.
Method
M 3 CoT
MMBench
SQA-IMG
V* Bench
Low-conf
45.8
72.8
71.0
42.9
Low-conf w/ PSP
47.6
74.3
72.0
44.5
Low-conf w/ VRG
47.8
75.1
72.1
45.0
Low-conf w/ PSP & VRG
48.4
74.9
72.8
45.5
Table 2: LaViDa’s ablation study results with generation length L=64 / step T=32 . Low-conf denotes the Low-confidence remasking strategy, and PSP and VRG are combined with Low-conf. Bold represents the best performance, and underline indicates the second-best performance.
Method
M 3 CoT
MMBench
SQA-IMG
V* Bench
Low-conf
45.8
72.8
71.0
42.9
Low-conf w/ PSP & VRG
48.4
74.9
72.8
45.5
Entropy
46.4
72.6
70.9
42.4
Entropy w/ PSP & VRG
48.0
74.9
72.6
46.0
Margin
46.3
72.5
71.0
43.4
Margin w/ PSP & VRG
48.1
74.8
72.9
45.0
Table 3: LaViDa’s experimental results across different remasking strategies with generation length L=64 / step T=32 . PSP and VRG are combined with each method.
Figure 5: Performance comparison of LaViDa under different hyperparameter settings. On the horizontal axis, the notation [x,y] represents the hyperparameter pair, where x denotes the penalty strength coefficient γ and y denotes the visual guidance scale svrg .
Method
64/32
128/64
256/128
Low-conf
4.01s
6.07s
14.48s
Low-conf w/ DDCoT
6.03s
9.99s
25.95s
Low-conf w/ CCoT
4.05s
6.09s
14.59s
Low-conf w/ PSP
4.03s
6.06s
14.51s
Low-conf w/ VRG
4.73s
8.46s
22.05s
Low-conf w/ PSP & VRG
4.65s
8.43s
22.29s
Table 4: Average time consumption of LaViDa on the M3CoT validation set. The default remasking strategy used was Low-confidence, and DDCoT, CCoT, PSP, VRG, and PSP & VRG were applied. The unit s denotes seconds.
Method
M 3 CoT
MMBench
SQA-IMG
V* Bench
Low-conf
45.8
72.8
71.0
42.9
Low-conf w/ L2R
47.2
74.0
71.1
43.4
Low-conf w/ PSP
47.6
74.3
72.0
44.5
Low-conf w/ L2R & VRG
47.6
74.6
71.3
44.5
Low-conf w/ PSP & VRG
48.4
74.9
72.8
45.5
Table 5: Comparison experimental results between PSP and L2R in LaViDa with generation length L=64 / step T=32 . Bold indicates the better result between L2R & VRG and PSP & VRG, while underline denotes the better result between L2R and PSP.
Figure 6: Comparison of PDM measurements on the M3CoT validation set between Low-conf and Low-conf + VRG using LaViDa. the visual guidance scale svrg = 0.5.
Figure 7: Example of a reasoning prompt based on each model’s reference implementation.
Figure 8: Results of the final answer generation step on the M3CoT validation set using MMaDa. The default remasking strategy is Low-confidence.
Figure 9: Comparison of PDM measurements on the M3CoT validation set between Low-conf and Low-conf + VRG using MMaDa.
Figure 10: Results of the final answer generation step on the M3CoT validation set using LaViDa, evaluated across different diffusion steps T .
Figure 11: Comparison of PDM measurements on the M3CoT validation set between Low-conf and Low-conf + VRG using LaViDa, evaluated across different diffusion steps T .
Figure 12: Observation 1 for LLaVA-Bench COCO.
MME Exist. ↑
MME Count ↑
MME Pos. ↑
MME Color ↑
MME Total ↑
LLaVA-Bench ↑
Low Conf.
183.33
133.33
86.67
141.67
545.00
16.5
CCoT
185.00
126.67
90.55
148.33
550.55
17.2
DDCoT
176.67
137.22
81.67
149.17
544.73
15.4
PSP & VRG
187.00
143.33
91.67
150.00
570.00
20.0
Table 7: Experimental results on the MME benchmark for LaViDa with the 64/128 setting.
Vision Language Models (VLMs) achieve strong reasoning with Chain-of-Thought (CoT) prompting but incur high sequential-generation cost, error accumulation, and limited self-correction. Diffusion Multimodal Large Language Models (dMLLMs) unmask tokens in an order-agnostic process, improving efficiency and enabling iterative refinement, yet their reasoning and how to enhance it remain underexplored. We propose a training-free method, Spatio-Temporal Token Veto (ST-Veto), which leverages the ability to observe all token positions at each diffusion step. Rather than relying only on current-step confidence, ST-Veto vetoes temporally unstable tokens via second-order Taylor prediction of confidence dynamics and filters weakly grounded tokens using image-attention mass, swapping them with safer candidates. Across multiple dMLLMs and multimodal reasoning benchmarks, ST-Veto consistently outperforms standard decoding policies and prior VLM reasoning methods, improving accuracy by up to 9% with no additional training or generation cost. Analyses show that ST-Veto steers generation toward higher-confidence, better-grounded paths.
Keuntae Kim, Beomseok Lee, Hyunwoo Kim +1
Hanyang University · LG Electronics · KT Corporation
Diffusion language models (dLLMs) recently emerged as a promising alternative to auto-regressive LLMs. The latest works further extended it to multimodal understanding and generation tasks. In this work, we propose LaViDa-R1, a multimodal, general-purpose reasoning dLLM. Unlike existing works that build reasoning dLLMs through task-specific reinforcement learning, LaViDa-R1 incorporates diverse multimodal understanding and generation tasks in a unified manner. In particular, LaViDa-R1 is built with a novel unified post-training framework that seamlessly integrates supervised finetuning (SFT) and multi-task reinforcement learning (RL). It employs several novel training techniques, including answer-forcing, tree search, and complementary likelihood estimation, to enhance effectiveness and scalability. Extensive experiments demonstrate LaViDa-R1's strong performance on a wide range of multimodal tasks, including visual math reasoning, reason-intensive grounding, and image editing.
Recently, Multimodal Large Language Models (MLLMs) have been widely integrated into diffusion frameworks primarily as text encoders to tackle complex tasks such as spatial reasoning. However, this paradigm suffers from two critical limitations: (i) MLLMs text encoder exhibits insufficient reasoning depth. Single-step encoding fails to activate the Chain-of-Thought process, which is essential for MLLMs to provide accurate guidance for complex tasks. (ii) The guidance remains invariant during the decoding process. Invariant guidance during decoding prevents DiT from progressively decomposing complex instructions into actionable denoising steps, even with correct MLLM encodings. To this end, we propose Endogenous Chain-of-Thought (EndoCoT), a novel framework that first activates MLLMs' reasoning potential by iteratively refining latent thought states through an iterative thought guidance module, and then bridges these states to the DiT's denoising process. Second, a terminal thought grounding module is applied to ensure the reasoning trajectory remains grounded in textual supervision by aligning the final state with ground-truth answers. With these two components, the MLLM text encoder delivers meticulously reasoned guidance, enabling the DiT to execute it progressively and ultimately solve complex tasks in a step-by-step manner. Extensive evaluations across diverse benchmarks (e.g., Maze, TSP, VSP, and Sudoku) achieve an average accuracy of 92.1%, outperforming the strongest baseline by 8.3 percentage points. The code and dataset are publicly available at https://internlm.github.io/EndoCoT/.
Xuanlang Dai, Yujie Zhou, Long Xing +6
Shanghai AI Laboratory · Xi’an Jiaotong University · Shanghai Jiaotong University +3