Diffusion large language models (dLLMs) are emerging as promising alternatives to autoregressive (AR) LLMs. Recently, this paradigm has been extended to multimodal tasks, leading to the development of diffusion multimodal large language models (dMLLMs). These models are expected to retain the reasoning capabilities of LLMs while enabling faster inference through parallel generation. However, when combined with Chain-of-Thought (CoT) reasoning, dMLLMs exhibit two critical issues. First, we observe that dMLLMs often generate the final answer token at a very early timestep. This trend indicates that the model determines the answer before sufficient reasoning, leading to degraded reasoning performance. Second, during the initial timesteps, dMLLMs show minimal dependency on visual prompts, exhibiting a fundamentally different pattern of visual information utilization compared to AR vision-language models. In summary, these findings indicate that dMLLMs tend to generate premature final answers without sufficiently grounding on visual inputs. To address these limitations, we propose Position and Step Penalty (PSP) and Visual Reasoning Guidance (VRG). PSP penalizes tokens in later positions during early timesteps, delaying premature answer generation and encouraging progressive reasoning across timesteps. VRG, inspired by classifier-free guidance, amplifies visual grounding signals to enhance the model's alignment with visual evidence. Extensive experiments across various dMLLMs demonstrate that our method achieves up to 7.5% higher accuracy while delivering more than 3x speedup compared to reasoning with four times more diffusion steps.
Figures & tables
Figure 1: Example responses of LaViDa (generation length = 64, diffusion steps = 32). State at answer generation refers to the output state at the moment when the final answer (A, B, C) is generated, while Final Response denotes the model’s final response generated by each method.
Figure 2: Overview of our method. (a) shows the base inference process, which exhibits early answer generation and less visual grounding. (b) illustrates our proposed inference process that introduces PSP and VRG to mitigate these two issues.
Figure 3: Results of the final answer generation step on the M3CoT validation set using LaViDa. The default remasking strategy is Low-confidence.
Figure 4: Comparison of PDM measurements on the M3CoT validation set between the autoregressive-based model LLaVA-1.5 and the diffusion-based model LaViDa.
Method
M 3 CoT
MMBench
SQA-IMG
V* Bench
Low-conf
45.8
72.8
71.0
42.9
Low-conf w/ PSP
47.6
74.3
72.0
44.5
Low-conf w/ VRG
47.8
75.1
72.1
45.0
Low-conf w/ PSP & VRG
48.4
74.9
72.8
45.5
Table 2: LaViDa’s ablation study results with generation length L=64 / step T=32 . Low-conf denotes the Low-confidence remasking strategy, and PSP and VRG are combined with Low-conf. Bold represents the best performance, and underline indicates the second-best performance.
Method
M 3 CoT
MMBench
SQA-IMG
V* Bench
Low-conf
45.8
72.8
71.0
42.9
Low-conf w/ PSP & VRG
48.4
74.9
72.8
45.5
Entropy
46.4
72.6
70.9
42.4
Entropy w/ PSP & VRG
48.0
74.9
72.6
46.0
Margin
46.3
72.5
71.0
43.4
Margin w/ PSP & VRG
48.1
74.8
72.9
45.0
Table 3: LaViDa’s experimental results across different remasking strategies with generation length L=64 / step T=32 . PSP and VRG are combined with each method.
Figure 5: Performance comparison of LaViDa under different hyperparameter settings. On the horizontal axis, the notation [x,y] represents the hyperparameter pair, where x denotes the penalty strength coefficient γ and y denotes the visual guidance scale svrg .
Method
64/32
128/64
256/128
Low-conf
4.01s
6.07s
14.48s
Low-conf w/ DDCoT
6.03s
9.99s
25.95s
Low-conf w/ CCoT
4.05s
6.09s
14.59s
Low-conf w/ PSP
4.03s
6.06s
14.51s
Low-conf w/ VRG
4.73s
8.46s
22.05s
Low-conf w/ PSP & VRG
4.65s
8.43s
22.29s
Table 4: Average time consumption of LaViDa on the M3CoT validation set. The default remasking strategy used was Low-confidence, and DDCoT, CCoT, PSP, VRG, and PSP & VRG were applied. The unit s denotes seconds.
Method
M 3 CoT
MMBench
SQA-IMG
V* Bench
Low-conf
45.8
72.8
71.0
42.9
Low-conf w/ L2R
47.2
74.0
71.1
43.4
Low-conf w/ PSP
47.6
74.3
72.0
44.5
Low-conf w/ L2R & VRG
47.6
74.6
71.3
44.5
Low-conf w/ PSP & VRG
48.4
74.9
72.8
45.5
Table 5: Comparison experimental results between PSP and L2R in LaViDa with generation length L=64 / step T=32 . Bold indicates the better result between L2R & VRG and PSP & VRG, while underline denotes the better result between L2R and PSP.
Figure 6: Comparison of PDM measurements on the M3CoT validation set between Low-conf and Low-conf + VRG using LaViDa. the visual guidance scale svrg = 0.5.
Figure 7: Example of a reasoning prompt based on each model’s reference implementation.
Figure 8: Results of the final answer generation step on the M3CoT validation set using MMaDa. The default remasking strategy is Low-confidence.
Figure 9: Comparison of PDM measurements on the M3CoT validation set between Low-conf and Low-conf + VRG using MMaDa.
Figure 10: Results of the final answer generation step on the M3CoT validation set using LaViDa, evaluated across different diffusion steps T .
Figure 11: Comparison of PDM measurements on the M3CoT validation set between Low-conf and Low-conf + VRG using LaViDa, evaluated across different diffusion steps T .
Figure 12: Observation 1 for LLaVA-Bench COCO.
MME Exist. ↑
MME Count ↑
MME Pos. ↑
MME Color ↑
MME Total ↑
LLaVA-Bench ↑
Low Conf.
183.33
133.33
86.67
141.67
545.00
16.5
CCoT
185.00
126.67
90.55
148.33
550.55
17.2
DDCoT
176.67
137.22
81.67
149.17
544.73
15.4
PSP & VRG
187.00
143.33
91.67
150.00
570.00
20.0
Table 7: Experimental results on the MME benchmark for LaViDa with the 64/128 setting.