Diffusion-CAM: Faithful Visual Explanations for dMLLMs

Haomin Zuo; Yidi Li; Luoxiao Yang; Xiaofeng Zhang

cs.AIApr 13, 2026

Diffusion-CAM: Faithful Visual Explanations for dMLLMs

Authors: Haomin Zuo, Yidi Li, Luoxiao Yang, Xiaofeng Zhang

Organizations: Sun Yat-sen University · Northwestern University · Technion - Israel Institute of Technology · Department of Automation and Intelligent Sensing, Shanghai Jiao Tong University

Abstract

While diffusion Multimodal Large Language Models (dMLLMs) have recently achieved remarkable strides in multimodal generation, the development of interpretability mechanisms has lagged behind their architectural evolution. Unlike traditional autoregressive models that produce sequential activations, diffusion-based architectures generate tokens via parallel denoising, resulting in smooth, distributed activation patterns across the entire sequence. Consequently, existing Class Activation Mapping (CAM) methods, which are tailored for local, sequential dependencies, are ill-suited for interpreting these non-autoregressive behaviors. To bridge this gap, we propose Diffusion-CAM, the first interpretability method specifically tailored for dMLLMs. We derive raw activation maps by differentiably probing intermediate representations in the transformer backbone, accordingly capturing both latent features and their class-specific gradients. To address the inherent stochasticity of these raw signals, we incorporate four key modules to resolve spatial ambiguity and mitigate intra-image confounders and redundant token correlations. Extensive experiments demonstrate that Diffusion-CAM significantly outperforms SoTA methods in both localization accuracy and visual fidelity, establishing a new standard for understanding the parallel generation process of diffusion multimodal systems.

Figures & tables

Figure 1: Visual comparison of (A) [LLaVA-CAM] and (B) [Diffusion-CAM] (ours). Our method generates more focused and clearer activation maps.

Figure 2: Illustration of the Raw CAM generation and Adaptive Denoising. (Left) Raw CAM Generation: We capture intermediate visual features and gradients from the LaViDa backbone during the denoising process via hook registration. (Right) Adaptive Kernel Denoising: A dynamic kernel $\mathbf{K}_{adapt}$ , calibrated by feature statistics (e.g., $\sigma$ ), drives a Rank Gaussian Filter to suppress stochastic noise and architectural artifacts, transforming noisy activations into precise, semantically coherent heatmaps.

Figure 3: Illustration of the other three modules of Diffusion-CAM. Distribution-Aware Confidence Gating: We calculate an adaptive threshold and process regions differently. Contextual Background Attenuation: identifies the foreground and background and calculates suppression factors. Single-Instance Causal Debiasing: uses repeated tokens and abnormally high activation mask to make heatmap clearer.

	COCO Caption				GranDf
Method	Obj-IoU(%)	Contrast	Concen.	F3-Score	Obj-IoU(%)	Contrast	Concen.	F3-Score
LLaVA-CAM	20.02	2.18 $\times$	41.22	18.08	18.16	1.41 $\times$	70.80	14.22
Grad-CAM	19.93	2.04 $\times$	41.91	17.43	17.82	1.48 $\times$	75.42	14.67
TAM	15.21	2.51 $\times$	40.10	17.61	20.39	1.36 $\times$	67.31	14.23
Diffusion-CAM (Ours)	30.10	2.58 $\times$	51.41	23.04	28.41	2.02 $\times$	86.14	20.53

Table 1: Comprehensive comparison with state-of-the-art methods on COCO Caption Chen et al. (2015) and GranDf datasets Rasheed et al. (2024) , and the best baseline results are underlined . The F3-Score (%) reflects the overall performance, calculated as the harmonic mean of the three core metrics.

Config.	Obj-IoU(%)	Contrast	Concen.	F3-Score
Baseline	20.12	2.19×	41.42	18.16
+ Denoising	25.85	2.28×	44.01	20.16
+ Gating	21.22	2.20×	47.94	18.88
+ Attenuation	21.42	2.41×	42.63	19.59
+ Debiasing	24.11	2.29×	44.31	19.82
Full	30.10	2.58×	51.44	23.04

Table 2: Ablation study on COCO Caption Chen et al. (2015) . Each row toggles one module on top of the baseline.

Figure 4: Visual comparison between our method (D) and SoTA approaches including (A) [Grad-CAM], (B) [LLaVA-CAM], and (C) [TAM] at different Transformer layers. Diffusion-CAM produces more precise activation maps compared to the baselines, exhibiting robust localization ability across all visualized layers.

Figure 5: Visual comparison regarding artifacts elimination. (A) [Original CAM], (B) [Diffusion-CAM].

Figure 6: Visual comparison between our method and SoTA methods. Diffusion-CAM demonstrates superior global semantic alignment and also suppresses some noise in the baseline methods

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Attribution step	feat_len	img_end	Valid
$t=0$ (Prefix)	579	551	PASS
$t=S/4$	64	551	FAIL
$t=S/2$	64	551	FAIL
$t=3S/4$	64	551	FAIL
Aggregate over steps	–	–	N/A

Appendix

Table 4: Per-step feasibility check for image-span CAM extraction under LaViDa’s Prefix-DLM + KV-cached generation. A denoising step is considered valid only if the hooked hidden-state length satisfies

\texttt{seq\_len}\geq\texttt{img\_end}

. Across all logged step-records (

6600=200

images

\times

(1

prefix

+S

denoising steps

)

), only the earliest conditioning step passes, yielding a valid ratio of

200/6600=3.0\%

Attribution step	Valid	Contrast $\uparrow$	Concen. $\uparrow$	Obj-IoU $\uparrow$	F3-score $\uparrow$
Prefix ( $t=0$ )	200/200	$2.549\pm 0.126$	$50.69\pm 1.741$	$0.307\pm 0.024$	$0.234\pm 0.018$
$t=S/4$	0/200	–	–	–	–
$t=S/2$	0/200	–	–	–	–
$t=3S/4$	0/200	–	–	–	–
Aggregate over steps	–	–	–	–	–

Appendix

Table 5: CAM quality at the valid attribution step. Since later denoising steps are infeasible for image-span CAM extraction under Prefix-DLM + KV cache, metrics are reported only for the valid prefix step.

$\delta_{\sigma}$	Obj-IoU $\uparrow$	Contrast $\uparrow$	Concen. $\uparrow$	F3-Score $\uparrow$	high_var	high_mean	low_mean	default
0.18	$0.201\pm 0.021$	$2.29\pm 0.13$	$41.89\pm 2.044$	$0.192\pm 0.013$	36.4%	14.1%	47.2%	3.3%
0.20	$0.208\pm 0.017$	$2.38\pm 0.08$	$42.77\pm 2.041$	$0.194\pm 0.012$	27.8%	16.6%	50.8%	4.8%
0.22	$\mathbf{0.215\pm 0.015}$	$\mathbf{2.44\pm 0.12}$	$\mathbf{43.21\pm 2.037}$	$\mathbf{0.196\pm 0.011}$	22.1%	19.3%	53.4%	5.2%
0.24	$0.206\pm 0.019$	$2.39\pm 0.12$	$42.69\pm 2.039$	$0.193\pm 0.015$	18.6%	21.0%	54.9%	5.6%
0.26	$0.202\pm 0.025$	$2.30\pm 0.14$	$42.01\pm 2.049$	$0.192\pm 0.011$	15.1%	22.3%	56.4%	6.2%

Appendix

Table 6:

\delta_{\sigma}

sweep for DACG branch-boundary sensitivity. (Baseline values are

\delta_{\sigma}=0.22

\delta_{\mu}=0.35

\delta^{\prime}_{\mu}=0.25

, with

\alpha_{1}=90

\alpha_{2}=75

\alpha_{3}=80

, and

\alpha_{4}=85

$\delta_{\mu}$	Obj-IoU $\uparrow$	Contrast $\uparrow$	Concen. $\uparrow$	F3-Score $\uparrow$	high_var	high_mean	low_mean	default
0.28	$0.199\pm 0.024$	$2.30\pm 0.15$	$41.75\pm 2.121$	$0.192\pm 0.037$	14.7%	37.2%	45.2%	2.9%
0.32	$0.205\pm 0.022$	$2.36\pm 0.18$	$42.41\pm 2.132$	$0.194\pm 0.034$	19.2%	26.4%	49.8%	4.6%
0.35	$\mathbf{0.215\pm 0.015}$	$\mathbf{2.44\pm 0.12}$	$\mathbf{43.21\pm 2.037}$	$\mathbf{0.196\pm 0.011}$	22.1%	19.3%	53.4%	5.2%
0.38	$0.207\pm 0.027$	$2.35\pm 0.19$	$42.60\pm 2.054$	$0.194\pm 0.031$	25.6%	11.8%	57.2%	5.4%
0.42	$0.201\pm 0.025$	$2.32\pm 0.19$	$42.01\pm 2.173$	$0.193\pm 0.039$	29.3%	3.1%	61.7%	5.9%

Appendix

Table 7:

\delta_{\mu}

sweep for DACG branch-boundary sensitivity.

$\delta^{\prime}_{\mu}$	Obj-IoU $\uparrow$	Contrast $\uparrow$	Concen. $\uparrow$	F3-Score $\uparrow$	high_var	high_mean	low_mean	default
0.20	$0.198\pm 0.025$	$2.28\pm 0.22$	$40.81\pm 2.201$	$0.192\pm 0.012$	26.0%	23.5%	43.6%	6.9%
0.23	$0.204\pm 0.023$	$2.31\pm 0.19$	$42.07\pm 2.147$	$0.194\pm 0.013$	23.4%	20.7%	49.9%	6.0%
0.25	$\mathbf{0.215\pm 0.015}$	$\mathbf{2.44\pm 0.12}$	$\mathbf{43.21\pm 2.037}$	$\mathbf{0.196\pm 0.011}$	22.1%	19.3%	53.4%	5.2%
0.28	$0.209\pm 0.021$	$2.39\pm 0.17$	$42.61\pm 2.094$	$0.195\pm 0.015$	19.2%	16.8%	60.2%	3.8%
0.30	$0.202\pm 0.031$	$2.25\pm 0.24$	$41.98\pm 2.188$	$0.193\pm 0.013$	14.9%	13.4%	69.3%	2.4%

Appendix

Table 8:

\delta^{\prime}_{\mu}

sweep for DACG branch-boundary sensitivity.

Setting	Obj-IoU $\uparrow$	Contrast $\uparrow$	Concen. $\uparrow$	F3-Score $\uparrow$	high_var	high_mean	low_mean	default
Dynamic (ours)	$\mathbf{0.215\pm 0.015}$	$\mathbf{2.44\pm 0.12}$	$\mathbf{43.21\pm 2.037}$	$\mathbf{0.196\pm 0.011}$	22.1%	19.3%	53.4%	5.2%
All $\alpha-5$	$0.201\pm 0.022$	$2.28\pm 0.26$	$41.77\pm 2.111$	$0.193\pm 0.008$	22.1%	19.3%	53.4%	5.2%
All $\alpha+5$	$0.203\pm 0.017$	$2.32\pm 0.28$	$41.62\pm 2.102$	$0.193\pm 0.013$	22.1%	19.3%	53.4%	5.2%
Fixed $\alpha=85$	$0.197\pm 0.019$	$2.23\pm 0.23$	$41.05\pm 1.966$	$0.192\pm 0.009$	22.1%	19.3%	53.4%	5.2%

Appendix

Table 9: Percentile sensitivity of DACG with branch boundaries fixed.

Figure 7: Visualization results of four caption variants under the same teacher-forced attribution protocol by Diffusion-CAM. It is evident that, following the addition of redundant function words, regions in the image unrelated to the answer keywords (e.g., "cup") began to exhibit high-intensity activation; furthermore, as the number of function words increased, these irrelevant activations became increasingly pronounced.

Stage	Time (ms)	Device	Complexity
Base attribution pass	4880.8	GPU	-
AKD	12.7	CPU	$O(HW\cdot k^{2})$
DACG	6.9	CPU	$O(HW)$
CBA	5.6	CPU	$O(HW)$
SICD	10.2	CPU	$O(HW)$
Total post-processing	35.4	CPU	$O(HW)$

Appendix

Table 10: Per-module runtime breakdown. The refinement modules are CPU-only and add negligible overhead relative to the attribution pass.

Method	Time (ms)	Peak Mem (GB)	F3-score	Notes
Diffusion-CAM (base)	4880.8	83.21	0.188	generate + contrastive backward
Base + AKD	4893.8	83.21	0.207	+ rank Gaussian filter
Base + DACG	4887.0	83.21	0.187	+ confidence gating
Base + CBA	4886.3	83.21	0.190	+ background attenuation
Base + SICD	4891.1	83.21	0.195	+ causal debiasing
Diffusion-CAM (full)	4914.4	83.21	0.225	all modules enabled

Appendix

Table 11: End-to-end computational overhead per image on 200 images (NVIDIA H20).

Variant	Contrast $\uparrow$	Concen. $\uparrow$	Obj-IoU $\uparrow$	F3-score $\uparrow$
concise	$2.3546\pm 0.1133$	$43.56\pm 1.968$	$0.2342\pm 0.0153$	$0.1942\pm 0.0071$
original	$2.1905\pm 0.1219$	$41.31\pm 2.029$	$0.2086\pm 0.0166$	$0.1846\pm 0.0085$
verbose	$2.1266\pm 0.1286$	$39.17\pm 2.031$	$0.1969\pm 0.0155$	$0.1829\pm 0.0087$
repeated	$2.0985\pm 0.1300$	$39.01\pm 2.027$	$0.1908\pm 0.0138$	$0.1811\pm 0.0092$

Appendix

Table 12: Controlled validation of the linguistic economy hypothesis on 200 COCO images. We compare four caption variants under the same teacher-forced attribution protocol. Increasing function-word redundancy leads to more diffuse and less object-specific CAMs.

Multimodal Model Interpretability

Multimodal model interpretability studies how AI systems combine and use images, text, audio, video, and other signals, and how to explain their decisions.

See all 129 papers

Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models
Apr 7, 2026Keuntae Kim, Mingyu Kang, Yong Suk ChoiVLM Reasoning Diffusion Language Models
Department of Computer Science, Hanyang University · Department of Artificial Intelligence, Hanyang University
Learning to Read the Contextual Tokens in Diffusion Transformers
Oct 5, 2026Omer Dahary, Etai Sella, Hadar Averbuch-Elor +2Diffusion Model Alignment Diffusion Transformer
Tel Aviv University · Cornell University
Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs
Aug 23, 2026Yikai Zhao, Qiyan Zhao, Jiaquan Zhang +3Diffusion Language Model Decoding Long-Form Document Generation
Sun Yat-sen University · Shanghai Jiao Tong University · University of Electronic Science and Technology of China +2

Apr 7, 2026J/K move · Enter open · S save

Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models

Keuntae Kim, Mingyu Kang, Yong Suk Choi

Department of Computer Science, Hanyang University · Department of Artificial Intelligence, Hanyang University

VLM Reasoning Diffusion Language Models Multimodal Diffusion Models Diffusion Model Guidance Vision-Language Grounding

Open paper AskPDF arXiv

Diffusion large language models (dLLMs) are emerging as promising alternatives to autoregressive (AR) LLMs. Recently, this paradigm has been extended to multimodal tasks, leading to the development of diffusion multimodal large language models (dMLLMs). These models are expected to retain the reasoning capabilities of LLMs while enabling faster inference through parallel generation. However, when combined with Chain-of-Thought (CoT) reasoning, dMLLMs exhibit two critical issues. First, we observe that dMLLMs often generate the final answer token at a very early timestep. This trend indicates that the model determines the answer before sufficient reasoning, leading to degraded reasoning performance. Second, during the initial timesteps, dMLLMs show minimal dependency on visual prompts, exhibiting a fundamentally different pattern of visual information utilization compared to AR vision-language models. In summary, these findings indicate that dMLLMs tend to generate premature final answers without sufficiently grounding on visual inputs. To address these limitations, we propose Position and Step Penalty (PSP) and Visual Reasoning Guidance (VRG). PSP penalizes tokens in later positions during early timesteps, delaying premature answer generation and encouraging progressive reasoning across timesteps. VRG, inspired by classifier-free guidance, amplifies visual grounding signals to enhance the model's alignment with visual evidence. Extensive experiments across various dMLLMs demonstrate that our method achieves up to 7.5% higher accuracy while delivering more than 3x speedup compared to reasoning with four times more diffusion steps.

Abstract

Multimodal Model Interpretability

Explore similar work

Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models

Learning to Read the Contextual Tokens in Diffusion Transformers

Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs