Multimodal (e.g. vision-language, in this work) attribution aims to interpret the models without access to ground-truth explanatory supervision. Existing multimodal attribution methods typically rely on paired modalities as proxy supervision, implicitly assuming that image-text pairs are semantically aligned. Information bottleneck (IB)-based attribution methods follow this general paradigm and instantiate it through a global sufficiency-compression objective. In realistic multimodal datasets, however, this assumption is frequently violated due to partial semantic misalignment and unreliable cross-modal correspondences. Through theoretical analysis, we show that, under such misalignment, information-bottleneck-based attribution objectives exhibit a structural limitation: by assuming equal reliability across all proxy supervision signals, they induce posterior over-contraction and forced explanations. To address this issue, we propose TIB, a sample-wise Tempered Information Bottleneck framework that adaptively modulates attribution strength according to cross-modal reliability, without assuming semantic alignment. Extensive experiments on controlled benchmarks and large-scale datasets show the effectiveness of the proposed approach, while also validating the identified failure mode.
Figures & tables
Figure 1: Qualitative comparison of multimodal attribution under completely aligned (top: ✓ ) and misaligned (bottom: ✗ ) image–text pairs. Our method suppresses attribution on completely mismatched image–text pairs.
Figure 2: Conceptual visualisation of our multimodal data generation assumption. We assume that each multimodal pair contains both semantically aligned and misaligned information.
Figure 3: Illustration of the IB fitting decomposition under misalignment. Top: mild contamination. Bottom: severe contamination. As contamination increases, the estimated optimisation direction deviates further from the semantic optimum.
Figure 4: GMM fit on a sampled subset of LAION-400M.
Figure 5: Mechanistic analysis of heteroscedastic gating ( σ2(xi) ). (a,b) Relation between predicted variance σ2 and attribution-mask gradient magnitude ∥∇λL∥ under V→T and T→V , comparing heteroscedastic and homoscedastic bottlenecks. Heteroscedastic IB (TIB) shows variance-dependent gradient flow, whereas homoscedastic IB (classical IB) does not. (c) Predicted variance is consistently larger for misaligned than aligned pairs, indicating higher uncertainty.
Figure 6: Risk–coverage curves for uncertainty-based reliability ranking. Lower is better.
Caption type
Grad-CAM [ 26 ]
RISE [ 20 ]
M2IB [ 32 ]
Fast-IG [ 8 ]
Chefer [ 4 ]
Grad-ECLIP [ 35 ]
FIxLIP [ 2 ]
NIB [ 37 ]
Ours (TIB)
Aligned
8.89±3.34
10.65±3.12
14.20±2.18
11.92±2.86
12.83±2.68
13.48±2.43
13.82±2.27
11.96±2.05
16.26±1.02
Selective-bias
7.42±3.28
8.91±3.05
11.76±2.12
9.84±2.79
10.53±2.61
11.18±2.38
11.47±2.21
9.72±2.01
14.21±1.01
Perturbation-bias
5.36±3.19
6.48±2.96
8.73±2.08
7.12±2.71
7.81±2.55
8.26±2.31
8.55±2.17
7.08±1.98
11.83±1.00
Table 1: Ground-truth attribution evaluation on the synthetic RefCOCO benchmark. We apply the same synthetic caption protocol to RefCOCO and report IoU ( ↑ ) as mean ± standard deviation across five random seeds. Coloured cells indicate caption types using the corresponding curve colours.
Dataset
Metric
Grad-CAM [ 26 ]
M2IB [ 32 ]
NIB [ 37 ]
Ours (TIB)
Drop / Incr.
cc3M-I
Drop ↓
8.07 6.02
1.11 1.04
1.03 0.95
1.00 0.91
cc3M-I
Incr ↑
7.20 10.21
37.20 39.43
38.80 41.90
41.21 44.77
cc3M-T
Drop ↓
1.79 1.24
0.90 0.85
1.13 1.04
1.10 0.96
cc3M-T
Incr ↑
33.90 35.12
37.80 40.07
40.50 43.74
41.11 44.11
Flickr8k-I
Drop ↓
9.19 7.23
1.59 1.49
1.49 1.37
1.19 1.13
Table 2: Quantitative comparison of interpretability methods across vision–language benchmarks. Results are reported as meanacc with acceptance rate κ=0.8 . Main values report full-sample means; subscripts report accepted-sample means at acceptance rate.
Figure 7: Predictive variance analysis. (a–b) Reducible uncertainty in hard-but-aligned pairs versus persistent uncertainty in misaligned pairs. (c–d) Discriminability of misalignment under different variance parameterisations.
Setting
Aligned
Selective
Perturbation
Random σ2
−3.62/+0.15
−4.44/+0.44
−4.74/+0.54
Fixed σ2
−4.19/+0.12
−3.79/+0.38
−4.05/+0.47
Inverted learned σ2
−3.41/+0.29
−8.17/+0.81
−8.40/+0.94
w/o stop-gradient
−0.60/+0.06
−1.44/+0.12
−1.50/+0.16
w/o variance regularisation
−1.86/+0.28
−4.96/+0.37
−3.99/+0.62
Table 3: Ablation of variance modelling in TIB. Each entry reports ΔIncr./ΔDrop relative to the default TIB setting with learned σ2 . Negative ΔIncr. and positive ΔDrop indicate performance degradation.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Directional structure of real-world multimodal data. Structural analysis on CC3M and Flickr8k shows that, even when paired retrieval remains strong, the feature space still exhibits substantial heterogeneity in local dispersion and cluster organisation, providing empirical support for our mixture-based multimodal misalignment assumption.
Dataset
Entropy ↑
Sim Std ↓
Text Consistency ↑
Cluster Var ↑
Silhouette ↑
Paired Top- k↑
Flickr8k
0.758
0.017
0.637
0.191
0.219
0.967
CC3M
0.730
0.017
0.670
0.172
0.280
0.933
LAION400M
0.445
0.024
0.564
0.236
0.224
0.972
Appendix
Table 4: Structural analysis of multimodal feature-space heterogeneity across datasets. Higher entropy and cluster variance indicate greater local dispersion, while silhouette score measures cluster separability. Paired Top- k reflects local image–text alignment through paired-caption retrieval. Despite high retrieval accuracy, substantial feature-space heterogeneity persists across datasets, supporting the proposed mixture-based misalignment assumption.
Figure 9: Roles of α and β in TIB. β controls the bottleneck capacity, while α controls the uncertainty-based split strength after compression.
Figure 10: β -sensitivity analysis. TIB remains near-optimal over a wide range of β , showing lower sensitivity than competing methods.
Figure 11: Qualitative comparison of multimodal attribution maps. The maps highlight image regions and text tokens contributing to predictions, illustrating how different methods attribute importance across modalities. More examples on the following pages.
Figure 12: Qualitative comparison of multimodal attribution maps (continued).
Figure 13: Qualitative comparison of multimodal attribution maps (continued).
Multimodal model interpretability studies how AI systems combine and use images, text, audio, video, and other signals, and how to explain their decisions.
A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.
Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4
Contrastive learning is effective for aligning paired views or modalities, but alignment beyond two modalities remains non-trivial and comparatively underexplored. Pairwise CLIP-style losses decompose multi-modal alignment into independent two-way comparisons and therefore do not explicitly model higher-order dependencies among multiple modalities. Recent beyond-pairwise objectives approach this problem from statistical or geometric perspectives, but arbitrary-modality alignment still lacks a principled criterion for defining what each modality should preserve and compress relative to the others. We revisit arbitrary-modality alignment through the Information Bottleneck principle. In multi-modal learning, sufficiency should preserve information predictable from the remaining modalities, while minimality should compress modality-specific information not supported by them. This naturally leads to a One-vs-All view, where each modality is characterized with respect to the remaining modalities. We propose OVA-IB, an Information Bottleneck framework for arbitrary-modality alignment. OVA-IB optimizes a tractable One-vs-All contrastive lower bound for sufficiency connected to a Dual Total Correlation-style objective, uses a parameter-free geometry-aware projection score, and derives a tractable upper-bound regularizer for minimality by bounding each representation's dependence on its own input with representation distributions induced by the remaining modalities. Experiments on classification, regression, modality-agnostic evaluation, and cross-modal retrieval benchmarks demonstrate strong and robust performance.
Tianchao Li, Shujian Yu, Xinrui Zu +4
Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway +3
Multimodal large language models (MLLMs) increasingly support high-stakes decision making by combining complementary information from images and text. While existing explainability methods identify influential image regions or text tokens, they cannot answer a fundamental question: which modality drives a prediction? Consequently, a model may produce the correct output while relying on the wrong source of evidence, masking shortcut learning and unsafe reasoning. We formulate modality attribution as a complementary explainability objective for multimodal foundation models and propose Counterfactual Modality Attribution (CMA), the first framework for quantifying modality-level contributions in MLLMs. CMA generates image-only, text-only, and joint multimodal counterfactuals using coupled diffusion priors and converts them into principled modality attribution scores through a cooperative game-theoretic formulation based on Shapley values. We evaluate CMA on controlled synthetic benchmarks with known ground-truth modality reliance and on a real-world multimodal clinical dataset. CMA correctly identifies the decision-driving modality in 98% of controlled cases and consistently outperforms baselines, revealing failures of cross-modal reasoning that remain invisible to predictive accuracy alone. Our results establish modality attribution as a complementary dimension of explainability beyond feature attribution, providing a principled framework for auditing multimodal foundation models in safety-critical applications.
Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera +2
Faculty of Electrical Engineering, University of Sarajevo, Bosnia and Herzegovina · King’s College London, United Kingdom