TIB: Sample-wise Tempered Information Bottleneck for Multimodal Attribution beyond Alignment Assumption
Organizations: The MIx group, School of Computer Science, University of Birmingham
Abstract
Multimodal (e.g. vision-language, in this work) attribution aims to interpret the models without access to ground-truth explanatory supervision. Existing multimodal attribution methods typically rely on paired modalities as proxy supervision, implicitly assuming that image-text pairs are semantically aligned. Information bottleneck (IB)-based attribution methods follow this general paradigm and instantiate it through a global sufficiency-compression objective. In realistic multimodal datasets, however, this assumption is frequently violated due to partial semantic misalignment and unreliable cross-modal correspondences. Through theoretical analysis, we show that, under such misalignment, information-bottleneck-based attribution objectives exhibit a structural limitation: by assuming equal reliability across all proxy supervision signals, they induce posterior over-contraction and forced explanations. To address this issue, we propose TIB, a sample-wise Tempered Information Bottleneck framework that adaptively modulates attribution strength according to cross-modal reliability, without assuming semantic alignment. Extensive experiments on controlled benchmarks and large-scale datasets show the effectiveness of the proposed approach, while also validating the identified failure mode.
Figures & tables
| Caption type | Grad-CAM [ 26 ] | RISE [ 20 ] | M2IB [ 32 ] | Fast-IG [ 8 ] | Chefer [ 4 ] | Grad-ECLIP [ 35 ] | FIxLIP [ 2 ] | NIB [ 37 ] | Ours (TIB) |
|---|---|---|---|---|---|---|---|---|---|
| Aligned | |||||||||
| Selective-bias | |||||||||
| Perturbation-bias |
| Dataset | Metric | Grad-CAM [ 26 ] | M2IB [ 32 ] | NIB [ 37 ] | Ours (TIB) |
| Drop / Incr. | |||||
| cc3M-I | Drop | 8.07 6.02 | 1.11 1.04 | 1.03 0.95 | 1.00 0.91 |
| cc3M-I | Incr | 7.20 10.21 | 37.20 39.43 | 38.80 41.90 | 41.21 44.77 |
| cc3M-T | Drop | 1.79 1.24 | 0.90 0.85 | 1.13 1.04 | 1.10 0.96 |
| cc3M-T | Incr | 33.90 35.12 | 37.80 40.07 | 40.50 43.74 | 41.11 44.11 |
| Flickr8k-I | Drop | 9.19 7.23 | 1.59 1.49 | 1.49 1.37 | 1.19 1.13 |
| Setting | Aligned | Selective | Perturbation |
|---|---|---|---|
| Random | |||
| Fixed | |||
| Inverted learned | |||
| w/o stop-gradient | |||
| w/o variance regularisation |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Entropy | Sim Std | Text Consistency | Cluster Var | Silhouette | Paired Top- |
|---|---|---|---|---|---|---|
| Flickr8k | 0.758 | 0.017 | 0.637 | 0.191 | 0.219 | 0.967 |
| CC3M | 0.730 | 0.017 | 0.670 | 0.172 | 0.280 | 0.933 |
| LAION400M | 0.445 | 0.024 | 0.564 | 0.236 | 0.224 | 0.972 |
Multimodal Model Interpretability
Multimodal model interpretability studies how AI systems combine and use images, text, audio, video, and other signals, and how to explain their decisions.
See all 138 papers