cs.CVNov 8, 2026

TIB: Sample-wise Tempered Information Bottleneck for Multimodal Attribution beyond Alignment Assumption

Authors: Qiming Huang, Peixi Liu, Jianbo Jiao

Organizations: The MIx group, School of Computer Science, University of Birmingham

Abstract

Multimodal (e.g. vision-language, in this work) attribution aims to interpret the models without access to ground-truth explanatory supervision. Existing multimodal attribution methods typically rely on paired modalities as proxy supervision, implicitly assuming that image-text pairs are semantically aligned. Information bottleneck (IB)-based attribution methods follow this general paradigm and instantiate it through a global sufficiency-compression objective. In realistic multimodal datasets, however, this assumption is frequently violated due to partial semantic misalignment and unreliable cross-modal correspondences. Through theoretical analysis, we show that, under such misalignment, information-bottleneck-based attribution objectives exhibit a structural limitation: by assuming equal reliability across all proxy supervision signals, they induce posterior over-contraction and forced explanations. To address this issue, we propose TIB, a sample-wise Tempered Information Bottleneck framework that adaptively modulates attribution strength according to cross-modal reliability, without assuming semantic alignment. Extensive experiments on controlled benchmarks and large-scale datasets show the effectiveness of the proposed approach, while also validating the identified failure mode.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

    May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal LearningMultimodal Learning

  2. OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment

    May 28, 2026Tianchao Li, Shujian Yu, Xinrui Zu +4Cross-Modal LearningCross-Modal Alignment

  3. Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs

    Jul 29, 2026Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera +2Multimodal Large Language ModelsMultimodal Model Interpretability