Modality Reliance in Multimodal Models

Latest papers

23 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information
Oct 17, 2026Xinyu Chen, Liangjian Wen, Jiang Duan +7Multimodal Fusion Modality Reliance in Multimodal Models
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · X-Humanoid +4
Reliability-Consistent Appearance Modeling for RGB-Infrared Multi-Object Tracking
Oct 16, 2026Bingyang Wang, Jiazuo Yu, Lu Zhang +3Multimodal Visual Object Tracking Modality Reliance in Multimodal Models
School of Information and Communication Engineering, Dalian University of Technology, Dalian 116024, China · Department of Engineering, Tsinghua University, Beijing 100084, China
Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts
Sep 5, 2026Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu +4Modality Reliance in Multimodal Models Multimodal Large Language Models
Characterizing Text Branch Sensitivity in Medical Vision-Language Segmentation via Evidence Decoupling
Sep 2, 2026Ziquan Liu, Zhewei Zhu, Xuyang ShiModality Reliance in Multimodal Models Medical Image Segmentation
School of Information and Control Engineering, Southwest University of Science and Technology, Mianyang 621010, China
Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting
Aug 23, 2026Karthik Sridhar, Atharva Gupta, Nishant Pradhan +3Modality Reliance in Multimodal Models Multimodal Foundation Models
Birla AI Labs, Mumbai, India · BITS Pilani, Pilani, India · KIIT, Bhubaneswar, India
Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation
Aug 10, 2026Guanqun Yang, Wenlong ZhangMissing-Modality Learning Multimodal Robustness
Stevens Institute of Technology Hoboken, NJ, USA
C $^3$ PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
Aug 5, 2026Swapnanil Mukherjee, Agyeya Negi, Tanuja Ganu +1Modality Imbalance Multimodal Reasoning
Microsoft Research India · IIIT Hyderabad, India
Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs
Jul 29, 2026Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera +2Multimodal Large Language Models Multimodal Model Interpretability
Faculty of Electrical Engineering, University of Sarajevo, Bosnia and Herzegovina · King’s College London, United Kingdom
See2Think: Do Multimodal Models Really Use Intermediate Visual States?
Jul 29, 2026Siyu Yan, Zhuoran Yan, Haiying Xu +10Visual Reasoning Multimodal Reasoning
1The Hong Kong University of Science and Technology · 3Shanghai AI Laboratory · 2Central South University +4
Should Missing Modalities Always Be Necessary to Repair for Multi-modal Sentiment Analysis?
Jul 19, 2026Yubo Gao, Haotian Wu, Xiaoyu Xu +9Missing-Modality Learning Multimodal Sentiment Analysis
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Nanyang Technological University +2
Attending to Multimodal Generation One Token at a Time
Jul 4, 2026Varun Gupta, Vineet Gandhi, Makarand TapaswiMultimodal Large Language Models Multimodal Generation
CVIT, IIIT Hyderabad
Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails
Jul 2, 2026Qianyu Chen, Canran Xiao, Runxuan TangMultimodal Grounding Multimodal Foundation Models
1Nanyang Technological University · 2Shenzhen Campus of Sun Yat-sen University
When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence
Jun 25, 2026Jaden Moon, Arvind Pillai, Andrew CampbellUncertainty-Aware Multimodal Fusion Multimodal Fusion
Dartmouth College, United States
Learning from Audio-Dependency Errors: Data Curation Strategies Based on Model Confusion Patterns in Audio Question Answering
Jun 20, 2026Hyeonuk NamAudio-Language Model Evaluation Modality Reliance in Multimodal Models
Independent Researcher · Seoul, South Korea
To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection
Jun 4, 2026Erfan Loweimi, Mengjie Qian, Kate Knill +7Cross-Modal Retrieval Missing-Modality Learning
University of Cambridge, UK · Southwest Jiaotong University, China · University of Surrey, UK +1
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Understanding Multimodal Grounding
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal Learning Multimodal Learning
1KU Leuven · 2MIT
Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs
May 10, 2026Jiafeng Liang, Zhihao Zhu, Zihan Zhang +7Modality Reliance in Multimodal Models Causal Discovery
Harbin Institute of Technology · Harvard University · Peking University +2
All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation
Apr 27, 2026Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2Audio-Language Model Evaluation Modality Reliance in Multimodal Models
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models
Apr 15, 2026Akshay Paruchuri, Ishan Chatterjee, Henry Fuchs +2Modality Reliance in Multimodal Models Multimodal Large Language Models
Università della Svizzera italiana · Stanford University · University of Washington +1
Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention
Mar 28, 2026Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan ParvezModality Reliance in Multimodal Models Multimodal Model Evaluation
University of California, Riverside · University of Maryland, Baltimore County · Qatar Computing Research Institute (QCRI)
FBCIR: Balancing Cross-Modal Focuses in Composed Image Retrieval
Mar 12, 2026Chenchen Zhao, Jianhuan Zhuo, Muxi Chen +6Modality Reliance in Multimodal Models Composed Image Retrieval
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Tencent AI Data Department · School of Mathematical Sciences, Dalian University of Technology +1
Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection
Sep 18, 2025Zhu Li, Xiyuan Gao, Yuqing Zhang +2Sarcasm Detection Multimodal Model Evaluation
University of Groningen, The Netherlands

Oct 17, 2026J/K move · Enter open · S save

MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information

Xinyu Chen, Liangjian Wen, Jiang Duan, Dongkai Wang+6

School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · X-Humanoid+4

Multimodal Fusion Modality Reliance in Multimodal Models Multimodal Optimization

Open paper AskPDF arXiv

In multimodal learning, modality imbalance often lets one modality dominate optimization and keeps the model from fully exploiting cross-modal interactions. Balance-oriented methods such as the Multimodal Competition Regularizer (MCR) perform well by alleviating this imbalance during training. These dynamic balancing methods, however, mainly adjust modality contributions and do not consider how imbalance affects the formation of synergistic information during fusion. Synergy is a major source of multimodal advantage, because it is predictive gain that cannot be recovered from unimodal evidence alone. We propose the Multimodal Synergistic Competition Regularizer (MSCR), a fusion method that jointly promotes balanced modality utilization and the discovery of synergistic information. MSCR has two complementary objectives. An anti-dominance objective suppresses excessive unimodal influence and leaves room for synergistic information, and a synergy objective encourages the fused predictor to outperform a non-synergistic unimodal reference. An adaptive gate further concentrates the regularization on samples that need stronger collaborative modeling. Experiments on five benchmarks show that MSCR consistently improves performance over MCR and strengthens synergistic capability. It raises conditional recovery, the accuracy of the fused prediction on samples that the unimodal predictors misclassify, as well as the share of the model's output that InterSHAP attributes to cross-modal interactions. Its gain in conditional recovery persists when the fusion module is Transformer-based. Code is available at https://github.com/chenxinyu991-collab/MSCR.