Modality Reliance in Multimodal Models

Latest papers

34 papers8 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning
Nov 5, 2026Liangjian Wen, Qun Dai, Jiang Duan +7Multimodal Learning Multimodal Fusion
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, China · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd. +7
What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis
Nov 3, 2026Christian Pipal, Eva-Maria Vogel, Morgan WackModality Reliance in Multimodal Models Video-Language Model Evaluation
Department of Communication and Media Research, University of Zurich, Zurich, Switzerland.
DGC-Net: Cross-Modal Dynamic Routing with Global-Guided Compensation in Multimodal Sentiment Analysis
Nov 3, 2026Linan Zhu, Huaijing Wu, Xiao Han +1Multimodal Sentiment Analysis Gated Multimodal Fusion
College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China · AIDC, Alibaba Group, China · Department of Data Science, City University of Hong Kong, Hong Kong SAR, China
Probe-Guided Gradient Balancing for Multimodal Learning
Nov 2, 2026Vu Vo, Haytham M. Fayek, Thuy NguyenModality Imbalance Modality Reliance in Multimodal Models
School of Science, Engineering & Technology, RMIT University · School of Computing Technologies, RMIT University
MOSAIC: Addressing Disproportionate Task-Modality Dependencies via Modality-Partitioned Gradient Routing
Oct 29, 2026Aryan Chowdhury, Yizheng Sun, Thomas Carroll +1Modality Reliance in Multimodal Models Multi-Task Learning
The University of Manchester
A Broader Look into MLLM's Retrieval Heads
Oct 23, 2026Jia Wang, Yuqing Li, Jiangnan Li +2Multimodal Large Language Models Multimodal IR
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Tencent
I $^2$ -KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis
Oct 22, 2026Xiaoting Xie, Feng Qiao, Xianwei Zhang +5Multimodal Sentiment Analysis Multimodal Fusion
School of Information Science and Engineering, Shandong University, Qingdao, Shandong, China · Department of Computer Science, University of Exeter, Exeter, UK · School of Computing Science, University of Glasgow, Glasgow, UK +1
TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
Oct 22, 2026Zhaolu Kang, Yidi Wang, Yile Li +4Multimodal Model Evaluation Multimodal QA
Hunyuan, Tencent · Peking University
Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis
Oct 20, 2026Xiaohua Huang, Bingrui Lei, Xiuyan Shao +4Multimodal Sentiment Analysis Modality Reliance in Multimodal Models
the School of Communication and Information Engineering, Nanjing Institute of Technology, Nanjing 210096, China. · the School of Economics and Management, Southeast University, Nanjing 211196, China. · the School of Computer Science and Engineering, Macau University of Science and Technology, Macau SAR, 999078, China. +2
EDMA: Entropy-Driven Multimodal Answering
Oct 19, 2026Emanuele Mezzi, Gertjan Burghouts, Fabio Massacci +1Multimodal QA Multimodal Reasoning
Vrije Universiteit Amsterdam · TNO · University of Trento
MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information
Oct 17, 2026Xinyu Chen, Liangjian Wen, Jiang Duan +7Multimodal Fusion Modality Reliance in Multimodal Models
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · X-Humanoid +4
Reliability-Consistent Appearance Modeling for RGB-Infrared Multi-Object Tracking
Oct 16, 2026Bingyang Wang, Jiazuo Yu, Lu Zhang +3Multimodal Visual Object Tracking Modality Reliance in Multimodal Models
School of Information and Communication Engineering, Dalian University of Technology, Dalian 116024, China · Department of Engineering, Tsinghua University, Beijing 100084, China
Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts
Sep 5, 2026Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu +4Modality Reliance in Multimodal Models Multimodal Large Language Models
Characterizing Text Branch Sensitivity in Medical Vision-Language Segmentation via Evidence Decoupling
Sep 2, 2026Ziquan Liu, Zhewei Zhu, Xuyang ShiModality Reliance in Multimodal Models Medical Image Segmentation
School of Information and Control Engineering, Southwest University of Science and Technology, Mianyang 621010, China
Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting
Aug 23, 2026Karthik Sridhar, Atharva Gupta, Nishant Pradhan +3Modality Reliance in Multimodal Models Multimodal Foundation Models
Birla AI Labs, Mumbai, India · BITS Pilani, Pilani, India · KIIT, Bhubaneswar, India
Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation
Aug 10, 2026Guanqun Yang, Wenlong ZhangMissing-Modality Learning Multimodal Robustness
Stevens Institute of Technology Hoboken, NJ, USA
C $^3$ PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
Aug 5, 2026Swapnanil Mukherjee, Agyeya Negi, Tanuja Ganu +1Modality Imbalance Multimodal Reasoning
Microsoft Research India · IIIT Hyderabad, India
Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs
Jul 29, 2026Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera +2Multimodal Large Language Models Multimodal Model Interpretability
Faculty of Electrical Engineering, University of Sarajevo, Bosnia and Herzegovina · King’s College London, United Kingdom
See2Think: Do Multimodal Models Really Use Intermediate Visual States?
Jul 29, 2026Siyu Yan, Zhuoran Yan, Haiying Xu +10Visual Reasoning Multimodal Reasoning
1The Hong Kong University of Science and Technology · 3Shanghai AI Laboratory · 2Central South University +4
Should Missing Modalities Always Be Necessary to Repair for Multi-modal Sentiment Analysis?
Jul 19, 2026Yubo Gao, Haotian Wu, Xiaoyu Xu +9Missing-Modality Learning Multimodal Sentiment Analysis
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Nanyang Technological University +2
Attending to Multimodal Generation One Token at a Time
Jul 4, 2026Varun Gupta, Vineet Gandhi, Makarand TapaswiMultimodal Large Language Models Multimodal Generation
CVIT, IIIT Hyderabad
Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails
Jul 2, 2026Qianyu Chen, Canran Xiao, Runxuan TangMultimodal Grounding Multimodal Foundation Models
1Nanyang Technological University · 2Shenzhen Campus of Sun Yat-sen University
When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence
Jun 25, 2026Jaden Moon, Arvind Pillai, Andrew CampbellUncertainty-Aware Multimodal Fusion Multimodal Fusion
Dartmouth College, United States
Learning from Audio-Dependency Errors: Data Curation Strategies Based on Model Confusion Patterns in Audio Question Answering
Jun 20, 2026Hyeonuk NamAudio-Language Model Evaluation Modality Reliance in Multimodal Models
Independent Researcher · Seoul, South Korea
To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection
Jun 4, 2026Erfan Loweimi, Mengjie Qian, Kate Knill +7Cross-Modal Retrieval Missing-Modality Learning
University of Cambridge, UK · Southwest Jiaotong University, China · University of Surrey, UK +1
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Understanding Multimodal Grounding
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal Learning Multimodal Learning
1KU Leuven · 2MIT
Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs
May 10, 2026Jiafeng Liang, Zhihao Zhu, Zihan Zhang +7Modality Reliance in Multimodal Models Causal Discovery
Harbin Institute of Technology · Harvard University · Peking University +2
All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation
Apr 27, 2026Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2Audio-Language Model Evaluation Modality Reliance in Multimodal Models
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models
Apr 15, 2026Akshay Paruchuri, Ishan Chatterjee, Henry Fuchs +2Modality Reliance in Multimodal Models Multimodal Large Language Models
Università della Svizzera italiana · Stanford University · University of Washington +1

Nov 5, 2026J/K move · Enter open · S save

InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning

Liangjian Wen, Qun Dai, Jiang Duan, Yong Dai+6

School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, China · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd.+7

Multimodal Learning Multimodal Fusion Modality Reliance in Multimodal Models

Open paper AskPDF arXiv

Multimodal models are expected to exploit synergistic information, that is, information available only when multiple modalities are observed together. In practice, however, a fusion module can fit the training objective by relying on a dominant modality or on information that modalities share, and the synergy between modalities is often left unused. We propose InfMasking, a unified multimodal learning framework based on Infinite Masking. Before fusion, InfMasking randomly masks the features of each modality; each sampled mask produces a different partial combination of cross-modal evidence and a fused representation of the corresponding modality interactions. All masked fused representations are required to remain consistent with a reference target: the unmasked fused view when training is unsupervised, and the class proxy of the task label when labels are available. Maintaining this consistency across diverse partial combinations discourages reliance on a dominant modality or on shared information alone, and drives the fusion module to learn cross-modal interaction patterns that remain meaningful under partial observations. Since the objective is an expectation over infinitely many masks, we model the masked fused representations as Gaussian and optimize closed-form variational bounds, whose population-moment gaps are controlled by the masking-induced covariance under this model. On the evaluated benchmarks, InfMasking improves predictive performance and yields higher mean PID estimates of task-label synergy in learned modality-specific representations, alongside higher InterSHAP interaction attribution in the trained predictors. Code is available at https://github.com/brightest66/InfMasking.