Modality Imbalance

Follow this topic Ask about this topic

The field studies why models combining inputs such as images, text, audio, and sensor data may let one modality dominate, weakening robustness when inputs are noisy, missing, or in conflict. This matters in settings from sentiment analysis and audio-visual speech recognition to remote sensing and vision-language systems, where useful information can be overlooked. Approaches include gradient and loss reweighting, reliability-aware gating, modality dropout and curricula, representation alignment, and benchmarks that measure each modality’s contribution. A central tension is whether to equalize modalities or prioritize the most useful one: gradient norms and confidence are imperfect proxies for value, and the best balance can depend on the task and data.

Latest papers

33 papers1 paper in the last four weeks, down 80% on the four weeks before. 0.0% of all new papers.

Layer Roles Matter: Mitigating Modality Imbalance in Multimodal Emotion Reasoning via Adaptive Layer Profiling
Nov 9, 2026Pengyu Huang, Peipei Song, Beier Zhu +3Modality Imbalance Multimodal Reasoning
University of Science and Technology of China, Hefei, China · Hefei University of Technology, Hefei, China
Probe-Guided Gradient Balancing for Multimodal Learning
Nov 2, 2026Vu Vo, Haytham M. Fayek, Thuy NguyenModality Imbalance Modality Reliance in Multimodal Models
School of Science, Engineering & Technology, RMIT University · School of Computing Technologies, RMIT University
Balancing Multimodal Learning via Functional Progress
Oct 2, 2026Zhongjing Gu, Fengqiang Wan, Yiming Cui +2Modality Imbalance Multimodal Optimization
Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification
Sep 23, 2026Longfei Huang, Xiangyu Wu, Yang YangModality Imbalance Multimodal Optimization
Nanjing University of Science and Technology · Alibaba Group
All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
Sep 23, 2026Ohad Rahamim, Dvir Samuel, Idan Schwartz +1Video Diffusion Models Modality Imbalance
Bar-Ilan University · NVIDIA
Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domains
Sep 20, 2026Kiran Naseer, Samreen Azhar, Dwarikanath MahapatraImage Captioning VLM Adaptation
Department of Computer Science, University of Gujrat, Pakistan · Department of Computer Science, Khalifa University, Abu Dhabi, UAE
GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning
Sep 20, 2026Zechang Xiong, Da Li, Rong Yin +7Modality Imbalance Geometric Representation Learning
Beihang University, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Amap, Alibaba Group, Beijing, China +3
The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods
Sep 10, 2026Ioanna Kaffeza, Efthymios Georgiou, Alexandros PotamianosModality Imbalance Multimodal Optimization
PERSEE Center, Mines Paris-PSL University, Sophia-Antipolis, France · National Technical University of Athens, Athens, Greece · University of Bern, Bern, Switzerland +2
MM-Spectrum: Multimodal Multi-spectral Molecular Structural Elucidation with a Stable MoE Framework
Aug 27, 2026Hai-tao Yu, Nan Min, Zheng Fang +4Multimodal Fusion Modality Imbalance
The Hong Kong University of Science and Technology (Guangzhou) · Southeast University · The Hong Kong University of Science and Technology +1
CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery
Aug 24, 2026Mahir Shahriar Tamim, Sharjil Khan, Md. Samiul Alim +3Multimodal Understanding Cross-Modal Learning
Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh
C $^3$ PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
Aug 5, 2026Swapnanil Mukherjee, Agyeya Negi, Tanuja Ganu +1Modality Imbalance Multimodal Reasoning
Microsoft Research India · IIIT Hyderabad, India
OPD-V: Visual On-Policy Self-Distillation with Modality Balance
Aug 5, 2026Aniri, Jinhe Bi, Peng Liao +5Modality Imbalance On-Policy Self-Distillation
2Ludwig Maximilian University of Munich · 3Munich Center for Machine Learning · 1National University of Singapore +1
Beyond Modality Fusion: Deep Ensembles for Multimodal Classification
Jul 6, 2026Ilya Burenko, Dmitry VetrovModality Imbalance Multimodal Classification
ScaDS.AI, Technische Universität Dresden · Dresden, Germany · Constructor University +1
Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration
Jun 15, 2026Xiwen Wei, Mark Nutter, Madhusudhanan Srinivasan +1Unified Multimodal Models Modality Imbalance
The University of Texas at Austin · Advanced Micro Devices, Inc.
Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
Jun 3, 2026Alessandro Gambetti, Qiwei Han, Cláudia Soares +1VLM Evaluation Modality Imbalance
NOVA School of Science and Technology · Lisbon, Portugal · Nova School of Business and Economics +2
A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis
May 27, 2026Jianheng Dai, Jiazhang Liang, Sijie MaiModality Imbalance Gradient Interference
School of Computer Science, South China Normal University, Guangzhou, Guangdong, China
Balancing Multimodal Learning through Label Space Reshaping
May 22, 2026Xiaoyu Ma, Weijie Zhang, Yuanhao Gao +3Cross-Modal Learning Cross-Modal Representation Learning
PREPRINT 2026.05
Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
May 22, 2026Liangwei Nathan Zheng, Wenhao Liang, Wei Emma Zhang +2Efficient Multimodal Inference Cross-Modal Representation Learning
Adelaide University
Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition
May 20, 2026Phuong-Anh Nguyen, The-Son Le, Duc-Trong Le +1Modality Imbalance Emotion Recognition in Conversations
Faculty of Information Technology, VNU University of Engineering and Technology, 100000, Hanoi, Vietnam.
Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration
May 5, 2026Xun Jiang, Yufan Gu, Disen Hu +5Multimodal Robustness Modality Imbalance
School of Computer Science and Technology, Tongji University · School of Computer Science and Engineering, University of Electronic Science and Technology of China · Independent Researcher +1
Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization
Apr 17, 2026Habibeh Naderi, Behrouz Haji Soleimani, Stan MatwinCross-Modal Learning Modality Imbalance
Dalhousie University, Halifax NS, Canada
Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Apr 14, 2026Kang He, Yuzhe Ding, Xinrong Wang +3Multimodal Sentiment Analysis Modality Imbalance
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute
PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization
Apr 7, 2026Shicai Wei, Chunbo Luo, Qiang Zhu +1Multimodal Learning Multimodal Optimization
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China. · School of Information and Communication Engineering, University of Electronic Science and Technology of China, Sichuan, China. · Peng Cheng Laboratory, Shenzhen, China.
Mind the Gap: A Framework for Assessing Pitfalls in Multimodal Active Learning
Mar 31, 2026Dustin Eisenhardt, Yunhee Jeong, Florian BuettnerActive Learning Modality Imbalance
German Cancer Research Center (DKFZ), Heidelberg, Germany · German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany · Institute of Informatics, Goethe University Frankfurt, Frankfurt am Main, Germany +3
CCF: Complementary Collaborative Fusion for Domain Generalized Multi-Modal 3D Object Detection
Mar 24, 2026Yuchen Wu, Kun Wang, Yining Pan +13D Object Detection Multimodal Object Detection
Singapore University of Technology and Design
BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates
Mar 20, 2026Phuong-Anh Nguyen, Tien Anh Pham, Duc-Trong Le +1Modality Imbalance Multimodal Learning
VNU University of Engineering and Technology, Hanoi, Vietnam
Unbiased Dynamic Multimodal Fusion
Mar 20, 2026Shicai Wei, Kaijie Zhang, Luyi Chen +2Multimodal Fusion Uncertainty-Aware Multimodal Fusion
University of Electronic Science and Technology of China
Parameter-Efficient Modality-Balanced Symmetric Fusion for Multimodal Remote Sensing Semantic Segmentation
Mar 18, 2026Haocheng Li, Juepeng Zheng, Shuangxi Miao +4Remote Sensing Image Segmentation Multimodal Fusion
the College of Land Science and Technology, China Agricultural University, and also with the Key Laboratory of Remote Sensing for Agri-Hazards, Ministry of Agriculture and Rural Affairs, Beijing 100083, China · the School of Artificial Intelligence, Sun Yat-Sen University, Zhuhai, China · Henan Polytechnic University, and also with the Key Laboratory of Spatio-Temporal Information and Ecological Restoration of Mines, Ministry of Natural Resources of the People’s Republic of China, Jiaozuo 454003, China +4
Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition
Mar 12, 2026Umberto Cappellazzo, Stavros Petridis, Maja PanticAudio-Visual Speech Recognition Modality Imbalance
Imperial College London, UK · NatWest AI Research, UK
MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities
Mar 10, 2026Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le +1Multimodal Model Evaluation Missing-Modality Learning
VNU University of Engineering and Technology, Hanoi, Vietnam

Nov 9, 2026J/K move · Enter open · S save

Layer Roles Matter: Mitigating Modality Imbalance in Multimodal Emotion Reasoning via Adaptive Layer Profiling

Pengyu Huang, Peipei Song, Beier Zhu, Zhiyuan Han+2

University of Science and Technology of China, Hefei, China · Hefei University of Technology, Hefei, China

Modality Imbalance Multimodal Reasoning Multimodal Emotion Recognition MLLM Reasoning Multimodal Learning

Open paper AskPDF arXiv

Multimodal Emotion Reasoning (MER) requires perceiving and integrating emotion cues from multiple modalities. Recent advances increasingly leverage Multimodal Large Language Models (MLLMs) to enhance reasoning capacity. However, due to the inherent heterogeneity in conveying emotions across modalities, these models often suffer from modality imbalance and confront flawed emotion understanding. In this paper, we conduct an in-depth analysis on MER models and uncover two key empirical insights. (i) In MER, modality imbalance manifests two distinct failure modes: unimodal over-reliance and insufficient cross-modal utilization, which call for different intervention strategies. (ii) MLLM layers exhibit implicit functional hierarchies in multimodal emotion processing, playing separable roles in stabilizing unimodal representations and integrating cross-modal information. Motivated by these observations, we propose a Layer-Aware Modality Rebalancing (LAMR) framework by explicitly aligning intervention strategies with layer roles. Specifically, we first introduce two quantized metrics for adaptive layer role profiling. In each layer, Modality Entropy (ME) measures the dispersion of modality contributions, and Cross-modal Intensity (CMI) captures the strength of cross-modal interactions. Then, we respectively enhance modality equilibrium in stabilization layers and cross-modal propagation in integration layers. Extensive experiments on two comprehensive MER benchmarks demonstrate that LAMR effectively mitigates modality imbalance and improves emotion reasoning across diverse MLLM backbones, without additional training.