Multimodal Optimization

Follow this topic Ask about this topic

The field covers both optimization of models that combine modalities such as text, images, audio, and video, and search over objectives with many distinct optima. For multimodal learning, methods tune data mixtures and model components, balance or prioritize modality-specific gradients, and use gating, distillation, or adaptive optimization; evaluation commonly spans multimodal understanding, generation, and domain-generalization benchmarks. For multimodal objective landscapes, population-based methods such as CMA-ES, Stein-based search, and learned proposal models aim to find high-quality solutions while preserving coverage of different peaks. A central tension is how to balance convergence speed and performance against robustness and diversity: equal treatment of modalities or solutions is not always best, but aggressive prioritization can neglect useful signals or optima.

Latest papers

32 papers2 papers in the last four weeks, down 75% on the four weeks before. 0.0% of all new papers.

MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information
Oct 17, 2026Xinyu Chen, Liangjian Wen, Jiang Duan +7Multimodal Fusion Modality Reliance in Multimodal Models
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · X-Humanoid +4
Balancing Multimodal Learning via Functional Progress
Oct 2, 2026Zhongjing Gu, Fengqiang Wan, Yiming Cui +2Modality Imbalance Multimodal Optimization
Contact-Implicit Stein Projected ADMM for Discovery of Diverse Contact-Rich Manipulation Strategies
Sep 23, 2026Hrishikesh Sathyanarayan, Christian Hughes, Ian AbrahamContact-Rich Robotic Manipulation Trajectory Optimization
Department of Mechanical Engineering, Yale University, New Haven, CT, USA. · Department of Electrical Engineering, University of Sydney, NSW, Australia.
Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification
Sep 23, 2026Longfei Huang, Xiangyu Wu, Yang YangModality Imbalance Multimodal Optimization
Nanjing University of Science and Technology · Alibaba Group
VCMM: Variance-Calibrated Momentum for Multimodal Learning
Sep 23, 2026Zhongjing Gu, Chenyang Huang, Yufa Feng +3Multimodal Optimization Adaptive Gradient Methods
GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning
Sep 20, 2026Zechang Xiong, Da Li, Rong Yin +7Modality Imbalance Geometric Representation Learning
Beihang University, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Amap, Alibaba Group, Beijing, China +3
From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning
Sep 16, 2026Pan Wang, Siwei Song, Hui Ji +12Efficient Multimodal Inference Multimodal Optimization
University of Pittsburgh · New York University · Carnegie Mellon University +6
PACE: Progressive Angular-to-Norm Contrastive Embedding
Sep 14, 2026Yanping Li, Wei Zhou, Yawen Liu +7Cross-Modal Representation Learning Multimodal Embedding
The Hong Kong University of Science and Technology · Alibaba Group · Huazhong University of Science and Technology +1
Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging
Sep 11, 2026Pengyang Zhou, Xiaobin Tu, Zhengxi Liu +7Multimodal Optimization Language Model Merging
Alibaba Cloud
The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods
Sep 10, 2026Ioanna Kaffeza, Efthymios Georgiou, Alexandros PotamianosModality Imbalance Multimodal Optimization
PERSEE Center, Mines Paris-PSL University, Sophia-Antipolis, France · National Technical University of Athens, Athens, Greece · University of Bern, Bern, Switzerland +2
CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery
Aug 24, 2026Mahir Shahriar Tamim, Sharjil Khan, Md. Samiul Alim +3Multimodal Understanding Cross-Modal Learning
Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh
Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence
Jul 23, 2026Jay Gor, Karm Dave, Akshita Abrol +3Efficient Multimodal Inference VLM Quantization
Nirma University · Singapore Institute of Technology, Singapore · Department of CE-AI and Big Data, Marwadi University
OPOD: On-Policy Omni Distillation
Jul 23, 2026Tong Zhao, Yuyang Hu, Yutao Zhu +5Cross-Modal Knowledge Distillation Unified Multimodal Models
1Renmin University of China · 2Tencent
S-CARD-CMSA: A Score-Aware Candidate Archive with Density-Filtered Reporting for Multimodal Optimization
Jul 15, 2026Dikshit ChauhanEvolutionary Optimization Multimodal Optimization
Department of Electrical and Computer Engineering, National University of Singapore
Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process
Jul 4, 2026Zican Hu, Xuyang Hu, Yiming Liu +10Unified Multimodal Models Multimodal Reasoning
Nanjing University · Shanghai AI Laboratory · Tsinghua University +5
\chisao{}: A GPU-Native Parallel Optimizer for Multimodal Black-Box Functions via Convergence-Anticonvergence Oscillation
Jun 24, 2026Ira WolfsonGPU Acceleration Black-Box Optimization
Department of Electronics and Electrical Engineering Braude College of Engineering, Karmiel, Israel
MSC-CMA-ES: Structure-Aware Restarts for CMA-ES via Cyclic Nearest-Better Basin Discovery
Jun 14, 2026Dimitar Nedanovski, Svetoslav Nenov, Dimitar PilevEvolutionary Optimization Covariance Matrix Adaptation Evolution Strategy
Faculty of Mathematics and Informatics Sofia University St. Kliment Ohridski James Baucher Blvd., 1164 Sofia, Bulgaria · Department of Mathematics University of Chemical Technology and Metallurgy Sofia, Bulgaria · Department of Informatics University of Chemical Technology and Metallurgy Sofia, Bulgaria
GLENS: Global Search via Learning from Solver Iterates with Diffusion Models
May 29, 2026Anjian Li, Bartolomeo Stellato, Ryne BeesonLearning to Optimize Diffusion Sampling
Department of Electrical and Computer Engineering, Princeton University, Princeton, 08544, NJ, USA. · Department of Operations Research and Financial Engineering, Princeton University, Princeton, 08544, NJ, USA. · Department of Mechanical and Aerospace Engineering, Princeton University, Princeton, 08544, NJ, USA.
DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement
May 25, 2026Renjie Lu, Xulong Zhang, Xiaoyang Qu +2Multimodal Disentangled Representation Learning Unified Multimodal Models
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · University of Science and Technology of China, Hefei, China.
Mapping the Fitness Landscape: A Structure-Guided Approach to Multi-Modal Optimization
May 18, 2026Meng Xiang, Pei YanFitness Landscape Analysis Evolutionary Optimization
University of Aizu Aizuwakamatsu, Japan
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
May 15, 2026Yishun Lu, Wes ArmourSecond-Order Optimization Multimodal Optimization
University of Oxford, Oxford, United Kingdom
MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling
May 12, 2026Zhong Li, Qi Huang, Yuxuan Zhu +6Multimodal Optimization Synthetic Benchmark Generation
Great Bay University · Leiden University · Rensselaer Polytechnic Institute +1
Stein Variational Black-Box Combinatorial Optimization
Apr 17, 2026Thomas Landais, Olivier Goudet, Adrien Goëffon +2Stochastic Optimization Black-Box Optimization
LERIA, Université d’Angers, 2 Boulevard Lavoisier, Angers 49045, France
PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization
Apr 7, 2026Shicai Wei, Chunbo Luo, Qiang Zhu +1Multimodal Learning Multimodal Optimization
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China. · School of Information and Communication Engineering, University of Electronic Science and Technology of China, Sichuan, China. · Peng Cheng Laboratory, Shenzhen, China.
MixAtlas: Uncertainty-aware Data Mixture Optimization for Multimodal LLM Midtraining
Apr 3, 2026Bingbing Wen, Sirajul Salekin, Feiyang Kang +4Data Mixture Optimization Multimodal Large Language Models
University of Washington · Work done during an internship at Apple · Apple +1
Joint Segmentation and Grading with Iterative Optimization for Multimodal Glaucoma Diagnosis
Mar 15, 2026Zhiwei Wang, Yuxing Li, Meilu Zhu +2Optic Disc Segmentation Glaucoma Detection
Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China · College of Information Engineering, Zhejiang University of Technology, Hangzhou, China
Balancing Multimodal Domain Generalization via Gradient Modulation and Projection
Mar 15, 2026Hongzhao Li, Guohao Shen, Shupan Li +2Multimodal Domain Generalization Domain Generalization
School of Computer and Artificial Intelligence, Zhengzhou University · Engineering Research Center of Intelligent Swarm Systems, Ministry of Education · National Supercomputing Center in Zhengzhou +2
Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning
Mar 12, 2026Rujie Wu, Haozhe Zhao, Hai Ci +1Multimodal Optimization Instruction-Tuning Data Selection
School of Computer Science, Peking University, Beijing, China · Institute for Artificial Intelligence, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China +2
MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search
Feb 28, 2026Minkyoung Cho, Insu Jang, Shuowei Jin +5Multimodal Large Language Models Multimodal Optimization
University of Michigan · NVIDIA
ModalImmune: Immunity Driven Unlearning via Self Destructive Training
Feb 18, 2026Rong Fu, WeiZhi Tang, Ziming Wang +6Multimodal Robustness Missing-Modality Learning
University of Macau · Zhejiang University · Renmin University of China +3

Oct 17, 2026J/K move · Enter open · S save

MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information

Xinyu Chen, Liangjian Wen, Jiang Duan, Dongkai Wang+6

School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · X-Humanoid+4

Multimodal Fusion Modality Reliance in Multimodal Models Multimodal Optimization

Open paper AskPDF arXiv

In multimodal learning, modality imbalance often lets one modality dominate optimization and keeps the model from fully exploiting cross-modal interactions. Balance-oriented methods such as the Multimodal Competition Regularizer (MCR) perform well by alleviating this imbalance during training. These dynamic balancing methods, however, mainly adjust modality contributions and do not consider how imbalance affects the formation of synergistic information during fusion. Synergy is a major source of multimodal advantage, because it is predictive gain that cannot be recovered from unimodal evidence alone. We propose the Multimodal Synergistic Competition Regularizer (MSCR), a fusion method that jointly promotes balanced modality utilization and the discovery of synergistic information. MSCR has two complementary objectives. An anti-dominance objective suppresses excessive unimodal influence and leaves room for synergistic information, and a synergy objective encourages the fused predictor to outperform a non-synergistic unimodal reference. An adaptive gate further concentrates the regularization on samples that need stronger collaborative modeling. Experiments on five benchmarks show that MSCR consistently improves performance over MCR and strengthens synergistic capability. It raises conditional recovery, the accuracy of the fused prediction on samples that the unimodal predictors misclassify, as well as the share of the model's output that InterSHAP attributes to cross-modal interactions. Its gain in conditional recovery persists when the fusion module is Transformer-based. Code is available at https://github.com/chenxinyu991-collab/MSCR.