Multimodal Learning

Follow this topic Ask about this topic

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

338 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

Learning to Feel Materials from Multisensory Tactile Data via Interpretable Models
May 28, 2026Li Zou, Yasemin VardarTactile Sensing Multimodal Learning
Delft University of Technology (TU Delft), Department of Cognitive Robotics, Delft, 2628 CD, The Netherlands
TRACER: Persistent Regularization for Robust Multimodal Finetuning
May 28, 2026Hesam Asadollahzadeh, Feng Liu, Christopher Leckie +1VLM Robustness Neural Network Robustness
School of Computing and Information Systems (CIS), Faculty of Engineering and IT (FEIT), University of Melbourne, Australia.
IPO-Mine: A Toolkit and Dataset for Section-Structured Analysis of Long, Multimodal IPO Documents
May 27, 2026Michael Galarnyk, Siddharth Lohani, Vidhyakshaya Kannan +8Document Information Extraction Multimodal Model Evaluation
Georgia Institute of Technology Atlanta, GA, USA · Sai University Chennai, Tamil Nadu, India · Duke University Durham, NC, USA
A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis
May 27, 2026Jianheng Dai, Jiazhang Liang, Sijie MaiModality Imbalance Gradient Interference
School of Computer Science, South China Normal University, Guangzhou, Guangdong, China
MeniOmni: A Structured Multimodal Benchmark for Holistic Meniscus Injury Assessment
May 27, 2026Shurui Xu, Siqi Yang, Weiping Ding +4Radiology Report Generation Medical Image Benchmarks
School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Belfast, UK · Radiology Department, Affiliated Nantong Clinical College of Nantong University, Nantong First People’s Hospital, School of Clinical Medicine, Nantong University, Nantong, Jiangsu, China · School of Artificial Intelligence and Computer Science, Nantong University, Nantong, China +5
Bounded-Compute Multimodal Regression for Product-Rating Prediction
May 26, 2026William Leach, Ru He, Sizhuo Ma +4Vision-Language Models VLM Adaptation
Snap Inc.
OnceSelect: Reusable Data Selection for Efficient Multimodal Instruction Tuning
May 26, 2026Mingkang Dong, Muxin Pu, Hongyi Cai +5Instruction-Tuning Data Selection Multimodal Large Language Models
Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China
Rethinking Gradient Modulation in Multimodal Regression
May 26, 2026Haojie Yin, Chengcheng Feng, Tianyi Liu +2Multimodal Regression Multimodal Learning
Duke Kunshan University, China · Xi’an Jiaotong-Liverpool University, China · Soochow University
Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
May 25, 2026Bonan Ding, Umair Nawaz, Ufaq Khan +5Multimodal Understanding Cross-Modal Learning
Mohamed bin Zayed University of Artificial Intelligence · Tianjin University · Chongqing University +1
MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models
May 25, 2026Shristi Das Biswas, Kaushik RoyVision-Language Models Coreset Selection
Purdue University
VEN-VL: A Visual Ensemble MoE Framework for Effective and Efficient Multi-Modal Understanding
May 25, 2026Yinghao Wu, Zhuoyan Luo, Yiyao Yu +3Efficient Multimodal Inference Mixture-of-Experts Models
Tsinghua University
DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement
May 25, 2026Renjie Lu, Xulong Zhang, Xiaoyang Qu +2Multimodal Disentangled Representation Learning Unified Multimodal Models
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · University of Science and Technology of China, Hefei, China.
Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs
May 23, 2026Hao Yan, Xuanru Wang, Jun Yin +3Graph Neural Networks Multimodal Graph Learning
School of Computer Science and Engineering, Central South University, Changsha, China · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China · School of Information and Communication Technology, Griffith University, Brisbane, Australia
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
May 23, 2026Juan Ignacio Bustos Gorostegui, Maria Elena BuemiEgocentric Video Understanding Multimodal Fusion
Univ. of Buenos Aires. Faculty of Exact and Natural Sciences. Dept. of Computer Science (DC). · CONICET-Univ. of Buenos Aires. Institute of Computer Sciences (ICC). Argentina.
Balancing Multimodal Learning through Label Space Reshaping
May 22, 2026Xiaoyu Ma, Weijie Zhang, Yuanhao Gao +3Cross-Modal Learning Cross-Modal Representation Learning
PREPRINT 2026.05
Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
May 22, 2026Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel +2Efficient Multimodal Inference Cross-Modal Representation Learning
Adelaide University
Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models
May 21, 2026Yuting He, Chenyu You, Shuo LiMedical Imaging Foundation Models Multimodal Pretraining
Case Western Reserve University · Stony Brook University
Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
May 20, 2026Herman Bergström, Aditya Mehrotra, Rahul G. KrishnanMultimodal Embedding Multimodal Classification
Chalmers University of Technology and University of Gothenburg · Vector Institute · University of Toronto
TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
May 20, 2026Andong Hua, Colton Bishop, Igor Mordatch +5Multimodal Large Language Models Modality Gap in VLMs
University of California, Santa Barbara · Google DeepMind
Identifiable Multimodal Causal Representation Learning under Partial Latent Sharing
May 18, 2026Manal Benhamza, Marianne Clausel, Myriam TamiLatent Variable Models Causal Representation Learning
Paris-Saclay University, CentraleSupélec, MICS Lab, France · Lorraine University, CRAN, France
CLIC: Contextual Language-Informed Cardiac Pathology Classification
May 18, 2026Giovani D. Lucafo, Rafael da Costa Silva, João Lucas Luz Lima Sarcinelli +2Electrocardiogram Classification Multimodal Learning
Institute of Mathematical and Computer Sciences Universidade de S˜ao Paulo S˜ao Carlos, S˜ao Paulo, Brazil
Lance: Unified Multimodal Modeling by Multi-Task Synergy
May 18, 2026Fengyi Fu, Mengqi Huang, Shaojin Wu +10Unified Multimodal Models Multi-Task Learning
Intelligent Creation Lab, ByteDance
Weakly Supervised Cross-Modal Learning for 4D Radar Scene Flow Estimation
May 18, 2026Jingyun Fu, Zhiyu Xiang, Na ZhaoCross-Modal Learning Autonomous Driving Perception
Zhejiang University, Hangzhou, Zhejiang, China · Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing, Zhejiang, China · Singapore University of Technology and Design, Singapore.
RAVE: Re-Allocating Visual Attention in Large Multimodal Models
May 18, 2026Xi Leng, Xinhong Ma, Ziqiang Dong +4Visual Attention Self-Attention
Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen · Beijing Institute of Technology
CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook
May 18, 2026Zeyu Chen, Jie Li, Kai HanCross-Modal Learning Multimodal Disentangled Representation Learning
Visual AI Lab, The University of Hong Kong
SkyNative: A Native Multimodal Framework for Remote Sensing Visual Evidence Reasoning
May 18, 2026Xiao Yang, Ronghao Fu, Zhiwen Lin +10VLM Evaluation VLM Robustness
College of Computer Science and Technology, Jilin University, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education
Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
May 16, 2026Firoj Alam, Shammur Absar Chowdhury, Enamul Hoque PrinceCross-Modal Alignment Multimodal Large Language Models
Qatar Computing Research Institute, HBKU, Qatar · York University, Canada
Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning
May 16, 2026Wen-Hsin Tsai, Chia-Ming Lee, Yuk-Ying TungCross-Modal Learning Multimodal Robustness
Institute of Education, National Cheng Kung University · Institute of Intelligent System, National Yang Ming Chiao Tung University · Department of Computer Science, University at Albany, State University of New York
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
May 15, 2026Yishun Lu, Wes ArmourSecond-Order Optimization Multimodal Optimization
University of Oxford, Oxford, United Kingdom

May 28, 2026J/K move · Enter open · S save

Learning to Feel Materials from Multisensory Tactile Data via Interpretable Models

Li Zou, Yasemin Vardar

Delft University of Technology (TU Delft), Department of Cognitive Robotics, Delft, 2628 CD, The Netherlands

Tactile Sensing Multimodal Learning

Open paper AskPDF arXiv

Human tactile perception of materials relies on complex multisensory touch cues, yet the relationship between low-level tactile signals and perceptual representations remains poorly understood. This knowledge gap hinders the integration of touch in digital environments and the development of robots capable of human-like tactile perception. Here, we present an interpretable computational framework for modeling human material perception and recognition using multisensory touch data. Our framework comprises three interconnected models: Model 1 maps finger-surface interaction features to psychophysical sensory attributes, Model 2 classifies materials based on these perceptual representations, and Model 3 directly classifies materials from tactile features. The results showed that combining information from pressing, static contact, and sliding interactions improves prediction accuracy, and that thermal cues are particularly informative for both perceptual modeling and material classification. These findings highlight the importance of thermal and compliance cues, which remain underrepresented in current robotic fingers and haptic displays. Incorporating such cues may enhance artificial systems' ability to approximate human material perception and guide the design of more perceptually grounded haptic interfaces.