Modalities

Follow this topic Ask about this topic

Momentum

33 papers in the last four weeks, up 154% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 217

All topics

RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities
Jun 3, 2026Vasiliki Rizou, Pascal Frossard, Dorina ThanouMultimodal Representations Representation Learning
EPFL
Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
Jun 3, 2026Alessandro Gambetti, Qiwei Han, Cláudia Soares +1Medical Vision-Language Models Vision-Language Alignment
NOVA School of Science and Technology · Lisbon, Portugal · Nova School of Business and Economics +2
An Empirical Study of Data Scale, Model Complexity, and Input Modalities in Visual Generalization
Jun 3, 2026Yidi ZhouluoStructural Complexity Modalities
School of Medical Information and Artificial Intelligence, Shandong First Medical University, Taian 271000, Shandong, China
scTranslation: A Comprehensive Benchmark for Single-Cell Multi-Omics Modality Translation
Jun 2, 2026Jiabei Cheng, Jingbo Zhou, Jun Xia +4Modalities
Westlake University Hangzhou, Zhejiang, China · Shanghai Jiao Tong University Shanghai, China · Zhejiang University Hangzhou, Zhejiang, China +3
Visual Instruction Tuning Aligns Modalities through Abstraction
Jun 2, 2026Luis Palacios, Lorenzo Basile, Diego Doimo +1Multimodal Continual Instruction Tuning Instruction Tuning
Area Science Park, Trieste, Italy
Any2Poster: Any-Source Poster Generation Across Modalities and Domains
Jun 1, 2026Amogh Vinaykumar, Aiden Li, Suozhi Huang +1Paperpilot Modalities
Flower Mound High School · University College London · Princeton University
Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals
Jun 1, 2026Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang +2Multimodal Fusion Multimodal Representations
Adelaide University Adelaide, Australia · Shandong University Jinan, China
When Tabular Foundation Models Transfer Across Modalities: A Systematic Evaluation Across 95 Datasets, 7 Modalities, and Two Regimes
Jun 1, 2026Julien LafranceTabular Foundation Models Multimodal Classification
Télécom Paris, Institut Polytechnique de Paris · Palaiseau, France
Boosting Multimodal Federated Learning via Chained Modality Optimization
Jun 1, 2026Zixin Zhang, Fan Qi, Shuai Li +2Fedavg Modalities
College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, China · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Large Language Models Modalities
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
Wavelet-Fusion Diffusion Model for Multimodal Brain MRI Synthesis with Modality and Metadata Conditioning
May 30, 2026Muhammad Nabi Yasinzai, Remika Mito, Mangor PedersenMedical Image Generation Multimodal Diffusion Models
Department of Psychology & Neuroscience, Auckland University of Technology, Auckland, New Zealand. · Department of Psychiatry, University of Melbourne, Melbourne, Victoria 3053, Australia. · Auckland University of Technology, Auckland, New Zealand.
Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models
May 29, 2026Zijie Zhou, Dandan Zhu, Hangxiangpan Wang +3Recent Vision-Language Models Cross-Modal
China University of Petroleum (Beijing) · Hainan Institute of China University of Petroleum (Beijing) · South China Normal University
AMix-2: Establishing Protein as a Native Modality in Large Language Models
May 29, 2026Keyue Qiu, Yixin Wu, Lihao Wang +19Protein Language Modeling
1Shanghai Artificial Intelligence Laboratory · Institute for AI Industry Research (AIR), Tsinghua University
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
May 28, 2026Feng Han, Zhixiong Zhang, Zheming Liang +2Recent Vision-Language Models Multimodal Benchmarks
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
May 25, 2026Bonan Ding, Umair Nawaz, Ufaq Khan +5Fine-Grained Video Understanding Cross-Modal Attention
Mohamed bin Zayed University of Artificial Intelligence · Tianjin University · Chongqing University +1
Zero-Shot Parkinson's Disease Detection from Speech: Comparing Large Audio and Language Models
May 24, 2026Muhammad Ashad Kabir, Sirajam MuniraLarge Audio Language Models Acoustic
School of Computing, Mathematics and Engineering, Charles Sturt University, NSW, Australia · Department of Computer Science, Rensselaer Polytechnic Institute, NY, USA
Balancing Multimodal Learning through Label Space Reshaping
May 22, 2026Xiaoyu Ma, Weijie Zhang, Yuanhao Gao +3Multimodal Learning Cross-Modal
PREPRINT 2026.05
Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
May 22, 2026Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel +2Multimodal Learning Mixture-Of-Experts
Adelaide University
Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs
May 22, 2026Casey Ford, Madison Van Doren, Sicheng Jin +1Large Language Model Jailbreaks Multilingual Safety
Appen
Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models
May 21, 2026Yuting He, Chenyu You, Shuo LiModalities Self-Supervised Learning
Case Western Reserve University · Stony Brook University
Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Multimodal Large Language Models Modalities
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling
May 19, 2026Zhiping Yu, Chenyang Liu, Jinqi Cao +4Remote Sensing Trustworthy Earth Observation Decisions
Department of Aerospace Intelligent Science and Tech-nology, School of Astronautics, Beihang University, Beijing 100191, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China · Shenyuan Honors College, Beihang University, Beijing 100191, China
Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
May 19, 2026Yi Zhong, Haotong Qin, Xindong Zhang +2Cross-Modal Modalities
VCIP, College of Computer Science, Nankai University · D-ITET, ETH Zürich · OPPO Research Institute +1
Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks
May 18, 2026Jue Jiang, Harini VeeraraghavanSwin Amodal Segmentation
Department of Medical Physics, Memorial Sloan Kettering Cancer Center
Modality vs. Morphology: A Framework for Time Series Classification for Biological Signals
May 18, 2026Jordan Tschida, Matthew Yohe, Edward Kane +10Physiological Signals Time-Series Classification
Oak Ridge National Laboratory, Oak Ridge, Tennessee
TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval
May 18, 2026Xinyu Sun, Huangyu Dai, Lingtao Mao +5Multimodal Retrieval Image-Text Pairs
Kuaishou Technology · Beijing, China
SynGR: Unleashing the Potential of Cross-Modal Synergy for Generative Recommendation
May 18, 2026Wei Chen, Xingyu Guo, Shuang Li +6Multimodal Recommendation Model Cross-Modal
School of Artificial Intelligence, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China
WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform
May 18, 2026Yu Shang, Yinzhou Tang, Yiding Ma +22Embodied Artificial Intelligence World Models
Tsinghua University · Shanghai Jiao Tong University · Zhejiang University +7
Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech
May 17, 2026Kaavya Chaparala, Thomas Thebaud, Jesús Villalba López +3Transcript Summarization
Johns Hopkins University 3400 N. Charles St., Baltimore, MD 21218
Medical Context Distorts Decisions in Clinical Vision Language Models
May 17, 2026David Restrepo, Ira Ktena, Maria Vakalopoulou +2Medical Vision-Language Models Multimodal Clinical Data
1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK +1
EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers
May 16, 2026Zongyuan Yang, Mingjing Yi, Wanli Ma +83D Generative Models Multimodal Large Language Models
SeeleAI Team
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
May 15, 2026Yishun Lu, Wes ArmourMultimodal Alignment Multimodal Model
University of Oxford, Oxford, United Kingdom
MuteBench: Modality Unavailability Tolerance Evaluation for Incomplete Multimodal Fusion
May 13, 2026Wugeng Zheng, Ziwen Kan, Tianlong Chen +2Missing Modalities Multimodal Fusion
University of Central Florida · University of North Carolina at Chapel Hill
Towards Robotic Dexterous Hand Intelligence: A Survey
May 13, 2026Weiguang Zhao, Tian Liang, Xihao Guo +3Robotic Hands Force
University of Liverpool · Xi’an Jiaotong-Liverpool University · Duke Kunshan University +1
WD-FQDet: Multispectral Detection Transformer via Wavelet Decomposition and Frequency-aware Query Learning
May 13, 2026Chunjin Yang, Xiwei Zhang, Yiming Xiao +1Infrared Small Target Detection Feature Pyramid
University of Electronic Science and Technology of China,Chengdu, Sichuan 611731, China
PRA-PoE: Robust Multimodal Alzheimer's Diagnosis with Arbitrary Missing Modalities
May 13, 2026Guangqian Yang, Ye Du, Wenlong Hou +2Missing Modalities Multimodal Learning
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Technology Management for Innovation, The University of Tokyo,2026 Japan · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China
BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability
May 13, 2026Guangqian Yang, Tong Ding, Wenlong Hou +4Multimodal Neuroimaging Multimodal Pretraining
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Technology Management for Innovation, The University of Tokyo, Japan · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China
SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning
May 13, 2026Truong Pham, Anay Majee, Rishabh IyerMultimodal Alignment Modality Alignment
The University of Texas at Dallas
Multitask Multimodal Fusion with Tabular Foundation Models for Peak and Durability Prediction of Pertussis Booster Response
May 13, 2026Divya SitaniInoculation Prompting Multimodal Fusion
Berlin, Germany
Towards Robust Federated Multimodal Graph Learning under Modality Heterogeneity
May 12, 2026Sirui Zhang, Haonan Wang, Xunkai Li +5Federated Graph Learning Graph Foundation Model
Beijing Institute of Technology
Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
May 12, 2026Souptik Sen, Raneen Younis, Zahra AhmadiCross-Modal Alignment Cross-Modal
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
Resilient Vision-Tabular Multimodal Learning under Modality Missingness
May 12, 2026Camillo Maria Caruso, Valerio Guarrasi, Paolo SodaMissing Modalities Multimodal Learning
Research Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Roma, Italy, Europe · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden, Europe
CMKL: Modality-Aware Continual Learning for Evolving Biomedical Knowledge Graphs
May 11, 2026Yousef A. Radwan, Yao Li, Qing Qing +5Knowledge Graph Embeddings Biomedical Knowledge Graph
Technology, Innovation, Entrepreneurship Department King Abdullah University of Science and Technology · School of Computing and Information Systems The University of Melbourne · College of Computer Science and Technology Jilin University +2
Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent
May 10, 2026Yihong Tang, Kehai Chen, Xuefeng Bai +1Multimodal Agents Modalities
Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute (SLAI), Shenzhen, China
From pre-training to downstream performance: Does domain-specific pre-training make sense?
May 9, 2026Felix KronesMultimodal Pretraining Pretraining
Oxford Internet Institute, University of Oxford, Oxford, UK
Encoder-Decoder Transformers: Logical Characterizations and Periodicity
May 8, 2026Veeti Ahvonen, Damian Heiman, Antti Kuusisto +2Transformer Encoder Transformer Architectures
Mathematics Research Centre, Tampere University, Tampere, Finland
PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition
May 8, 2026Yuchen He, Jing ZhangVideo Object Segmentation Modality-Specific Encoders
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China
Data Language Models: A New Foundation Model Class for Tabular Data
May 7, 2026Eda Erol, Giuliano Pezzoli, Ozer Cem KelahmetTabular Learning Tabular Data
SchemaLabs
LARGO: Low-Rank Hypernetwork for Handling Missing Modalities
May 7, 2026Niels Vyncke, Pooya Ashtari, Aleksandra PižuricaMissing Modalities Modalities
Department of Telecommunications and Information Processing Ghent University Ghent, Belgium
To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition
May 6, 2026Yangchen Yu, Qian Chen, Jia Li +5Emotion Recognition Cross-Modal
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computing and Information Systems, Singapore Management University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore +1
Information Coordination as a Bridge: A Neuro-Symbolic Architecture for Reliable Autonomous Driving Scene Understanding
May 6, 2026Shuo Liu, Lei Shi, Haowen Liu +3Autonomous Driving Collaborative Perception
School of Computer and Artificial Intelligence, Zhengzhou University, 450001, Zhengzhou, China · School of Cyber Science and Engineering, Zhengzhou University, 450001, Zhengzhou, China · Songshan Laboratory, 450003, Zhengzhou, China
Triple Spectral Fusion for Sensor-based Human Activity Recognition
May 4, 2026Ye Zhang, Longguang Wang, Qing Gao +3Human Activity Recognition Two-Level Spatial-Frequency Fusion Strategy
School of Electronics and Communication Engineering, the Shenzhen Campus of Sun Yat-sen University, Sun Yat-sen University, Shenzhen 518107, China · Aviation University of Air Force, Changchun 130012, China · College of Computer Science, Chongqing University, Chongqing 400044, China +1
Structured Diffusion Bridges: Inductive Bias for Denoising Diffusion Bridges
May 3, 2026Eitan Kosman, Gabriele Serussi, Chaim BaskinDiffusion Bridges Cross-Modal
Ben-Gurion University of the Negev, Israel.
Quantifying Multimodal Capabilities: Formal Generalization Guarantees in Pairwise Metric Learning
May 2, 2026Richeng Zhou, Xuelin Zhang, Liyuan LiuMultimodal Learning Modalities
Multimodal Data Curation Through Ranked Retrieval
May 1, 2026Pratyush Muthukumar, Harshil Kotamreddy, Sarah Amiraslani +4Multimodal Embeddings Cross-Modal
NVIDIA
Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation
May 1, 2026Suryakant Singh, Saarthak Kapse, Joel Saltz +1Computational Pathology Whole-Slide Images
Stony Brook University
EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness
May 1, 2026Yueru Sun, Yimeng Zhang, Haoyu Gu +5Emotion Recognition Multimodal Large Language Models
School of Future Technology, South China University of Technology, Guangzhou, China
Energy-Based Constraint Networks: Learning Structural Coherence Across Modalities
May 1, 2026Chirag ShindeDual-Encoder Architectures Frozen Encoder
Independent Researcher
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Large Language Model Jailbreaks Safety Alignment
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration
May 1, 2026Chunlei Meng, Pengbin Feng, Rong Fu +7Multimodal Learning Cross-Modal
The College of Intelligent Robotics and Advanced Manufacturing, Fudan University · University of Southern California · University of Macau +5

Jun 3, 2026, cs.LGJ/K move · Enter open · S save

RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities

Vasiliki Rizou, Pascal Frossard, Dorina Thanou

EPFL

Multimodal Representations Representation Learning Modalities Self-Supervised Learning Recast

Open paper AskPDF arXiv

To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information. Learning disentangled representations is a principled way to identify these underlying shared and unique factors that are hidden in observational data. However, while multimodal disentanglement is a compelling paradigm, existing methods are largely confined to the two-modality regime due to its inherent scalability bottleneck. To address this, we propose RePercENT, a self-supervised framework designed to surpass these limitations and unlocks scalable pairwise disentanglement beyond two modalities. Through a multimodal `plug-and-play' architecture, our approach operates directly on pre-extracted embeddings, eliminating the need for extensive joint pre-training while making no assumptions regarding the underlying modalities or foundation model backbones. Moreover, we introduce a joint optimization objective for simultaneously deriving the shared and unique components, and provide formal theoretical guarantees that characterize the optimality of our solution. Across diverse modalities and tasks, RePercENT successfully recovers disentangled components while maintaining competitive performance and significantly reducing computational complexity.

Momentum

Related topics

Latest papers 217