Cross-Modal Representation Learning

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 136

All topics

Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
May 22, 2026Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel +2Efficient Multimodal Inference Cross-Modal Representation Learning
Adelaide University
GeoMamba: A Geometry-driven MambaVision Framework and Dataset for Fine-grained Optical-SAR Object Retrieval
May 19, 2026Tiantong Fang, Xiuwei Wang, Jing Xiao +3Cross-Modal Representation Learning Geometric Representation Learning
School of Artificial Intelligence, Wuhan University · School of Artificial Intelligence and Information Engineering, Zhejiang University of Science & Technology · Hangzhou Institute of Technology, Xidian University +1
Neuroscience-inspired Staged Representation Learning with Disentangled Coarse- and Fine-Grained Semantics for EEG Visual Decoding
May 16, 2026Xiang Gao, Hui Tian, Yanming Zhu +2Disentangled Representation Learning Cross-Modal Representation Learning
School of Information and Communication Technology, Griffith University, Parklands Drive, Southport, 4222, QLD, Australia
Mind the Gap: Learning Modality-Agnostic Representations with a Cross-Modality UNet
May 16, 2026Xin Niu, Enyi Li, Jinchao Liu +3Cross-Modal Learning Cross-Modal Representation Learning
Tianjin Key Laboratory of Intelligent Robotics, College of Artificial Intelligence, Nankai University, China, and also with Engineering Research Center of Trusted Behavior Intelligence, Ministry of Education, Nankai University, China. · VisionMetric Ltd, Canterbury, Kent, UK. · Department of Computer Science, Haifa University, Israel.
Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Data
May 16, 2026Shiwon Kim, Yu Rang ParkCross-Modal Learning Representation Learning
Yonsei University
Cross-Domain Molecular Relational Learning: Leveraging Chemical Structure-Activity Analysis
May 16, 2026Peiliang Zhang, Jingling Yuan, Shiqing Wu +4Domain-Adversarial Learning Cross-Modal Representation Learning
Wuhan University of Technology Wuhan, Hubei, China · Hubei Key Laboratory of Transportation Internet of Things State Key Laboratory of Silicate Materials for Architectures Wuhan University of Technology Wuhan, Hubei, China · Yonsei University Seoul, Republic of Korea +3
AIM-DDI: A Model-Agnostic Multimodal Integration Module for Drug-Drug Interaction Prediction
May 14, 2026Yerin Park, Sangseon LeeCross-Modal Representation Learning Multimodal Fusion
Department of Artificial Intelligence, Inha University
VoxCor: Training-Free Volumetric Features for Multimodal Voxel Correspondence
May 13, 2026Guney Tombak, Ertunc Erdil, Ender KonukogluMedical Imaging Foundation Models Cross-Modal Representation Learning
Biomedical Image Computing Group, ETH Zurich, Zurich, Switzerland · The LOOP Zurich – Medical Research Center, Zurich, Switzerland
STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
May 13, 2026Hongli Liu, Yu Wang, Shengjie ZhaoCross-Modal Representation Learning Few-Shot Learning
School of Computer Science and Technology, Tongji University, Shanghai, China · Engineering Research Center of Key Software Technologies for Smart City Perception and Planning, Ministry of Education
Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
May 12, 2026Souptik Sen, Raneen Younis, Zahra AhmadiCross-Modal Learning Representation Learning
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
Interactive State Space Model with Cross-Modal Local Scanning for Depth Super-Resolution
May 12, 2026Chen Wu, Ling Wang, Zhuoran Zheng +4Cross-Modal Representation Learning Visual SSMs
National University of Defense Technology · The Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University +2
STAGE: Tackling Semantic Drift in Multimodal Federated Graph Learning
May 12, 2026Zekai Chen, Xun Wu, Xunkai Li +3Multimodal Federated Learning Cross-Modal Representation Learning
Beijing Institute of Technology, Beijing, 100081, China · Minzu University of China, Beijing, 100081, China
Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining
May 12, 2026Yucheng Xing, Pei Liu, Jingying Ma +6Computational Pathology Cross-Modal Representation Learning
National University of Singapore, Singapore · Hunan University, China · Peking Union Medical College Hospital (PUMCH), China +1
Physical probes expose and alleviate chemical-environment collapse in molecular representations
May 11, 2026Jiebin Fang, Zidi Yan, Churu Mao +7Contrastive Learning Cross-Modal Representation Learning
Hainan Institute, Zhejiang University, Sanya 572025, China · Institute of Marine Biology and Pharmacology, Ocean College, Zhejiang University, Zhoushan 316021, China · School of Food and Pharmacy, Zhejiang Ocean University, Zhoushan 316021, China +2
Biosignal Fingerprinting: A Cross-Modal PPG-ECG Foundation Model
May 10, 2026Zhangdaihong Liu, Chang Liu, Fenglin Liu +4Cross-Modal Representation Learning Masked Autoencoders
Department of Engineering Science, University of Oxford, Oxford OX3 7DQ, UK · Oxford Suzhou Centre for Advanced Research, Suzhou, 215123, China · School of Public Health, Shanghai Jiao Tong University, Shanghai, China
Learning Displacement-Aware WiFi Representations for Weakly Supervised Relative Localization
May 9, 2026Tzu-Ti Wei, Po-Cheng Chen, Yu-Chee Tseng +1Cross-Modal Representation Learning Wireless Localization
College of AI, National Yang Ming Chiao Tung University, Taiwan
jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
May 8, 2026Florian Hönicke, Michael Günther, Andreas Koukounas +4Cross-Modal Learning Cross-Modal Representation Learning
Jina by Elastic
PRISM: Iterative Cross-Modal Posterior Refinement for Dynamic Text-Attributed Graphs
May 7, 2026Trimble Chang, Yihang Liu, Mingjing Han +1Cross-Modal Representation Learning Temporal Link Prediction
College of Artificial Intelligence Nankai University
Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
May 5, 2026Hahyeon Choi, Nojun KwakCross-Modal Representation Learning Sparse Mixture-of-Experts
Seoul National University, Seoul, South Korea.
SplAttN: Bridging 2D and 3D with Gaussian Soft Splatting and Attention for Point Cloud Completion
May 2, 2026Zhaoyang Li, Zhichao You, Tianrui LiCross-Modal Learning Point Cloud Completion
School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China
Physiology-Aware Masked Cross-Modal Reconstruction for Biosignal Representation Learning
May 1, 2026Hao Zhou, Simon A. Lee, Cyrus Tanade +12Representation Learning Cross-Modal Representation Learning
Samsung Research America · Algorithm Team, Digital Health Lab, Samsung Research America · The Pennsylvania State University +1
CGM-JEPA: Learning Consistent Continuous Glucose Monitor Representations via Predictive Self-Supervised Pretraining
May 1, 2026Hada Melino Muhammad, Zechen Li, Flora Salim +1Cross-Modal Representation Learning Self-Supervised Time Series Representation Learning
University of New South Wales · Google Research
REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception
Apr 30, 2026Vincenzo Polizzi, David B. Lindell, Jonathan KellyCross-Modal Learning Event-Based Vision
University of Toronto, Robotics Institute · University of Toronto, Department of Computer Science
Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints
Apr 29, 2026Wasim Ahmad, Wei Zhang, Xuerui MaoAudio Deepfake Detection Cross-Modal Representation Learning
School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
Apr 29, 2026Junwon You, Mihyun Jang, Sangwoo Mo +1Cross-Modal Learning Cross-Modal Representation Learning
1KAIST · 2POSTECH
Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data
Apr 25, 2026Long Jing, Zhixiong Yang, Yajun Zhang +1Cross-Modal Representation Learning Human Activity Recognition
College of Computer Science, Northwest University, Xi’an, China · Xinjiang University, Urumqi, China
AnalogRetriever: Learning Cross-Modal Representations for Analog Circuit Retrieval
Apr 25, 2026Yihan Wang, Lei Li, Yao Lai +2Cross-Modal Representation Learning Electronic Design Automation
The University of Hong Kong Hong Kong, China · University of Cambridge Cambridge, United Kingdom · Tsinghua University Beijing, China
CHRep: Cross-modal Histology Representation and Post-hoc Calibration for Spatial Gene Expression Prediction
Apr 23, 2026Changfan Wang, Xinran Wang, Donghai Liu +4Spatial Transcriptomics Post-Hoc Calibration
Beijing University of Posts and Telecommunications, Beijing 100876, China, and also with the Beijing Key Laboratory of Network System and Network Culture, Beijing 100876, China · Peking University Third Hospital, Beijing 100191, China
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Apr 20, 2026A. Sophia Koepke, Daniil Zverev, Shiry Ginosar +1Cross-Modal Learning Cross-Modal Representation Learning
UC Berkeley · Technical University Munich, MCML · University of Tübingen, Tübingen AI Center +1
ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrieval
Apr 20, 2026Zixu Li, Yupeng Hu, Zhiwei Chen +4Cross-Modal Learning Cross-Modal Representation Learning
School of Software, Shandong University · School of Computer Science and Technology, Shandong Jianzhu University

May 22, 2026, cs.LGJ/K move · Enter open · S save

Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey

Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel, Lin Yue+1

Adelaide University

Efficient Multimodal Inference Cross-Modal Representation Learning Modality Imbalance Mixture-of-Experts Models Multimodal Fusion Multimodal Learning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) presents a naturally compatible and scalable framework for multimodal learning, demonstrating strong adaptability across diverse modalities and tasks. Despite its growing success, a comprehensive and systematic review on the MoE metho addressing multimodal challenges remains lacking. Existing surveys tend to evaluate either multimodal learning or MoE independently from method taxonomy, overlooking the unique interplay between them. This survey fills that gap by answering a central question: \textit{How does MoE effectively resolve multimodal challenges?} We approach this from three key perspectives: (1) \textbf{MoE as an Efficient Multimodal Engine:} enabling scalable multimodal modeling by decoupling computational cost from parameter growth and mitigating modality redundancy through selective expert activation; (2) \textbf{MoE as a Multimodal Representation Learner:} integrating complementary multi-opinion expert knowledge to enrich alignment and interaction representations; and (3) \textbf{MoE as a Multimodal Adapter:} providing a modular and flexible mechanism to model imperfect data scenarios such as modality imbalance and missing modality. Through our extensive literature review, we identify critical research gaps, including interpretable routing, expert communication, modality integration, and lifelong multimodal learning. We position this survey as a foundation for future research toward interpretable and sustainable multimodal Mixture-of-Experts system.

Momentum

Related topics

Latest papers 136