Cross-Modal Representation Learning

Follow this topic Ask about this topic

The field studies how models can represent different kinds of data so that related observations can be matched, combined, or used to transfer knowledge. Common approaches include CLIP-style contrastive alignment, masked cross-modal reconstruction and predictive learning, teacher–student distillation, and methods that separate shared from modality-specific features; anchors and optimal transport can help when paired examples are scarce. These representations support image-text retrieval, cross-sensor recognition, and transfer across domains or missing-modality settings. A central tension is that shared spaces can improve transfer but discard useful modality-specific structure, and apparent semantic alignment does not always imply fine-grained correspondence; the best choice between alignment and prediction depends on data quality and cross-modal relationships.

Latest papers

210 papers22 papers in the last four weeks, up 29% on the four weeks before. 0.1% of all new papers.

Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data
Apr 25, 2026Long Jing, Zhixiong Yang, Yajun Zhang +1Cross-Modal Representation Learning Human Activity Recognition
College of Computer Science, Northwest University, Xi’an, China · Xinjiang University, Urumqi, China
AnalogRetriever: Learning Cross-Modal Representations for Analog Circuit Retrieval
Apr 25, 2026Yihan Wang, Lei Li, Yao Lai +2Cross-Modal Representation Learning Electronic Design Automation
The University of Hong Kong Hong Kong, China · University of Cambridge Cambridge, United Kingdom · Tsinghua University Beijing, China
CHRep: Cross-modal Histology Representation and Post-hoc Calibration for Spatial Gene Expression Prediction
Apr 23, 2026Changfan Wang, Xinran Wang, Donghai Liu +4Spatial Transcriptomics Post-Hoc Calibration
Beijing University of Posts and Telecommunications, Beijing 100876, China, and also with the Beijing Key Laboratory of Network System and Network Culture, Beijing 100876, China · Peking University Third Hospital, Beijing 100191, China
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Apr 20, 2026A. Sophia Koepke, Daniil Zverev, Shiry Ginosar +1Cross-Modal Learning Cross-Modal Representation Learning
UC Berkeley · Technical University Munich, MCML · University of Tübingen, Tübingen AI Center +1
ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrieval
Apr 20, 2026Zixu Li, Yupeng Hu, Zhiwei Chen +4Cross-Modal Learning Cross-Modal Representation Learning
School of Software, Shandong University · School of Computer Science and Technology, Shandong Jianzhu University
Intervention-Aware Multiscale Representation Learning from Imaging Phenomics and Perturbation Transcriptomics
Apr 19, 2026Jiayuan Chen, Ruoqi Liu, Zishan Gu +1Cross-Modal Knowledge Distillation Cross-Modal Representation Learning
The Ohio State University
TriTS: Time Series Forecasting from a Multimodal Perspective
Apr 17, 2026Xiang AoLong-Term Time Series Forecasting Cross-Modal Representation Learning
School of Software Engineering, Beijing Jiaotong University Beijing 100044, China
MMGait: Towards Multi-Modal Gait Recognition
Apr 17, 2026Chenye Wang, Qingyuan Cai, Saihui Hou +2Cross-Modal Representation Learning Gait Analysis
School of Artificial Intelligence, Beijing Normal University
Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
Apr 17, 2026Irem Ulku, Erdem Akagündüz, Ömer Özgür TanrıöverMissing-Modality Learning Remote Sensing Image Segmentation
Department of Computer Engineering, Ankara University, Ankara, Türkiye · Department of Modeling and Simulation, Graduate School of Informatics, METU, Ankara, Türkiye
Sketch and Text Synergy: Fusing Structural Contours and Descriptive Attributes for Fine-Grained Image Retrieval
Apr 17, 2026Siyuan Wang, Hanchen Gao, Guangming Zhu +5Cross-Modal Representation Learning Fine-Grained Image Retrieval
Xidian University · First Aircraft Design Institute
SIMPLER: H&E-Informed Representation Learning for Structured Illumination Microscopy
Apr 11, 2026Abu Zahid Bin Aziz, Syed Fahim Ahmed, Gnanesh Rasineni +8Medical Image Representation Learning Microscopy Image Analysis
Kahlert School of Computing, University of Utah, Salt Lake City, UT 84112, USA · Scientific Computing & Imaging Institute, University of Utah, Salt Lake City, UT 84112, USA · Instapath Inc., Houston, TX 77021 +1
Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
Apr 9, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Cross-Modal Representation Learning Unified Multimodal Models
HKUST · Tencent Hunyuan
Beyond Pedestrians: Caption-Guided CLIP Framework for High-Difficulty Video-based Person Re-Identification
Apr 9, 2026Shogo Hamano, Shunya Wakasugi, Tatsuhito Sato +1Person Re-Identification Cross-Modal Representation Learning
Sony Group Corporation
The Indra Representation Hypothesis for Multimodal Alignment
Apr 6, 2026Jianglin Lu, Hailing Wang, Kuo Yang +3Representation Learning Cross-Modal Alignment
Department of Electrical and Computer Engineering, Northeastern University · Adobe Research · Khoury College of Computer Science, Northeastern University
A General Framework for Generative Self-supervised Learning in Non-invasive Estimation of Physiological Parameters Using Photoplethysmography
Apr 3, 2026Zexing Zhang, Huimin Lu, Songzhe Ma +4Self-Supervised Time Series Representation Learning Photoplethysmography
School of Computer Science and Engineering, Changchun University of Technology, Jilin, Changchun, 130102, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Jilin, Changchun, 130012, China · Smart Health Joint Innovation Laboratory for the New Generation of AI, Jilin, Changchun, 130102, China
CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics
Mar 31, 2026Andrew Jeong, Jaemin Kim, Sebin Lee +1Robot Motion Planning Cross-Modal Representation Learning
KAIST
SegRGB-X: General RGB-X Semantic Segmentation Model
Mar 30, 2026Jiong Liu, Yingjie Xu, Xingcheng Zhou +4Multimodal Semantic Segmentation Semantic Segmentation
Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Munich, Germany
TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution
Mar 29, 2026Yinlin Zhu, Xunkai Li, Di Wu +3Multimodal Graph Learning Graph Structure Learning
Sun Yat-sen University, Guangzhou, China · Beijing Institute of Technology, Beijing, China
Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval
Mar 25, 2026Xintao Zong, Xian Zhong, Wenxuan Liu +3Image-Text Retrieval Spiking Neural Networks
Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology · State Key Laboratory for Multimedia Information Processing, Peking University
Generative Event Pretraining with Foundation Model Alignment
Mar 24, 2026Jianwen Cao, Jiaxu Xing, Nico Messikommer +1Event-Based Vision Vision Foundation Model Adaptation
Robotics and Perception Group, University of Zurich
Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements
Mar 23, 2026Omkar Sawant, Luca Zanatta, Grzegorz Malczyk +1Robot Navigation Cross-Modal Learning
Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU), Norway
NOUS: Video-Driven 3D Human Reaction Generation via Observation-Reaction Mutual Steering
Mar 20, 2026Yuan Zhou, Luanyuan Dai, Yongzhi Li +7Cross-Modal Representation Learning Human Motion Generation
College of Computing and Data Science, Nanyang Technological University, Singapore 639798 · School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China · School of Computer Science and Information Engineering (School of Artificial Intelligence), Hefei University of Technology, Hefei 230601, China +1
Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus
Mar 19, 2026Mohamed Badi, Chaouki Ben Issaid, Mehdi BennisMultimodal Federated Learning Federated Learning
Center for Wireless Communications, University of Oulu, Oulu 90014, Finland
Early Intervention for VFM-based Multimodal Medical Image Classification
Mar 18, 2026Qijie Wei, Hailan Lin, Xirong LiCross-Modal Representation Learning Vision Foundation Model Adaptation
Renmin University of China, Beijing 100872, China
Mutually Causal Semantic Distillation Network for Zero-Shot Learning
Mar 18, 2026Shiming Chen, Shuhuang Chen, Guo-Sen Xie +1Zero-Shot Learning Cross-Modal Representation Learning
Huazhong University of Science and Technology, Wuhan, China · National Anti-Counterfeit Engineering Research Center, Wuhan, Chna · Nanjing University of Science and Technology, Nanjing, China
$x^2$ -Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space
Mar 17, 2026Ruishan Guo, Ciyu Ruan, Haoyang Wang +3Scene Flow Estimation Optical Flow
Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology · The University of Hong Kong
Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
Mar 17, 2026Omnilingual SONAR Team, João Maria Janeiro, Pere-Lluís Huguet Cabot +17Sentence Embeddings Cross-Lingual Representation Alignment
FAIR at Meta
Cross-modal learning for plankton recognition
Mar 17, 2026Joona Kareinen, Veikka Immonen, Tuomas Eerola +5Cross-Modal Representation Learning Self-Supervised Learning
Lappeenranta-Lahti University of Technology LUT, Lappeenranta, Finland · Finnish Environment Institute, Helsinki, Finland · Faculty of Information Technology, Brno University of Technology, Czechia
BIT: Matching-based Bi-directional Interaction Transformation Network for Visible-Infrared Person Re-Identification
Mar 15, 2026Haoxuan Xu, Guanglin NiuVisible-Infrared Person Re-Identification Person Re-Identification
School of Artificial Intelligence, Beihang University
Multimodal Protein Language Models for Enzyme Kinetic Parameters: From Substrate Recognition to Conformational Adaptation
Mar 13, 2026Fei Wang, Xinye Zheng, Kun Li +5Molecular Property Prediction Cross-Modal Representation Learning
School of Computer Science and Information Engineering, Hefei University of Technology · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · CVLab, College of Information Technology, United Arab Emirates University +2

Apr 25, 2026J/K move · Enter open · S save

Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data

Long Jing, Zhixiong Yang, Yajun Zhang, Xinlong Feng

College of Computer Science, Northwest University, Xi’an, China · Xinjiang University, Urumqi, China

Cross-Modal Representation Learning Human Activity Recognition Multimodal Contrastive Learning Multimodal Fusion Data-Efficient Learning

Open paper AskPDF arXiv

Human activity recognition serves as the foundation for various emerging applications. In recent years, researchers have used collaborative sensing of multi-source sensors to capture complex and dynamic human activities. However, multimodal human activity sensing typically encounters highly heterogeneous data across modalities and label scarcity, resulting in an application gap between existing solutions and real-world needs. In this paper, we propose CLMM, a general contrastive learning framework for human activity recognition that achieves effective multimodal recognition with limited labeled data. CLMM employs a novel two-stage training strategy. In the first stage, CLMM employs a CNN-DiffTransformer encoder to capture cross-modal shared information by extracting local and global features. Meanwhile, a hard-positive samples weighting algorithm enhances gradient propagation to reinforce shared learning. In the second stage, a dual-branch architecture combining quality-guided attention and bidirectional gated units captures modality-specific information, while a primary-auxiliary collaborative training strategy fuses both shared and modality-specific information. Experimental results on three public datasets demonstrate that CLMM significantly improves state-of-the-art baselines in both recognition accuracy and convergence performance.