Cross-Modal Representation Learning

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 136

All topics

3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy
Jun 22, 2026Amirhossein Kardoost, Lion Gleiter, Tingying Peng +1Cross-Modal Representation Learning Masked Autoencoders
Institute of AI for Health & Helmholtz AI, Computational Health Center, Helmholtz Munich – German Research Center for Environmental Health, Neuherberg, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Department of Medicine III, Ludwig-Maximilian-University Hospital, Munich, Germany +2
Retrieval-Augmented Multimodal Learning for Enzyme-Substrate Interaction Prediction Under Low-Homology Shift
Jun 22, 2026Chen Liu, Bingxin Zhou, Xinyuan Wang +3Cross-Modal Representation Learning Domain Generalization
School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China · Institute of Natural Sciences and Zhangjiang Institute for Advanced Study, Shanghai Jiao Tong University, Shanghai 200240, China · School of Life Sciences and Biotechnology, Shanghai Jiao Tong University. Shanghai 200240, China +1
T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition
Jun 19, 2026Di Yang, Mahmoud Ali, Quan Kong +2Skeleton-Based Action Recognition Representation Learning
Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China · Inria Center at Universit´e Cˆote d’Azur, Valbonne, France · Woven by Toyota, Tokyo, Japan +1
MV-WAM: Manifold-Aware World Action Model with Value Augmentation
Jun 19, 2026Jintao Chen, Peidong Jia, Qingpo Wuwu +13RL Control Cross-Modal Representation Learning
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics
FUSE: Frequency-domain Unification and Spectral Energy Alignment for Multi-modal Object Re-Identification
Jun 18, 2026Xuanhao Qi, Tom H. Luan, Yukang Zhang +4Cross-Modal Representation Learning Multimodal Robustness
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, China · School of Informatics, Xiamen University, Xiamen, China · National University of Singapore, Singapore
TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer
Jun 17, 2026Arunim Joarder, Arjun Bhardwaj, René Zurbrügg +6Cross-Modal Representation Learning Tactile Sensing
Robotic Systems Lab, ETH Zürich · ETH AI Center · NVIDIA +1
CAP: Towards PPG Universal Representation Learning with Patient-level Supervision
Jun 13, 2026Chenyang He, Xinyi Shao, Shun Huang +4Representation Learning Cross-Modal Representation Learning
Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China · Peking University Beijing, Beijing, China · Independent Researcher +1
EyeMVP: OCT-Informed Fundus Representation Learning via Paired CFP--OCT Pretraining
Jun 13, 2026Zhuo Deng, Ruiheng Zhang, Ziheng Zhang +23Medical Imaging Foundation Models Cross-Modal Representation Learning
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Beijing Tongren Eye Center, Beijing Tongren Hospital, Capital Medical University, Beijing, China · Liangxiang Hospital of Beijing Fangshan District, Capital Medical University, Beijing, China +6
HiST: A Hierarchical Sparse Transformer for Cross-Modal Spatial Transcriptomics Modeling
Jun 12, 2026Weiyi Wu, Xinwen Xu, Xingjian Diao +4Computational Pathology Spatial Transcriptomics
Dartmouth College · Mass General Hospital · New Jersey Institute of Technology +1
When to Align, When to Predict: A Phase Diagram for Multimodal Learning
Jun 9, 2026Ilay Kamai, Hugues Van Assel, Aviv Regev +2Cross-Modal Representation Learning Cross-Modal Alignment
Technion · Genentech · Brown University
AnyMod-LLVE: Low-Light Video Enhancement with Modality-Agnostic Inference
Jun 9, 2026Hangfeng Liang, Yutao Hu, Yanhan Hu +3Missing-Modality Learning Cross-Modal Representation Learning
School of Computer Science and Engineering, Southeast University · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Shanghai Innovation Institute +1
RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities
Jun 3, 2026Vasiliki Rizou, Pascal Frossard, Dorina ThanouDisentangled Representation Learning Cross-Modal Representation Learning
EPFL
COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations
Jun 3, 2026Zixu Li, Yupeng Hu, Zhiwei Chen +3Cross-Modal Representation Learning Cross-Modal Alignment
School of Software, Shandong University, Jinan, 250100, China · School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, 518000, China · Department of Data Science, City University of Hong Kong, Hong Kong 523808, China +1
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Jun 3, 2026Shuwen Yu, Zhanxuan Hu, Yi Zhao +2Cross-Modal Representation Learning Vision Foundation Model Adaptation
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China
Building The Ph(ysical)AI Layer Of Machine Intelligence
Jun 2, 2026Ulbert Jose Botero, Liam Smith, Brooks Olney +5Cross-Modal Learning Cross-Modal Representation Learning
MIT Lincoln Laboratory
SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition
Jun 2, 2026Yanan Liu, Anqi Zhu, Jingmin Zhu +6Skeleton-Based Action Recognition Cross-Modal Representation Learning
Yunnan University, Kunming, China · Monash University · Lancaster University +1
Channel-Oriented Design for EEG-to-Music Reconstruction
Jun 2, 2026Jiaxin Qing, Junwei Lu, Lexin LiElectroencephalography Cross-Modal Representation Learning
UC Berkeley · Harvard University
Spatial Representation Learning Beyond Pixels: Unifying Raster Data and Vector Semantics for Human-Centric Geospatial Foundation Models
Jun 1, 2026Steffen Knoblauch, Hao Li, Gengchen Mai +3Remote Sensing Image Understanding Representation Learning
Heidelberg University · National University of Singapore · University of Texas at Austin +3
Towards Resolving Optimization Conflicts Between Image- and Text-Based Person Re-Identification
Jun 1, 2026Karina Kvanchiani, Timur MamedovCross-Modal Representation Learning Multi-Task Learning
1Tevian, Russia · 2Lomonosov Moscow State University, Russia
CR-JEPA: Cross-Modal Joint-Embedding Predictive Learning for Remote Sensing Image Retrieval
May 30, 2026Md Aminur Hossain, Ayush V. Patel, Nitant Dube +1Cross-Modal Representation Learning Joint-Embedding Predictive Architecture
Space Applications Centre, Indian Space Research Organisation, Ahmedabad, India · Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay, Mumbai, India
HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learning
May 29, 2026Md Aminur Hossain, Ayush V. Patel, Sanjay K. Singh +1Representation Learning Cross-Modal Representation Learning
Space Applications Centre, Indian Space Research Organisation, Ahmedabad, India · Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay, Mumbai, India
UniRTL: Unifying Code and Graph for Robust RTL Representation Learning
May 29, 2026Yi Liu, Hongji Zhang, Lei Chen +2Representation Learning Cross-Modal Representation Learning
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR · Noah’s Ark Lab, Huawei, Hong Kong SAR
Variational Adapter for Cross-modal Similarity Representation
May 29, 2026WenZhang Wei, Zhipeng Gui, Dehua Peng +2Cross-Modal Representation Learning VLM Adaptation
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China · School of Data Science and Engineering, East China Normal University, Shanghai 200062, China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China
ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment
May 29, 2026Jun-Hak Yun, Seung-Bin Kim, Seong-Whan LeeCross-Modal Representation Learning TTS Synthesis
Department of Artificial Intelligence, Korea University, Seoul, Korea
Improving Relative Representations with Learned Anchors and Whitened Inner Products
May 28, 2026Oscar Thorsted Svendsen, Nikolaj Holst Jakobsen, Fabian Mager +1Representation Learning Cross-Modal Representation Learning
1Technical University of Denmark
A Shared Valence Axis Across Modern LLMs and Human EEG: The Saturation Regularity
May 28, 2026Yousef A. Radwan, Xuhui Liu, Kilichbek Haydarov +2Cross-Modal Representation Learning Neural Decoding
King Abdullah University of Science and Technology (KAUST)
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
May 28, 2026Jusuk Lee, Seungjae Lee, Jonghun Shin +6Cross-Modal Learning Cross-Modal Representation Learning
Seoul National University · University of Maryland, College Park · Georgia Institute of Technology
Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals
May 26, 2026Phu X. Nguyen, Konstantinos Kontras, Wei Dai +5Representation Learning Cross-Modal Representation Learning
1KU Leuven · 3MIT · 4DFKI +1
MAIL++: Multi-Modal Bi-directional Agent Layer for Vision-Language Models
May 25, 2026Kaixiang Chen, Pengfei Fang, Hui XueCross-Modal Learning Cross-Modal Representation Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Balancing Multimodal Learning through Label Space Reshaping
May 22, 2026Xiaoyu Ma, Weijie Zhang, Yuanhao Gao +3Cross-Modal Learning Cross-Modal Representation Learning
PREPRINT 2026.05

Jun 22, 2026, cs.LGJ/K move · Enter open · S save

3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy

Amirhossein Kardoost, Lion Gleiter, Tingying Peng, Carsten Marr

Institute of AI for Health & Helmholtz AI, Computational Health Center, Helmholtz Munich – German Research Center for Environmental Health, Neuherberg, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Department of Medicine III, Ludwig-Maximilian-University Hospital, Munich, Germany+2

Cross-Modal Representation Learning Masked Autoencoders 3D Representation Learning Self-Supervised Visual Representation Learning Microscopy Image Analysis

Open paper AskPDF arXiv

Self-supervised learning in fluorescence microscopy often relies on 2D projections, despite the inherently three-dimensional nature of cells. We present a systematic comparison of 2D and 3D masked autoencoders (MAE-2D vs. MAE-3D) on volumetric microscopy data. Under matched architectures and training protocols, MAE-3D consistently outperforms 2D max-projection and slice-based variants on downstream single-cell tasks. We further align visual representations with a pretrained protein language model (ESM2) and show that cross-modal supervision yields larger gains for volumetric models. Channel cross-attention and frequency-domain regularization are critical for leveraging 3D spatial context. On protein--protein interaction prediction, our best model achieves a ROC--AUC of 0.86, while on protein localization it reaches an AUC

_{\text{micro}}

of 0.95 and an F1

_{\text{micro}}

of 0.74, demonstrating competitive performance on both tasks. Overall, our findings highlight the potential of volumetric modeling and multimodal alignment for representation learning in single-cell microscopy.

Momentum

Related topics

Latest papers 136