Cross-Modal Representation Transfer

Follow this topic Ask about this topic

Latest papers

34 papers12 papers in the last four weeks, up 140% on the four weeks before. 0.1% of all new papers.

Do Pretrained Backbones Transfer to Physical Sensors? A Systematic Study of Multimodal Initialization Across the Sensor Stack
Nov 2, 2026Fabio Montello, Ronja Güldenring, Lucas Mogensen Sandby +2Multimodal Pretraining Cross-Modal Representation Transfer
Technical University of Denmark
Latent Spectral Prompting for Multimodal Hyperspectral Object Tracking
Nov 2, 2026Wangshu Cai, Jun Zhou, Lei Wang +3Multimodal Visual Object Tracking Cross-Modal Representation Transfer
Griffith University, Australia · East China Normal University, China · Bangladesh Agricultural University, Bangladesh
GaitFM: A Cross-Modal Foundation Model for Wearable Gait Assessment
Nov 1, 2026Yize Cai, Zhiqing HongGait Analysis Wearable Sensing
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · The Hong Kong University of Science and Technology, Hong Kong, China.
Transferring Peptide-Target Interaction Patterns to Small Molecules by Joint Guidance for Diffusion and Flow Matching
Oct 30, 2026Jian Lee, Changyun Cho, Sun Kim3D Molecular Generation Drug Discovery
AIGENDRUG Co., Ltd., Seoul · Interdisciplinary Program in Artificial Intelligence, Seoul National University · Department of Computer Science and Engineering, Seoul National University +1
DiFuse: Fusing Native Diffusion Representations for Spatial Reasoning in Unified Multimodal Models
Oct 29, 2026Ruixin Fan, Weining Wang, Jiahui Sun +2Multimodal Spatial Reasoning Unified Multimodal Models
Institute of Automation, Chinese Academy of Sciences
BridgeRouter: Gain-Aware Instance Routing over Cross-Modal Bridge Portfolios
Oct 28, 2026Ruoxi Piao, Alicia Lu, Yunzhen Zhang +2Multimodal Model Routing Cross-Modal Representation Transfer
Duke Kunshan University · Ankara University
Towards Generalizing Pretrained All-Atom Representations to Unified Binder Design
Oct 27, 2026Maojiang Su, Jiale Chen, Qijie Zhu +6Protein Binder Design Cross-Modal Representation Transfer
Center for Foundation Models and Generative AI, Northwestern University · Department of Statistics and Data Science, Northwestern University
Cross-modal Steering Vectors Exist Across Both Image and Text Generation
Oct 26, 2026Donghao Zeng, Nishant Subramani, Tamar Rott Shaham +1Cross-Modal Representation Transfer Unified Multimodal Models
Mila, McGill University · Carnegie Mellon University · MIT CSAIL +1
Correspondence Is Not Transfer: A Pairing-Controlled Study of Clinical Audio-Metadata Alignment
Oct 26, 2026Liu He, Xi Wang, Jiahong Yuan +1Cross-Modal Alignment Audio Understanding
University of Science and Technology of China, China · Independent Researcher · University of Cambridge, UK
The Geometry of Meaning in Neural Manifolds
Oct 22, 2026Valentino MaiorcaNeural Representation Geometry Semantic Correspondence
Department of Computer Science, Faculty of Information Engineering, Informatics, and Statistics, Sapienza University of Rome
VitronEdit: Universal Visual Editing across Images, Videos, and 3D Scenes
Oct 19, 2026Jing Wang, Yanlin Li, Mong-Li Lee +33D Scene Editing Image Editing
National University of Singapore · University of Oxford
A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help
Oct 18, 2026Shashwat Singh, He He, Tal LinzenVision-Language Alignment Cross-Modal Representation Transfer
New York University
CIIR - Class-Indexed Interface Representation Learning
Oct 16, 2026Doo Yon Kim, Christian WallravenCross-Modal Representation Learning Cross-Modal Representation Transfer
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Oct 7, 2026Dominik Schnaus, Thomas Dagès, Daniel Cremers +2Cross-Modal Alignment Cross-Modal Representation Transfer
TU Munich · MCML · Ulm University +2
Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
Oct 3, 2026Hoigi Seo, Byung Hyun Lee, Minjun Kim +3Cross-Modal Alignment Multimodal Large Language Models
Dept. of ECE · INMC & AIIS · IPAI, Seoul National University, Republic of Korea
Learning Through Game: Skewed Transfer of Tabular Knowledge to Strengthen Image Model
Sep 26, 2026Longfei Huang, Shangdong Yang, Yang YangCross-Modal Representation Transfer Multimodal Learning
Nanjing University of Science and Technology · Nanjing University of Posts and Telecommunications
Hub-Spectral Activation of Latent Multimodal Knowledge
Sep 15, 2026Ying Guo, Haidong Chen, Linrui Xu +7Cross-Modal Representation Learning Cross-Modal Alignment
Beijing Key Laboratory of Key Technologies for AI+ Domain Applications, North China University of Technology, Beijing 100144, China · School of Geosciences and Info-Physics, Central South University, Changsha 410083, China · NExT++ Research Centre, National University of Singapore, Singapore +2
Multimodal Floorplan Encoding: Learning Dense Modality-Invariant Representations
Sep 11, 2026Xavier Anadón, Rémi Pautrat, Rui WangLatent Space Alignment Cross-Modal Representation Learning
University of Zaragoza, Spain · Microsoft, Switzerland
Cross-modal learning for SAR target recognition using optical vision foundation models
Sep 7, 2026Lucas Hirsch, James R. Hopgood, Javid Khan +2Cross-Modal Alignment Cross-Modal Representation Transfer
School of Engineering, University of Edinburgh, United Kingdom · Leonardo UK, United Kingdom · School of Engineering and Physical Sciences, Heriot-Watt University, United Kingdom
CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space
Aug 31, 2026Paul Schneider, Nazim HaouchineCross-Modal Representation Learning Multimodal Disentangled Representation Learning
Harvard Medical School, Brigham and Women’s Hospital, Boston, MA, USA · Technical University of Munich, Munich, Germany
Giraffe: A Mapping Architecture from Hidden Text Representations to Visual Embeddings for Efficient Graphic Design
Aug 25, 2026Nejla GhaboosiCross-Modal Representation Transfer Multimodal Large Language Models
Canva Research
LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents
Aug 7, 2026Ivan Majic, Zexian Huang, Franziska Hübl +5Cross-Modal Alignment Multimodal Large Language Models
Graz University of Technology, Austria · University of Vienna, Austria · University of Liverpool, UK
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Aug 5, 2026Junlin Han, Shengbang Tong, David Fan +4Unified Multimodal Models Multimodal Pretraining
1FAIR · University of Oxford · 2Reality Labs
Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding
May 29, 2026Xiang Fang, Daizong Liu, Wanlong Fang +4Temporal Video Grounding Cross-Modal Representation Transfer
Hubei Key Laboratory of Distributed System Security, Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology · Peking University · Henan University +3
Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
Mind the Gap: Learning Modality-Agnostic Representations with a Cross-Modality UNet
May 16, 2026Xin Niu, Enyi Li, Jinchao Liu +3Cross-Modal Learning Cross-Modal Representation Learning
Tianjin Key Laboratory of Intelligent Robotics, College of Artificial Intelligence, Nankai University, China, and also with Engineering Research Center of Trusted Behavior Intelligence, Ministry of Education, Nankai University, China. · VisionMetric Ltd, Canterbury, Kent, UK. · Department of Computer Science, Haifa University, Israel.
Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
May 12, 2026Souptik Sen, Raneen Younis, Zahra AhmadiMultimodal Learning Representation Learning
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings
May 11, 2026Liuyin Yang, Qiang Sun, Bob Van Dyck +2Brain-Computer Interfaces Neural Decoding
Laboratory for Neuro- & Psychophysiology, Department of Neurosciences, KU Leuven
EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models
Apr 13, 2026Jincheng Xie, Xingchen Xiao, Runheng Liu +3Cross-Modal Learning Multimodal Embedding
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JD iCity, JD Technology, JD Intelligent Cities Research, Beijing, China
Improving Generalization on Cybersecurity Tasks with Multi-Modal Contrastive Learning
Mar 20, 2026Jianan Huang, Rodolfo V. Valentim, Luca Vassio +4Cross-Modal Representation Transfer Cybersecurity
Politecnico di Torino, Italy · Universit`a di Torino, Italy · Huawei Paris Research Center, France

Nov 2, 2026J/K move · Enter open · S save

Do Pretrained Backbones Transfer to Physical Sensors? A Systematic Study of Multimodal Initialization Across the Sensor Stack

Fabio Montello, Ronja Güldenring, Lucas Mogensen Sandby, Christoffer Koo Øhrstrøm+1

Technical University of Denmark

Multimodal Pretraining Cross-Modal Representation Transfer Masked Autoencoders Multimodal Sensor Fusion

Open paper AskPDF arXiv

Transfer learning from pretrained backbones has become standard practice in multi-sensor computer vision---yet two questions remain open: which pretrained weights transfer best to multi-sensor setups, and can in-domain pretraining outperform off-the-shelf solutions? Frameworks like MultiMAE demonstrate the utility of masked autoencoding in a multi-modal context, but rely entirely on dense, model-derived modalities. It is unclear how robustly these methods translate to real-world applications that use physical sensors, where the aggregation of diverse public datasets for large-scale pretraining inherently introduces modality sparsity. To investigate this systematically, we curate a multi-sensor autonomous driving collection of

2.82M

images from 16 publicly available datasets, incorporating both physical hardware modalities (RGB, LWIR, NIR, Event) and model-derived modalities (Stereo Depth, Segmentation). We use this collection for in-domain pretraining based on the MultiMAE approach. We adapt MultiMAE's loss formulation to accommodate sparse samples, and study the effect of an enforced encoder bottleneck for modality alignment and of additional losses specific to dense reconstruction (SSIM and FFT). We then systematically compare against common pretrained backbones (ViT, MAE, DINO and MultiMAE) across downstream sensor fusion tasks. We find that MultiMAE is a surprisingly strong out-of-the-box baseline, transferring effectively even to modalities it has never seen, while our targeted modifications yield incremental but consistent improvements on multi-sensor transfer and overall robustness.