Cross-Modal Representation Transfer

Follow this topic Ask about this topic

Latest papers

22 papers4 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

CIIR - Class-Indexed Interface Representation Learning
Oct 16, 2026Doo Yon Kim, Christian WallravenCross-Modal Representation Learning Cross-Modal Representation Transfer
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Oct 7, 2026Dominik Schnaus, Thomas Dagès, Daniel Cremers +2Cross-Modal Alignment Cross-Modal Representation Transfer
TU Munich · MCML · Ulm University +2
Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
Oct 3, 2026Hoigi Seo, Byung Hyun Lee, Minjun Kim +3Cross-Modal Alignment Multimodal Large Language Models
Dept. of ECE · INMC & AIIS · IPAI, Seoul National University, Republic of Korea
Learning Through Game: Skewed Transfer of Tabular Knowledge to Strengthen Image Model
Sep 26, 2026Longfei Huang, Shangdong Yang, Yang YangCross-Modal Representation Transfer Multimodal Learning
Nanjing University of Science and Technology · Nanjing University of Posts and Telecommunications
Hub-Spectral Activation of Latent Multimodal Knowledge
Sep 15, 2026Ying Guo, Haidong Chen, Linrui Xu +7Cross-Modal Representation Learning Cross-Modal Alignment
Beijing Key Laboratory of Key Technologies for AI+ Domain Applications, North China University of Technology, Beijing 100144, China · School of Geosciences and Info-Physics, Central South University, Changsha 410083, China · NExT++ Research Centre, National University of Singapore, Singapore +2
Multimodal Floorplan Encoding: Learning Dense Modality-Invariant Representations
Sep 11, 2026Xavier Anadón, Rémi Pautrat, Rui WangLatent Space Alignment Cross-Modal Representation Learning
University of Zaragoza, Spain · Microsoft, Switzerland
Cross-modal learning for SAR target recognition using optical vision foundation models
Sep 7, 2026Lucas Hirsch, James R. Hopgood, Javid Khan +2Cross-Modal Alignment Cross-Modal Representation Transfer
School of Engineering, University of Edinburgh, United Kingdom · Leonardo UK, United Kingdom · School of Engineering and Physical Sciences, Heriot-Watt University, United Kingdom
CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space
Aug 31, 2026Paul Schneider, Nazim HaouchineCross-Modal Representation Learning Multimodal Disentangled Representation Learning
Harvard Medical School, Brigham and Women’s Hospital, Boston, MA, USA · Technical University of Munich, Munich, Germany
Giraffe: A Mapping Architecture from Hidden Text Representations to Visual Embeddings for Efficient Graphic Design
Aug 25, 2026Nejla GhaboosiCross-Modal Representation Transfer Multimodal Large Language Models
Canva Research
LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents
Aug 7, 2026Ivan Majic, Zexian Huang, Franziska Hübl +5Cross-Modal Alignment Multimodal Large Language Models
Graz University of Technology, Austria · University of Vienna, Austria · University of Liverpool, UK
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Aug 5, 2026Junlin Han, Shengbang Tong, David Fan +4Unified Multimodal Models Multimodal Pretraining
1FAIR · University of Oxford · 2Reality Labs
Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding
May 29, 2026Xiang Fang, Daizong Liu, Wanlong Fang +4Temporal Video Grounding Cross-Modal Representation Transfer
Hubei Key Laboratory of Distributed System Security, Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology · Peking University · Henan University +3
Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
Mind the Gap: Learning Modality-Agnostic Representations with a Cross-Modality UNet
May 16, 2026Xin Niu, Enyi Li, Jinchao Liu +3Cross-Modal Learning Cross-Modal Representation Learning
Tianjin Key Laboratory of Intelligent Robotics, College of Artificial Intelligence, Nankai University, China, and also with Engineering Research Center of Trusted Behavior Intelligence, Ministry of Education, Nankai University, China. · VisionMetric Ltd, Canterbury, Kent, UK. · Department of Computer Science, Haifa University, Israel.
Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
May 12, 2026Souptik Sen, Raneen Younis, Zahra AhmadiMultimodal Learning Representation Learning
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings
May 11, 2026Liuyin Yang, Qiang Sun, Bob Van Dyck +2Brain-Computer Interfaces Neural Decoding
Laboratory for Neuro- & Psychophysiology, Department of Neurosciences, KU Leuven
EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models
Apr 13, 2026Jincheng Xie, Xingchen Xiao, Runheng Liu +3Cross-Modal Learning Multimodal Embedding
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JD iCity, JD Technology, JD Intelligent Cities Research, Beijing, China
Improving Generalization on Cybersecurity Tasks with Multi-Modal Contrastive Learning
Mar 20, 2026Jianan Huang, Rodolfo V. Valentim, Luca Vassio +4Cross-Modal Representation Transfer Cybersecurity
Politecnico di Torino, Italy · Universit`a di Torino, Italy · Huawei Paris Research Center, France
MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization
Mar 13, 2026Chenyang Zhu, Hongxiang Li, Xiu Li +1Cross-Modal Representation Transfer T2I Generation
Tsinghua University · HKUST
Any2Any: Unified Arbitrary Modality Translation for Remote Sensing
Mar 4, 2026Haoyang Chen, Jing Zhang, Hebaixu Wang +7Remote Sensing Image Understanding Image-to-Image Translation
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University, and Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. 100094 · School of Electronic Information, Wuhan University, Wuhan, China +2
Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer
Feb 22, 2026Chenhang Cui, An Zhang, Yuxin Chen +5Cross-Modal Representation Transfer VLM Reasoning
National University of Singapore, Singapore · University of Science and Technology of China, Hefei, China · Nanyang Technological University, Singapore
Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
Feb 20, 2026Jingyang Qiao, Zhizhong Zhang, Xin Tan +3Cross-Modal Representation Transfer Multimodal Large Language Models
East China Normal University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +1

Oct 16, 2026J/K move · Enter open · S save

CIIR - Class-Indexed Interface Representation Learning

Doo Yon Kim, Christian Wallraven

Cross-Modal Representation Learning Cross-Modal Representation Transfer Multimodal Learning Cross-Modal Alignment

Open paper AskPDF arXiv

Multimodal learning commonly assumes paired or synchronized observations, yet independently collected datasets rarely provide such correspondence. We introduce CIIR, a modular encoder--decoder framework that connects heterogeneous datasets through a class-indexed address space while preserving each dataset's native input format and output module. A known local index selects address

k

, and an input-dependent 128-dimensional address representation Z_k(x) is written to that address. To demonstrate CIIR, we learn this address space from handwritten digits, clothing images, spoken-digit spectrograms, and hand signs without synchronized or instance-matched samples. Newly trained label-free receivers recover the requested addresses with 94.7% macro-average Address Realization Accuracy. We then attach the unseen SVHN dataset after freezing the base address space and train only lightweight adapters, achieving 90.2% MNIST>SVHN and 82.3% SVHN>MNIST end-to-end Address Transfer Accuracy without target-side decoder labels. To demonstrate generalizability of CIIR, we reuse the ten addresses as code elements in collision-free arithmetic and hierarchical address codes for all 100 CIFAR-100 classes and evaluate direct arithmetic-code realization through the existing receivers. These results establish CIIR as a compact interface for modular cross-dataset communication and post-hoc dataset attachment without exhaustive paired multimodal supervision.