Cross-Modal Representation Learning

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 136

All topics

Intervention-Aware Multiscale Representation Learning from Imaging Phenomics and Perturbation Transcriptomics
Apr 19, 2026Jiayuan Chen, Ruoqi Liu, Zishan Gu +1Cross-Modal Knowledge Distillation Cross-Modal Representation Learning
The Ohio State University
TriTS: Time Series Forecasting from a Multimodal Perspective
Apr 17, 2026Xiang AoLong-Term Time Series Forecasting Cross-Modal Representation Learning
School of Software Engineering, Beijing Jiaotong University Beijing 100044, China
MMGait: Towards Multi-Modal Gait Recognition
Apr 17, 2026Chenye Wang, Qingyuan Cai, Saihui Hou +2Cross-Modal Representation Learning Gait Analysis
School of Artificial Intelligence, Beijing Normal University
Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
Apr 17, 2026Irem Ulku, Erdem Akagündüz, Ömer Özgür TanrıöverMissing-Modality Learning Remote Sensing Image Segmentation
Department of Computer Engineering, Ankara University, Ankara, Türkiye · Department of Modeling and Simulation, Graduate School of Informatics, METU, Ankara, Türkiye
Sketch and Text Synergy: Fusing Structural Contours and Descriptive Attributes for Fine-Grained Image Retrieval
Apr 17, 2026Siyuan Wang, Hanchen Gao, Guangming Zhu +5Cross-Modal Representation Learning Fine-Grained Image Retrieval
Xidian University · First Aircraft Design Institute
Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
Apr 9, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Cross-Modal Representation Learning Unified Multimodal Models
HKUST · Tencent Hunyuan
NOUS: Video-Driven 3D Human Reaction Generation via Observation-Reaction Mutual Steering
Mar 20, 2026Yuan Zhou, Luanyuan Dai, Yongzhi Li +7Cross-Modal Representation Learning Human Motion Generation
College of Computing and Data Science, Nanyang Technological University, Singapore 639798 · School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China · School of Computer Science and Information Engineering (School of Artificial Intelligence), Hefei University of Technology, Hefei 230601, China +1
SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
Feb 19, 2026Lorenzo Caselli, Marco Mistretta, Simone Magistri +1Cross-Modal Learning Cross-Modal Representation Learning
University of Florence, Media Integration and Communication Center (MICC), Italy
Semantic Purification for Conditional Representation Learning
Feb 5, 2026Jiaquan Wang, Yan Lyu, Chen Li +1Disentangled Representation Learning Representation Learning
School of Computer Science and Engineering, Southeast University, China · Biomedicine Discovery Institute, Department of Biochemistry and Molecular Biology, Monash University, Australia
Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation
Feb 2, 2026Xinshun Wang, Peiming Li, Ziyi Wang +5Cross-Modal Representation Learning Unified Multimodal Models
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School · Sony R&D Center 3Nanyang Technological University
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Jan 31, 2026Yaohua Liu, Rong Fu, Amir H. Gandomi +2Representation Learning Cross-Modal Representation Learning
Chinese Medicine Guangdong Laboratory, Hengqin, Zhuhai 519031, China · Institute of Collaborative Innovation, University of Macau, Macau 999078, China · Faculty of Engineering & IT, University of Technology Sydney, Ultimo, NSW 2007, Australia +3
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
Aug 17, 2025Xuhui Zhan, Tyler DerrCross-Modal Learning Cross-Modal Representation Learning
Data Science Institute, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States. · Computer Science Department, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States.
Platonic Representations for Poverty Mapping: Unified Vision-Language Codes or Agent-Induced Novelty?
Aug 1, 2025Satiyabooshan Murugaboopathy, Connor T. Jerzak, Adel DaoudCross-Modal Representation Learning Multimodal Learning
AI and Global Development Lab, aidevlab.org · Institute for Analytical Sociology, Linköping University, Sweden · Institute of Computer Science, University Leipzig, Germany +1
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaCross-Modal Learning Multimodal IR
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg
A quantitative analysis of semantic information in deep representations of text and images
May 21, 2025Santiago Acevedo, Andrea Mascaretti, Riccardo Rende +3Representational Similarity Analysis Cross-Modal Representation Learning
Scuola Internazionale Superiore di Studi Avanzati (SISSA) · Universitat Pompeu Fabra (UPF) · Catalan Institute of Research and Advanced Studies (ICREA) and Universitat Pompeu Fabra (UPF)
Early Intervention for VFM-based Multimodal Medical Image Classification
Date pendingQijie Wei, Hailan Lin, Xirong LiCross-Modal Representation Learning Vision Foundation Model Adaptation
Renmin University of China, Beijing 100872, China

Apr 19, 2026, cs.CVJ/K move · Enter open · S save

Intervention-Aware Multiscale Representation Learning from Imaging Phenomics and Perturbation Transcriptomics

Jiayuan Chen, Ruoqi Liu, Zishan Gu, Ping Zhang

The Ohio State University

Cross-Modal Knowledge Distillation Cross-Modal Representation Learning AI-Assisted Scientific Research Drug Discovery

Open paper AskPDF arXiv

Microscopy-based phenotypic profiling is scalable for drug discovery but lacks the mechanistic depth of transcriptomics, which remains costly and scarce. Existing multimodal approaches either use images to support other modalities or naively align representations by sample identity, ignoring cell-type and dose variations in weakly paired data-limiting generalization to unseen interventions. In this paper, we introduce an intervention-aware distillation framework that leverages perturbational transcriptomics to guide image representation learning. A transcriptome-conditioned teacher integrates gene expression and intervention metadata to produce soft distributions over a chemistry-aware codebook organized by drug similarity. The teacher employs a fine-tuned single-cell foundation model to encode cell-type context and disentangle dose effects. An image-only student learns to predict these distributions from microscopy alone, distilling mechanistic knowledge while operating independently at test time. This design emphasizes intervention semantics rather than identity alignment and explicitly handles dose and cell-type mismatches. We provide theoretical guarantees showing that transcriptomic guidance tightens the risk bound for image-based prediction. On Cell Painting and RxRx datasets paired with L1000, our method significantly improves one-shot transfer to unseen interventions and drug-target gene discovery compared to self-supervised and alignment baselines.

Momentum

Related topics

Latest papers 136