Cross-Modal Learning

Follow this topic Ask about this topic

Cross-modal learning studies how models can connect information from different data types, such as images and text, audio and video, or medical images and clinical records. Common approaches learn shared embeddings with contrastive objectives, exchange information through attention or prompts, or transfer knowledge using translation, distillation, and optimal transport. These methods support tasks including retrieval, multimodal reasoning, diagnosis, forecasting, and robotics, especially when one modality is incomplete or unreliable. A central challenge is balancing shared semantics with modality-specific detail: stronger alignment can aid transfer but may erase useful distinctions or create interference, while limited paired data and uncertain evidence make evaluation difficult.

Latest papers

144 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Sounding Surgical Instrument Localisation through Audio-Visual Correspondence
Oct 23, 2026Dominic Behling, Jonas Hein, Matthias SeiboldSurgical Instrument Tracking Audio-Visual Understanding
Research in Orthopedic Computer Science (ROCS), Balgrist University Hospital, University of Zurich, Zurich, Switzerland. · Eberhard Karls Universität Tübingen, Germany.
More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training
Oct 14, 2026Haoyuan Wu, Aoqi Wu, Jiani Hu +1Multimodal Pretraining Cross-Modal Learning
The Chinese University of Hong Kong · Foundation Model Department, Tencent
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
Oct 7, 2026William Chen, Prem Seetharaman, Ke Chen +10Instruction-Based Video Editing Unified Multimodal Models
Adobe Research · Carnegie Mellon University
GCTAuto-encoder: A Cross modal Framework for Security Flaw Detection in IoT Networks
Oct 5, 2026Najmieh Sadat SafarabadiCross-Modal Learning IoT Intrusion Detection
Department of Computational Linguistics, University of Stuttgart, Stuttgart, Germany.
Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?
Sep 28, 2026Bohao Xing, Xin Liu, Kaishen Yuan +5Cross-Modal Learning Vision-Language Models
Lappeenranta-Lahti University of Technology LUT · Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou) +3
Cross-modal Translation via Conditional Latent Denoising for Video Deepfake Detection
Sep 27, 2026Xinzhe Li, Youzhi Tu, Kong Aik LeeCross-Modal Learning Cross-Modal Alignment
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR
Subject-Invariant Cross-Modal Decoding of Perceived Speech from Brain Recordings
Sep 25, 2026Aoke Zhang, Jing ChenNeural Decoding Brain-Computer Interfaces
School of Intelligence Science and Technology, Peking University, Beijing, China · Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China · National Key Laboratory of General Artificial Intelligence, Beijing, China
ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models
Sep 24, 2026Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud AbdarVLM Adaptation Visual Prompt Learning
University of Kurdistan, Sanandaj, Iran · Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · CHIRP, Child Health Research Centre, The University of Queensland, Brisbane, Australia
JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics
Aug 25, 2026Kehan Wen, Ziming Li, Siyuan Luo +1Latent Dynamics Modeling Joint-Embedding Predictive Architecture
Department of Electrical and Computer Engineering, NUS
CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery
Aug 24, 2026Mahir Shahriar Tamim, Sharjil Khan, Md. Samiul Alim +3Multimodal Understanding Cross-Modal Learning
Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh
Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning
Aug 7, 2026Chen Ling, Hanqian Li, Dongnan Liu +9Cross-Modal Learning Visual Reasoning
Zhejiang University · Ant Group · The Hong Kong University of Science and Technology (Guangzhou) +2
Deep Multimodal Fusion Detection through Spatial Mask and Channel Competition
Aug 3, 2026Guandi Wang, Ming Li, Yunsen Xing +1Cross-Modal Learning Cross-Modal Attention
KTH Royal Institute of Technology Stockholm, Sweden · University of Maryland, College Park, USA
Scaling Native Multimodal Pre-Training From Scratch
Jul 24, 2026Haoyuan Wu, Aoqi Wu, Hai Wang +3Cross-Modal Learning Multimodal Pretraining
The Chinese University of Hong Kong · LLM Department, Tencent
FlowMimic: Mask-free Visual Editing and Generation with Pixel-pair Warped Flow Field for Online Video Editing Data Generation and Modality Mimicry
Jul 20, 2026Dingyun Zhang, Lixue Gong, Wei LiuCross-Modal Learning Video Editing
1ByteDance
Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
Jul 16, 2026Kunal Pratap Singh, Ali Garjani, Rishubh Singh +6Cross-Modal Learning Cross-Modal Representation Learning
Swiss Federal Institute of Technology Lausanne (EPFL) · Apple
Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts
Jul 7, 2026Andrei-George Durdun, Victor Constantinescu, Radu Tudor IonescuCross-Modal Learning Sentiment Analysis
Department of Computer Science, University of Bucharest, 15G Iuliu Maniu, Bucharest 061075, Romania · Department of Data Science, PPC Romania, 30 Mircea Voda, Bucharest 030667, Romania
Learning When to Listen: Gated Affect Fusion for Human Motion Prediction
Jul 1, 2026Jingni HuangCross-Modal Learning Gated Multimodal Fusion
University of Oxford
MSNN-LINet: Cross-Modal Learning via Continuous Linear Integration
Jun 30, 2026Gabriel ClingerCross-Modal Learning Multimodal Fusion
The George Washington University
Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality
Jun 11, 2026Wei Li, Zhen Huang, Xinmei TianCross-Modal Learning Vision-Language Models
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Independent Researcher
Building The Ph(ysical)AI Layer Of Machine Intelligence
Jun 2, 2026Ulbert Jose Botero, Liam Smith, Brooks Olney +5Cross-Modal Learning Cross-Modal Representation Learning
MIT Lincoln Laboratory
When Tabular Foundation Models Transfer Across Modalities: A Systematic Evaluation Across 95 Datasets, 7 Modalities, and Two Regimes
Jun 1, 2026Julien LafranceCross-Modal Learning Tabular Foundation Models
Télécom Paris, Institut Polytechnique de Paris · Palaiseau, France
A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception
May 31, 2026Weitung Chen, Phil Tinn, Per Gunnar Auran +2Cross-Modal Learning Underwater Robotics
Massachusetts Institute of Technology (MIT), Cambridge, MA, USA · SINTEF, Trondheim, Norway · Department of Marine Technology, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
May 28, 2026Jusuk Lee, Seungjae Lee, Jonghun Shin +6Cross-Modal Learning Cross-Modal Representation Learning
Seoul National University · University of Maryland, College Park · Georgia Institute of Technology
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
May 28, 2026Feng Han, Zhixiong Zhang, Zheming Liang +2Cross-Modal Learning VLM Robustness
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment
May 28, 2026Tianchao Li, Shujian Yu, Xinrui Zu +4Cross-Modal Learning Cross-Modal Alignment
Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway +3
Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language
May 28, 2026Xiang Fang, Daizong Liu, Wanlong Fang +5Cross-Modal Learning Temporal Video Grounding
Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science of Technology Wuhan, China · Peking University · Henan University +4
AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling
May 28, 2026Yiheng Li, Zhuo Li, Ruibing Hou +4Cross-Modal Learning Unified Multimodal Models
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China · University of Chinese Academy of Sciences, China · Independent Author
ProgVLA: Progress-Aware Robot Manipulation Skill Learning
May 27, 2026Seungsu Kim, Jinyoung Choi, Seungmin Baek +1Cross-Modal Learning Efficient Multimodal Inference
NAVER LABS Europe · NAVER LABS
LAST: Bridging Vision-Language and Action Manifolds via Gromov-Wasserstein Alignment
May 27, 2026Huaihai Lyu, Chaofan Chen, Yuheng Ji +4Cross-Modal Learning Vision-Language-Action Models
1MAIS, Institute of Automation, Chinese Academy of Sciences. · 2Beijing Academy of Artificial Intelligence. · 3Peking University..
Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
May 26, 2026Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang +86Cross-Modal Learning Unified Multimodal Models

Oct 23, 2026J/K move · Enter open · S save

Sounding Surgical Instrument Localisation through Audio-Visual Correspondence

Dominic Behling, Jonas Hein, Matthias Seibold

Research in Orthopedic Computer Science (ROCS), Balgrist University Hospital, University of Zurich, Zurich, Switzerland. · Eberhard Karls Universität Tübingen, Germany.

Surgical Instrument Tracking Audio-Visual Understanding Cross-Modal Learning Weakly Supervised Learning

Open paper AskPDF arXiv

Surgical scene understanding is a prerequisite for context-aware and intelligent systems in the operating room. Current methods rely mostly on visual data, while acoustic sensing has shown potential to provide complementary information on surgical activity. First approaches for the localisation of surgical activity relied on microphone arrays and acoustic beamforming. This work investigates whether the active instrument can be localized from a single microphone and a camera by learning the correspondence between the appearance and the sound of surgical instruments, without annotating its position. We adapt weakly supervised audio-visual localisation, which learns from the co-occurrence of image and sound, to the surgical setting. We record simulated surgical actions, sawing, drilling, and chiseling, and train the model on these recordings without any spatial annotation to produce a heatmap over the sounding tool. We extend the training objective by two terms: a silence term, which suppresses the output for silence and background noise, and a composite term, which trains the model to select the sounding one of two visible instruments. The model localizes the active instrument on an independent corpus recorded with a different camera and microphone, in which instrument positions were obtained by optical tracking. It separates tool activity from silence and background noise and can attribute sound correctly when two tools are present. Audio-visual correspondence learned from in-domain recordings connects the sound of a surgical action to the instrument that produces it, using a single microphone and a camera. This proof of concept introduces a novel paradigm that enables the detection and localisation of surgical activity without manual annotations.