Cross-Modal Learning

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 127

All topics

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
May 11, 2026Xuanchen Li, Yuheng Lu, Chenrui Cui +6Cross-Modal Learning Multimodal QA
Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China +1
FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning
May 10, 2026Xing Han, Shravan Chaudhari, Tanvi Ranade +2Cross-Modal Learning Healthcare
Department of Computer Science Johns Hopkins University · Department of ECE Johns Hopkins University · Department of Computer Science Johns Hopkins University, Bayesian Health
SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning
May 10, 2026Kun Xiang, Terry Jingchen Zhang, Zirong Liu +15Cross-Modal Learning Multimodal Grounding
Sun Yat-sen University · ETH Zurich · Huawei Technologies Ltd. +1
Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval
May 8, 2026Mingyu Liu, Sihan Huang, Yijia Fan +4Cross-Modal Learning VLM Adaptation
1Sun Yat-sen University · 2Guangdong Province Key Laboratory of Information Security Technology, China · 3Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
May 8, 2026Florian Hönicke, Michael Günther, Andreas Koukounas +4Cross-Modal Learning Cross-Modal Representation Learning
Jina by Elastic
Anisotropic Modality Align
May 8, 2026Xiaomin Yu, Yijiang Li, Yuhui Zhang +8Cross-Modal Learning Cross-Modal Alignment
HKUST(GZ) · NUS · Stanford +2
Emergent Symbolic Structure in Health Foundation Models: Extraction, Alignment, and Cross-Modal Transfer
May 8, 2026Gajendra Katuwal, Advait Koparkar, Salar Abbaspourazad +2Cross-Modal Learning Representation Learning
Apple
Weather-Robust Scene Semantics with Vision-Aligned 4D Radar
May 8, 2026Kali Hamilton, Christoffer HeckmanCross-Modal Learning VLM Robustness
Autonomous Robotics and Perception Group at the University of Colorado Boulder, Boulder, CO 80309, USA.
Cross-Modal Navigation with Multi-Agent Reinforcement Learning
May 7, 2026Shuo Liu, Xinzichen Li, Christopher AmatoCross-Modal Learning Multi-Agent Collaboration
Khoury College of Computer Sciences Northeastern University Boston, MA, 02120
Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval
May 7, 2026Jun Li, Peifeng Lai, Xuhang Lou +5Cross-Modal Learning Evidential Deep Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China
Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images
May 6, 2026Aneesh Rangnekar, Joao Miranda, Natally Horvat +13Cross-Modal Learning Cross-Modality Medical Image Segmentation
Department of Medical Physics, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA · Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA · Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA +2
Direct Product Flow Matching: Decoupling Radial and Angular Dynamics for Few-Shot Adaptation
May 6, 2026Hongxu Chen, Yanghao Wang, Bowei Zhu +6Cross-Modal Learning Flow Matching
HKUST · USTC · Huawei Research
HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
May 6, 2026Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang +2Cross-Modal Learning Image Matching
Institute for Natural Language Processing, University of Stuttgart
To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition
May 6, 2026Yangchen Yu, Qian Chen, Jia Li +5Cross-Modal Learning Multimodal Robustness
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computing and Information Systems, Singapore Management University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore +1
Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
May 5, 2026Lingyi Hong, Jinglun Li, Xinyu Zhou +6Cross-Modal Learning Multimodal Robustness
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University
Structured Diffusion Bridges: Inductive Bias for Denoising Diffusion Bridges
May 3, 2026Eitan Kosman, Gabriele Serussi, Chaim BaskinCross-Modal Learning Cross-Modal Alignment
Ben-Gurion University of the Negev, Israel.
OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder
May 2, 2026Detao Bai, Shimin Yao, Weixuan Chen +4Cross-Modal Learning Video Understanding
1Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology
SplAttN: Bridging 2D and 3D with Gaussian Soft Splatting and Attention for Point Cloud Completion
May 2, 2026Zhaoyang Li, Zhichao You, Tianrui LiCross-Modal Learning Point Cloud Completion
School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China
Multimodal Data Curation Through Ranked Retrieval
May 1, 2026Pratyush Muthukumar, Harshil Kotamreddy, Sarah Amiraslani +4Cross-Modal Learning Cross-Modal Alignment
NVIDIA
Let ViT Speak: Generative Language-Image Pre-training
May 1, 2026Yan Fang, Mengcheng Lan, Zilong Huang +7Cross-Modal Learning Vision Transformer
1Beijing Jiaotong University · 2ByteDance · 3Nanyang Technological University
EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure
May 1, 2026Zihao Ding, Beining Wu, Jun HuangCross-Modal Learning Multimodal Federated Learning
Department of Electrical Engineering and Computer Science South Dakota State University Brookings, SD 57007, USA
Towards Customized Multimodal Role-Play
May 1, 2026Chao Tang, Jianzong Wu, Qingyu Shi +5Cross-Modal Learning Cross-Modal Alignment
1Peking University · 2Alibaba Group · 3Zhejiang University
Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration
May 1, 2026Chunlei Meng, Pengbin Feng, Rong Fu +7Cross-Modal Learning Multi-Agent Collaboration
The College of Intelligent Robotics and Advanced Manufacturing, Fudan University · University of Southern California · University of Macau +5
Borrowed Geometry: Cross-Distribution Head-Importance Fingerprints of Frozen Pretrained Gemma 4 31B
May 1, 2026Abay BektursunCross-Modal Learning Attention Head Analysis
Independent research
REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception
Apr 30, 2026Vincenzo Polizzi, David B. Lindell, Jonathan KellyCross-Modal Learning Event-Based Vision
University of Toronto, Robotics Institute · University of Toronto, Department of Computer Science
One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubness
Apr 30, 2026Hiroyuki Deguchi, Katsuki Chousa, Yusuke SakaiCross-Modal Learning Multimodal Robustness
NTT, Inc. · Nara Institute of Science and Technology
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
Apr 29, 2026Junwon You, Mihyun Jang, Sangwoo Mo +1Cross-Modal Learning Cross-Modal Representation Learning
1KAIST · 2POSTECH
VitaminP: cross-modal learning enables whole-cell segmentation from routine histology
Apr 26, 2026Yasin Shokrollahi, Karina B. Pinao Gonzales, Elizve N. Barrientos Toro +5Cross-Modal Learning Computational Pathology
Department of Translational Molecular Pathology, Division of Pathology and Laboratory Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. · Institute for Data Science in Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. · The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
Apr 26, 2026Zhen Ye, Xu Tan, Aoxiong Yin +8Cross-Modal Learning Audio-Video Generation
Hong Kong University of Science and Technology · Independent Researcher · Zhejiang University +3
Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization
Apr 24, 2026Hefeng Zhou, Xuan Liu, Sicheng Chen +7Cross-Modal Learning Multimodal Federated Learning
Shanghai Jiao Tong University · Hohai University · Shenzhen Univ of Advanced Technology

May 11, 2026, cs.AIJ/K move · Enter open · S save

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang+5

Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China+1

Cross-Modal Learning Multimodal QA Multimodal Hallucination Audio-Visual QA Multimodal CoT Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Audio and vision provide complementary evidence for audio-visual question answering, yet current audio-visual large language models may suffer from cross-modal interference: information from one modality misguides the interpretation of another, thereby inducing hallucinations. We attribute this issue to uncontrolled cross-modal interactions during intermediate reasoning. To mitigate this, we propose Separate First, Fuse Later (SFFL), an audio-visual reasoning framework designed to reduce cross-modal interference. SFFL enforces modality-specific chain-of-thought reasoning, producing separate audio and visual reasoning traces and integrating evidence for answering. We construct modality-preference labels via a data pipeline under different modality input settings. We use these labels as an auxiliary reward in reinforcement learning to encourage a instance-dependent preference for modality cues when answering. We further introduce a modality-specific reasoning mechanism that preserves modality isolation during the separated reasoning stage while enabling full access to cross-modal information at the evidence fusion stage. Experiments demonstrate consistent improvements in both accuracy and robustness, yielding an average relative gain of 5.16% on general AVQA benchmarks and 11.17% on a cross-modal hallucination benchmark.

Momentum

Related topics

Latest papers 127