Cross-Modal Attention

Ask about this topic

Overview

Cross-modal attention studies how information from one modality—such as text, images, audio, video, or sensor data—can guide the representation of another. It matters for tasks including visual question answering, retrieval, medical diagnosis, speech separation, and autonomous systems, where useful evidence may be distributed across inputs. Common approaches include bidirectional or asymmetric cross-attention, query- and register-based fusion, and sparse, local, or temporally guided attention, sometimes combined with confidence gating or feature disentanglement. Performance has improved across many benchmarks, but reliable fusion remains difficult when modalities are misaligned, unevenly informative, or prone to dominating one another; attention patterns also do not always guarantee faithful grounding or reasoning.

Latest papers

CardsList

90 papers9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

  1. Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss

    Apr 25, 2026Meizhu Liu, Matthew Rowe, Amit Agarwal +8Multimodal RobustnessMultimodal Embedding

  2. DARC-CLIP: Dynamic Adaptive Refinement with Cross-Attention for Meme Understanding

    Apr 25, 2026Qiyuan JinVLM AdaptationHate Speech Detection

    The Hong Kong University of Science and Technology

  3. CMTM: Cross-Modal Token Modulation for Unsupervised Video Object Segmentation

    Apr 16, 2026Inseok Jeon, Suhwan Cho, Minhyeok Lee +6Video Object SegmentationCross-Modal Attention

    Yonsei University, Republic of Korea

  4. CoFusion: Multispectral and Hyperspectral Image Fusion via Spectral Coordinate Attention

    Apr 12, 2026Baisong LiMulti-Source Image FusionHyperspectral Image Super-Resolution

    School of Integrated Circuits, Tsinghua University, Beijing 100084, China

  5. Descriptor-Injected Cross-Modal Learning: A Systematic Exploration of Audio-MIDI Alignment via Spectral and Melodic Features

    Apr 11, 2026Mariano Fernández MéndezCross-Modal RetrievalCross-Modal Attention

    Asociación Civil AlterMundi, Córdoba, Argentina

  6. A Dual Cross-Attention Graph Learning Framework For Multimodal MRI-Based Major Depressive Disorder Detection

    Apr 11, 2026Nojod M. Alotaibi, Areej M. AlhothaliDepression DetectionMultimodal Fusion

    Department of Computer Science, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia

  7. MSCT: Differential Cross-Modal Attention for Deepfake Detection

    Apr 9, 2026Fangda Wei, Miao Liu, Yingxue Wang +3Deepfake DetectionMultimodal Deepfake Detection

    Beijing Institute of Technology, China · China Academy of Electronics and Information Technology, China

  8. Multi-modal user interface control detection using cross-attention

    Apr 8, 2026Milad Moradi, Ke Yan, David Colwell +2Multimodal FusionCross-Modal Attention

    AI Research Lab, Tricentis, Vienna, Austria · AI Research Lab, Tricentis, Sydney, Australia · Institute of Artificial Intelligence, Center for Medical Statistics, Informatics, and Intelligent Systems, Medical University of Vienna, Vienna, Austria

  9. Video-guided Machine Translation with Global Video Context

    Apr 8, 2026Jian Chen, JinZe Lv, Zi Long +1Machine TranslationCross-Modal Attention

    Shenzhen University · Shenzhen Technology University

  10. Attention at Rest Stays at Rest: Breaking Visual Inertia to Mitigate Relation Hallucinations

    Apr 2, 2026Boyang Gong, Yu Zheng, Fanye Kong +2Emotional Hallucination in VLMsVLM Hallucination Mitigation

    Tsinghua University

  11. CGCMA: Conditionally-Gated Cross-Modal Attention for Event-Conditioned Asynchronous Fusion

    Apr 1, 2026Yunxiang GuoMultimodal FusionCross-Modal Attention

    Independent Researcher, China

  12. Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations

    Mar 31, 2026Ni Ou, Zhuo Chen, Xinru Zhang +1Lidar-Camera Extrinsic CalibrationCamera Calibration

    Ni Ou, Junzheng Wang are with the School of Automation, Beijing Institute of Technology, Beijing, 100081, China. · Zhuo Chen is with the Robot Perception Lab, Centre for Robotics Research, Department of Engineering, King’s College London, London WC2R 2LS, United Kingdom. · Xinru Zhang is with the School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, 100081, China.

  13. A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data

    Mar 20, 2026Aram Ansary Ogholbake, Hannah Choi, Spencer Brandenburg +10Medical Image ClassificationMultimodal Classification

  14. Improving Joint Audio-Video Generation with Cross-Modal Context Learning

    Mar 19, 2026Bingqi Ma, Linlong Lang, Ming Zhang +5Audio-Video GenerationMultimodal Diffusion Models

    Vivix Group Limited

  15. PC-CrossDiff: Point-Cluster Dual-Level Cross-Modal Differential Attention for Unified 3D Referring and Segmentation

    Mar 18, 2026Wenbin Tan, Jiawen Lin, Fangyong Wang +43D Referring Segmentation3D Visual Grounding

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University, Xiamen, China · School of Big Data, Tongren University, Tongren, China · Hanjiang National Laboratory, Wuhan, China +2

  16. Multimodal Connectome Fusion via Cross-Attention for Autism Spectrum Disorder Classification Using Graph Learning

    Mar 16, 2026Ansar Rahman, Hassan Shojaee-Mend, Sepideh HatamikiaMultimodal Graph LearningMultimodal Fusion

    Department of Medicine, Danube Private University (DPU), Krems, Austria · Department of Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria · Department of Medical Informatics, Faculty of Medicine, Gonabad University of Medical Sciences, Gonabad, Iran +1

  17. Bidirectional Cross-Attention Fusion of High-Resolution RGB and Low-Resolution Hyperspectral Inputs for Multimodal Semantic Segmentation

    Mar 14, 2026Jonas V. Funk, Lukas Roming, Andreas Michel +4Multimodal Semantic SegmentationSemantic Segmentation

    KIT, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany · Fraunhofer IOSB, Fraunhofer Institute of Optronics, System Technologies and Image Exploitation, 76131 Karlsruhe, Germany

  18. Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modeling

    Mar 12, 2026Junhyeong Byeon, Jeongyeol Kim, Sejoon LimMultimodal Emotion RecognitionFacial Expression Recognition

    Kookmin University, Republic of Korea

  19. Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition

    Mar 11, 2026Inyong Koo, yeeun Seong, Minseok Son +2Multimodal Emotion RecognitionAudio-Visual Alignment

    School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea · Graduate School of Green Growth and Sustainability, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

  20. SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving

    Mar 9, 2026Zihan You, Hongwei Liu, Chenxu Dang +4VLMs for Autonomous DrivingVision-Language-Action Models

    Institute for AI Industry Research (AIR), Tsinghua University · School of Instrument Science and Engineering, Southeast University · Zhili College, Tsinghua University +3

  21. Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout

    Mar 9, 2026Jun Yu, Naixiang Zheng, Guoyuan Wang +5Multimodal Emotion RecognitionFacial Expression Recognition

    University of Science and Technology of China · Unisound AI Technology Co., Ltd.

  22. CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

    Feb 23, 2026Chunlei Meng, Guanhong Huang, Rong Fu +3Cross-Modal Representation LearningMultimodal Learning

    Fudan University · Shantou University · University of Macau +1

  23. Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis

    Feb 23, 2026Chunlei Meng, Jiabin Luo, Zhenglin Yan +4Multimodal Sentiment AnalysisMultimodal Disentangled Representation Learning

    Fudan University · Peking University · University of Macau

  24. Retrieval Augmented Enhanced Dual Co-Attention Framework for Target Aware Multimodal Bengali Hateful Meme Detection

    Feb 22, 2026Raihan Tanvir, Md. Golam Rabiul AlamHate Speech DetectionLow-Resource Language Processing

    Department of Computer Science and Engineering, BRAC University, Dhaka 1212, Bangladesh · Department of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka 1208, Bangladesh

  25. HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis

    Feb 18, 2026J. Dhar, M. K. Pandey, D. Chakladar +4Multimodal FusionMultimodal Learning

    Indian Institute of Technology Ropar, India · RoentGen Health, India · Lulea University of Technology, Sweden +4

  26. VIPA: Visual Informative Part Attention for Referring Image Segmentation

    Feb 16, 2026Yubin Cho, Hyunwoo Yu, Kyeongbo Kong +3Referring Image SegmentationCross-Modal Attention

    School of Artificial Intelligence, Sogang University · AI Lab of CTO division, LG Electronics · School of Electronic Engineering, Sogang University +2

  27. Hierarchical Vision-Language Interaction for Facial Action Unit Detection

    Feb 16, 2026Yong Li, Yi Ren, Yizhe Zhang +5Facial Action Unit DetectionVision-Language Models

    the School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 210096, China. · the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China. · Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China. +1

  28. GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

    Nov 2, 2025Shijie Zhou, Viet Dac Lai, Hao Tan +4Multimodal GroundingGUI Grounding

    University at Buffalo, USA · Adobe Research, USA

  29. Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

    Oct 3, 2025Kaisi Guan, Xihua Wang, Zhengfeng Lai +5Audio-Video GenerationVideo Diffusion Models

    1Renmin University of China · 2Apple

  30. NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding

    Aug 6, 2025Zelin Peng, Yichen Zhao, Yu Huang +5Cross-Modal LearningVLM Adaptation

    MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Department of Radiology, The First Affiliated Hospital, School of Medicine, Zhejiang University · Mohamed bin Zayed University of Artificial Intelligence +1