Audio-Visual Understanding

Overview

Audio-visual understanding asks how AI can use sound and images or video together to identify speech, people, sound sources, events, spatial relations, and social interactions. Methods range from cross-attention and contrastive fusion to audio-visual segmentation, source localization, and multimodal language models that use temporal reasoning, memory, or tools. Benchmarks such as AVE, AVSBench, and VGGSound test event recognition and localization, while newer long-video and streaming evaluations probe evidence tracking across time. Models have improved on many task-specific benchmarks, but reliable audio use, temporal grounding, and robustness to noise, missing or conflicting modalities, and long contexts remain open challenges.

Latest papers

CardsList

161 papers13 papers in the last four weeks, down 62% on the four weeks before. 0.1% of all new papers.

  1. Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio

    Nov 17, 2026Siddharth Ravi, Anders R. Bargum, Hugo Anselme +5Multimodal Model EvaluationAudio-Visual Understanding

  2. Video-as-Experience: Learning Computer Use via Audio-Visual Understanding

    Nov 11, 2026Guangzhi Sun, Zhonghao Zhai, Yixuan Li +1Computer-Use AgentsLearning from Demonstration

  3. CineMME: From Dialogue Grounding to Narrative Reasoning in Multimodal Video Understanding

    Nov 11, 2026Jin Liu, Dawei Du, Yexiang Liu +7Video-Language Model EvaluationAudio-Visual Understanding

  4. Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

    Nov 6, 2026Junbo Zhang, Hang Su, Zhaofan Li +2Audio-Visual UnderstandingMultimodal Semantic Segmentation

  5. SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?

    Nov 4, 2026Peiran Wu, Fangfei Gou, Zhaoxi Mu +6Video-Language ModelsAudio-Visual Understanding

  6. Which One Is Sounding? Binaural Sounding Instance Disambiguation for Audio-Visual Segmentation

    Nov 4, 2026Haoze Ma, Yidi Li, Yu Ma +5Audio-Visual UnderstandingInstance Segmentation

  7. AVPS-360: A Spatial Audio-Visual Benchmark for Instance Segmentation and Tracking of Sound-Producing Persons

    Oct 29, 2026Yifu Sun, Parthasaarathy Sudarsanam, Tuomas Virtanen +1Audio-Visual UnderstandingInstance Segmentation

  8. AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360∘^\circ Videos

    Oct 25, 2026Qingyu Luo, Peng Zhang, Davide Berghi +2Spatial AudioAudio-Visual Understanding

  9. Sounding Surgical Instrument Localisation through Audio-Visual Correspondence

    Oct 23, 2026Dominic Behling, Jonas Hein, Matthias SeiboldSurgical Instrument TrackingAudio-Visual Understanding

  10. Beyond Visible Speakers: Context-Aware Active Speaker Detection with Speaker Registration

    Oct 23, 2026Pu Wang, Hugo Van hammeAudio-Visual Understanding

  11. Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports

    Oct 22, 2026Jiahao Meng, Qize Yang, Yuxuan Wang +4Deep Research AgentsAudio-Visual Understanding

  12. EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

    Oct 20, 2026Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi +17Audio-Visual UnderstandingAudio-Visual QA

  13. AVTime: Reinforcing Dense Audio-Visual Captions with Bidirectional Alignment

    Oct 19, 2026Mingyang Wu, Kaituo Feng, Bohao Li +3Dense Video CaptioningAudio-Visual Understanding

  14. Local Evidence, Global Failure: Diagnosing Speech-Vision Temporal Composition in Audio-Visual Language Models

    Oct 18, 2026Jaehong Myung, Seunghun Oh, Unsang ParkAudio-Visual UnderstandingMultimodal Large Language Models

  15. ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark

    Oct 16, 2026Joel Julin, Souraja Kundu, Liza Dahiya +6Multimodal Model EvaluationMultimodal Grounding

  16. AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

    Oct 15, 2026Yuruo Wu, Kaihang Pan, Kai Shen +5Video Generation EvaluationAudio-Video Generation

  17. EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment

    Oct 15, 2026Jiapeng Shi, Qiuxia Hou, Jiaqi Leng +5Video CaptioningAudio-Visual Understanding

  18. Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs

    Oct 15, 2026Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung +4Audio-Visual UnderstandingMultimodal Large Language Models

  19. Audio-Visual Dog Identification and Tracking in Real-World Videos

    Oct 14, 2026Aryahvishwa Babu, Kingsley Jin, Lillian Wang +5Animal Re-IdentificationAudio-Visual Understanding

  20. OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

    Oct 13, 2026Xuyang Liu, Shihao Wang, Xinyu Wei +5Video-Language Model EvaluationMultimodal Grounding

  21. EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

    Oct 13, 2026Songruo Chen, Yuqian Fu, Yu Li +7Egocentric Video UnderstandingEgocentric VQA

  22. SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention

    Oct 11, 2026Arnav Sharma, Anand Mishra, Renu RameshanAudio-Visual UnderstandingSlot Attention

  23. SIMPLE: Semantic Structure-Aligned Prompt Learning for Open-Vocabulary Audio-Visual Event Localization

    Oct 11, 2026Hanyu Xuan, Haoran Xu, Min Yin +7Audio-Visual UnderstandingPrompt Learning

  24. CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks

    Oct 11, 2026Mengzhe Geng, Max W. Y. Lam, Yuewen Cao +4Multimodal Model EvaluationAudio-Visual Understanding

  25. Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation

    Oct 8, 2026Kun Li, Sami Sebastian Brandt, Michael Ying YangAudio-Visual Understanding

  26. OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

    Oct 8, 2026Zhongyu Yang, Jiale Tao, Ruitao Chen +9Multimodal Model EvaluationAudio-Visual Understanding

  27. Perception Test 2026: Challenge Summary and Extension to City-scale Audio-Visual Reasoning

    Oct 8, 2026Fedor Kitashov, João Carreira, Shiry Ginosar +3Spatial Reasoning BenchmarksAudio-Visual Understanding

  28. Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

    Oct 8, 2026Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi +2Audio-Visual UnderstandingMultimodal Fusion

  29. FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs

    Oct 8, 2026Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh3D Spatial ReasoningAudio-Visual Understanding