Audio-Visual Understanding

Ask about this topic

Momentum

18 papers in the last four weeks, up 260% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 94

All topics
CardsList

  1. MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos

    Oct 6, 2026Souptik Sen, Zahra AhmadiMissing-Modality LearningAudio-Visual Understanding

    Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed), Hannover, Germany

  2. CueRator: Agentic Search for Symbolic Rules to Adapt Frozen Multimodal Encoders

    Oct 6, 2026Sunchan Park, Beomkwon Cho, Kyeongbo KongAudio-Visual UnderstandingAgentic Search

    Pusan National University

  3. OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

    Oct 1, 2026Haibo Wang, Jiteng Mu, Jialu Li +5Audio-Visual UnderstandingRL for Language Model Reasoning

    University of California, Davis · Adobe Research

  4. AVSD-Scenes: A Dataset for Audio-Visual Description of Urban Scenes

    Oct 1, 2026Dhanunjaya Varma Devalraju, Arshdeep Singh, Mark D. PlumbleyAudio-Visual UnderstandingCross-Modal Retrieval

    IIT Madras, India · King’s College London, United Kingdom

  5. Supervising Sound Localization by In-the-wild Egomotion

    Oct 1, 2026Anna Min, Ziyang Chen, Hang Zhao +1Audio-Visual UnderstandingDirection-of-Arrival Estimation

    Tsinghua University · University of Michigan · Shanghai Qi Zhi Institute

  6. OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning

    Sep 30, 2026Xingming Shui, Dapeng Chen, Bowei Liu +5Audio-Visual UnderstandingMultimodal Robustness

    Tsinghua Shenzhen International Graduate School, Tsinghua University · Independent Researcher

  7. Uncovering Ordinal-Matching Bias in Audio-Visual LLMs

    Sep 28, 2026Jihoo Jung, Youngjoon Jang, Hyebin Cho +2Audio-Visual UnderstandingMultimodal Large Language Models

    Korea Advanced Institute of Science and Technology, South Korea · University of Oxford, United Kingdom

  8. Exploring a Single Autoregressive LLM for Unified Target Speech Extraction across Synchronous and Asynchronous Cues

    Sep 24, 2026Wenxuan Wu, Shuhan Zhang, Shuai Wang +1Audio-Visual UnderstandingTarget Speaker Extraction

  9. Less is More: Encoder-only Audio-Visual Segmentation

    Sep 24, 2026Ilpo Viertola, Vladimir Iashin, Sophie Tötterström +1Audio-Visual UnderstandingEfficient ViTs

    Tampere University

  10. ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

    Sep 22, 2026Pu Wang, Hugo Van hammeAudio-Visual UnderstandingMultimodal Robustness

    KU Leuven, Department of Electrical Engineering, Leuven, Belgium

  11. TV-AudioRemover: Joint Text-Visual Guided Sound Removal with Multi-Task Hard-Mixture Curriculum

    Sep 22, 2026Xinyue Guo, Jianxuan Yang, Daiguo Zhou +4Audio-Visual UnderstandingAudio Source Separation

    MiLM Plus, Xiaomi Inc.

  12. Omni2Web: Benchmarking Audiovisual Website Development

    Sep 20, 2026Minghao Han, Zhenghao Xing, Xize Cheng +7Multimodal GroundingAudio-Visual Understanding

    FDU · Alibaba Token Hub, Alibaba Group · CUHK +2

  13. OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents

    Sep 20, 2026Ruixun Liu, Yuxuan Wang, Jiacheng Xie +10Audio-Visual UnderstandingSpatial Reasoning Benchmarks

    School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group +1

  14. E-AVI: Evidence-Grounded Multimodal Assessment for Automated Video Interviews

    Sep 17, 2026Haoshen Wang, Dongbo Che, Zeyi Xie +3Multimodal GroundingAudio-Visual Understanding

    The Hong Kong Polytechnic University · Independent Researcher

  15. AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models

    Sep 17, 2026Longyin Zhang, Parth Sakhare Mahendra, Chengwei Wei +4Temporal Video GroundingAudio-Visual Understanding

    Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Centre for Frontier AI Research (CFAR), A*STAR, Singapore

  16. G-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancement

    Sep 16, 2026Guo-Ruei Tseng, Hung-Shin Lee, Hsin-Min Wang +1Audio-Visual UnderstandingSpeech Enhancement

    Dept. of Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · Grad. Inst. of AI Interdisciplinary Applied Technology, National Taiwan Normal University, Taiwan · Inst. of Information Science, Academia Sinica, Taiwan

  17. Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?

    Sep 15, 2026Zhaoyang Wei, Zipeng Wang, Yushe Cao +10Audio-Visual UnderstandingVideo-Language Model Evaluation

    University of Chinese Academy of Sciences, China · Tsinghua University, China · Tencent, China

  18. Omni-Streaming Thinking

    Sep 14, 2026Enjun Du, Siyi Liu, Ziyu Zheng +4Audio-Visual UnderstandingStreaming Video Understanding

    1The University of Hong Kong · 2The Hong Kong University of Science and Technology (Guangzhou) · University of Sussex +2

  19. Realtime-Venus: A full-duplex interaction system with asynchronous delegation

    Sep 12, 2026Ant GroupAudio-Visual UnderstandingTool-Using Language Agents

    Venus Team, Ant Group · Tsinghua University

  20. Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

    Sep 9, 2026Killian Steunou, Yannis Tevissen, Mounîm A. El YacoubiAudio-Visual UnderstandingEfficient VLM Inference

    SAMOVAR Laboratory, T´el´ecom SudParis, Institut Polytechnique de Paris, Palaiseau, France · Moments Lab, Paris, France

  21. From Visual Cues to Spoken Narration: Rethinking Audio Description

    Sep 1, 2026Akshita Gupta, Aditya Arora, Federico Tombari +2Audio-Visual Understanding

    TU Darmstadt & hessian.AI, Darmstadt, Germany · Google Research · TU Munich

  22. TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film

    Aug 30, 2026Kaishuu Shinozaki-Conefrey, Olivier Pascaud, Robin Courant +3Audio-Visual UnderstandingVideo-Language Model Evaluation

    LIX, Ecole Polytechnique, IP Paris, Palaiseau, France · New York University, New York, NY, USA · Ecole nationale supérieure Louis-Lumière, Saint-Denis, France +1

  23. Towards Expert-level Medical AI for Real-time Video Consultations

    Aug 10, 2026Mahvish Nagda, Jihyeon Lee, Matthew Thompson +37Audio-Visual UnderstandingMedical VLMs

    Google Research · Google DeepMind

  24. Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework

    Aug 10, 2026Dongxu Ge, Shansong Liu, Cheng Gong +3Audio-Visual UnderstandingConditional Image Generation

    University of Science and Technology of China Hefei, China · Institute of Artificial Intelligence, China Telecom (TeleAI) Beijing, China · Northwest Polytechnical University Xi’an, China

  25. Agreement-Based Audio-Visual Segmentation:Champion Report for the MeViS-Audio Track in the 8th LSVOS Challenge

    Aug 10, 2026Yiwen Ren, Jianing Liu, Yingxin Wang +4Audio-Visual UnderstandingReferring Video Object Segmentation

    National 111 Project Base of Intelligent Information Processing

  26. SCoPE: Training-Free Audio-Visual Event Perception via Sparse Cross-Modal Prior Exchange

    Aug 8, 2026Jaemo Jeong, Junho Yoon, Hyunju Kim +1Audio-Visual Understanding

    KAIST

  27. AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

    Aug 7, 2026Mingyang Wu, Kaituo Feng, Bohao Li +3Audio-Visual UnderstandingGroup Relative Policy Optimization

    MMLab, CUHK · Peking University

  28. OmniReasoner: Thinking with Long Audio-Video via Native Tool Use

    Jul 21, 2026Yu Chen, Caorui Li, Ziyu Xiong +8Temporal Video GroundingAudio-Visual Understanding

    University of Chinese Academy of Sciences · Institute of Automation, CAS · Southeast University +3

  29. Efficient Audio-Visual Event Recognition via Knowledge Distillation and Dynamic INT8 Quantization of a Hybrid Cross-Attention Network

    Jul 18, 2026Parinaz Binandeh Dehaghani, Danilo Pena, A. Pedro AguiarAudio-Visual UnderstandingNeural Network Compression

    University of Porto, Porto, Portugal · ResoSight, Montreal, Canada

  30. Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

    Jul 17, 2026Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar +19Audio-Visual UnderstandingMultimodal Large Language Models

    1NVIDIA, USA · University of Maryland, USA

  31. Video = World + Event Stream

    Jul 16, 2026Lianghua Huang, Zhi-Fan Wu, Yupeng Shi +24Audio-Visual UnderstandingStreaming Video Understanding

    Wan Team, Alibaba Group

  32. AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning

    Jul 14, 2026Yanghai Wang, Jiahao Wang, Jiafu Tang +9Audio-Visual UnderstandingAudio-Visual Alignment

    NJU-LINK Team, Nanjing University · Kling Team, Kuaishou Technology · Institute of Automation, Chinese Academy of Sciences

  33. Empowering Long-form Omni-modal Understanding with Robust Audio Perception

    Jul 11, 2026Kaiying Yan, Luoyi Sun, Xiao Zhou +1Audio-Visual UnderstandingMultimodal Pretraining

    SAI, Shanghai Jiao Tong University, shanghai, China · Zhejiang University, Hangzhou, Zhejiang, china · Shanghai AI Lab, shanghai, China

  34. Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas

    Jul 2, 2026Yuxuan Li, Lingxi Xie, Xinyue Huo +6Audio-Visual UnderstandingSpeaker Diarization

    Tsinghua University, China · Huawei Inc., China · University of Chinese Academy of Sciences, China +1

  35. Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning

    Jul 2, 2026Chen Zhao, Jiajun Ma, Qilong Huang +6Audio-Visual UnderstandingVideo Captioning

    Meituan, China · Nanjing University, State Key Laboratory for Novel Software Technology, China

  36. MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

    Jun 29, 2026Yuxuan Fan, Gyusik Seo, Jing Hao +3Audio-Visual UnderstandingMultimodal Large Language Models

    1NTU Singapore · 2The University of Hong Kong · 3Johns Hopkins University +2

  37. Agent-Computer Observation Interfaces Enable Dynamic Computer Use

    Jun 28, 2026Bojie Li, Noah ShiAudio-Visual UnderstandingComputer-Use Agents

    Pine AI · University of Washington

  38. AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression

    Jun 23, 2026Yijing Chen, Wenhui Tan, Xiaoyi Yu +7Audio-Visual UnderstandingMultimodal Large Language Models

    Gaoling School of Artificial Intelligence, Renmin University of China · Huawei Inc.

  39. Unlocking Spatial Grounding in Large Audio-Visual Retrieval models

    Jun 22, 2026Hugo Malard, Michel Olvera, Sanjeel Parekh +3Multimodal GroundingAudio-Visual Understanding

    LTCI, Télécom Paris, Institut Polytechnique de Paris, France · Meta, Reality Labs Research · NVIDIA, France +1

  40. Towards Accurate and Robust Surveillance Roadside IVD via Trackletized Audio-Visual Reasoning

    Jun 21, 2026Xiwen Li, Xiaoya Tang, Bodong Zhang +1Audio-Visual UnderstandingVideo Surveillance

    Kahlert School of Computing, University of Utah · Scientific Computing and Imaging Institute, University of Utah · Department of Electrical and Computer Engineering, University of Utah

  41. Native Active Perception as Reasoning for Omni-Modal Understanding

    Jun 17, 2026Zhenghao Xing, Ruiyang Xu, Yuxuan Wang +8Audio-Visual UnderstandingActive Perception

    1The Chinese University of Hong Kong · †Work done during an internship at Qwen Team, Alibaba Group. · 2Shanghai Jiao Tong University +2

  42. MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild

    Jun 15, 2026Haotian Qi, Gabriel SkantzeMultimodal GroundingAudio-Visual Understanding

    Department of Speech Music and Hearing, KTH Stockholm, Sweden

  43. Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning

    Jun 9, 2026Zihan Meng, Dexiang Hong, Weidong Chen +3Audio-Visual UnderstandingMultimodal Token Compression

    University of Science and Technology of China

  44. Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization

    Jun 5, 2026Zhe Yang, Ruyi Zhang, Hongtao Chen +4Audio-Visual UnderstandingHeterogeneous GNNs

    Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China · Peng Cheng Laboratory, Shenzhen 518000, China · Harbin Institute of Technology Suzhou Research Institute, Suzhou 215000, China

  45. To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection

    Jun 4, 2026Erfan Loweimi, Mengjie Qian, Kate Knill +7Missing-Modality LearningAudio-Visual Understanding

    University of Cambridge, UK · Southwest Jiaotong University, China · University of Surrey, UK +1

  46. AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs

    Jun 1, 2026Yaoting Wang, Ziyi Zhang, Wenming Tu +10Audio-Visual UnderstandingMultimodal Robustness

    Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Huazhong University of Science and Technology · Shanghai Jiaotong University +5

  47. AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

    Jun 1, 2026Yaoting Wang, Yun Zhou, Zipei Zhang +1Audio-Visual UnderstandingInstance Segmentation

    Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China.

  48. SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models

    May 31, 2026Chenshuang Zhang, Kyeong Seon Kim, Chengxin Liu +1Audio-Visual UnderstandingMultimodal Large Language Models

    KAIST

  49. Head-Pose-Aware Visual Speech Recognition with FiLM Modulation

    May 30, 2026Matthew Kit Khinn Teng, Haibo Zhang, Takeshi SaitohAudio-Visual UnderstandingVisual Speaker Recognition

    Department of Artificial Intelligence, Kyushu Institute of Technology, Iizuka, 820-8502, Fukuoka, Japan.

  50. Sandboxed Coding Agents are Competitive Omni-modal Task Solvers

    May 30, 2026Dongping Chen, Xuanao Huang, Zhihan Hu +3Audio-Visual UnderstandingAI Coding Agents

    University of Maryland · MBZUAI

  51. Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning

    May 27, 2026Ke Xu, Yuhao Wang, Ziyang Cheng +3Audio-Visual UnderstandingActive Perception

    Shanghai Jiao Tong University

  52. O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding

    May 26, 2026Peiran Wu, Yunze Liu, Chi-Hao Wu +2Audio-Visual UnderstandingEfficient Multimodal Inference

    University of Bristol · Memories.ai Research · University of Central Florida

  53. StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering

    May 25, 2026Ming Xie, Zizheng Huang, Xudong Tan +6Audio-Visual UnderstandingStreaming Video Understanding

    Shanghai Innovation Institute · Fudan University · Nanjing University +2

  54. MTLLFM: Multimodal-Temporal Laughter Localization: UR-FUNNY-Temporal and SMILE-Temporal Benchmarks with an Adaptive Multimodal Fusion Model

    May 25, 2026Eyal Hanania, Nadav Kirsch, Daniel Arkushin +4Audio-Visual UnderstandingWeakly Supervised Learning

    WSC-Sports

  55. VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding

    May 21, 2026Haichen He, Jiayi Zhou, Sifeng Shang +3VLM EvaluationVideo Understanding

    Hong Kong Baptist University · GVC Lab, Great Bay University · S-Lab, Nanyang Technological University

  56. LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

    May 21, 2026Yifan Dai, Zhenhua Wu, Bohan Zeng +18Cross-Modal LearningAudio-Visual Understanding

    School of AI, Shanghai Jiao Tong University · 2Kling Team, Kuaishou Technology · 3Peking University +5

  57. OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding

    May 18, 2026Ruixiang Zhao, Jie Yang, Zijie Xin +4Audio-Visual UnderstandingStreaming Video Understanding

    Renmin University of China · WeChat Vision, Tencent Inc.

  58. Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks

    May 18, 2026Yajing Zhou, Xiangyu KongAudio-Visual UnderstandingVisual Spatial Reasoning

    College of Computer Science, Beijing Information Science and Technology University, No.55 Taihang Road, Changping District, Beijing, 102206, China

  59. IsoNet: Spatially-aware audio-visual target speech extraction in complex acoustic environments

    May 14, 2026Dinanath Padhya, Sajen Maharjan, Binita Adhikari +1Audio-Visual UnderstandingTarget Speaker Extraction

    Department of Electronics and Computer Engineering, Thapathali Campus, Institute of Engineering, Tribhuvan University, Kathmandu, Nepal

  60. When Vision Speaks for Sound

    May 13, 2026Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6Audio-Language Model EvaluationAudio-Visual Understanding

    dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1