Audio-Visual Understanding

Follow this topic Ask about this topic

Audio-visual understanding asks how AI can use sound and images or video together to identify speech, people, sound sources, events, spatial relations, and social interactions. Methods range from cross-attention and contrastive fusion to audio-visual segmentation, source localization, and multimodal language models that use temporal reasoning, memory, or tools. Benchmarks such as AVE, AVSBench, and VGGSound test event recognition and localization, while newer long-video and streaming evaluations probe evidence tracking across time. Models have improved on many task-specific benchmarks, but reliable audio use, temporal grounding, and robustness to noise, missing or conflicting modalities, and long contexts remain open challenges.

Latest papers

155 papers33 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers
Apr 17, 2026Edson Araujo, Saurabhchand Bhati, M. Jehanzeb Mirza +5Cross-Modal Learning Audio-Visual Understanding
University of Tübingen, Germany · MIT, Cambridge MA, USA · IBM Research, USA +2
Qwen3.5-Omni Technical Report
Apr 17, 2026Qwen TeamMultimodal Grounding Audio-Visual Understanding
Qwen Team
Why Your Tokenizer Fails in Information Fusion: A Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization
Apr 13, 2026Xiangyu Zhang, Benjamin John Southwell, Siqi Pan +3Audio Tokenization Multimodal Fusion
School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, Australia.
OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering
Apr 9, 2026Yiduo Jia, Muzhi Zhu, Hao Zhong +7Audio-Visual Understanding Multimodal Reasoning
Zhejiang University · Xiaomi Inc.
Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning
Apr 8, 2026Subin Park, Jung Uk KimAudio-Visual Understanding
Kyung Hee University
A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoning
Apr 5, 2026Tianle Chen, Deepti GhadiyaramAdversarial Attacks on Unified Multimodal Models Adversarial Attacks
Department of Computer Science, Boston University
Do Audio-Visual Large Language Models Really See and Hear?
Apr 3, 2026Ramaneswaran Selvakumar, Kaousheik Jayakumar, S Sakshi +3Multimodal Large Language Models Audio-Visual Understanding
University of Maryland, College Park
AViS-Mamba: Adaptive Visual Steering of Audio State-Space Dynamics for Violence Detection
Apr 2, 2026Damith Chamalke Senadeera, Dimitrios Kollias, Gregory SlabaughAudio-Visual Understanding Audio-Visual Representation Learning
Queen Mary University of London, UK · Digital Environment Research Institute (DERI), London, UK
Omni-MMSI: Toward Identity-attributed Social Interaction Understanding
Mar 31, 2026Xinpeng Li, Bolin Lai, Hardy Chen +5Multimodal Understanding Social Reasoning
University of Texas at Dallas · Georgia Institute of Technology · University of California, Santa Cruz +1
SIREN: Spatially-Informed Reconstruction of Binaural Audio with Vision
Mar 31, 2026Mingyeong Song, Seoyeon Ko, Junhyug NohSpatial Audio Audio-Visual Understanding
Ewha Womans University, Seoul, Korea
Counting Without Numbers and Finding Without Words
Mar 25, 2026Badri Narayana PatroAnimal Re-Identification Audio-Visual Understanding
Microsoft
SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts
Mar 24, 2026Khanh Binh Nguyen, Chae Jung ParkAudio-Visual Understanding Visual Prompt Learning
Deakin University, Australia · National Cancer Center, South Korea
HumanOmni-Speaker: Identifying Who said What and When
Mar 23, 2026Detao Bai, Zhiheng Ma, Xihan WeiAudio-Visual Understanding Visual Speaker Recognition
Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology · Guangdong Provincial Key Laboratory of Computility Microelectronics +1
LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs
Mar 19, 2026Keda Tao, Yuhua Zheng, Jia Xu +13Audio-Visual Understanding Long-Video Understanding
Zhejiang University · Westlake University · Ant Group +4
SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models
Mar 17, 2026Tianyu Xie, Jinfa Huang, Yuexiao Ma +11Audio-Visual Understanding Spoken Dialogue Systems
Xiamen University · University of Rochester · Nanyang Technological University +1
DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression
Mar 15, 2026Bingzhou Li, Tao HuangAudio-Visual Understanding Efficient Multimodal Inference
Shanghai Jiao Tong University, Shanghai, China · Tongji University, Shanghai, China
Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation
Mar 15, 2026Kai Peng, Yunzhe Shen, Miao Zhang +6Audio-Visual Understanding Multimodal Fusion
Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Yongri Piao, and Huchuan Lu are with Dalian University of Technology, Dalian 116024, China · Wei Ji is with the School of Medicine, Yale University, New Haven, CT 06520 USA · Jingjing Li is with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada
MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
Mar 14, 2026Arushi Goel, Sreyan Ghosh, Vatsal Agarwal +16Multimodal Understanding Multimodal Reasoning
NVIDIA, USA · University of Maryland, College Park, USA
RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding
Mar 10, 2026Muyi Sun, Yixuan Wang, Hong Wang +5Audio-Visual Understanding
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China. · Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology, Hong Kong, China. · Institute of Automation, Chinese Academy of Sciences, and also with the University of Chinese Academy of Sciences, Beijing, China
Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation
Mar 9, 2026Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn +1Audio-Visual Understanding Continual Learning
Purdue University, West Lafayette, IN 47906, USA
AV-Unified: A Unified Framework for Audio-visual Scene Understanding
Mar 6, 2026Guangyao Li, Xin Wang, Wenwu ZhuAudio-Visual Understanding Unified Multimodal Models
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China · Beijing National Research Center for Information Science and Technology, Beijing 100084, China
Interpretable Perception and Reasoning for Audiovisual Geolocation
Mar 5, 2026Yiyang Su, Xiaoming LiuAudio-Visual Understanding
Michigan State University, East Lansing, MI 48824, USA
Crab $^{+}$ : A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation
Mar 4, 2026Dongnuan Cai, Henghui Du, Chang Zhou +5Audio-Visual Understanding Unified Multimodal Models
Gaoling School of Artificial Intelligence, Renmin University of China · Institute of Artificial Intelligence of China Telecom (TeleAI) · AI Technology Center, Online Video Business Unit, Tencent PCG +2
Audio-Visual Continual Test-Time Adaptation without Forgetting
Feb 20, 2026Sarthak Kumar Maharana, Akshay Mehra, Bhavya Ramakrishna +2Cross-Modal Learning Source-Free Domain Adaptation
The University of Texas at Dallas, Richardson, TX 75080, USA · Dolby Laboratories, Inc., San Francisco, CA 94103, USA
TAC: Timestamped Audio Captioning
Feb 17, 2026Sonal Kumar, Prem Seetharaman, Ke Chen +8Audio Captioning Temporal Audio Grounding
University of Maryland, College Park, USA · Adobe Research, USA · OpenAI, USA (work done while at Adobe).
TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
Feb 9, 2026Linli Yao, Yuancheng Wei, Yaojie Zhang +12Audio-Visual Understanding Video Captioning
School of Computer Science, Peking University · South China University of Technology · University of Electronic Science and Technology of China +3
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Multimodal Understanding Multimodal Grounding
Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
Jan 25, 2026Xilin Jiang, Qiaolin Wang, Junkai Wu +30Audio-Visual Understanding Multimodal Large Language Models
Columbia University · University of Washington · University of Illinois Urbana-Champaign +5
Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
Taming Modality Entanglement in Continual Audio-Visual Segmentation
Oct 20, 2025Yuyang Hong, Qi Yang, Tao Zhang +5Audio-Visual Understanding Replay-Based Continual Learning
School of Artificial Intelligence, UCAS · 2MAIS, Institute of Automation · School of Intelligent Science and Technology, University of Science and Technolog Beijing

Apr 17, 2026J/K move · Enter open · S save

AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers

Edson Araujo, Saurabhchand Bhati, M. Jehanzeb Mirza, Brian Kingsbury+4

University of Tübingen, Germany · MIT, Cambridge MA, USA · IBM Research, USA+2

Cross-Modal Learning Audio-Visual Understanding Multimodal Reasoning Multimodal Instruction Tuning Multimodal Understanding

Open paper AskPDF arXiv

Recent advances in reasoning models have shown remarkable progress in text-based domains, but transferring those capabilities to multimodal settings, e.g., to allow reasoning over audio-visual data, still remains a challenge, in part because of the limited availability of high-quality reasoning data in targeted multimodal combinations. To address this problem, we introduce AVRT, a novel framework that generates high-quality audio-visual reasoning traces from single-modality teacher models. We generate independent vision- and audio-reasoning traces via models specialized to reason over their respective modalities and merge the resulting traces with an LLM merger model. The resulting multimodal traces are used in a supervised fine-tuning (SFT) cold start to adapt the target model to audio-visual reasoning traces first, before training it in a second reinforcement learning stage on larger-scale data. Evaluated on seven audio-visual and audio benchmarks, our 3B and 7B parameter models achieve state-of-the-art results among models of comparable size including OmniBench and DailyOmni for audio-visual and MMAR for audio-only reasoning, showing that cross-modal training also transfers to single-modality tasks and establishing a new training pipeline for multimodal reasoning models.