Audio-Visual Understanding

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 260% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models
May 12, 2026Yuchen Deng, Zidang Cai, Hai-Tao Zheng +3Audio-Visual Understanding Efficient Multimodal Inference
Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory
Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs
May 12, 2026Chaeyoung Jung, Kyeongha Rho, Joon Son ChungAudio-Visual Understanding Efficient Multimodal Inference
Korea Advanced Institute of Science and Technology (KAIST)
TB-AVA: Text as a Semantic Bridge for Audio-Visual Parameter Efficient Finetuning
May 12, 2026Seongah Kim, Dinh Phu Tran, Hyeontaek Hwang +3Audio-Visual Understanding Audio-Visual Representation Learning
AI2 Lab, KAIST
Towards Conversational Medical AI with Eyes, Ears and a Voice
May 10, 2026Meet Shah, Jason Gusdorf, Anil Palepu +50Audio-Visual Understanding Healthcare
LightAVSeg: Lightweight Audio-Visual Segmentation
May 9, 2026Qing Zhong, Guodong Ding, Lingqiao Liu +3Audio-Visual Understanding Audio-Visual Alignment
College of Informatics, Huazhong Agricultural University, Wuhan, China · School of Computing, National University of Singapore, Singapore · School of Computer Science, Adelaide University, Australia +2
EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing
May 9, 2026Huilai Li, Xiaomeng Di, Ying Xing +3Audio-Visual Understanding Weakly Supervised Learning
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications, Beijing 100876, China
TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos
May 8, 2026Hengyi Feng, Hao Liang, Mingrui Chen +6Audio-Visual Understanding Multimodal Hallucination
University of Electronic Science and Technology of China · Peking University · Zhongguancun Academy +1
MMTB: Evaluating Terminal Agents on Multimedia-File Tasks
May 8, 2026Chiyeong Heo, Jaechang Kim, Junhyuk Kwon +4Computer-Use Agent Benchmarks Audio-Visual Understanding
GSAI, POSTECH · National AI Research Lab · Krafton AI +1
PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition
May 8, 2026Yuchen He, Jing ZhangAudio-Visual Understanding Referring Expression Segmentation
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China
Audio-Visual Intelligence in Large Foundation Models
May 5, 2026You Qin, Kai Liu, Shengqiong Wu +12Audio-Visual Understanding Audio-Video Generation
National University of Singapore · University of Oxford · University of Toronto +5
KARMA-MV: A Benchmark for Causal Question Answering on Music Videos
May 5, 2026Archishman Ghosh, Abhinaba Roy, Dorien HerremansAudio-Visual Understanding Video QA
AMAAI Lab, Singapore University of Technology and Design
Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning
May 3, 2026Xinmeng Xu, Haoran Xie, S. Joe Qin +3Audio-Visual Understanding Audio-Visual Representation Learning
Division of Artificial Intelligence, Lingnan University, Tuen Mun, Hong Kong SAR · School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, China · School of Mathematics, Physics and Computing, University of Southern Queensland, Toowoomba, Australia +1
OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder
May 2, 2026Detao Bai, Shimin Yao, Weixuan Chen +4Cross-Modal Learning Video Understanding
1Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology
Are DeepFakes Realistic Enough? Exploring Semantic Mismatch as a Novel Challenge
Apr 30, 2026Sharayu Nilesh Deshmukh, Kailash A. Hambarde, Joana C. Costa +2Audio-Visual Understanding Audio Deepfake Detection
Instituto de Telecomunicações, Universidade da Beira Interior, Portugal
Report of the 5th PVUW Challenge: Towards More Diverse Modalities in Pixel-Level Understanding
Apr 28, 2026Chang Liu, Henghui Ding, Nikhila Ravi +40Audio-Visual Understanding Video Object Segmentation
Omni-o3: Deep Nested Omnimodal Deduction for Deliberative Audio-Visual Reasoning
Apr 27, 2026Zhicheng Zhang, Wentao Gu, Weicheng Wang +5Audio-Visual Understanding Tree Search
Nankai University · Kuaishou Technology · Pengcheng Laboratory +1
Exploring Audio Hallucination in Egocentric Video Understanding
Apr 26, 2026Ashish Seth, Xinhao Mei, Changsheng Zhao +9Audio-Visual Understanding Hallucination in Language Models
Meta · University Of Maryland, College Park
APRVOS: 1st Place Winner of 5th PVUW MeViS-Audio Track
Apr 20, 2026Deshui Miao, Yameng Gu, Chao Yang +3Audio-Visual Understanding Video Object Segmentation
Pengcheng Laboratory · Harbin Institute of Technology · University of California at Merced
AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers
Apr 17, 2026Edson Araujo, Saurabhchand Bhati, M. Jehanzeb Mirza +5Cross-Modal Learning Audio-Visual Understanding
University of Tübingen, Germany · MIT, Cambridge MA, USA · IBM Research, USA +2
Qwen3.5-Omni Technical Report
Apr 17, 2026Qwen TeamMultimodal Grounding Audio-Visual Understanding
Qwen Team
AViS-Mamba: Adaptive Visual Steering of Audio State-Space Dynamics for Violence Detection
Apr 2, 2026Damith Chamalke Senadeera, Dimitrios Kollias, Gregory SlabaughAudio-Visual Understanding Audio-Visual Representation Learning
Queen Mary University of London, UK · Digital Environment Research Institute (DERI), London, UK
HumanOmni-Speaker: Identifying Who said What and When
Mar 23, 2026Detao Bai, Zhiheng Ma, Xihan WeiAudio-Visual Understanding Visual Speaker Recognition
Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology · Guangdong Provincial Key Laboratory of Computility Microelectronics +1
SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models
Mar 17, 2026Tianyu Xie, Jinfa Huang, Yuexiao Ma +11Audio-Visual Understanding Spoken Dialogue Systems
Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China · Department of Computer Science, University of Rochester, Rochester, NY, USA +2
DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression
Mar 15, 2026Bingzhou Li, Tao HuangAudio-Visual Understanding Efficient Multimodal Inference
Shanghai Jiao Tong University, Shanghai, China · Tongji University, Shanghai, China
Audio-Visual Continual Test-Time Adaptation without Forgetting
Feb 20, 2026Sarthak Kumar Maharana, Akshay Mehra, Bhavya Ramakrishna +2Cross-Modal Learning Source-Free Domain Adaptation
The University of Texas at Dallas · Dolby Laboratories
TAC: Timestamped Audio Captioning
Feb 17, 2026Sonal Kumar, Prem Seetharaman, Ke Chen +8Audio Captioning Temporal Audio Grounding
University of Maryland, College Park, USA · Adobe Research, USA · OpenAI, USA (work done while at Adobe).
TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
Feb 9, 2026Linli Yao, Yuancheng Wei, Yaojie Zhang +12Audio-Visual Understanding Video Captioning
School of Computer Science, Peking University · South China University of Technology · University of Electronic Science and Technology of China +3
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Audio-Language Model Evaluation Multimodal Grounding
Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
Jan 25, 2026Xilin Jiang, Qiaolin Wang, Junkai Wu +30Audio-Visual Understanding Multimodal Large Language Models
Columbia University · University of Washington · University of Illinois Urbana-Champaign +5
Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China

May 12, 2026, cs.AIJ/K move · Enter open · S save

OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models

Yuchen Deng, Zidang Cai, Hai-Tao Zheng, Jie Wang+2

Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory

Audio-Visual Understanding Efficient Multimodal Inference Audio Token Compression Multimodal Large Language Models Multimodal Token Compression Multimodal Foundation Models+2

Open paper AskPDF arXiv

Omnimodal large language models (Omni-LLMs) show strong capability in audio-video understanding, but their practical deployment remains limited by high inference cost of long video streams and dense audio sequences. Despite recent progress, existing compression methods for Omni-LLMs typically rely on fixed or native compression units, which can disrupt cross-modal correspondence and the complementary information required for audio-video reasoning, making it difficult to improve inference efficiency while stably preserving performance. To address this, we propose OmniRefine, a training-free two-stage framework for efficient audio-visual token compression in Omni-LLMs. First, Correspondence-Preserving Chunk Refinement refines native chunk boundaries into cross-modally aligned compression units through frame-audio similarity and dynamic programming. Second, Modality-Aware Cooperative Compression jointly compresses video and audio tokens within each refined unit to reduce redundancy while preserving critical evidence. Extensive experiments show that OmniRefine achieves a better efficiency-performance trade-off than strong baselines and maintains stable performance under lower compression ratios. On WorldSense, it still reaches 46.7% accuracy at a 44% token retention ratio, nearly matching the full-token baseline. The code and interface will be released to facilitate further research.

Momentum

Related topics

Latest papers 95