Do Audio-Visual Large Language Models Really See and Hear?

Ramaneswaran Selvakumar; Kaousheik Jayakumar; S Sakshi; Sreyan Ghosh; Ruohan Gao; Dinesh Manocha

cs.AIApr 3, 2026

Do Audio-Visual Large Language Models Really See and Hear?

Authors: Ramaneswaran Selvakumar, Kaousheik Jayakumar, S Sakshi, Sreyan Ghosh, Ruohan Gao, Dinesh Manocha

Organizations: University of Maryland, College Park

Abstract

Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM's audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.

Figures & tables

Figure 1: Illustration of visual bias. AVLLMs exhibit a critical modality bias, often prioritizing visual cues over vital audio cues. The diagram illustrates a counterfactual scene, visible objects (a blue car and a woman walking a dog) are silent and the only audible sound is an out-of-view ambulance siren. When prompted to describe the scene, the AVLLM hallucinates audio events (car engine, dog barking) and misses the actual siren sound.

Figure 2: Audio Understanding Performance. Audio understanding severely degrades under audio-visual conflict.

Figure 3: Caption evaluation. We use an open-source reasoning LLM to evaluate audio-visual captions by assessing temporal sequences, object attributes, and cross-modal relationships. This approach is interpretable (explicit reasoning for scores) and flexible, we can calibrate it using in-context examples.

Figure 4: Mean attention from generated to input tokens. Generated tokens allocate high attention to audio in early layers (40-50% in layers 0-5), which drops to near-zero afterward. Video attention steadily increases through deeper layers, reaching 20-40% in layers 15-30.

Figure 5: Probing Audio Representations. We decode intermediate layer audio representations using the base LLM’s unembedding matrix and observe that they decode into meaningful concepts describing sound events and their sources, in multiple languages (e.g., 键盘/keyboard, typing).

Figure 6: Blocking audio-visual information flow to generated tokens We block attention from generated tokens to vision ( G $\nrightarrow$ V ) or audio ( G $\nrightarrow$ A ) and measure impact on caption fidelity. 1) G $\nrightarrow$ A for factual samples does not degrade video understandingⒶ as expected (ideally audio cues should not be used for video understanding) 2) surprisingly audio understandingⒷ does not degrade either, (AVLLM instead correlates audio cues from vision). On the contrary, G $\nrightarrow$ V , especially in deeper layers, in counterfactual samples, improves audio understanding Ⓓ (misleading vision features dominate over audio in cross-modal integration to generated text)

Figure 7: Illustration of AVLLM captions and comparison with LVLMs. We compare token distributions between Qwen2.5Omni and Qwen2.5VL, and measure shift ( $\eta$ ) in token distribution unshifted tokens ( $\eta=1$ ), marginal tokens ( $1<\eta\leq 3$ ), and shifted tokens ( $\eta>3$ ) (Refer Sec 8 ). Left: The model strongly attends to visual cues and predicts “helicopter” sound event confidently asserting it’s ”clear and distinct.”. Notably, when the model correctly identifies “child speaking” from audio, those tokens shift in LVLM predictions, indicating geuine audio understanding. Right: the model hallucinates that it can ”hear a man speaking”. These outputs are largely unshifted compared to Qwen2.5VL demonstrating that the AVLLM’s predictions remain visually driven despite having access to audio.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Factual Sample A video where the visual content and audio content are highly correlated. The audio events in a factual sample can potentially be inferred by using visual cues

Figure 10: World Sense Task An example task from World Sense. These tasks couple perception and reasoning.

Figure 11: AVHBench Task An example task from AVHBench. These tasks evaluate perception capabilities.

Panel of Figure 12 — Figure 12: Comparison of Information Flow Blocking. Blocking audio-visual information flow, for both factual and counterfactual samples in VideoLlama 2.1 (top) versus MiniCPM-o2.6 (bottom).

Figure 13: The prompt used to measure audio caption fidelity.

Figure 14: The prompt used to measure video caption fidelity.

Figure 15: Qualitative examples . We illustrate some outputs generated by VideoLlama along with its LLM-judge score and reasoning, for factual and counterfactal samples and under attention knockout. The examples in the first row capture audio fidelity and the ones below capture video fidelity

Figure 16: Qualitative examples . We illustrate some outputs generated by MiniCPM-2.6-o along with its LLM-judge score and reasoning, for factual and counterfactal samples and under attention knockout. The examples in the first row capture audio fidelity and the ones below capture video fidelity

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
Jun 8, 2026Wish Suharitdamrong, Muhammad Awais, Xiatian Zhu +1Efficient Multimodal Inference Multimodal Large Language Models
Surrey Institute for People-Centred AI (PAI), University of Surrey, UK · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK
Probing Cross-modal Information Hubs in Audio-Visual LLMs
May 11, 2026Jihoo Jung, Chaeyoung Jung, Ji-Hoon Kim +1Cross-Modal Learning Multimodal Large Language Models
Department of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea · The Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University, Seoul, Republic of Korea
Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
May 11, 2026Xuanchen Li, Yuheng Lu, Chenrui Cui +6Cross-Modal Learning Multimodal QA
Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China · Tencent, China · Huiyan Technology Company, Ltd., Tianjin, China +1

Jun 8, 2026J/K move · Enter open · S save

From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs

Wish Suharitdamrong, Muhammad Awais, Xiatian Zhu, Sara Atito

Surrey Institute for People-Centred AI (PAI), University of Surrey, UK · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK

Efficient Multimodal Inference Multimodal Large Language Models Neural Network Interpretability Multimodal Model Interpretability Multimodal Understanding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) can listen and see, but how do audio and visual signals actually travel through the network to shape an answer? Despite their growing role in research and real-world applications, the internal pathways through which audio and visual tokens influence the final prediction remain poorly understood. In this study, we examine audio-visual information flow inside Audio-Visual Large Language Models (AVLLMs), tracing how AVLLMs route, utilize, and integrate audio and visual information across two input configurations, audio-visual video and multiple interleaved audio-visual items. We find that for audio-visual video, AVLLMs follow the sequential information flow pathway established for VLMs and VideoLLMs, with audio and visual contribution flowing along this pathway in proportion to the task's reliance on each modality. In settings with multiple interleaved audio-visual items, this routing shifts to different parallel streams. Furthermore, we demonstrate that audio-visual and other token types can be discarded once their information is transferred to LLM, with minimal impact on the model's prediction or even slight improvement, generalizing across multiple tasks and datasets, enabling more efficient inference. These findings hold across multiple models and scales, Qwen2.5-Omni and Video-SALMONN2 Plus at 3B and 7B scales, leading to hypotheses on why these flow structures emerge. Together, these results deliver the first coherent picture of how AVLLMs orchestrate sound and sight inside the network and lay the groundwork for the next wave of interpretability, design, and efficiency advances in audio-visual and broader MLLMs.

Abstract

Multimodal Large Language Models

Explore similar work

From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs

Probing Cross-modal Information Hubs in Audio-Visual LLMs

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought