Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs

Jihoo Jung; Youngjoon Jang; Joon Son Chung

cs.CVSep 25, 2026

Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs

Authors: Jihoo Jung, Youngjoon Jang, Joon Son Chung

Organizations: KAIST · VGG, University of Oxford

Abstract

Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues. In such videos, resolving "who says what" is crucial, which necessitates trimodal (text-audio-visual) binding. Motivated by these challenges, we systematically investigate how this trimodal binding is achieved in AVLLMs. Specifically, we identify emergent symbolic trimodal binding mechanisms in AVLLMs that utilize modality-specific symbolic variables. By encoding auditory and visual components into symbolic variables-capturing temporal utterance sequences and spatial entity coordinates, respectively-the model establishes cross-modal linking within this abstract space. Crucially, we reveal that when trimodal binding fails, the breakdown predominantly stems from misaligned audio-visual connections. To overcome this bottleneck, we introduce an audio-visual prompting method utilizing an off-the-shelf Active Speaker Detection (ASD) model. By simply overlaying visual bounding boxes on active speakers, this training-free approach yields immediate performance gains across four conversation-centric benchmarks. Moreover, lightweight fine-tuning of fewer than 300 steps on these ASD-prompted-videos extends these gains to three general AV benchmarks, suggesting the generalizability of our method.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Trimodal binding consists of three stages: anchor ID retrieval, target ID selection, and feature retrieval. (b) The vanilla model fails to accurately associate the speaker to their corresponding utterance, whereas the ASD method successfully establishes this connection.

Panel of Figure 2 — Figure 2: RSA results at anchor attribute and last prompt tokens for AAVR and VAAR tasks. At the anchor token, anchor attribute symbolic IDs strongly correlate in mid-to-late layers (anchor ID retrieval). At the last token, target attribute symbolic IDs correlate strongly in late layers (target ID selection), while semantic information peaks in the deepest layers (feature retrieval).

Figure 3: Illustration of CMA for the AAVR task using three manipulated contexts. We construct modified contexts $c_{2}$ via temporal ID, position ID, and semantic content manipulations, and illustrate the resulting patched run $c_{1}^{*}$ obtained by patching $c_{2}$ outputs into $c_{1}$ after the anchor ID retrieval, target ID selection, and feature retrieval stages, respectively.

Panel of Figure 4 — Figure 4: CMA results at anchor attribute and last prompt tokens for AAVR and VAAR tasks. At the anchor token, anchor ID manipulation exhibit high CM scores in mid-to-late layers (anchor ID retrieval). At the last token, target ID manipulation exhibit high CM scores in late layers (target ID selection), while semantic content manipulation peaks in the deepest layers (feature retrieval).

Panel of Figure 5 — Figure 5: RSA results in the VAAR task for a real-world dataset. The symbolic trimodal binding mechanism generalizes to real-world settings. Position IDs exhibit strong correlations at the anchor token (anchor ID retrieval), whereas temporal IDs dominate at the last token (target ID selection).

Panel of Figure 6 — Figure 6: Target ID selection stage identified as the locus of failure. (a) The primed-unprimed representational gap peaks at the target ID selection stage in AAVR task. (b) Patching correct symbolic IDs at target ID selection stage significantly improves accuracy.

Model	Converse-Centric					General
	AVSpeaker [ 36 ]	DailyOmni [ 63 ]	SocialOmni [ 52 ]	DiaDemBench [ 7 ]		OmniBench [ 31 ]	DAVE [ 41 ]	WorldSense [ 23 ]
	Acc(%) $\uparrow$	Acc(%) $\uparrow$	Acc(%) $\uparrow$	REF $\uparrow$	ASR $\uparrow$	Acc(%) $\uparrow$	Acc(%) $\uparrow$	Acc(%) $\uparrow$
Qwen2.5-Omni [ 53 ] (7B)	44.86	64.33	38.95	19.1 $\pm$ 0.4	28.1 $\pm$ 0.1	50.79	30.75	50.06
+ ASD	45.59	64.66	41.90	22.5 $\pm$ 0.2	33.0 $\pm$ 0.2	50.79	30.75	50.06
+ ASD-FT	48.79	71.09	44.25	20.7 $\pm$ 0.2	32.3 $\pm$ 0.1	52.80	33.91	53.73
MiniCPM-o-4.5 [ 11 ] (9B)	49.41	61.99	46.05	6.3 $\pm$ 0.5	7.4 $\pm$ 0.7	47.99	55.80	55.73

Table 1: Results of the proposed methods. We evaluate three recent AVLLMs across four conversation-centric and three general benchmarks. For Qwen2.5-Omni and MiniCPM-o-4.5, training-free ASD improves on conversation-centric tasks (For general benchmarks, which often lack speech or visible speakers, we do not apply ASD to draw bounding boxes, making ASD identical to the vanilla setting.). ASD-FT improves bounding box grounding capabilities, yielding consistent performance gains across all three AVLLMs on all seven benchmarks.

Table 8

	Converse-Centric		General
Method	AV- Speaker	Daily- Omni	Omni- Bench	DAVE	World- Sense
Vanilla	44.86	64.33	50.79	30.75	50.06
+ FT	46.89	66.67	51.05	35.15	52.05
+ ASD-FT	48.79	71.09	52.80	33.91	53.73

Table 4: Ablation on finetuning methods. ASD-FT yields greater gains than baseline FT.

Panel of Figure 7 — Figure 7: Target ID selection after ASD-FT in the unprimed setting. Compared to the vanilla model (dashed), ASD-FT (solid) shows stronger correlation with the target ID.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Unprimed	Primed	Primed prompt only
video-SALMONN2+ (7B)	$27.0\%$	$99.2\%$	$1.00\%$
Qwen2.5-Omni (7B)	$29.6\%$	$100.0\%$	$12.50\%$
Qwen2.5-Omni (3B)	$28.7\%$	$99.94\%$	$12.50\%$
MiniCPM-o-4.5 (9B)	$0.0\%$	$99.31\%$	$0.06\%$

Appendix

Table 5: Accuracy across the four AVLLMs in three conditions: unprimed, primed, and primed prompt only. All values are averaged over

800

samples per model.

Panel of Figure 8 — Figure 8: RSA and CMA results for Qwen2.5-Omni(7B)

Panel of Figure 9 — Figure 9: RSA and CMA results for Qwen2.5-Omni(3B)

Panel of Figure 10 — Figure 10: RSA and CMA results for MiniCPM-o-4.5(9B)

Data	Token	AAVR			VAAR
Data	Token	Semantic	Position	Temporal	Semantic	Position	Temporal
5spk	Anchor	0.03@L20	0.07@L20	0.52@L18	0.15@L20	0.66@L18	0.17@L20
5spk	Last	0.83@L23	0.47@L20	0.27@L18	0.81@L23	0.32@L18	0.34@L20
6spk	Anchor	0.03@L20	0.06@L20	0.45@L18	0.09@L20	0.57@L18	0.10@L20
6spk	Last	0.65@L24	0.34@L20	0.22@L18	0.74@L23	0.32@L18	0.26@L20
5spk, 20dB	Anchor	0.04@L20	0.08@L20	0.52@L18	0.15@L20	0.66@L18	0.18@L20

Appendix

Table 6: RSA in five-/six-speaker settings (video-SALMONN2+, 7B). Each cell reports the peak correlation and its layer. Bold : highest among the three spaces; underline : second highest.

Panel of Figure 11 — Figure 11: RSA results primed vs unprimed settings for video-SALMONN2+(7B)

Panel of Figure 12 — Figure 12: RSA results primed vs unprimed settings for Qwen2.5-Omni(7B)

Panel of Figure 13 — Figure 13: RSA results primed vs unprimed settings for Qwen2.5-Omni(3B)

Panel of Figure 14 — Figure 14: RSA results primed vs unprimed settings for MiniCPM-o-4.5

Panel of Figure 15 — Figure 15: RSA results for video-SALMONN2+(7B) on real-world data.

Panel of Figure 16 — Figure 16: RSA results for Qwen2.5-Omni(7B) on real-world data.

Panel of Figure 17 — Figure 17: RSA results for Qwen2.5-Omni(3B) on real-world data.

Panel of Figure 18 — Figure 18: RSA results for MiniCPM-o-4.5(9B) on real-world data.

Panel of Figure 19 — Figure 19: Examples of the three video configurations in our training corpus. The active speaker at each moment is marked with a red bounding box.

# People	Shot type	# Videos
2	Single-shot	200
4	Single-shot	100
2	Multi-shot	100
Total		400

Appendix

Table 7: Video configurations in the training set. Each unique video yields multiple training samples by pairing it with different question types and target utterances.

Setting	AAVR	VAAR	Captioning	Total
Two people, single-shot	400	400	13	813
Four people, single-shot	400	400	13	813
Two people, multi-shot	200	200	14	414
Total	1,000	1,000	40	2,040

Appendix

Table 8: Distribution of training samples by video setting and question type. AAVR and VAAR are 2-way multiple choice for the two-people settings and 4-way multiple choice for the four-people setting; Captioning is open-ended.

Figure 20: Illustration of trimodal symbolic binding mechanisms in VAAR task.

Figure 21: $c_{2}$ =temporal ID manipulation, patching at the anchor ID retrieval stage

Figure 22: $c_{2}$ =position ID manipulation, patching at the anchor ID retrieval stage

Figure 23: $c_{2}$ =semantic content manipulation, Patching at the anchor ID retrieval stage

Figure 24: $c_{2}$ =temporal ID manipulation, patching at the target ID selection stage

Figure 25: $c_{2}$ =position ID manipulation, patching at the target ID selection stage

Figure 26: $c_{2}$ =semantic content manipulation, patching at the target ID selection stage

Figure 27: $c_{2}$ =temporal ID manipulation, patching at the feature retrieval stage

Figure 28: $c_{2}$ =position ID manipulation, patching at the feature retrieval stage

Figure 29: $c_{2}$ =semantic content manipulation, patching at feature retrieval stage

Panel of Figure 30 — Figure 30: Qualitative comparison on AVSpeaker [ 36 ] with Qwen2.5-Omni (7B). The vanilla model selects the wrong answer, while the ASD/ASD-FT variant answers correctly.

Figure 31: Qualitative comparison on a multi-speaker captioning example from DiaDemBench [ 7 ] with Qwen2.5-Omni (7B). The vanilla model produces speaker-speech mismatches, attributing utterances to the wrong characters in the scene. In contrast, the ASD variant generates a caption in which each utterance is consistently aligned with the correct speaker.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,092 papers

Do Audio-Visual Large Language Models Really See and Hear?
Apr 3, 2026Ramaneswaran Selvakumar, Kaousheik Jayakumar, S Sakshi +3Multimodal Large Language Models Audio-Visual Understanding
University of Maryland, College Park
Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition
Jul 10, 2026Xugang Lu, Peng Shen, Yu Tsao +1Automatic Speech Recognition Audio-Visual Speech Recognition
National Institute of Information and Communications Technology, Kyoto, Japan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan
Probing Cross-modal Information Hubs in Audio-Visual LLMs
May 11, 2026Jihoo Jung, Chaeyoung Jung, Ji-Hoon Kim +1Cross-Modal Learning Multimodal Large Language Models
Department of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea · The Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University, Seoul, Republic of Korea

Apr 3, 2026J/K move · Enter open · S save

Do Audio-Visual Large Language Models Really See and Hear?

Ramaneswaran Selvakumar, Kaousheik Jayakumar, S Sakshi, Sreyan Ghosh+2

University of Maryland, College Park

Multimodal Large Language Models Audio-Visual Understanding Multimodal Model Interpretability

Open paper AskPDF arXiv

Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM's audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.

Abstract

Multimodal Large Language Models

Explore similar work

Do Audio-Visual Large Language Models Really See and Hear?

Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

Probing Cross-modal Information Hubs in Audio-Visual LLMs