Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs

Kim Sung-Bin; Lee Jung-Mok; Jinwoo Jung; Oh Hyun-Bin; Hyeonggon Ryu; David Harwath; Tae-Hyun Oh

cs.CVOct 15, 2026

Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs

Authors: Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung, Oh Hyun-Bin, Hyeonggon Ryu, David Harwath, Tae-Hyun Oh

Organizations: POSTECH · KAIST · Hankuk University of Foreign Studies · The University of Texas at Austin

Abstract

Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.

Figures & tables

Figure 1: Overview of HLR-AVSceneQA. We introduce HLR-AVSceneQA, a comprehensive benchmark for spatial audio-visual scene understanding organized around a hear-localize-reason hierarchy. For a queried sound source, the model recognizes what is heard, localizes where it originates in egocentric and allocentric views, and uses the localized source together with visual scene evidence to reason about how it relates to the surrounding scene.

Benchmark	A+V	Multi- source	Far-field ASR	Sound event	Ego.	Allo.	AV Corr.	Cross- room	Hidden Source	Proximity	AV Caption
BAT Zheng et al. (2024)	✗	✓	✗	✓	✓	✗	✗	✗	✗	✗	✗
SpatialWhisper Tang et al. (2024)	✗	✓	✓	✗	✓	✗	✗	✗	✗	✗	✗
Hear You Are Ryu et al. (2026)	✓	✗	✗	✓	✓	✗	✓	✗	✗	✗	✗
SAVVY-Bench Chen et al. (2025)	✓	✗	✗	✗	✓	✓	✗	✓	✗	✗	✗
SpatialSceneQA Liu et al. (2026)	✓	✓	✗	✗	✓	✗	✓	✗	✗	✗	✗
HLR-AVSceneQA	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓

Table 1: Evaluation scope of spatial audio and audio-visual benchmarks. We compare whether each benchmark explicitly evaluates key spatial audio-visual capabilities. A+V denotes audio-visual input; Multi-source denotes simultaneous sound sources; Far-field ASR and Sound event denote speech recognition from far-field audio and non-speech sound recognition; Ego./Allo. denote egocentric/allocentric reasoning; and Corr. denotes audio-visual correspondence. Cross-room, Hidden Source, Proximity, and AV Caption denote reasoning about across-room sources, invisible or occluded sources, nearest-object relations, and open-ended audio-visual captioning, respectively.

Task family	Subtask	Representative example
Spatial hearing	Sound event	Q : What sound do you hear? A : Hair dryer drying .
	Far-field ASR	Q : Write the exact phrase spoken. A : ‘‘Yes it is suppressed’’ .
Egocentric grounding	Direction	Q : From the current viewpoint, which direction is the sound coming from? A : front-left .
	Distance	Q : From the current viewpoint, how far is the sound source? A : 2.5--3 m .
Allocentric grounding	Direction	Q : If you stand at the sofa and face the TV, which direction is the sound coming from? A : back-left .
	Distance	Q : If you stand at the sofa and face the TV, how far is the sound source? A : 0--1.0 m .

Table 2: Representative QA examples in HLR-AVSceneQA. Most tasks use controlled answer formats, such as multiple choice or discretized bins; simplified prompts and answers are shown for clarity. The final row shows one CoT supervision example used for rationale-augmented training.

Figure 2: Examples of rendered scenes and metadata. Our dataset covers diverse scenarios that require spatially aware audio-visual reasoning, including (a) sound sources occluded by objects and (b) ambiguous sound sources among same-category instances. Each sample is summarized with structured scene metadata. Yellow denotes existing scene objects, pink denotes inserted objects, blue denotes sounding objects, and green denotes the agent.

Figure 3: Architecture and training curriculum of HLR-LLM. (a) HLR-LLM augments a strong AV-LLM with a binaural spatial audio branch, enabling the language model to jointly attend to text, visual, downmixed monaural audio, and binaural spatial audio tokens. (b) The model is trained through a three-stage curriculum: spatial audio alignment, spatial audio-visual instruction tuning, and final-stage CoT tuning for structured spatial reasoning.

Model	Modality	Spatial Hearing		Egocentric Grounding		Allocentric Grounding
Model	Modality	Sound (Acc. %)	Speech (WER $\downarrow$ )	Direction	Distance	Direction	Distance
Random chance		10.00	-	12.50	10.00	12.50	10.00
Zero-shot models
MiniCPM-o-v2.6 Yao et al. (2024)	V + M	35.26 $\pm$ 0.32	126.10 $\pm$ 5.51	12.83 $\pm$ 0.48	14.26 $\pm$ 0.66	11.92 $\pm$ 0.36	18.42 $\pm$ 0.51
VideoLLaMA2 Cheng et al. (2024)	V + M	54.54 $\pm$ 0.11	113.54 $\pm$ 0.82	12.15 $\pm$ 0.41	7.73 $\pm$ 0.05	15.84 $\pm$ 0.36	7.19 $\pm$ 0.17
Video-SALMONN 2 Tang et al. (2025)	V + M	32.20 $\pm$ 0.13	$\mathbf{13.06}$ $\pm$ 0.02	17.00 $\pm$ 0.22	7.62 $\pm$ 0.07	17.55 $\pm$ 0.20	6.98 $\pm$ 0.12

Table 3: Evaluation on spatial hearing and spatial grounding tasks. Sound event recognition and grounding tasks are reported in accuracy (%), while speech recognition is reported in WER (%, lower is better). V , M , and B denote visual input, monaural audio, and binaural audio, respectively. M L and M R denote independently encoded left- and right-channel audio inputs. Best and second-best mean results are shown in bold and underlined, respectively.

Model	Modality	Spatial Relational Reasoning (Acc. %)					Scene Captioning
Model	Modality	Visibility	Correspondence	Hidden source	Loudness	Proximity	CIDEr-D	METEOR	SPIDEr
Random chance		50.00	9.49	20.00	50.00	28.68	-	-	-
Zero-shot models
MiniCPM-o-v2.6 Yao et al. (2024)	V + M	75.20 $\pm$ 0.14	22.52 $\pm$ 1.09	11.65 $\pm$ 0.62	47.29 $\pm$ 0.48	26.83 $\pm$ 0.62	0.078 $\pm$ 0.004	0.173 $\pm$ 0.001	0.136 $\pm$ 0.002
VideoLLaMA2 Cheng et al. (2024)	V + M	68.37 $\pm$ 0.22	19.77 $\pm$ 0.14	21.25 $\pm$ 0.26	39.18 $\pm$ 0.20	25.52 $\pm$ 0.30	0.112 $\pm$ 0.002	0.131 $\pm$ 0.001	0.079 $\pm$ 0.000
Video-SALMONN 2 Tang et al. (2025)	V + M	75.56 $\pm$ 0.00	7.97 $\pm$ 0.22	22.00 $\pm$ 0.31	51.89 $\pm$ 0.00	30.10 $\pm$ 0.05	0.075 $\pm$ 0.001	0.094 $\pm$ 0.001	0.074 $\pm$ 0.000

Table 4: Evaluation on spatial relational reasoning and scene captioning tasks. Relational reasoning tasks are reported in accuracy (%). Scene captioning is evaluated with CIDEr-D, METEOR, and SPIDEr. V , M , and B denote visual input, monaural audio, and binaural audio, respectively. M L and M R denote independently encoded left- and right-channel audio inputs. Best and second-best mean results are shown in bold and underlined, respectively.

ID	Variant	Modality	Egocentric Grounding		Allocentric Grounding		Spatial Relational Reasoning
ID	Variant	Modality	Direction	Distance	Direction	Distance	Correspondence	Hidden source	Proximity
Modality ablation
(a)	Without vision	M + B	$74.72\pm 0.36$	$50.64\pm 0.54$	$21.51\pm 0.41$	$26.63\pm 0.62$	$57.11\pm 0.11$	$66.62\pm 0.21$	$34.13\pm 0.12$
(b)	Visual only	V	$31.20\pm 0.14$	$20.33\pm 0.49$	$28.11\pm 0.36$	$22.96\pm 0.22$	$26.71\pm 0.02$	$24.16\pm 0.43$	$37.09\pm 0.55$
(c)	Without binaural	V + M	$32.69\pm 0.26$	$30.92\pm 0.42$	$26.26\pm 0.18$	$27.73\pm 1.22$	$55.73\pm 0.07$	$51.24\pm 0.58$	$47.11\pm 0.27$
Training ablation

Table 5: Ablation study. We ablate input modalities and training strategies to validate the contribution of each component. All metrics are reported in accuracy (%). V , M , and B denote visual input, monaural audio, and binaural audio, respectively. Results are mean

\pm

standard deviation over three seeds. Best and second-best mean results are shown in bold and underlined, respectively. The full ablation is provided in Appendix B .

ID	Model	Pretrain	Sound	Direction	Distance	Visibility
(a)	Hear You Are LLM Ryu et al. (2026)	–	$38.66\pm 0.16$	$15.25\pm 0.57$	$\underline{34.87}\pm 0.35$	$62.35\pm 0.07$
(b)	Hear You Are LLM Ryu et al. (2026)	✓	$40.80\pm 0.76$	$\underline{16.15}\pm 0.36$	$26.93\pm 0.45$	$63.80\pm 0.48$
(c)	HLR-LLM	–	$\underline{44.59}\pm 0.13$	$16.13\pm 0.04$	$28.11\pm 0.04$	$\underline{69.67}\pm 0.18$
(d)	HLR-LLM	✓	$\mathbf{53.93}\pm 0.14$	$\mathbf{29.92}\pm 0.13$	$\mathbf{35.13}\pm 0.07$	$\mathbf{71.68}\pm 0.09$

Table 6: Real-world adaptation on STARSS23. “Pretrain” denotes HLR-AVSceneQA pretraining before STARSS23 fine-tuning. “Sound” denotes sound event recognition; direction, distance, and visibility are egocentric. All metrics are accuracy (%). Results are mean

\pm

standard deviation over three seeds. Best and second-best mean results are shown in bold and underlined, respectively.

Figure 4: Qualitative examples of spatial grounding. HLR-LLM correctly identifies the sound direction in a scene with multiple similar objects (left) and localizes a non-visible sound source as hidden behind the bed (right), outperforming the comparison models in the hidden-source example.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Variant	Modality	Spatial Hearing		Egocentric Grounding		Allocentric Grounding
ID	Variant	Modality	Sound (Acc. %)	Speech (WER $\downarrow$ )	Direction	Distance	Direction	Distance
Modality ablation
(a)	Without vision	M + B	$78.44\pm 0.11$	$14.62\pm 2.81$	$74.72\pm 0.36$	$50.64\pm 0.54$	$21.51\pm 0.41$	$26.63\pm 0.62$
(b)	Visual only	V	$29.16\pm 0.04$	$250.14\pm 15.37$	$31.20\pm 0.14$	$20.33\pm 0.49$	$28.11\pm 0.36$	$22.96\pm 0.22$
(c)	Without binaural	V + M	$80.57\pm 0.27$	$15.33\pm 2.33$	$32.69\pm 0.26$	$30.92\pm 0.42$	$26.26\pm 0.18$	$27.73\pm 1.22$
(d)	Without monaural	V + B	$80.81\pm 0.03$	$361.05\pm 4.00$	$77.87\pm 0.14$	$47.78\pm 0.49$	$\underline{50.23}\pm 0.41$	$37.95\pm 0.24$

Appendix

Table S1: Ablation study on spatial hearing and spatial grounding tasks. Sound event recognition and grounding tasks are reported in accuracy (%), while speech recognition is reported in WER (%, lower is better). V , M , and B denote visual input, monaural audio, and binaural audio, respectively. Results are mean

\pm

standard deviation over three seeds. Best and second-best mean results are shown in bold and underlined, respectively.

ID	Variant	Modality	Spatial Relational Reasoning (Acc. %)					Scene Captioning
ID	Variant	Modality	Visibility	Correspondence	Hidden source	Loudness	Proximity	CIDEr-D	METEOR	SPIDEr
Modality ablation
(a)	Without vision	M + B	$65.43\pm 0.42$	$57.11\pm 0.11$	$66.62\pm 0.21$	$69.37\pm 0.14$	$34.13\pm 0.12$	$5.390\pm 0.014$	$0.441\pm 0.001$	$3.070\pm 0.007$
(b)	Visual only	V	$53.21\pm 0.10$	$26.71\pm 0.02$	$24.16\pm 0.43$	$52.97\pm 0.34$	$37.09\pm 0.55$	$1.757\pm 0.006$	$0.316\pm 0.001$	$1.171\pm 0.002$
(c)	Without binaural	V + M	$62.72\pm 0.10$	$55.73\pm 0.07$	$51.24\pm 0.58$	$68.84\pm 0.32$	$47.11\pm 0.27$	$4.796\pm 0.014$	$0.420\pm 0.000$	$2.751\pm 0.007$
(d)	Without monaural	V + B	$75.67\pm 0.07$	$66.58\pm 0.09$	$66.48\pm 0.10$	$77.64\pm 0.14$	$55.23\pm 0.62$	$5.475\pm 0.016$	$0.440\pm 0.001$	$3.091\pm 0.008$

Appendix

Table S2: Ablation study on spatial relational reasoning and scene captioning tasks. Relational reasoning tasks are reported in accuracy (%). Scene captioning is evaluated with CIDEr-D, METEOR, and SPIDEr. V , M , and B denote visual input, monaural audio, and binaural audio, respectively. Results are mean

\pm

standard deviation over three seeds. Best and second-best mean results are shown in bold and underlined, respectively.

Model	Sound	Direction	Distance	Visibility
VideoLLaMA2 [ 2 ]	$34.52\pm 0.31$	$12.61\pm 0.16$	$4.91\pm 0.17$	$61.80\pm 0.31$
Video-SALMONN 2 [ 3 ]	$35.30\pm 0.11$	$14.36\pm 0.10$	$4.80\pm 0.29$	$66.99\pm 0.00$
Qwen2.5-Omni [ 4 ]	$\mathbf{43.22}\pm 0.79$	$14.66\pm 0.76$	$3.62\pm 0.17$	$67.61\pm 0.25$
HLR-LLM (Ours)	$33.99\pm 0.34$	$\mathbf{17.48}\pm 1.17$	$\mathbf{19.76}\pm 3.42$	$\mathbf{68.61}\pm 0.09$

Appendix

Table S3: Zero-shot evaluation on STARSS23. We evaluate direct transfer to real-world spatial audio-visual recordings without any STARSS23 fine-tuning. Sound denotes sound event recognition; direction, distance, and visibility are egocentric. All metrics are reported in accuracy (%).

Figure S1: Qualitative chain-of-thought reasoning examples from HLR-AVSceneQA. We compare Qwen2.5-Omni with HLR-LLM on proximity and audio-visual correspondence tasks. HLR-LLM generates explicit CoT rationales that localize the sound source, relate it to visible objects, and derive the correct answer.

Figure S2: Qualitative examples from the STARSS23 real-world evaluation. We show two representative task types, sound-event recognition and visibility prediction, together with the midpoint video frame, annotated sound-source locations, ground-truth answers, and model predictions.

Figure S3: Failure cases. We show representative failure cases where models make fine-grained spatial errors, such as confusing back-left with left, or struggle with multi-source scenes in which sound sources are spatially close.

Figure S4: Counterfactual analysis with conflicting audio-visual cues. The image is kept fixed with the dog visible at its original location, while the binaural source position is changed. HLR-LLM follows the counterfactual spatial audio and correspondingly updates its localization and downstream loudness reasoning.

Figure S5: Modality-specific attention visualization. From left to right: input image, visual-to-visual, audio-to-visual, and spatial-audio-to-visual attention from the 18th transformer layer. Spatial-audio tokens attend more selectively to the ground-truth sound-source regions (red boxes).

Type	Bin	Range
Azimuth	front	$337.5^{\circ}$ – $22.5^{\circ}$
	front-right	$22.5^{\circ}$ – $67.5^{\circ}$
	right	$67.5^{\circ}$ – $112.5^{\circ}$
	back-right	$112.5^{\circ}$ – $157.5^{\circ}$
	back	$157.5^{\circ}$ – $202.5^{\circ}$
	back-left	$202.5^{\circ}$ – $247.5^{\circ}$

Appendix

Table S4: Azimuth and distance bins used in HLR-AVSceneQA. Azimuth angles increase clockwise from the agent’s forward direction.

Figure S6: Examples of 3D object meshes. The first three rows show meshes retrieved from Objaverse-XL, while the last three rows show meshes generated by generative models.

Figure S7: Example illustration. This example data sample is used to facilitate understanding of our task definitions and details.

Task family	Task	Total	Passed	Failed	Pass rate
Spatial hearing	Sound event recognition	112	112	0	100.0%
Spatial hearing	Far-field ASR	29	29	0	100.0%
Egocentric grounding	Direction	98	98	0	100.0%
Egocentric grounding	Distance	98	98	0	100.0%
Allocentric grounding	Direction	148	146	2	98.6%
Allocentric grounding	Distance	148	148	0	100.0%

Appendix

Table S5: Human verification of HLR-AVSceneQA annotations. Three human annotators verify 1,144 QA pairs from 150 sampled scenes. We report the number of passed and failed QA pairs and the corresponding pass rate for each task.

Task family	Task	Train			Train-CoT			Test
Task family	Task	Single	Multi	Total	Single	Multi	Total	Single	Multi	Total
Spatial hearing	Sound event	14,794	7,054	21,848	–	–	–	1,498	772	2,270
Spatial hearing	Far-field ASR	2,708	867	3,575	–	–	–	380	128	508
Spatial grounding	Egocentric direction	13,770	6,500	20,270	–	–	–	1,442	632	2,074
	Egocentric distance	13,770	6,500	20,270	–	–	–	1,442	632	2,074
	Allocentric direction	8,410	3,909	12,319	8,086	3,735	11,821	1,454	461	1,915

Appendix

Table S6: Dataset statistics by task split. Each entry reports the number of QA pairs. For Train and Test, Single and Multi are computed from the source-count annotations. For Train-CoT, Single and Multi report the number of CoT QA pairs in each source-count group after task-specific filtering.

Statistic	Train	Test	Train–Test Overlap
Hidden-source distribution
Off-screen (front)	2,681 (34.1%)	408 (27.6%)	–
Off-screen (back)	2,752 (35.0%)	518 (35.1%)	–
Adjacent room	959 (12.2%)	237 (16.1%)	–
Hidden behind wall	418 (5.3%)	123 (8.3%)	–
Hidden behind object	1,056 (13.4%)	190 (12.9%)	–

Appendix

Table S7: Additional dataset statistics and train–test overlap. We report label distributions for hidden-source and visibility tasks, followed by statistics on the train and test splits. Percentages are computed within each corresponding task split. Train and test environments, rooms, and object mesh instances are disjoint, while test semantic categories are covered by the training split.

Figure S8: Distance annotation statistics. We show histograms of distance-bin annotations for egocentric and allocentric perspectives in both the training and test sets.

Figure S9: Direction annotation statistics. We show log-scaled distributions of direction-bin annotations for egocentric and allocentric perspectives in the training and test sets.

Figure S10: Top 30 inserted object categories. We show the 30 most frequent object categories inserted into training and test scenes.

Figure S11: Top 30 active sound categories. We show the 30 most frequent sound categories used as active sound sources in training and test scenes.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models
Aug 10, 2026Zhi Zeng, Cheng Zhang, Zesheng Yang +9Unified Multimodal Models Audio-Visual QA
Xi’an Jiaotong University · The Hong Kong Polytechnic University · National University of Singapore +2
FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs
Oct 8, 2026Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh3D Spatial Reasoning Audio-Visual Understanding
Korea Advanced Institute of Science and Technology
Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding
Jun 9, 2026Zhiyuan Zhu, Yixuan Chen, Yiwen Shao +13Multimodal Large Language Models Audio Understanding
Zhejiang University · Tencent Hunyuan

Aug 10, 2026J/K move · Enter open · S save

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi+8

Xi’an Jiaotong University · The Hong Kong Polytechnic University · National University of Singapore+2

Unified Multimodal Models Audio-Visual QA Spatial Audio

Open paper AskPDF arXiv

Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83% average semantic accuracy across the four levels, compared with 37.28% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.

Abstract

Audio-Visual Understanding

Explore similar work

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs

Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding