Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints, and datasets are available at https://github.com/liuzhan22/JAEGER.
Figures & tables
Figure 1: Overview of the SpatialSceneQA 61k dataset. . Left: Example question-answer pairs demonstrating diverse spatial tasks, including sound source localization (azimuth/elevation), visual grounding (bounding boxes), and overlapping sound source identification. Right: The data synthesis pipeline leveraging Habitat-Sim and SoundSpaces 2.0 . The process consists of four stages: (1) selecting an HM3D scene, (2) sampling random source and receiver poses, (3) inserting 3D visual sound sources (e.g., speakers generated by Hunyuan3D-1.0), and (4) exporting synchronized RGB-D frames, FOA audio, and semantic and camera intrinsic/extrinsic metadata.
Task Type
#Sources
#Samples
Question & Answer Example
Perception
A: Single-Source Audio DoA
1
32K
Q: Based on the audio cues, please output the precise azimuth and elevation.
A: azimuth: -7; elevation: -22
B: Overlap-Source Audio DoA
2
30K
Q: Where is the female voice coming from? Output azimuth and elevation.
A: azimuth: 28; elevation: -15
C: Visual Grounding
1
17K
Q: Identify the 3D box for the speaker in this scene. A: bbox_0=Bbox(speaker,0.14,−0.48,−1.15,0.33,0.88,0.32)
Table 1: Statistics of the SpatialSceneQA 61K dataset. Tasks are grouped into “Perception” and “Reasoning categories. #Sources denotes active audio sources for A–B and inserted loudspeaker assets for C–E. Regarding definitions of different types of tasks, A–B estimate source azimuth and elevation; C predicts the precise Bounding Box of the inserted speaker; D–E identify the correspondence between a target audio speaker and a visual loudspeaker based on spatial audio-visual cues.
Figure 2: Comparisons between Classical IV and Neural IV.
Figure 3: Overview of the JAEGER Architecture. The framework processes RGB-D and FOA inputs. (1) Visual Stream: RGB features are fused with 3D-aware positional encodings derived from depth-projected Point Clouds. (2) Audio Stream: Semantic features are extracted from the omnidirectional channel (FOA W). For spatial cues, we compare Classical IV with Neural IV (N. IV). Specifically, IV derives features via STFT followed by fetching real parts and normalization, whereas Neural IV extracts geometric features from raw waveforms using channel-wise 1D-CNNs followed by an MLP.
Model
Modality
Perception
Reasoning
Audio DoA ↓
Overlap DoA ↓
3D IoU ↑
Visual Offset ↓
1-speaker ↑
2-speaker ↑
Random
N/A
90 °
90 °
0.00
∞
45.6
47.4
Open-source omni model
Qwen2.5-Omni
RGB+Mono
–
–
0.00
2.40
35.8
44.0
Open-source specialized models
BAT
Binaural
2.16 °
19.09 °
–
–
–
–
Table 2: Main results compared with baselines. Task Definitions & Metrics: (1) Audio DoA : Detection of sound source arrival direction. Metric: Median Angular Error (MAE) in degrees ( ∘ ). (2) Overlap Audio DoA : MAE of a specific audio source’s arrival angle when two audio clips overlap. (3) 3D IoU : Mean Intersection over Union of the predicted 3D bounding box. (4) Visual Offset : Median distance deviation of the predicted object center coordinates from the ground truth. (5) Reasoning 1-speaker : Localizing one of multiple visible loudspeakers in the scene based on spatial audio cues from a single speaker. (6) Reasoning 2-speaker : Identifying a specific loudspeaker among multiple candidates based on the spatial location cues of a target speech in a two-speaker scenario. Reasoning tasks are evaluated by Accuracy . “–” indicates the model is incapable of performing the task.
Table 3: Cross-evaluation matrix (Median Angular Error in ∘ ). We compare Classical IV and Neural IV by training and testing on matched or mismatched source scenarios.
Model Setting
3D IoU ↑
Visual Offset ↓
Mean
Median
Mean
Median
Full Model (w/ Depth)
0.32
0.31
0.22
0.16
w/o Depth Encoding
0.29
0.27
0.24
0.18
Table 4: Ablation study on Depth Encoding for visual grounding. We report both Mean and Median values to better analyze the distribution of errors.
Model Setting
Accuracy (%)
1-Speaker ↑
2-Speaker ↑
Ours (Neural IV)
99.5
99.2
Ours (Classical IV)
99.5
98.6
Ours (Neural) w/o Depth
96.9
94.9
Ours (Classical) w/o Depth
99.2
98.7
Ours w/o FOA Encoder
43.8
47.6
Table 5: Component analysis on Reasoning Tasks . We evaluate the impact of audio encoders (Neural/Classical), Depth, and FOA on 1-speaker and 2-speaker scenarios.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
4 Candidates ↑
5 Candidates ↑
6 Candidates ↑
JAEGER (Classical IV)
87.5
82.5
80.0
JAEGER (Neural IV)
95.0
77.5
72.5
Appendix
Table 9: Single-source audio-visual reasoning accuracy under harder settings with more visible speaker candidates.
Figure 4: Visualization of the diversity in generated speaker point clouds. We display 32 randomly selected samples from the 120 generated instances. Despite using the same text prompt, varying the random seed results in distinct structural and morphological variations.
While 3D spatial reasoning in dynamic egocentric environments is crucial for embodied intelligence, audio-visual large language models (AV-LLMs) lack explicit mechanisms to process and internalize global geometry directly from raw sensory streams. Existing approaches either require costly fine-tuning or underutilize the model's cross-modal reasoning capacities. In this paper, we propose FloorSAV, a novel framework that explicitly grounds spatial audio-visual context by rendering a dynamic 2D floormap. By integrating 3D point clouds, camera trajectories, spatial audio cues, and semantically grounded object landmarks, we inject this floormap into the AV-LLM as a synchronized stream with an egocentric video. AV-LLMs utilize their multi-modal capabilities to jointly reason over visual, auditory, and geometric cues in a single inference with floormap interpretation guidance. We further introduce SAVED-Bench (Spatial Audio-Visual Egocentric Benchmark with Dynamic Agents), constructing essential tasks of spatial capability in real-world scenarios: dynamic relativity, regional, and path reasoning QAs. FloorSAV improves AV-LLMs' spatial reasoning on various tasks from both SAVED-Bench and SAVVY-Bench. Studies with ground-truth floormaps demonstrate the substantial potential of FloorSAV with accurate spatial information.
Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh
Korea Advanced Institute of Science and Technology
Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose OmniEcho, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges. Our code and data will be available in https://github.com/PKU-VaLuE-Lab/OmniEcho/tree/main
Ruixun Liu, Yuxuan Wang, Jiacheng Xie +10
School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group +1
Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities. Existing benchmarks often rely on simulated scenes, evaluate isolated spatial skills, and provide limited diagnostic insight into failure modes. We introduce SAVU-Bench, a real-world benchmark that systematically evaluates spatial audio-visual understanding across three capability levels and seven evaluation tasks. We further introduce SAVU-Diag, a scene-linked diagnostic set that decomposes reasoning questions into their prerequisite grounding and alignment sub-tasks. Evaluation of 12 representative models on SAVU-Bench reveals that while visual spatial grounding is relatively mature, spatial perception involving audio remains a primary bottleneck. SAVU-Diag further demonstrates that most reasoning errors co-occur with failures on these prerequisite tasks, though reasoning gaps persist even when prerequisites are correctly resolved. Motivated by these findings, we introduce SAVU-EA, a training-free evidence-augmented baseline that makes spatial cues more explicit. While SAVU-EA substantially improves spatial grounding and joint matching, high-level spatial reasoning remains challenging. Our findings highlight the urgent need for both robust spatial audio perception and deeper integration of cross-modal spatial relations.
Yu Chen, Ruihang Liu, Yangguang Xu +5
University of Western Australia · University of Science and Technology Beijing · Monash University