Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints, and datasets are available at https://github.com/liuzhan22/JAEGER.
Figures & tables
Figure 1: Overview of the SpatialSceneQA 61k dataset. . Left: Example question-answer pairs demonstrating diverse spatial tasks, including sound source localization (azimuth/elevation), visual grounding (bounding boxes), and overlapping sound source identification. Right: The data synthesis pipeline leveraging Habitat-Sim and SoundSpaces 2.0 . The process consists of four stages: (1) selecting an HM3D scene, (2) sampling random source and receiver poses, (3) inserting 3D visual sound sources (e.g., speakers generated by Hunyuan3D-1.0), and (4) exporting synchronized RGB-D frames, FOA audio, and semantic and camera intrinsic/extrinsic metadata.
Task Type
#Sources
#Samples
Question & Answer Example
Perception
A: Single-Source Audio DoA
1
32K
Q: Based on the audio cues, please output the precise azimuth and elevation.
A: azimuth: -7; elevation: -22
B: Overlap-Source Audio DoA
2
30K
Q: Where is the female voice coming from? Output azimuth and elevation.
A: azimuth: 28; elevation: -15
C: Visual Grounding
1
17K
Q: Identify the 3D box for the speaker in this scene. A: bbox_0=Bbox(speaker,0.14,−0.48,−1.15,0.33,0.88,0.32)
Table 1: Statistics of the SpatialSceneQA 61K dataset. Tasks are grouped into “Perception” and “Reasoning categories. #Sources denotes active audio sources for A–B and inserted loudspeaker assets for C–E. Regarding definitions of different types of tasks, A–B estimate source azimuth and elevation; C predicts the precise Bounding Box of the inserted speaker; D–E identify the correspondence between a target audio speaker and a visual loudspeaker based on spatial audio-visual cues.
Figure 2: Comparisons between Classical IV and Neural IV.
Figure 3: Overview of the JAEGER Architecture. The framework processes RGB-D and FOA inputs. (1) Visual Stream: RGB features are fused with 3D-aware positional encodings derived from depth-projected Point Clouds. (2) Audio Stream: Semantic features are extracted from the omnidirectional channel (FOA W). For spatial cues, we compare Classical IV with Neural IV (N. IV). Specifically, IV derives features via STFT followed by fetching real parts and normalization, whereas Neural IV extracts geometric features from raw waveforms using channel-wise 1D-CNNs followed by an MLP.
Model
Modality
Perception
Reasoning
Audio DoA ↓
Overlap DoA ↓
3D IoU ↑
Visual Offset ↓
1-speaker ↑
2-speaker ↑
Random
N/A
90 °
90 °
0.00
∞
45.6
47.4
Open-source omni model
Qwen2.5-Omni
RGB+Mono
–
–
0.00
2.40
35.8
44.0
Open-source specialized models
BAT
Binaural
2.16 °
19.09 °
–
–
–
–
Table 2: Main results compared with baselines. Task Definitions & Metrics: (1) Audio DoA : Detection of sound source arrival direction. Metric: Median Angular Error (MAE) in degrees ( ∘ ). (2) Overlap Audio DoA : MAE of a specific audio source’s arrival angle when two audio clips overlap. (3) 3D IoU : Mean Intersection over Union of the predicted 3D bounding box. (4) Visual Offset : Median distance deviation of the predicted object center coordinates from the ground truth. (5) Reasoning 1-speaker : Localizing one of multiple visible loudspeakers in the scene based on spatial audio cues from a single speaker. (6) Reasoning 2-speaker : Identifying a specific loudspeaker among multiple candidates based on the spatial location cues of a target speech in a two-speaker scenario. Reasoning tasks are evaluated by Accuracy . “–” indicates the model is incapable of performing the task.
Table 3: Cross-evaluation matrix (Median Angular Error in ∘ ). We compare Classical IV and Neural IV by training and testing on matched or mismatched source scenarios.
Model Setting
3D IoU ↑
Visual Offset ↓
Mean
Median
Mean
Median
Full Model (w/ Depth)
0.32
0.31
0.22
0.16
w/o Depth Encoding
0.29
0.27
0.24
0.18
Table 4: Ablation study on Depth Encoding for visual grounding. We report both Mean and Median values to better analyze the distribution of errors.
Model Setting
Accuracy (%)
1-Speaker ↑
2-Speaker ↑
Ours (Neural IV)
99.5
99.2
Ours (Classical IV)
99.5
98.6
Ours (Neural) w/o Depth
96.9
94.9
Ours (Classical) w/o Depth
99.2
98.7
Ours w/o FOA Encoder
43.8
47.6
Table 5: Component analysis on Reasoning Tasks . We evaluate the impact of audio encoders (Neural/Classical), Depth, and FOA on 1-speaker and 2-speaker scenarios.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
4 Candidates ↑
5 Candidates ↑
6 Candidates ↑
JAEGER (Classical IV)
87.5
82.5
80.0
JAEGER (Neural IV)
95.0
77.5
72.5
Appendix
Table 9: Single-source audio-visual reasoning accuracy under harder settings with more visible speaker candidates.
Figure 4: Visualization of the diversity in generated speaker point clouds. We display 32 randomly selected samples from the 120 generated instances. Despite using the same text prompt, varying the random seed results in distinct structural and morphological variations.
School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group +1