SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning

Chenlong Wei; Hanling Wang; Mingjin Lou; Yufan Shi; Yandong Sun; Sheldon Yu; Xiaohui Zhu; Xiaobo Jin

cs.SDOct 12, 2026

SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning

Authors: Chenlong Wei, Hanling Wang, Mingjin Lou, Yufan Shi, Yandong Sun, Sheldon Yu, Xiaohui Zhu, Xiaobo Jin, +1 more

Organizations: Xi’an Jiaotong-Liverpool University · University of California, San Diego · Imperial College London · Lyuliang University · Southeast University

Abstract

Dynamic spatial audio reasoning requires a model to associate changing acoustic observations with the correct source and recover its spatial state at a queried time or interval, a challenge intensified by overlapping speech and source motion. To study this problem, we introduce Sonic-Bench, comprising 2.8K first-order Ambisonics clips and 28.8K target-specific question-answer pairs across 480 simulated indoor scenes. The benchmark covers localization, distance estimation, and motion reasoning in single- and dual-source settings. We further propose SonicWorld, which integrates physical, learned acoustic, and semantic features into persistent source states through learned transitions and recurrent updates, and projects them into an Omni LLM for spatial reasoning. Compared with Spatial-Omni on the Sonic-Bench test split, SonicWorld improves direction accuracy from 31.98% to 34.69% and reduces azimuth error from 57.22° to 48.01°. In zero-shot transfer to Spatial LibriSpeech, SonicWorld retains strong localization performance, achieving 73.98% direction accuracy and 19.29° azimuth error, compared with 27.14% and 70.74° for Spatial-Omni.

Figures & tables

Figure 1: Task overview of Sonic-Bench. Given an FOA recording, the task is to infer a specified source’s spatial attributes at a queried time or its movement direction over a queried interval. Time-aligned source trajectories provide ground-truth answers, illustrated here with two moving sources and a fixed receiver.

Figure 2: SonicWorld architecture. Physical, neural, and frozen-BEATs semantic evidence are concatenated before recurrent source binding. Smoothed states and decoded geometry are projected into spatial tokens, which join audio and question tokens at the language model.

	Single source					Dual source
Model	Dir. $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Dist. $\downarrow$	Mot. $\uparrow$	Dir. $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Dist. $\downarrow$	Mot. $\uparrow$
Closed-source models
Gemini 3 Flash	12.81	88.55	4.40	1.11	7.81	15.47	88.23	4.49	2.71	8.28
Gemini 3.1 Pro	9.06	90.87	15.78	1.86	12.19	14.53	90.32	28.88	2.03	14.38
Open-source models
Qwen2.5-Omni-7B	12.50	91.18	84.91	98.83	51.88	12.66	88.33	83.58	34.30	48.59

Table 1: Experimental results on Sonic-Bench. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B. Dir./Mot. are label accuracies (%); Az./El. are MAE in degrees and Dist. is MAE in meters. Best values in each column are bold.

	SpatialSceneQA		SLS
Model	Az. $\downarrow$	El. $\downarrow$	Dir. $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Dist. $\downarrow$
Open-source models
Qwen2.5-Omni-7B	60.47	80.14	12.69	90.26	88.47	55.04
InternOmni	80.54	78.05	12.38	89.69	57.59	98.83
Phi-4-MM-5.6B	61.86	18.31	12.61	89.43	7.08	1.99
Audio Flamingo 3	132.66	24.92	11.87	89.92	28.08	33.40

Table 2: Zero-shot transfer to SpatialSceneQA and SLS. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B.

Physical	Neural	BEATs	BiGRU	Dir. $\uparrow$	Mot. $\uparrow$	Az20 $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Dist. $\downarrow$
4	8	8	4	28.33	52.40	25.73	55.68	0.97	0.62
4	4	8	4	28.65	52.19	26.88	55.59	0.99	0.62
4	4	4	8	32.19	51.98	33.13	48.11	0.96	0.63
4	4	4	4	34.69	53.44	34.90	48.01	0.92	0.59

Table 3: Module ablations with Qwen2.5-Omni-7B. 4 denotes enabled and 8 disabled.

Variant	Dir. $\uparrow$	Mot. $\uparrow$	Az20 $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Dist. $\downarrow$
w/o transition MLP	33.44	51.46	32.71	49.94	0.96	0.64
w/o prior-state supervision	24.48	51.67	22.19	61.25	1.05	0.65
w/o masked-state loss	33.02	52.81	30.21	50.66	0.98	0.64
w/o persistent source slots	33.65	52.81	25.94	54.13	1.09	0.68
Full model	34.69	53.44	34.90	48.01	0.92	0.59

Table 4: Ablations of source-state modeling with Qwen2.5-Omni-7B.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Unit	Train	Test	Total
Scenes	400	80	480
Single-source clips	800	160	960
Dual-source clips	1,600	320	1,920
Source trajectories	4,000	800	4,800
Question–answer pairs	24,000	4,800	28,800
Audio duration	6 h 40 min	1 h 20 min	8 h

Appendix

Table A.1 : Sonic-Bench statistics by split. Each clip contains two target query times and five questions per query time.

Resource	Format	Scale	Motion	Spatial supervision	Queries
SpatialSoundQA [ 7 ]	Binaural	872K QA	Static	Static source geometry	Source attributes / relations
SpatialSceneQA [ 9 ]	FOA, RGB-D	61K samples	Static	Static 3D poses	Source identity / visual grounding
SAVVY-Bench [ 12 ]	7-ch audio, video	1,522 QA	Static / Moving	Camera-derived speaker tracks	Sound event-anchored
STAR-Bench [ 5 ]	Binaural	2,353 QA	Static / Moving	Motion / trajectory QA labels	Motion-pattern reasoning
SO-Bench [ 10 ]	FOA	7K QA	Static / Moving	Linked SO-Dataset tracks	Task-dependent source / time
Sonic-Bench (ours)	FOA	28.8K QA	Static / Moving	Full-clip source tracks	Target- and time-specific

Appendix

Table A.2 : Comparison with spatial-audio resources. Scale uses each resource’s reporting unit; Spatial supervision refers to geometry available during construction or training.

Parameter	Setting
Duration / sample rate	10 s / 16 kHz
Geometric sampling	12 Hz; 121 frames (including endpoints)
Receiver height	1.50 m above NavMesh
Source height	1.55 m above NavMesh
Source–receiver range	1.5–4 m along the path
Moving-path length	2–5 m

Appendix

Table A.3 : Scene and acoustic generation parameters.

Figure A.1 : Target-bearing sectors and clip distribution. Each sector spans

45^{\circ}

and contains 12.5% of the clips in each split.

Task	Example question	Answer
Position direction	At 3.00 s, which direction is the female voice coming from?	Front
Azimuth	At 6.00 s, what is the azimuth of the female voice?	$-9.47^{\circ}$
Elevation	At 4.50 s, what is the vertical angle of the male voice?	$+2.25^{\circ}$
Distance	At 7.00 s, how far away is the male voice?	$3.12$ m
Movement direction	Between 1.50 and 2.50 s, which direction did the female source move relative to the listener’s fixed orientation?	Front-Right

Appendix

Table A.5 : Examples of target-specific spatial questions.

Quantity	Statistic	Train	Test
Source range (m)	Minimum	1.50	1.50
	Maximum	4.00	4.00
	Mean	2.85	2.82
	Median	2.90	2.87
Elevation ( ∘ )	Minimum	-16.76	-13.45
	Maximum	19.94	20.30

Appendix

Table A.6 : Source geometry and trajectory statistics.

Subset	S	F	FL	L	BL	B	BR	R	FR	Total
Single	160	15	30	14	23	23	21	13	21	320
Dual	320	55	34	42	37	46	38	40	28	640
Total	480	70	64	56	60	69	59	53	49	960

Appendix

Table A.7 : Test movement-direction distribution. Counts denote questions in the single- and dual-source subsets. S denotes Stationary; F/B/L/R denote Front/Back/Left/Right, with combined letters indicating diagonal directions. Stationary labels account for half of each subset.

Figure A.2 : Target geometry at query times. Distributions use 4,800 training and 960 test query times, expressed as percentages within each split. The upper panels group azimuth into

45^{\circ}

bins starting at

0^{\circ}

and distance into 0.5 m bands from 1.5 to 4.0 m. The lower panels show elevation below

0^{\circ}

, in

[0^{\circ},2^{\circ})

, and at least

2^{\circ}

Layer	Channels	Kernel	Stride
1	$1\to 48$	10	5
2	$48\to 96$	7	4
3	$96\to 128$	5	4
4	$128\to 192$	5	4

Appendix

Table B.1 : Shared waveform encoder. Each convolution uses one-group normalization and GELU.

Input	Dimensions
Prior, observation, recurrent state, refined state, observation minus prior	$5\times 256$
Direction anchor; predicted direction and distance	$3+3+1$
Source-presence probability; positional log-variance	$1+1$
Auxiliary features	6
Total	1,295

Appendix

Table B.2 : Projector input per source and time step.

Module	Linear widths
Frame encoder	$1295\to 512\to 512$
Local temporal compressor	$2048\to 2048\to 512$
Language-space projector	$512\to 2048\to 3584$

Appendix

Table B.3 : Projector MLPs. Each uses layer normalization at its input and output, with SiLU between the two linear layers.

Objective	Weights
Trajectory supervision	Direction 1; log-distance 1; presence 1; uncertainty 0.05; prior 0.20
Masked center prediction	Direction 1; log-distance 0.3; presence 0.2
Projector training	Answer cross-entropy 1; geometric alignment 0.5
Projector + LoRA	Answer cross-entropy 1; geometric alignment 0.25

Appendix

Table C.1 : Loss coefficients.

Representation	Shape
FOA waveform, $[W,Y,Z,X]$	$[B,4,160000]$
Physical input features	$[B,13,64,121]$
Physical tokens	$[B,121,64,192]$
Neural tokens / semantic tokens	$[B,121,1,192]$
Combined acoustic tokens	$[B,121,66,192]$
Slot–token weights	$[B,3,66]$

Appendix

Table C.2 : Model representations from input to output.

B

denotes batch size. Slot–token weights include two source slots and one background slot; subsequent source representations retain only the two source slots.

Stage	Epochs	Accum.	Updates	Trainable modules and peak learning rate
Source states	25	2	30,000	Physical and neural branches, semantic pooling, recurrent filter, bidirectional GRU, state heads: $10^{-4}$
Projector	5	8	15,000	Projector and alignment heads: $10^{-4}$
Projector + LoRA	3	8	9,000	Projector : $3\times 10^{-5}$ ; LoRA: $5\times 10^{-5}$
GPUs				$3\times$ NVIDIA RTX 4090; $6\times$ NVIDIA RTX 3090

Appendix

Table C.3 : SonicWorld training stages. Updates count optimizer steps.

Backbone	Dir. $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Dist. $\downarrow$	Mot. $\uparrow$
Single source
Qwen2.5-Omni-7B	50.00 ( $+37.50$ )	30.85 ( $-60.33$ )	0.91 ( $-84.00$ )	0.56 ( $-98.27$ )	52.50 ( $+0.62$ )
InternOmni	51.88 ( $+39.07$ )	36.49 ( $-57.23$ )	0.92 ( $-21.73$ )	0.61 ( $-39.97$ )	55.94 ( $+47.81$ )
Phi-4-MM-5.6B	48.75 ( $+35.31$ )	32.03 ( $-49.18$ )	0.92 ( $-2.37$ )	0.53 ( $-1.40$ )	53.44 ( $+47.19$ )
Dual source
Qwen2.5-Omni-7B	27.03 ( $+14.37$ )	56.59 ( $-31.74$ )	0.93 ( $-82.65$ )	0.60 ( $-33.70$ )	53.91 ( $+5.32$ )

Appendix

Table D.1 : Effect of SonicWorld on different backbones on Sonic-Bench. Values in parentheses indicate changes relative to the corresponding original backbone without SonicWorld.

	SpatialSceneQA		SLS
Backbone	Az. $\downarrow$	El. $\downarrow$	Dir. $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Dist. $\downarrow$
Qwen2.5-Omni-7B	21.35 ( $-39.12$ )	20.51 ( $-59.63$ )	73.98 ( $+61.29$ )	19.29 ( $-70.97$ )	7.09 ( $-81.38$ )	0.89 ( $-54.16$ )
InternOmni	22.57 ( $-57.97$ )	20.58 ( $-57.47$ )	66.80 ( $+54.42$ )	22.20 ( $-67.49$ )	7.08 ( $-50.51$ )	1.24 ( $-97.59$ )
Phi-4-MM-5.6B	19.79 ( $-42.07$ )	20.34 ( $+2.03$ )	73.44 ( $+60.83$ )	18.22 ( $-71.21$ )	7.95 ( $+0.87$ )	0.81 ( $-1.18$ )

Appendix

Table D.2 : Effect of SonicWorld on zero-shot transfer across backbones. Values in parentheses indicate changes relative to the corresponding original backbone without SonicWorld.

	Single source			Dual source
Model	Az20 $\uparrow$	El1 $\uparrow$	D0.5 $\uparrow$	Az20 $\uparrow$	El1 $\uparrow$	D0.5 $\uparrow$
Closed-source models
Gemini 3 Flash	10.00	30.31	25.94	11.72	27.34	26.41
Gemini 3.1 Pro	12.19	31.88	1.88	12.03	18.59	6.25
Open-source models
Qwen2.5-Omni-7B	14.06	1.88	0.00	11.72	1.09	0.00

Appendix

Table D.3 : Localization tolerance accuracies. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B. Az20, El1, and D0.5 denote azimuth, elevation, and distance accuracies within

20^{\circ}

1^{\circ}

, and 0.5 m, respectively.

	Sonic- Bench	Spatial SceneQA	SLS
Evaluated clips	480	622	8,512
Single-source clips	160	247	8,512
Dual-source clips	320	375	0
Motion state	Stationary / Moving	Stationary	Stationary
Questions per evaluated clip	10	2	4
Single-source questions	1,600	494	34,048

Appendix

Table D.4 : Evaluation-set scale and selection. Sonic-Bench uses its full test set; SpatialSceneQA and SLS use the selected test subsets. Dir., Az., El., Dist., and Mot. denote direction, azimuth, elevation, distance, and movement direction.

	Single source				Dual source
Model	Az. $\downarrow$	El. $\downarrow$	Az20 $\uparrow$	El1 $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Az20 $\uparrow$	El1 $\uparrow$
Open-source audio models
Qwen2.5-Omni-7B	20.54	100.49	53.85	0.00	86.77	66.73	23.20	0.00
InternOmni	75.35	83.90	5.26	0.00	83.89	74.26	4.53	0.00
Phi-4-MM-5.6B	78.49	16.42	10.93	0.40	47.69	21.14	30.13	0.00
Audio Flamingo 3	117.98	27.04	14.17	0.40	145.38	22.64	3.47	0.00

Appendix

Table D.5 : Single- and dual-source SpatialSceneQA results. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B.

Model	Dir. $\uparrow$	Az. $\downarrow$	El. $\downarrow$	Dist. $\downarrow$
Open-source models
Qwen2.5-Omni-7B	12.01	90.68	14.40	19.61
InternOmni	12.76	89.19	54.55	7.12
Phi-4-MM-5.6B	12.48	90.42	7.04	0.96
Audio Flamingo 3	12.31	89.46	16.39	3.11
Kimi-Audio-7B-Instruct	11.83	90.81	8.59	2.81

Appendix

Table D.6 : Results on SLS with official spatial interference. Evaluation covers 8,512 clips and 34,048 questions. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B. Dir. is accuracy (%); Az./El. are MAE in degrees and Dist. is MAE in meters. Best values are bold.

Figure E.1 : Movement predictions with a stationary interferer.

Figure E.2 : Movement predictions for a male target with a moving interferer.

Figure E.3 : Decoded states of a single moving source. Solid lines show decoded trajectories and dashed lines show ground truth. The scene view shows the spatial paths; the time series show receiver-centered front and left coordinates and source distance. Circles and triangles mark trajectory starts and ends.

Figure E.4 : Decoded states of two moving sources. Colors distinguish the sources. Solid lines show decoded trajectories and dashed lines show ground truth.

Figure E.5 : Decoded states of a moving source and a stationary source. Colors distinguish the sources. Solid lines show decoded trajectories and dashed lines show ground truth.

Variant	ADE $\downarrow$	ID retention $\uparrow$
Stationary interferer
w/o transition MLP	0.82	0.00
w/o masked-state loss	0.66	9.09
Full SonicWorld	0.52	30.30
Moving interferer
w/o transition MLP	0.33	14.81

Appendix

Table E.1 : Source-state continuity across speech pauses. ADE (m) and ID retention (%) are evaluated after speech resumes, with pre-pause slot assignments fixed. ID retention uses distance thresholds of 0.5 m and 0.25 m for stationary and moving interferers, respectively.

Figure E.6 : Tracking across a speech pause with a stationary interferer. Blue/gold/green show the decoded target trajectory before/during/after the pause; A/B mark pause onset/resumption. Dark dashed/purple dotted lines show target/interferer ground truth. Right panels show speech activity, receiver-centered front coordinate, and target position error.

Figure E.7 : Tracking across a speech pause with a moving interferer. Both speakers continue moving during the target’s 1.08 s speech pause. Colors and markers follow Figure E.6 . The coordinate panel shows the receiver-centered left position.

Explore similar work

Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
Jun 12, 2026Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida +6Audio-Language Model Evaluation Audio Representation Learning
1POSTECH · 2Sony AI · 4Sungkyunkwan University +2
Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models
Aug 10, 2026Zhi Zeng, Cheng Zhang, Zesheng Yang +9Unified Multimodal Models Audio-Visual QA
Xi’an Jiaotong University · The Hong Kong Polytechnic University · National University of Singapore +2
Spatial Audio Question Answering and Reasoning on Dynamic Source Movements
Feb 18, 2026Arvind Krishna Sridhar, Yinyi Guo, Erik VisserSpatial Audio Audio Reasoning
Qualcomm Technologies Inc. San Diego, CA

Jun 12, 2026J/K move · Enter open · S save

Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources

Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida, Kim Sung-Bin+5

1POSTECH · 2Sony AI · 4Sungkyunkwan University+2

Audio-Language Model Evaluation Audio Representation Learning Audio Reasoning Audio Understanding Audio-Language Models Spatial Audio

Open paper AskPDF arXiv

Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To address this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and questions about what is sounding, where it is, how it moves, and how sources relate. We further propose ST-Audio Encoder, a time-resolved FOA audio encoder that learns event semantics together with source trajectories, and ST-AudioLM, which connects the audio tokens from the encoder to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the semantic-localization tradeoff and yields stronger reasoning performance than static spatial and localization-oriented baselines.

Abstract

Explore similar work

Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

Spatial Audio Question Answering and Reasoning on Dynamic Source Movements