PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation

Lingfeng Yao; Chenpei Huang; Xingke Yang; Ziye Geng; Changqing Luo; Hao Wang; Jiang Liu; Miao Pan

cs.SDAug 30, 2026

PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation

Authors: Lingfeng Yao, Chenpei Huang, Xingke Yang, Ziye Geng, Changqing Luo, Hao Wang, Jiang Liu, Miao Pan

Organizations: University of Houston · Stevens Institute of Technology · Waseda University

Abstract

Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.

Figures & tables

Figure 1: Dataset construction pipeline. Captioned monaural audio is paired with sampled trajectories and rendered into FOA audio through simulation.

Figure 2: Overview of PhysWave. A spatial caption is parsed into an acoustic caption and a waypoint trajectory. Text, waypoint, and temporal tokens jointly condition an FOA latent diffusion model, which is trained with latent denoising loss and physics-consistency losses on the decoded FOA waveform.

Variant	Audio quality					Spatial fidelity
Variant	FD $\downarrow$	FAD CLAP $\downarrow$	CLAP $\uparrow$	KL $\downarrow$	IS $\uparrow$	Static ( ∘ ) $\downarrow$	Moving ( ∘ ) $\downarrow$	InvSq Err. (dB) $\downarrow$	InvSq Corr. $\uparrow$
Trajectory text	23.13	0.20	0.32	1.74	8.05	7.33	17.62	4.59	0.62
PhysWave (w/o physics)	21.92	0.20	0.33	1.69	8.22	2.05	6.50	5.15	0.48
PhysWave (w/o $\mathcal{L}_{\mathrm{dist}}$ )	21.79	0.20	0.33	1.68	8.15	1.75	4.81	4.79	0.59
PhysWave (w/o $\mathcal{L}_{\mathrm{dir}}$ )	21.34	0.20	0.33	1.67	8.25	2.47	6.60	3.78	0.74
PhysWave	21.22	0.21	0.33	1.66	8.31	1.73	4.65	4.06	0.73

Table 1: Ablation study of PhysWave. All variants use the same generative backbone and caption condition. Angular errors are reported in degrees, and InvSq Err. is reported in dB.

Model	Control	Static ( ∘ ) $\downarrow$	Moving ( ∘ ) $\downarrow$
ID-D	Static text	7.07	–
ID-P	Static param.	2.79	–
SM-D	Circular text	20.82	31.09
SM-P	Circular param.	2.41	19.22
PhysWave	Text + Waypoints	1.73	5.78

Table 2: Reference comparison with published results from prior FOA generation methods. ID and SM denote ImmerseDiffusion and SonicMotion, respectively. D and P denote descriptive and parametric conditioning.

Figure 3: Qualitative trajectory comparison between PhysWave with and without physics-consistency losses.

Model	FD $\downarrow$	FAD CLAP $\downarrow$	CLAP $\uparrow$	KL $\downarrow$	IS $\uparrow$
AudioLDM	31.67	0.33	0.33	2.37	6.96
AudioLDM 2	24.13	0.12	0.33	2.11	8.62
Make-An-Audio	15.44	0.11	0.37	1.93	8.79
Make-An-Audio 2	13.86	0.14	0.40	1.73	10.85
Stable Audio Open	30.84	0.31	0.30	2.30	11.09
ImmerseDiffusion	20.45	0.14	0.35	1.66	9.55

Table 3: Audio quality comparison on the omnidirectional

W

channel.

Parser	Spatial Faith. $\uparrow$	Semantic Pres. $\uparrow$
Qwen3.5-0.8B	1.89 $\pm$ 0.20	3.40 $\pm$ 0.38
Llama-3.2-3B-Instruct	2.40 $\pm$ 0.24	4.18 $\pm$ 0.29
Qwen3.5-4B	3.88 $\pm$ 0.28	4.72 $\pm$ 0.17

Table 4: LLM parser evaluation on spatial captions. Scores are reported as mean

\pm

standard deviation.

Source	MOS-	MOS-	MOS-
Source	Event $\uparrow$	Trajectory $\uparrow$	Realism $\uparrow$
Real FOA Recording	4.89	4.56	4.79
AudioLDM2	2.70	2.07	2.07
Stable Audio Open	3.40	2.21	2.51
PhysWave	4.32	3.90	3.48

Table 5: Subjective evaluation of generated FOA audio. MOS scores are obtained from 14 participants.

Figure 4: Effect of inference-time physics guidance.

Method	Format	Conditioning	Spatial Source		Trajectory Control	Physics-Guided
Method	Format	Conditioning	Static	Moving	Trajectory Control	Physics-Guided
AudioLDM ( Liu et al., 2023 )	Mono	Text	–	–	–	✗
AudioLDM 2 ( Liu et al., 2024 )	Mono	Text	–	–	–	✗
Make-An-Audio ( Huang et al., 2023b )	Mono	Text	–	–	–	✗
Make-An-Audio 2 ( Huang et al., 2023a )	Mono	Text	–	–	–	✗
TANGO ( Ghosal et al., 2023 )	Mono	Text	–	–	–	✗

Table 6: Functional comparison with prior audio generation methods. Mono, Stereo, Binaural, and FOA denote output formats. ✓ means supported, ✗ means not supported, and “–” means not applicable. For SpatialSonic, trajectory control refers to azimuth-level control, while PhysWave supports listener-relative waypoint trajectories with azimuth, elevation, and distance. Physics-guided means that explicit acoustic priors are used during generation.

Family	Ratio	Sampling Range	Description
Static	$50.0\%$	$\theta\in[-180^{\circ},180^{\circ}]$ , $\phi\in[-35^{\circ},35^{\circ}]$ , $r\in[0.5,5]$ m	Fixed direction and distance
Linear pass-by	$\approx 16.7\%$	$v\in[1,25]$ m/s, $d_{\min}\in[1,8]$ m	Coupled direction and distance changes
Circular	$\approx 16.7\%$	$r\in[0.5,5]$ m, angular span $\geq 30^{\circ}$	Direction changes at near-constant distance
Approach/recede	$\approx 16.7\%$	$v\in[2,6]$ m/s, $r\in[1,60]$ m	Distance changes with fixed direction

Table 7: Trajectory sampling configuration. Each retained monaural clip is rendered once as a static source and once as a moving source.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Data Source	Train		Test
Data Source	Clips	Duration (h)	Clips	Duration (h)
AudioCaps	46,696	129.7	4,239	11.8
WavCaps	107,850	299.6	–	–
Clotho	2,863	8.0	1,021	2.8
Total monaural	157,409	437.2	5,260	14.6
Rendered FOA	314,818	874.5	10,520	29.2

Appendix

Table 8: Statistics of source data after preprocessing. Each retained monaural clip is rendered twice, once as a static source and once as a moving source.

Panel of Figure 5 — Figure 5: Prompt used by the LLM spatial parser. The parser outputs a compact trajectory representation, which is converted into the $M=10$ waypoint condition used by the diffusion model.

Panel of Figure 6 — Figure 6: Example outputs of the LLM spatial parser. Each output keeps the non-spatial acoustic content and converts the spatial description into a parametric trajectory.

Panel of Figure 7 — Figure 7: Prompt used for LLM-as-a-judge parser evaluation. The judge compares the original spatial caption with the parsed acoustic caption and converted waypoints.

Model	STFT $\downarrow$	Mel $\downarrow$	L1 $(\theta)\downarrow$	L1 $(\phi)\downarrow$	$\Delta_{\textbf{angle}}\downarrow$
FOA VAE	1.44	1.11	0.87 ∘	1.45 ∘	1.92 ∘

Appendix

Table 9: FOA VAE reconstruction quality.

Figure 8: More qualitative trajectory comparison between PhysWave with and without physics-consistency losses.

Figure 9: Energy envelope comparison. PhysWave better follows the target inverse-square log-energy profile, while the model without physics losses shows larger temporal deviations.

Spatial Audio

Spatial audio studies how sound is captured, represented, generated, and perceived with direction, distance, motion, and room acoustics.

See all 100 papers

Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer
May 29, 2026Ke Lei, Yu Zhang, Changhao Pan +4Text-to-Audio Generation Audio-Video Generation
1Zhejiang University, China · 2ByteDance
Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion
Oct 25, 2026Kam Man Wu, Yu Zhang, Pengjun Fang +4Spatial Audio Audio Editing
HKUST · Noiz AI · University of Washington
PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation
Sep 30, 2026Tiernon Riesenmy, You Zhang, Gautam Bhattacharya +1Text-to-Audio Generation Audio-Video Generation
The Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, USA · Advanced Technology Group, Dolby Laboratories, San Francisco, CA, USA

May 29, 2026J/K move · Enter open · S save

Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer

Ke Lei, Yu Zhang, Changhao Pan, Xueyi Pu+3

1Zhejiang University, China · 2ByteDance

Text-to-Audio Generation Audio-Video Generation Diffusion Transformer Autoregressive Diffusion Spatial Audio Audio Generation+1

Open paper AskPDF arXiv

Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose SwanSphere, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. SwanSphere mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video-Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization (ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that SwanSphere achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Demos can be found at: https://swanaigc.github.io.

Abstract

Spatial Audio

Explore similar work

Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer

Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation