Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

Kam Man Wu; Yu Zhang; Pengjun Fang; Mingfei Chen; Weijia Chen; Zeyue Tian; Qifeng Chen

cs.SDOct 25, 2026

Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

Authors: Kam Man Wu, Yu Zhang, Pengjun Fang, Mingfei Chen, Weijia Chen, Zeyue Tian, Qifeng Chen

Organizations: HKUST · Noiz AI · University of Washington

Abstract

Spatial audio authoring coordinates source decisions with soundfield synthesis. For editing, plans specify what to change and retain, while reference audio supplies further acoustic detail. We present AMBIT, a unified framework for native First-Order Ambisonics (FOA) generation and editing with three connected contributions. First, a shared Transfusion backbone couples autoregressive planning with rectified-flow rendering through executable ScenePlans that specify source content, activity, and 3D trajectories. For editing, the plan describes the full target scene, including retained sources; contrastive features ground the requested change, while full reference latents supply residual acoustic detail for preservation. Second, an FOA-aware continuous VAE combines shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to preserve content and inter-channel structure. Third, joint Transfusion on-policy self-distillation (OPSD) refines planning on student-generated prefixes and rendering at sampling states reached under student plans, with target-conditioned teachers supervising both modes of the shared student. Evaluations across speech, music, and sound show improved generation content and spatial quality, acoustic fidelity during instruction-guided editing, and unedited-source preservation over the compared baselines. Project page: https://wjm-george.github.io/AMBIT/

Figures & tables

Figure 1: Overview of Ambit capabilities. (1) Spatial audio generation creates multi-source 3D scenes from text, controlling content, timing, and trajectories. (2) Instruction-guided spatial and temporal editing modifies selected sources, preserving unmodified content and spatial structure.

Figure 2: Overview of Ambit . (a) A shared Transformer predicts executable ScenePlan s and renders native FOA through latent flow conditioned on compiled plans. Editing uses source latents in both modes and Ambit CLAP features in planning. (b) OPSD distills token and clean latent predictions on student prefixes and visited states, jointly updating the shared Transformer and AR/DiT modules. (c) The VAE combines shared-decoder auxiliary $W$ reconstruction, grouped KL, and spatial covariance supervision.

System/configuration	$W$ SI-SDR (dB) $\uparrow$	Direction ( ∘ ) $\downarrow$	Intensity coherence error $\downarrow$	$W$ paired CLAP $\uparrow$
DAC	5.57	5.13	0.08	0.88
EnCodec	0.41	12.25	0.15	0.85
WavTokenizer	$-19.41$	56.66	0.35	0.63
Ambit VAE	6.39	4.96	0.04	0.90
Phase (base)	5.63	6.51	0.06	0.86
+ SCM	6.08	5.14	0.05	0.88

Table 1: FOA reconstruction and cumulative ablation. The upper block compares codecs on native scenes; each “+” below adds to the previous row, with grouped KL replacing uniform KL. Ablations use matched updates from scratch and 24 kHz evaluation. Bold marks the best per block.

Domain	System	Paired CLAP $\uparrow$	FAD $\downarrow$	KL PANNs $\downarrow$	KL PaSST $\downarrow$	FD PaSST $\downarrow$	FD OpenL3 $\downarrow$
Music	AudioX-Turbo ( Tian et al., 2026a )	0.36	1.07	0.68	1.58	234.65	37.40
	AudioX ( Tian et al., 2026b )	0.35	1.30	0.74	1.56	243.68	46.67
	Stable Audio Open ( Evans et al., 2025 )	0.18	2.08	0.90	1.81	410.64	68.72
	TangoFlux ( Hung et al., 2026 )	0.36	1.01	0.59	1.43	319.37	92.09
	BEWO ( Sun et al., 2025 )	0.26	2.05	0.56	1.40	566.57	57.30
	Ambit	0.46	0.71	0.51	0.95	261.45	44.59

Table 2: Generation content quality. Bold and underlined values indicate the best and second-best results within each evaluated domain; displayed ties receive the same marking.

System	Paired CLAP $\uparrow$	FD CLAP $\downarrow$	FAD $\downarrow$	FD PANNs $\downarrow$	KL $\downarrow$	LSD $\downarrow$
Audio-Omni ( Tian et al., 2026c )	0.46	0.23	0.91	23.75	2.04	2.09
MMEdit ( Tao et al., 2025 )	0.48	0.31	1.15	22.73	1.37	2.45
SwanWeave ( Lei et al., 2026a )	0.68	0.15	0.57	11.40	0.92	1.65
SmartDJ ( Lan et al., 2025 )	0.63	0.17	0.70	19.30	1.16	1.89
ZETA ( Manor and Michaeli, 2024 )	0.41	0.38	1.82	33.56	1.63	2.86
AudioEditor–BEWO ( Jia et al., 2025 )	0.44	0.36	1.87	25.12	1.39	2.43

Table 3: Editing content quality and spectral reconstruction. The KL here is KL-PANNs. Bold and underlined values indicate the best and second-best results. Spatial metrics are reported in Table 5 .

	Scene spatial quality
System	GCC $\downarrow$	CRW $\downarrow$	FSAD $\downarrow$	DOA $\downarrow$	Azimuth MAE $\downarrow$	Ang. traj. RMSE $\downarrow$	wCS $\uparrow$
BEWO + AS	34.71	73.22	0.03	1.04	1.00	1.13	0.11
TangoFlux + AS	36.42	67.88	0.04	1.03	0.98	1.12	0.09
Qwen3.5-4B + DiT	31.07	64.36	0.03	1.07	1.02	1.18	0.27
Ambit	12.59	8.75	0.02	0.51	0.40	0.63	0.56

Table 4: Spatial generation at scene and object levels. Scene direction errors use mixtures; object metrics use isolated source intervals. Angular errors are in radians. Bold marks the best score.

	Scene spatial quality
System	GCC $\downarrow$	CRW $\downarrow$	FSAD $\downarrow$	DOA $\downarrow$	Az. MAE $\downarrow$	Ang. traj. RMSE $\downarrow$	wCS $\uparrow$
SwanWeave	30.77	41.11	0.04	1.11	1.06	1.25	0.27
Qwen3-Omni + DiT	20.45	26.82	0.02	0.86	0.85	1.01	0.36
Ambit	14.81	11.95	0.01	0.39	0.36	0.52	0.71

Table 5: Spatial editing: scene metrics (upper), preservation and object metrics on isolated intervals (lower). Angular errors are in radians; level changes in dB. Bold marks the best score.

	Content quality					Reconstruction and binaural
System	Paired CLAP $\uparrow$	FD CLAP $\downarrow$	FAD $\downarrow$	FD PANNs $\downarrow$	KL $\downarrow$	LSD $\downarrow$	GCC $\downarrow$	CRW $\downarrow$	FSAD $\downarrow$
Ambit	0.839	0.047	0.242	6.012	0.455	1.268	14.805	11.947	0.0142
Ambit + SFT	0.855	0.042	0.232	5.398	0.402	1.110	11.897	7.518	0.0119
Ambit + ReFL	0.853	0.044	0.233	5.505	0.391	1.146	12.053	7.845	0.0116
Ambit + GRPO	0.851	0.043	0.228	5.553	0.417	1.149	12.553	8.923	0.0117
Ambit + AR OPSD	0.852	0.042	0.237	5.472	0.413	1.109	11.903	8.193	0.0117

Table 6: Editing quality after post-training. AR OPSD and DiT OPSD retain GT supervision in the other branch. Bold and underlined values indicate the best and second-best scores per metric.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Representation and constraints
Duration	At most 648 latent frames at a 1,024-sample hop (approximately 15.05 s)
Room	Dry, moderate, reverberant, or outdoor
Sources	One to four plan local source IDs.
Content	Source kind, description, and verbatim transcript for speech
Activity	Interval $[o_{i},f_{i})$ with $0\leq o_{i}<f_{i}\leq d$
Trajectory	Static pose $(\theta,\psi,\rho)$ or linear motion with 2 to 8 keyframes

Appendix

Table 7: ScenePlan field ranges. Times, angles, and distances use seconds, degrees, and metres.

Class	Dimensions $L\times W\times H$ (m)	RT60 (s)	Image order
Dry	$[6,8]\times[5,7]\times[3,4]$	$[0.16,0.28]$	5 to 7
Moderate	$[8,12]\times[7,10]\times[3.2,4.8]$	$[0.38,0.62]$	6 to 9
Reverberant	$[12,18]\times[10,15]\times[4.5,7]$	$[0.78,1.20]$	6 to 9
Outdoor	$[10,16]\times[10,16]\times[5,8]$	—	0

Appendix

Table 8: Room sampling ranges for synthetic scenes. Outdoor scenes use no reflections.

Partition	Train	Validation	Test
Generation examples	1.6M	32k	8k
Editing pairs	1M	20k	5k

Appendix

Table 9: Dataset partitions. Generation training combines synthetic scenes and real recordings. The 8k generation test examples include 50 FOA clips of 10 s each from the annotated STARSS23 dev-test split.

Plan fields	AR mechanism	Prediction and completion
Room and duration	AR logits	Four room classes and 1 to 648 duration frames.
Source inventory	Inventory head	Predict source count and sound, music, or speech categories.
Description and transcript	Learned span selectors	Select literal request spans and encode them with the text codec.
Motion and azimuth	Qualitative head	Static or linear; eight compass categories for start and end directions.
Onset and offset	Qualitative head	Five relative time phases; select a feasible phase pair and complete integer frame values.
Radial motion	Qualitative head	Free, approaching, or receding; complete endpoint distances.

Appendix

Table 10: Source and scene fields completed by the AR planner.

Component	Optimization objective	Learning rates
Ambit VAE	FOA reconstruction	VAE $1.5\times 10^{-4}$ ; discriminator $3\times 10^{-4}$
Transfusion	Joint AR and DiT training	Adapters and heads $10^{-4}$ ; shared blocks $2\times 10^{-6}$

Appendix

Table 11: Learning rates for VAE pretraining and the joint optimization stage. All updates use AdamW.

System	Quality MOS	Instruction adherence	Spatial adherence	Spatial naturalness
BEWO + AS	2.93	2.96	2.83	2.89
TangoFlux + AS	2.77	2.84	2.63	2.76
Ambit	3.11	3.25	3.37	3.03

Appendix

Table 12: Subjective evaluation of FOA generation and editing. All dimensions use a five-point scale; higher is better. Entries report mean ratings over valid responses. The highest mean in each column is highlighted in bold.

Planner	Executable $\uparrow$	Source count accuracy $\uparrow$
Generation
Ambit AR	100.00	100.00
Qwen3.5-0.8B	22.88	22.69
Qwen3.5-2B	20.19	20.38
Qwen3.5-4B	98.08	98.06
Qwen3.5-9B	87.81	87.83

Appendix

Table 13: Planner executability and source count accuracy (%).

Teacher	Update budget	Batch	Learning rates	Loss weights
AR	1,000	128	$5\times 10^{-6}$ / $3.75\times 10^{-7}$	$(a,b,c)=(0.1,0.3,0.5)$
DiT	3,000	128	$3\times 10^{-6}$ / $10^{-4}$	RF: $1$

Appendix

Table 14: Teacher preparation budgets and settings. Batches are global.

Spatial Audio

Spatial audio studies how sound is captured, represented, generated, and perceived with direction, distance, motion, and room acoustics.

See all 100 papers

One-Stage Multi-Task Instruction-Guided 3D Spatial Audio Editing
Sep 4, 2026Ke Lei, Chenyuhao Wen, Yu Zhang +9Audio Editing Spatial Audio
Zhejiang University · ByteDance
PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation
Aug 30, 2026Lingfeng Yao, Chenpei Huang, Xingke Yang +5Spatial Audio Text-to-Audio Generation
University of Houston · Stevens Institute of Technology · Waseda University
Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer
May 29, 2026Ke Lei, Yu Zhang, Changhao Pan +4Text-to-Audio Generation Audio-Video Generation
1Zhejiang University, China · 2ByteDance

Sep 4, 2026J/K move · Enter open · S save

One-Stage Multi-Task Instruction-Guided 3D Spatial Audio Editing

Ke Lei, Chenyuhao Wen, Yu Zhang, Wenxiang Guo+8

Zhejiang University · ByteDance

Audio Editing Spatial Audio

Open paper AskPDF arXiv

Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.

Abstract

Spatial Audio

Explore similar work

One-Stage Multi-Task Instruction-Guided 3D Spatial Audio Editing

PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation

Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer