Spatial audio authoring coordinates source decisions with soundfield synthesis. For editing, plans specify what to change and retain, while reference audio supplies further acoustic detail. We present AMBIT, a unified framework for native First-Order Ambisonics (FOA) generation and editing with three connected contributions. First, a shared Transfusion backbone couples autoregressive planning with rectified-flow rendering through executable ScenePlans that specify source content, activity, and 3D trajectories. For editing, the plan describes the full target scene, including retained sources; contrastive features ground the requested change, while full reference latents supply residual acoustic detail for preservation. Second, an FOA-aware continuous VAE combines shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to preserve content and inter-channel structure. Third, joint Transfusion on-policy self-distillation (OPSD) refines planning on student-generated prefixes and rendering at sampling states reached under student plans, with target-conditioned teachers supervising both modes of the shared student. Evaluations across speech, music, and sound show improved generation content and spatial quality, acoustic fidelity during instruction-guided editing, and unedited-source preservation over the compared baselines. Project page: https://wjm-george.github.io/AMBIT/
Figures & tables
Figure 1: Overview of Ambit capabilities. (1) Spatial audio generation creates multi-source 3D scenes from text, controlling content, timing, and trajectories. (2) Instruction-guided spatial and temporal editing modifies selected sources, preserving unmodified content and spatial structure.
Figure 2: Overview of Ambit . (a) A shared Transformer predicts executable ScenePlan s and renders native FOA through latent flow conditioned on compiled plans. Editing uses source latents in both modes and Ambit CLAP features in planning. (b) OPSD distills token and clean latent predictions on student prefixes and visited states, jointly updating the shared Transformer and AR/DiT modules. (c) The VAE combines shared-decoder auxiliary W reconstruction, grouped KL, and spatial covariance supervision.
System/configuration
W SI-SDR (dB) ↑
Direction ( ∘ ) ↓
Intensity coherence error ↓
W paired CLAP ↑
DAC
5.57
5.13
0.08
0.88
EnCodec
0.41
12.25
0.15
0.85
WavTokenizer
−19.41
56.66
0.35
0.63
Ambit VAE
6.39
4.96
0.04
0.90
Phase (base)
5.63
6.51
0.06
0.86
+ SCM
6.08
5.14
0.05
0.88
Table 1: FOA reconstruction and cumulative ablation. The upper block compares codecs on native scenes; each “+” below adds to the previous row, with grouped KL replacing uniform KL. Ablations use matched updates from scratch and 24 kHz evaluation. Bold marks the best per block.
Domain
System
Paired CLAP ↑
FAD ↓
KL PANNs ↓
KL PaSST ↓
FD PaSST ↓
FD OpenL3 ↓
Music
AudioX-Turbo ( Tian et al., 2026a )
0.36
1.07
0.68
1.58
234.65
37.40
AudioX ( Tian et al., 2026b )
0.35
1.30
0.74
1.56
243.68
46.67
Stable Audio Open ( Evans et al., 2025 )
0.18
2.08
0.90
1.81
410.64
68.72
TangoFlux ( Hung et al., 2026 )
0.36
1.01
0.59
1.43
319.37
92.09
BEWO ( Sun et al., 2025 )
0.26
2.05
0.56
1.40
566.57
57.30
Ambit
0.46
0.71
0.51
0.95
261.45
44.59
Table 2: Generation content quality. Bold and underlined values indicate the best and second-best results within each evaluated domain; displayed ties receive the same marking.
System
Paired CLAP ↑
FD CLAP ↓
FAD ↓
FD PANNs ↓
KL ↓
LSD ↓
Audio-Omni ( Tian et al., 2026c )
0.46
0.23
0.91
23.75
2.04
2.09
MMEdit ( Tao et al., 2025 )
0.48
0.31
1.15
22.73
1.37
2.45
SwanWeave ( Lei et al., 2026a )
0.68
0.15
0.57
11.40
0.92
1.65
SmartDJ ( Lan et al., 2025 )
0.63
0.17
0.70
19.30
1.16
1.89
ZETA ( Manor and Michaeli, 2024 )
0.41
0.38
1.82
33.56
1.63
2.86
AudioEditor–BEWO ( Jia et al., 2025 )
0.44
0.36
1.87
25.12
1.39
2.43
Table 3: Editing content quality and spectral reconstruction. The KL here is KL-PANNs. Bold and underlined values indicate the best and second-best results. Spatial metrics are reported in Table 5 .
Scene spatial quality
System
GCC ↓
CRW ↓
FSAD ↓
DOA ↓
Azimuth MAE ↓
Ang. traj. RMSE ↓
wCS ↑
BEWO + AS
34.71
73.22
0.03
1.04
1.00
1.13
0.11
TangoFlux + AS
36.42
67.88
0.04
1.03
0.98
1.12
0.09
Qwen3.5-4B + DiT
31.07
64.36
0.03
1.07
1.02
1.18
0.27
Ambit
12.59
8.75
0.02
0.51
0.40
0.63
0.56
Table 4: Spatial generation at scene and object levels. Scene direction errors use mixtures; object metrics use isolated source intervals. Angular errors are in radians. Bold marks the best score.
Scene spatial quality
System
GCC ↓
CRW ↓
FSAD ↓
DOA ↓
Az. MAE ↓
Ang. traj. RMSE ↓
wCS ↑
SwanWeave
30.77
41.11
0.04
1.11
1.06
1.25
0.27
Qwen3-Omni + DiT
20.45
26.82
0.02
0.86
0.85
1.01
0.36
Ambit
14.81
11.95
0.01
0.39
0.36
0.52
0.71
Table 5: Spatial editing: scene metrics (upper), preservation and object metrics on isolated intervals (lower). Angular errors are in radians; level changes in dB. Bold marks the best score.
Content quality
Reconstruction and binaural
System
Paired CLAP ↑
FD CLAP ↓
FAD ↓
FD PANNs ↓
KL ↓
LSD ↓
GCC ↓
CRW ↓
FSAD ↓
Ambit
0.839
0.047
0.242
6.012
0.455
1.268
14.805
11.947
0.0142
Ambit + SFT
0.855
0.042
0.232
5.398
0.402
1.110
11.897
7.518
0.0119
Ambit + ReFL
0.853
0.044
0.233
5.505
0.391
1.146
12.053
7.845
0.0116
Ambit + GRPO
0.851
0.043
0.228
5.553
0.417
1.149
12.553
8.923
0.0117
Ambit + AR OPSD
0.852
0.042
0.237
5.472
0.413
1.109
11.903
8.193
0.0117
Table 6: Editing quality after post-training. AR OPSD and DiT OPSD retain GT supervision in the other branch. Bold and underlined values indicate the best and second-best scores per metric.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Representation and constraints
Duration
At most 648 latent frames at a 1,024-sample hop (approximately 15.05 s)
Room
Dry, moderate, reverberant, or outdoor
Sources
One to four plan local source IDs.
Content
Source kind, description, and verbatim transcript for speech
Activity
Interval [oi,fi) with 0≤oi<fi≤d
Trajectory
Static pose (θ,ψ,ρ) or linear motion with 2 to 8 keyframes
Appendix
Table 7: ScenePlan field ranges. Times, angles, and distances use seconds, degrees, and metres.
Class
Dimensions L×W×H (m)
RT60 (s)
Image order
Dry
[6,8]×[5,7]×[3,4]
[0.16,0.28]
5 to 7
Moderate
[8,12]×[7,10]×[3.2,4.8]
[0.38,0.62]
6 to 9
Reverberant
[12,18]×[10,15]×[4.5,7]
[0.78,1.20]
6 to 9
Outdoor
[10,16]×[10,16]×[5,8]
—
0
Appendix
Table 8: Room sampling ranges for synthetic scenes. Outdoor scenes use no reflections.
Partition
Train
Validation
Test
Generation examples
1.6M
32k
8k
Editing pairs
1M
20k
5k
Appendix
Table 9: Dataset partitions. Generation training combines synthetic scenes and real recordings. The 8k generation test examples include 50 FOA clips of 10 s each from the annotated STARSS23 dev-test split.
Plan fields
AR mechanism
Prediction and completion
Room and duration
AR logits
Four room classes and 1 to 648 duration frames.
Source inventory
Inventory head
Predict source count and sound, music, or speech categories.
Description and transcript
Learned span selectors
Select literal request spans and encode them with the text codec.
Motion and azimuth
Qualitative head
Static or linear; eight compass categories for start and end directions.
Onset and offset
Qualitative head
Five relative time phases; select a feasible phase pair and complete integer frame values.
Radial motion
Qualitative head
Free, approaching, or receding; complete endpoint distances.
Appendix
Table 10: Source and scene fields completed by the AR planner.
Component
Optimization objective
Learning rates
Ambit VAE
FOA reconstruction
VAE 1.5×10−4 ; discriminator 3×10−4
Transfusion
Joint AR and DiT training
Adapters and heads 10−4 ; shared blocks 2×10−6
Appendix
Table 11: Learning rates for VAE pretraining and the joint optimization stage. All updates use AdamW.
System
Quality MOS
Instruction adherence
Spatial adherence
Spatial naturalness
BEWO + AS
2.93
2.96
2.83
2.89
TangoFlux + AS
2.77
2.84
2.63
2.76
Ambit
3.11
3.25
3.37
3.03
Appendix
Table 12: Subjective evaluation of FOA generation and editing. All dimensions use a five-point scale; higher is better. Entries report mean ratings over valid responses. The highest mean in each column is highlighted in bold.
Planner
Executable ↑
Source count accuracy ↑
Generation
Ambit AR
100.00
100.00
Qwen3.5-0.8B
22.88
22.69
Qwen3.5-2B
20.19
20.38
Qwen3.5-4B
98.08
98.06
Qwen3.5-9B
87.81
87.83
Appendix
Table 13: Planner executability and source count accuracy (%).
Teacher
Update budget
Batch
Learning rates
Loss weights
AR
1,000
128
5×10−6 / 3.75×10−7
(a,b,c)=(0.1,0.3,0.5)
DiT
3,000
128
3×10−6 / 10−4
RF: 1
Appendix
Table 14: Teacher preparation budgets and settings. Batches are global.
Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.
Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.
Lingfeng Yao, Chenpei Huang, Xingke Yang +5
University of Houston · Stevens Institute of Technology · Waseda University
Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose SwanSphere, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. SwanSphere mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video-Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization (ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that SwanSphere achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Demos can be found at: https://swanaigc.github.io.