cs.SDNov 2, 2026

Spatial Transfusion: From Structured Spatial Scene Understanding to Generation and Editing

Authors: Jaeyeon Kim, Sebastian Braun, Hannes Gamper

Organizations: Carnegie Mellon University · Microsoft Corporation

Abstract

Spatial audio conveys not only what sounds are present, but where they originate and how they interact with the environment. Recent audio-language models increasingly understand and reason about such scenes, yet spatial audio understanding, generation, and editing remain largely separate. We present Spatial Transfusion, a unified first-order ambisonics (FOA) framework that extends a pretrained spatial audio understanding model to generation and editing through flow matching over VAE latents. We formulate editing as a structured scene transformation over a taxonomy of atomic edits. Keeping the pretrained backbone frozen, we introduce modality-based LoRA routing and a curriculum that progressively adds generation, editing, and natural-language interaction. Experiments show that Spatial Transfusion largely preserves spatial understanding while outperforming full fine-tuning in generation and editing. The resulting model supports controllable FOA generation and editing through both structured and natural-language interfaces. Audio samples are available at https://microsoft.github.io/spatial-transfusion/

Figures & tables

Explore similar work

CardsList
  1. Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

    Oct 25, 2026Kam Man Wu, Yu Zhang, Pengjun Fang +4Spatial AudioAudio Editing

  2. PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation

    Aug 30, 2026Lingfeng Yao, Chenpei Huang, Xingke Yang +5Spatial AudioText-to-Audio Generation