As generative audio models grow in complexity, the computational and ecological costs of synthesizing everyday sounds have become increasingly prohibitive, often requiring industrial-scale resources and massive datasets. In this paper, we present SCAPES: a Semantically Conditioned Autoregressive Prior for Environmental Sounds. SCAPES is a lightweight, resource-efficient generative model designed to synthesize high-fidelity environmental textures through high-level semantic control. By operating on the continuous latent manifold of a neural audio codec, our approach bypasses the rigid structural constraints inherent to discrete tokenization. We propose a segmentation strategy that decomposes audio into overlapping segments, enabling a Continuous Normalizing Flow (CNF) to model the evolution of latent trajectories using Flow Matching. Our experiments demonstrate that a 36-million parameter instance of SCAPES can be trained on limited, uncurated datasets using a single consumer-grade GPU. Notably, convergence is achieved after training for approximately twice the source audio duration, yielding high-fidelity outputs with robust long-term stability and semantic consistency. Furthermore, we showcase the model's capacity for smooth semantic interpolation, providing a flexible and accessible tool for open research and creative sound design. Code, pretrained weights, audio examples, and an interactive demo are publicly available on our project page https://cordutie.github.io/projects/scapes.html
Figures & tables
Figure 1: Data representation and annotations. Source audio (top row waveform) is decomposed into overlapping segments (pink and red waveforms) and encoded with EnCodec into atoms (pink and red matrices). Each atom is annotated with a semantic embedding (cyan vector) computed from a context window (cyan waveform), extended through randomized repetitions to satisfy CLAP’s minimum input length. During training, the model is tasked to predict the target atom (red matrix) from the preceding atoms in the memory buffer and its corresponding semantic embedding.
Figure 2: Inference architecture. The Transformer-based vector field Fθ is conditioned on past atomic memory via cross-attention and semantic context via AdaLN. An ODE solver integrates the field from t=0 to t=1 to generate the target atom.
FAD (VGGish)
KAD (CLAP)
KAD (TexStat)
Source
SCAPES
RAVE
SCAPES
RAVE
SCAPES
RAVE
Applause
4.35
39.06
33.16
47.77
38.1
65.32
Bonfire
3.66
3.65
49.37
37.64
13.28
15.45
Bubbles
5.03
21.05
31.89
47.82
10.9
37.87
Forests
11.05
3.49
61.71
52.56
37.28
16.38
Helicopter
6.9
8.0
22.65
27.8
11.17
28.88
Table 1: Quantitative comparison of generation quality across various metrics and models. The best results for each metric (lower is better) are highlighted in bold .
Figure 3: Class-wise distance matrices for generated versus original audio. Lower values indicate greater acoustic similarity. From left to right: FAD (VGGish), KAD (CLAP), and KAD (TexStat), latter two sharing a colorbar. FAD and KAD are not directly comparable. The KAD scores, however, share the same scale due to the use of a common scale invariant kernel, although their interpretations differ.
Cosine Similarity (CLAP)
Source
30 s
60 s
120 s
Applause
0.76±0.12
0.71±0.09
0.78±0.09
Bonfire
0.73±0.11
0.73±0.12
0.74±0.12
Bubbles
0.86±0.04
0.89±0.03
0.88±0.04
Forests
0.75±0.06
0.81±0.07
0.77±0.06
Helicopter
0.93±0.02
0.93±0.03
0.86±0.04
Table 2: Long-term stability analysis. Values represent the mean cosine similarity ( ± standard deviation, where 1 denotes a perfect match) between the target context embedding and 5-second segments of generation after different periods of time.
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
We present an agentic soundscape construction framework for controllable compositional audio generation that makes explicit the scene planning, source selection, temporal layout, and rendering steps typically handled implicitly by single-shot text-to-audio models. An LLM-based agent converts user intent into an executable scene plan, acquires assets through retrieval and on-demand generation, renders controllable multi-event mixtures, and exports aligned scene metadata. The framework also supports human-in-the-loop interaction through user-guided tool selection and editable scene plans. Together, these components provide an inspectable and reusable approach to controllable soundscape synthesis and scalable audio-language data construction. Listener studies and objective metrics demonstrate competitive generation performance against text-to-audio baselines, while models trained with agent-generated data consistently outperform real-only baselines in downstream audio reasoning. Code, demos, and listening-test results are available at https://haozhang6720.github.io/SoundscapeAgentDemoPage/.
Hao Zhang, Yiwen Zhao, Yixuan Zhang +2
Wuhan University, Wuhan, China · Tencent Hunyuan, Bellevue, WA, USA
Audio generation has long been fragmented, with speech, music, and sound effects produced by domain-specific models that fail to jointly generate coherent audio scenes from a single description. The key obstacles are insufficient fine-grained supervision for real-world mixed audio and limited acoustic representations for modeling concurrent audio components. We present Dasheng AudioGen, a unified framework for generating general mixed-audio scenes from text. Dasheng AudioGen introduces structured multi-view captions, which explicitly decouple complex acoustic scenes into complementary description views, thereby enabling fine-grained control over audio layers. Furthermore, we employ a high-dimensional unified semantic-acoustic representation as the shared latent space. It injects semantic priors that facilitate cross-modal training convergence, while its high-dimensional feature space provides sufficient capacity to disentangle and fuse concurrent audio components effectively. With these designs, a simple flow-matching DiT achieves high-quality end-to-end audio scene generation. We also establish a comprehensive evaluation pipeline for audio scene generation. Experiments demonstrate that Dasheng AudioGen achieves performance approaching real-world recordings in mixed-audio categories, while remaining competitive with specialized models in single-type generation tasks. Demos are available at https://nieeim.github.io/Dasheng-AudioGen-Web/.
Jiahao Mei, Heinrich Dinkel, Yadong Niu +7
1X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · 2MiLM Plus, Xiaomi Inc., Beijing, China
Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: https://noizai.github.io/WorldSonus/
Pengjun Fang, Jingyi Fa, Kam Man Wu +9
The Hong Kong University of Science and Technology · Noiz AI · MetaX +1