AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing

William Chen; Prem Seetharaman; Rithesh Kumar; Oriol Nieto; Shinji Watanabe; Justin Salamon; Zeyu Jin

cs.SDFeb 19, 2026

AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing

Authors: William Chen, Prem Seetharaman, Rithesh Kumar, Oriol Nieto, Shinji Watanabe, Justin Salamon, Zeyu Jin

Organizations: Adobe Research · Language Technologies Institute, Carnegie Mellon University · OpenAI

Abstract

Despite recent breakthroughs, audio foundation models struggle in processing complex multi-source acoustic scenes. We refer to this challenging domain as audio stories, which can have multiple speakers and background/foreground sound effects. Compared to traditional audio processing tasks, audio stories introduce new layers of semantic, temporal, and physical complexity. To address this challenge, we propose AudioChat, a framework for developing audio foundation models that can generate, edit, and understand audio stories. AudioChat introduces a new paradigm in which LLM-based toolcalling agents simulate interactions between users and the system, and these simulated dialogues are used as training data. We also introduce a novel Audio Transfusion Forcing objective to train the AudioChat model, allowing it to simultaneously decompose high-level instructions via structured chain-of-thought reasoning and perform interactive multi-turn audio understanding/generation. To evaluate generation and editing performance, we develop three new metrics that directly measure task performance instead of relying upon distribution-based scoring. We highly encourage readers to visit our demo to better understand the capabilities of AudioChat: https://wanchichen.github.io/audiochat/.

Figures & tables

Figure 1: Overview of AudioChat’s capabilities in multi-source audio storytelling and editing. AudioChat leverages structured CoT reasoning to break down the user prompt into individual sound effects, allowing for fine-grained control and interpretability.

Figure 2: Overview of data generation with AudioCopilot. AudioCopilot seeds its dialogue simulation with a text string and performs tool-calling to render and mix each separate sound source.

Figure 3: Model architecture overview. Tokens are input through a modality-specific tokenizer, the output of which is concatenated and input into the LM. The tokens are then routed through a modality-specific prediction head.

Figure 4: Visualization of Transfusion Forcing: a different diffusion timestep is sampled for each audio span. Audio tokens can attend to all other tokens within a span or in previous spans. Text tokens can only attend to previous text and audio tokens.

		Quality		Consistency		Instruction
	Difference	KAD( $\downarrow$ )	FAD( $\downarrow$ )	Human( $\uparrow$ )	$\Delta$ multiFLAM( $\downarrow$ )	Human( $\uparrow$ )	editFLAM( $\uparrow$ )
Topline
Ground Truth	-	-	-	4.12 $\pm 0.05$	8.2	3.68 $\pm 0.06$	19.4
Baselines
DiT	Architecture	1.74	0.07	3.71 $\pm 0.05$	12.5	2.21 $\pm 0.06$	2.7
Diffusion LLM	No CoT	3.81	0.11	2.32 $\pm 0.06$	27.8	2.39 $\pm 0.06$	18.1

Table 1: Audio Editing Results: Audio generation quality is measured in KAD/FAD, consistency with original audio via human ratings (1-5) and

\Delta

multiFLAM, and edit instruction following with human ratings (1-5) and editFLAM.

Figure 5: editFLAM( $\uparrow$ ) and multiFLAM( $\downarrow$ ) results on the three semantic editing tasks in StoryGen-Eval.

	KAD( $\downarrow$ )	multiFLAM( $\uparrow$ )	Latency( $\downarrow$ )
SAO	7.98	35.6	9.1
LLM+SAO	10.77	32.2	11.5
WavJourney	10.82	47.1	628
AudioChat	2.52	44.1	32

Table 2: Audio Storytelling Results

	Diar.	AC
	tcpWER	multiFLAM
WhisperX	55.9	-
Whisper-Story	5.5	88.1
AudioChat	9.7	86.3

Table 3: Fine-grained Audio Understanding results

	T2A		Editing
	FAD	CLAP	$\Delta$ multiFLAM	editFLAM
SAO	0.13	63.4	-	-
TangoFlux	0.23	56.1	-	-
DiT	0.14	61.6	12.5	2.7
Dense	0.12	65.3	15.5	17.2
MoT	0.13	65.8	16.4	9.6

Table 4: Architecture Ablations on Storytelling and Editing

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Overview of the MoT architecture.

	Stage 1	Stage 2	Stage 3
Data Sources	ASR, TTS, AC, T2A, TextLM	Stage 1 + Stories	Stage 1 + Stories HQ
Optimization
Training Steps	100K	100K	20K
Optimizer	AdamW	AdamW	AdamW
Learning Rate	2.5e-5	2.5e-5	1.0e-4
Scheduler	Cosine Decay	Cosine Decay	Cosine Decay

Appendix

Table 5: AudioChat Training Settings

Figure 7: Visualization of AudioChat’s fine-grained audio captioning capabilities.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
Feb 5, 2026Jinchuan Tian, Haoran Wang, Bo-Hao Su +14Audio Understanding Audio-Language Models
1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research
Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text
May 27, 2026Jiahao Mei, Heinrich Dinkel, Yadong Niu +7Text-to-Audio Generation Audio Representation Learning
1X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · 2MiLM Plus, Xiaomi Inc., Beijing, China
Unified Audio Intelligence Without Regressing on Text Intelligence
Jul 6, 2026Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim +17TTS Synthesis Unified Multimodal Models
NVIDIA

Feb 5, 2026J/K move · Enter open · S save

Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions

Jinchuan Tian, Haoran Wang, Bo-Hao Su, Chien-yu Huang+13

1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research

Audio Understanding Audio-Language Models Audio Captioning Audio Generation

Open paper AskPDF arXiv

Current audio foundation models typically rely on rigid, task-specific supervision, addressing isolated factors of audio rather than the whole. In contrast, human intelligence processes audio holistically, seamlessly bridging physical signals with abstract cognitive concepts to execute complex tasks. Grounded in this philosophy, we introduce Bagpiper, an 8B audio foundation model that interprets physical audio via rich captions, i.e., comprehensive natural language descriptions that encapsulate the critical cognitive concepts inherent in the signal (e.g., transcription, audio events). By pre-training on a massive corpus of 600B tokens, the model establishes a robust bidirectional mapping between raw audio and this high-level conceptual space. During fine-tuning, Bagpiper adopts a caption-then-process workflow, simulating an intermediate cognitive reasoning step to solve diverse tasks without task-specific priors. Experimentally, Bagpiper outperforms Qwen-2.5-Omni on MMAU and AIRBench for audio understanding and surpasses CosyVoice3 and TangoFlux in generation quality, capable of synthesizing arbitrary compositions of speech, music, and sound effects. To the best of our knowledge, Bagpiper is among the first works that achieve unified understanding generation for general audio. Model, data, and code are available at Bagpiper Home Page.

Abstract

Audio-Language Models

Explore similar work

Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions

Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text

Unified Audio Intelligence Without Regressing on Text Intelligence