Organizations: Hong Kong University of Science and Technology, Hong Kong SAR, China · WeChat Vision, Tencent Inc, China · Peking University, Beijing, China
Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni.
Figures & tables
Figure 1. An overview of the Audio-Omni framework and its capabilities. (Top) Our decoupled architecture connects a frozen MLLM for understanding with a trainable DiT for audio synthesis via a feature projector. (Middle) A showcase of the model’s unified capabilities across understanding, generation, and editing. (Bottom) A demonstration of remarkable emergent abilities inherited from the MLLM.
Figure 2. Overview of the hybrid pipeline for constructing our AudioEdit dataset. The pipeline consists of two parallel branches to ensure both data authenticity and scale. The Real Data Branch (left) mines editing pairs from real-world datasets (e.g., VGGSound) by first using an MLLM (Gemini) for category identification, followed by a dedicated segmentation model (SAM-Audio) for source separation. Concurrently, the Synthesis Data Branch (right) leverages the Scaper toolkit to programmatically generate a large volume of precisely annotated editing scenarios. This hybrid strategy yields a dataset that combines the acoustic fidelity of natural audio with the large-scale diversity needed for robust model training.
Task Type
Data Source
Train Samples
Test Samples
Add
Real Data
50K
500
Synthesis Data
150K
-
Remove
Real Data
50K
500
Synthesis Data
150K
-
Extract
Real Data
50K
500
Synthesis Data
150K
-
Table 1. Statistics of our proposed audio editing dataset AudioEdit.
Figure 3. The Audio-Omni Framework. Our framework utilizes a decoupled design with two distinct conditioning streams to guide a trainable DiT backbone. The High-Level Semantic Features stream provides global, instructional guidance. It is formed by concatenating features from a frozen MLLM (MM Features) with character-level embeddings from a trainable Transcript Encoder. The Low-Level Signal Features stream offers precise, temporal guidance for editing and synchronization. It combines features from Synchformer and Mel Encoder. These two streams are injected into the DiT via different mechanisms: the high-level stream as context for cross-attention, and the low-level stream concatenated with the input noise and time embedding.
Methods
MMSU ↑
MMAU ↑
Specialized Models
Audio Flamingo3 ( Goel et al., 2025 )
61.40
72.42
Qwen2-Audio-Instruct ( Chu et al., 2024 )
53.27
57.40
Qwen2.5-Omni-3B ( Xu et al., 2025 )
56.83
63.30
Unified Models
Ming-Omni ( AI et al., 2025 )
47.53
70.80
Table 2. Quantitative results on multimodal understanding benchmarks.
Methods
T2A_FAD ↓
T2M_FAD ↓
V2A_FAD ↓
V2M_FAD ↓
TTS_WER ↓
Specialized Models
Tango2 ( Majumder et al., 2024 )
3.20
-
-
-
-
MMAudio ( Cheng et al., 2025 )
4.71
-
2.04
-
-
Stable-Audio-Open ( Evans et al., 2024 )
3.15
3.23
-
-
-
MusicGen ( Copet et al., 2024 )
-
3.94
-
-
-
VATT ( Liu et al., 2024b )
-
-
2.55
-
-
Table 3. Quantitative results on multimodal generation benchmarks.
Methods
AE_FAD ↓
AE_LSD ↓
CLAP ↑
Specialized Models
ZETA ( Manor and Michaeli, 2024 )
3.81
3.80
0.30
SDEdit ( Meng et al., 2021 )
3.51
4.40
0.22
MMEDIT ( Tao et al., 2025 )
3.95
4.05
0.15
Unified Models
Audio-Omni
3.27
2.27
0.32
Table 4. Quantitative results on audio editing benchmarks.
Figure 4. Qualitative showcase of Audio-Omni’s capabilities , including (a) knowledge-augmented generation, (b) in-context generation, (c) zero-shot voice conversion, and (d) zero-shot speech editing.
Setting
KL ↓
IS ↑
FD ↓
FAD ↓
LSD ↓
Only Real
1.27
5.54
20.69
2.67
1.84
Only Synthesis
1.93
5.04
37.96
3.80
5.17
Syn. + Real (Ours)
1.30
5.93
20.48
2.48
1.82
Table 5. Ablation study on dataset composition for audio editing training.
Context
Cat.
T2A ↓
V2A ↓
TTS ↓
AE ↓
mm, trans, sync, mel
none
40.91
25.61
32.46
5.47
mm, trans, sync
mel
38.26
26.46
25.88
4.03
mm, trans
sync, mel
28.90
18.55
19.33
3.62
mm
trans, sync, mel
60.58
56.88
59.20
4.88
Table 6. Ablation study on conditioning injection strategies.
Feature Source
T2A
T2M
IS ↑
FD ↓
IS ↑
FAD ↓
Last Layer ( -1 )
9.36
4.21
2.90
3.26
Penultimate Layer ( -2 )
11.26
2.75
3.46
3.05
MetaQuery
7.44
8.34
2.38
8.15
Query
8.55
5.31
2.69
4.22
Table 7. Ablation study on the source of conditional features from the MLLM.
Task
Encoder
KL ↓
IS ↑
FD ↓
FAD ↓
T2A
T5
1.51
11.27
15.78
2.75
CLAP
1.81
9.64
16.58
1.84
Qwen
1.25
11.65
12.21
2.14
V2A
CLIP
2.20
7.42
12.21
2.80
VideoMAE
2.29
6.58
14.72
3.06
Qwen
2.00
8.08
8.50
2.23
Table A1. Ablation study on different encoders. We evaluate the impact of different encoders for text-to-audio (T2A) and video-to-audio (V2A) tasks.
Table A3. Detailed results on audio editing tasks. We report FAD/LSD for each task and their average.
Task
Hours
Datasets
T2A
1.4k
AudioCaps ( Kim et al., 2019 ) WavCaps ( Mei et al., 2024 ) AudioSetCaps ( Bai et al., 2025 ) AudioTime ( Xie et al., 2025a ) IF-Caps ( Tian et al., 2025b )
V2A
0.7k
VGGSound ( Chen et al., 2020 ) AudioSet Strong ( Hershey et al., 2021 )
T2M
17k
IF-Caps ( Tian et al., 2025b ) MUCaps ( Liu et al., 2024a )
V2M
16k
V2M ( Tian et al., 2025c ) MMTrail ( Chi et al., 2024 )
Speech
6k
Audio-FLAN (English subset) ( Xue et al., 2025 )
Editing
3k
AudioEdit (Sec. 3 )
Table A4. Training data summary across tasks.
Method
KL ↓
IS ↑
FD ↓
FAD ↓
Text-to-Audio (T2A)
Tango2 ( Majumder et al., 2024 )
1.11
10.37
12.22
3.20
MMAudio ( Cheng et al., 2025 )
1.35
12.03
12.63
4.71
Stable-Audio-Open ( Evans et al., 2024 )
2.01
10.37
29.01
3.15
Unified-IO2 ( Lu et al., 2024 )
2.72
5.44
37.95
7.81
AudioX ( Tian et al., 2025b )
1.34
12.09
11.83
1.86
Table A5. Detailed quantitative results on multimodal generation benchmarks with multiple metrics.
Task
Method
OVL ↑
REL ↑
T2A
Tango2 ( Majumder et al., 2024 )
72.3
74.1
AudioX ( Tian et al., 2025b )
81.5
83.2
Audio-Omni
78.6
83.5
T2M
MusicGen ( Copet et al., 2024 )
70.8
72.5
AudioX ( Tian et al., 2025b )
79.2
81.3
Audio-Omni
82.7
81.6
Table A6. Human evaluation results on generation and editing tasks.
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
With the growing focus on audio in multimedia applications, numerous advanced works on audio generation have emerged. Existing studies typically treat text-to-audio (TTA) and other related audio generation tasks, such as instruction-based audio editing, as independent challenges, adopting task-specific architectures or modules. This absence of a unified modeling paradigm substantially increases the overhead and complexity of building a system for both audio generation and editing, while also leading to limited scalability. To address this issue, we introduce AudioWeave, a unified model for TTA and audio editing without additional task-specific components. Specifically, we propose a joint condition modeling approach with a factorized position embedding, enabling the diffusion transformer backbone to operate under heterogeneous inputs of TTA and audio editing. We further propose a progressive multistage training strategy to mitigate task competition and catastrophic forgetting caused by interference among multiple tasks. This in turn helps maintain the performance of each individual task and may even lead to improvements in certain aspects. Experimental results on TTA task and six audio editing tasks show that our unified model achieves competitive performance with task-specific models, laying a groundwork for further exploration of unified audio generation models.
Haocheng Dong, Yuheng Lu, Cheng Gong +3
Department of Electronic Engineering and Information Science, University of Science and Technology of China, China · Institute of Artificial Intelligence of China Telecom (TeleAI), China · Tianjin Key Laboratory of Cognitive Computing and Application, School of Artificial Intelligence, Tianjin University, China
We present UNISON, a latent diffusion framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound generation, scene-level audio editing, speech-in-scene editing, and timed temporal composition, all of which share a single set of weights. Our architecture features two core designs: (1) Layer-wise deep LLM fusion, which injects hidden states from uniformly sampled layers of a frozen MLLM into corresponding MM-DiT blocks via learned projections, providing depth-matched semantic conditioning that improves instruction following over single-layer baselines; and (2) a unified multi-task architecture where task identity is encoded solely by a channel-wise mask and source audio is provided through VAE-encoded channel concatenation. Training is stabilized by an online GPU-side multi-task data synthesis pipeline with task-homogeneous batching and a two-stage curriculum. With 621M--732M trainable parameters, UNISON achieves results competitive with or exceeding task-specialist models across evaluated domains, while being roughly 4× smaller than comparable unified systems.
Zhaoqing Li, Haoning Xu, Jingran Su +9
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · City University of Hong Kong +3
Recent unified audio generation models can support diverse tasks across speech, sound effects, and music, but most of them still focus on isolated task-level synthesis. However, real video production often requires multiple components of a complete audio track to be generated jointly and consistently for the same video. We present Foley-Omni, a unified multimodal audio generation model that extends isolated task-level synthesis to complete video soundtrack generation by jointly modeling speech, sound effects, and music within a shared latent generation process. To support training and reproducible evaluation, we develop an audiovisual data curation pipeline and introduce V2ST-Bench, a benchmark for holistic video soundtrack generation evaluation. Experiments show that Foley-Omni achieves competitive performance with expert systems on individual synthesis tasks, while improving speech intelligibility, audiovisual consistency and perceptual quality for mixed soundtrack generation.
Ye Tao, Lupeng Liu, Xuenan Xu +6
School of Intelligence Science and Technology, Nanjing University · Video Rebirth · Shanghai Jiao Tong University +2