Organizations: Hong Kong University of Science and Technology, Hong Kong SAR, China · WeChat Vision, Tencent Inc, China · Peking University, Beijing, China
Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni.
Figures & tables
Figure 1. An overview of the Audio-Omni framework and its capabilities. (Top) Our decoupled architecture connects a frozen MLLM for understanding with a trainable DiT for audio synthesis via a feature projector. (Middle) A showcase of the model’s unified capabilities across understanding, generation, and editing. (Bottom) A demonstration of remarkable emergent abilities inherited from the MLLM.
Figure 2. Overview of the hybrid pipeline for constructing our AudioEdit dataset. The pipeline consists of two parallel branches to ensure both data authenticity and scale. The Real Data Branch (left) mines editing pairs from real-world datasets (e.g., VGGSound) by first using an MLLM (Gemini) for category identification, followed by a dedicated segmentation model (SAM-Audio) for source separation. Concurrently, the Synthesis Data Branch (right) leverages the Scaper toolkit to programmatically generate a large volume of precisely annotated editing scenarios. This hybrid strategy yields a dataset that combines the acoustic fidelity of natural audio with the large-scale diversity needed for robust model training.
Task Type
Data Source
Train Samples
Test Samples
Add
Real Data
50K
500
Synthesis Data
150K
-
Remove
Real Data
50K
500
Synthesis Data
150K
-
Extract
Real Data
50K
500
Synthesis Data
150K
-
Table 1. Statistics of our proposed audio editing dataset AudioEdit.
Figure 3. The Audio-Omni Framework. Our framework utilizes a decoupled design with two distinct conditioning streams to guide a trainable DiT backbone. The High-Level Semantic Features stream provides global, instructional guidance. It is formed by concatenating features from a frozen MLLM (MM Features) with character-level embeddings from a trainable Transcript Encoder. The Low-Level Signal Features stream offers precise, temporal guidance for editing and synchronization. It combines features from Synchformer and Mel Encoder. These two streams are injected into the DiT via different mechanisms: the high-level stream as context for cross-attention, and the low-level stream concatenated with the input noise and time embedding.
Methods
MMSU ↑
MMAU ↑
Specialized Models
Audio Flamingo3 ( Goel et al., 2025 )
61.40
72.42
Qwen2-Audio-Instruct ( Chu et al., 2024 )
53.27
57.40
Qwen2.5-Omni-3B ( Xu et al., 2025 )
56.83
63.30
Unified Models
Ming-Omni ( AI et al., 2025 )
47.53
70.80
Table 2. Quantitative results on multimodal understanding benchmarks.
Methods
T2A_FAD ↓
T2M_FAD ↓
V2A_FAD ↓
V2M_FAD ↓
TTS_WER ↓
Specialized Models
Tango2 ( Majumder et al., 2024 )
3.20
-
-
-
-
MMAudio ( Cheng et al., 2025 )
4.71
-
2.04
-
-
Stable-Audio-Open ( Evans et al., 2024 )
3.15
3.23
-
-
-
MusicGen ( Copet et al., 2024 )
-
3.94
-
-
-
VATT ( Liu et al., 2024b )
-
-
2.55
-
-
Table 3. Quantitative results on multimodal generation benchmarks.
Methods
AE_FAD ↓
AE_LSD ↓
CLAP ↑
Specialized Models
ZETA ( Manor and Michaeli, 2024 )
3.81
3.80
0.30
SDEdit ( Meng et al., 2021 )
3.51
4.40
0.22
MMEDIT ( Tao et al., 2025 )
3.95
4.05
0.15
Unified Models
Audio-Omni
3.27
2.27
0.32
Table 4. Quantitative results on audio editing benchmarks.
Figure 4. Qualitative showcase of Audio-Omni’s capabilities , including (a) knowledge-augmented generation, (b) in-context generation, (c) zero-shot voice conversion, and (d) zero-shot speech editing.
Setting
KL ↓
IS ↑
FD ↓
FAD ↓
LSD ↓
Only Real
1.27
5.54
20.69
2.67
1.84
Only Synthesis
1.93
5.04
37.96
3.80
5.17
Syn. + Real (Ours)
1.30
5.93
20.48
2.48
1.82
Table 5. Ablation study on dataset composition for audio editing training.
Context
Cat.
T2A ↓
V2A ↓
TTS ↓
AE ↓
mm, trans, sync, mel
none
40.91
25.61
32.46
5.47
mm, trans, sync
mel
38.26
26.46
25.88
4.03
mm, trans
sync, mel
28.90
18.55
19.33
3.62
mm
trans, sync, mel
60.58
56.88
59.20
4.88
Table 6. Ablation study on conditioning injection strategies.
Feature Source
T2A
T2M
IS ↑
FD ↓
IS ↑
FAD ↓
Last Layer ( -1 )
9.36
4.21
2.90
3.26
Penultimate Layer ( -2 )
11.26
2.75
3.46
3.05
MetaQuery
7.44
8.34
2.38
8.15
Query
8.55
5.31
2.69
4.22
Table 7. Ablation study on the source of conditional features from the MLLM.
Task
Encoder
KL ↓
IS ↑
FD ↓
FAD ↓
T2A
T5
1.51
11.27
15.78
2.75
CLAP
1.81
9.64
16.58
1.84
Qwen
1.25
11.65
12.21
2.14
V2A
CLIP
2.20
7.42
12.21
2.80
VideoMAE
2.29
6.58
14.72
3.06
Qwen
2.00
8.08
8.50
2.23
Table A1. Ablation study on different encoders. We evaluate the impact of different encoders for text-to-audio (T2A) and video-to-audio (V2A) tasks.
Table A3. Detailed results on audio editing tasks. We report FAD/LSD for each task and their average.
Task
Hours
Datasets
T2A
1.4k
AudioCaps ( Kim et al., 2019 ) WavCaps ( Mei et al., 2024 ) AudioSetCaps ( Bai et al., 2025 ) AudioTime ( Xie et al., 2025a ) IF-Caps ( Tian et al., 2025b )
V2A
0.7k
VGGSound ( Chen et al., 2020 ) AudioSet Strong ( Hershey et al., 2021 )
T2M
17k
IF-Caps ( Tian et al., 2025b ) MUCaps ( Liu et al., 2024a )
V2M
16k
V2M ( Tian et al., 2025c ) MMTrail ( Chi et al., 2024 )
Speech
6k
Audio-FLAN (English subset) ( Xue et al., 2025 )
Editing
3k
AudioEdit (Sec. 3 )
Table A4. Training data summary across tasks.
Method
KL ↓
IS ↑
FD ↓
FAD ↓
Text-to-Audio (T2A)
Tango2 ( Majumder et al., 2024 )
1.11
10.37
12.22
3.20
MMAudio ( Cheng et al., 2025 )
1.35
12.03
12.63
4.71
Stable-Audio-Open ( Evans et al., 2024 )
2.01
10.37
29.01
3.15
Unified-IO2 ( Lu et al., 2024 )
2.72
5.44
37.95
7.81
AudioX ( Tian et al., 2025b )
1.34
12.09
11.83
1.86
Table A5. Detailed quantitative results on multimodal generation benchmarks with multiple metrics.
Task
Method
OVL ↑
REL ↑
T2A
Tango2 ( Majumder et al., 2024 )
72.3
74.1
AudioX ( Tian et al., 2025b )
81.5
83.2
Audio-Omni
78.6
83.5
T2M
MusicGen ( Copet et al., 2024 )
70.8
72.5
AudioX ( Tian et al., 2025b )
79.2
81.3
Audio-Omni
82.7
81.6
Table A6. Human evaluation results on generation and editing tasks.
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
Department of Electronic Engineering and Information Science, University of Science and Technology of China, China · Institute of Artificial Intelligence of China Telecom (TeleAI), China · Tianjin Key Laboratory of Cognitive Computing and Application, School of Artificial Intelligence, Tianjin University, China