Organizations: X-LANCE Lab, Shanghai Jiao Tong University, China · Shanghai AI Lab, China · KlingAI Research, China · Peking University, Shenzhen, China
Speech-conditioned generation maps speech signals to target audio outputs. Prior end-to-end speech-conditioned generative models achieve high fidelity in clean acoustic conditions, but their performance degrades substantially under environmental noise. While prior noise-robust approaches have achieved significant success in feature-level purification, maintaining consistent generation under clean and noisy conditioning remains a key challenge. To address this, we propose AlignAudio, a noise-robust generative framework based on dual alignment: (i) aligning the representations of clean and noisy speech to preserve semantic cues, and (ii) enforcing generation-level consistency within a flow matching paradigm to improve conditioning fidelity. Experiments on speech-to-audio (STA) and voice conversion (VC) tasks demonstrate that AlignAudio maintains performance comparable to baselines on clean speech while significantly outperforming them across various noisy conditions. Furthermore, the proposed dual-alignment strategy is applicable to both single- and multi-stream speech conditioning and can be integrated into different flow matching-based generation frameworks and post-training settings. Audio demonstrations and qualitative comparisons: \href{https://alignaudio.github.io/AlignAudio}{\textcolor{cyan}{*https://alignaudio.github.io/AlignAudio*}}.
Figures & tables
Figure 1: AlignAudio generates audio with better noise robustness, preserved acoustic events, and higher semantic fidelity compared to traditional speech-conditioned generation systems that suffer from semantic interference and trajectory shifts under noise.
Figure 2: Overview of the AlignAudio architecture. Representation-level alignment (Stage 1) uses lightweight adapters in a pre-trained speech encoder to distill noisy speech embeddings toward clean ones via L2 loss, while generation-level alignment (Stage 2) regularizes the noisy-conditioned velocity field in DiT blocks toward the clean one using directional similarity.
System
Clean
Noisy
OOD
Recorded
STA
FAD ↓
CLAP ↑
OVL ↑
FAD ↓
CLAP ↑
OVL ↑
FAD ↓
CLAP ↑
OVL ↑
FAD ↓
CLAP ↑
STAR [ 29 ]
2.23
0.422
2.68
2.23
0.382
2.36
2.16
0.386
2.54
1.98
0.394
ST5 + FM
1.72
0.412
2.54
2.28
0.339
1.92
1.76
0.387
2.32
2.23
0.367
SE → ST5 + FM
1.92
0.405
2.51
2.35
0.367
2.90
1.79
0.403
2.72
1.85
0.402
AlignAudio-STA
1.96
0.418
2.86
1.88
0.421
3.12
1.84
0.421
2.62
1.78
0.414
EzAudio [ 7 ] (GT text)
1.68
0.427
3.02
–
–
–
–
–
–
–
–
Table 1: Speech-to-audio (STA) and voice conversion (VC) results. Bold and underlined entries indicate the best and second-best speech-conditioned results for each task, condition, and metric, excluding EzAudio with ground-truth (GT) text. AlignAudio-VC is the full model with Flow-GRPO; * denotes its variant without GRPO. Recorded uses a separate test set of AudioCaps captions read by participants in real-world scenarios. WER and SECS are reported in percent; ↓ / ↑ indicate lower/higher is better. – denotes unavailable results.
Configuration
STA
VC
FAD ↓
CLAP ↑
WER ↓
SECS ↑
Full dual-alignment
1.88
0.421
32.36
53.44
w/o Stage 1
2.24
0.384
76.55
30.57
w/o Stage 2
2.28
0.387
87.54
32.89
w/o both stages
2.28
0.339
108.93
28.77
Noisy augmentation
2.11
0.398
56.28
40.16
Table 2: Ablation under noisy conditions; VC uses AlignAudio-VC* (without GRPO). Stages 1/2: representation/generation alignment. Full STA uses Early- t ; weighting variants apply only to STA. Bold / italics mark the best/second-best per column.
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. However, existing open-source methods mainly rely on either dual-tower designs with posterior alignment or fully unified tri-modal designs that mix textual context, audio and video in one shared space. The former weakens fine-grained audio-video co-evolution, while the latter couples semantic conditioning with low-level synchronization. To address these limitations, we propose NAVA, a Native Audio-Visual Alignment framework for joint audio-video generation. NAVA is built upon context-conditioned native audio-visual alignment: it first establishes audio-video correspondence in a dedicated interaction space, and then uses external context to condition the joint denoising process. Specifically, NAVA is instantiated with an Align-then-Fuse MMDiT architecture, which transitions from modality-aware audio-video alignment to modality-shared joint denoising. Furthermore, we introduce Timbre-in-Context Conditioning to associate reference timbre cues with corresponding speech spans to achieve controllable speech timbre. Experiments on Verse-Bench and Seed-TTS, together with a user study, demonstrate that NAVA achieves superior video quality, precise audio-visual synchronization, competitive audio quality, and stronger reference-timbre controllability using only 6.3B parameters.
Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at https://github.com/xiaomi-research/midashenglm-gen and https://huggingface.co/mispeech/midashenglm-gen, and the demo page is available at https://xingws.github.io/midashenglm-gen-demo/.
Xingwei Sun, Heinrich Dinkel, Gang Li +7
MiLM Plus, Xiaomi Inc., Beijing, China · X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China
Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. However, reconstruction-oriented representations may be suboptimal for generation, motivating joint representation and generative learning. To this end, we introduce Unite-Audio, to our knowledge, is the first to jointly learn continuous audio representations and latent flow matching for TTA. By coupling reconstruction with self-supervised generative prediction, Unite-Audio allows the generative objective to directly shape the latent space rather than treating it as a fixed intermediate representation. We further employ Flow-GRPO post-training to improve text-conditioned generation. Experiments show competitive TTA performance with a compact latent flow model, while ablation studies confirm the benefit of jointly learning the audio representation and generative model. Audio samples are available at https://runwushi.github.io/Unite-Audio.
Runwu Shi, Kai Li, Yujin Wang +8
Institute of Science Tokyo · Tsinghua University · Wuhan University +2