Resonate: Reinforcing Text-to-Audio Generation via Online Feedback from Large Audio Language Models

Xiquan Li; Junxi Liu; Wenxi Chen; Haina Zhu; Ziyang Ma; Xie Chen

cs.SDMar 12, 2026

Resonate: Reinforcing Text-to-Audio Generation via Online Feedback from Large Audio Language Models

Authors: Xiquan Li, Junxi Liu, Wenxi Chen, Haina Zhu, Ziyang Ma, Xie Chen

Organizations: X-LANCE Lab, Shanghai Jiao Tong University, China · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China

Abstract

Reinforcement Learning (RL) has become an effective paradigm for enhancing Large Language Models (LLMs) and visual generative models. However, its application in text-to-audio (TTA) generation remains largely under-explored. Prior work typically employs offline methods like Direct Preference Optimization (DPO) and leverages Contrastive Language-Audio Pretraining (CLAP) models as reward functions. In this study, we investigate the integration of online Group Relative Policy Optimization (GRPO) into TTA generation. We adapt the algorithm for Flow Matching-based audio models and demonstrate that online RL significantly outperforms its offline counterparts. Furthermore, we incorporate rewards derived from Large Audio Language Models (LALMs), which can provide fine-grained scoring signals that are better aligned with human perception. With only 470M parameters, our final model, Resonate, establishes a new SOTA on TTA-Bench in terms of both audio quality and semantic alignment.

Figures & tables

Figure 1: Left: the model architecture of Resonate. Right: The employed Flow-GRPO training paradigm.

Model	Params	NFE	Objective Metrics						Subjective Metrics
Model	Params	NFE	CE $\uparrow$	CU $\uparrow$	PC $\uparrow$	PQ $\uparrow$	CLAP $\uparrow$	AQAScore $\uparrow$	OVL	REL
AudioLDM-L-Full [ 1 ]	739M	200	3.275	5.137	3.219	5.854	0.441	0.574	-	-
AudioLDM-2-Large [ 31 ]	718M	200	3.441	5.440	2.957	5.984	0.415	0.589	-	-
Tango-Full [ 3 ]	866M	200	3.264	5.151	3.360	5.954	0.440	0.599	-	-
Tango-2-Full [ 9 ]	866M	200	3.468	5.196	3.800	5.895	0.467	0.702	-	-
MeanAudio-L-Full [ 10 ]	480M	25	3.304	5.151	2.995	5.743	0.469	0.729	-	-

Table 1: Evaluation results on the Accuracy subset of TTA-Bench

Models	LCC $\uparrow$	SRCC $\uparrow$	KTAU $\uparrow$
LAION-CLAP-630K-Best	0.472	0.477	0.337
Qwen2.5-Omni-7B	0.518	0.589	0.429
Qwen3-Omni-Instruct	0.575	0.586	0.429

Table 2: Correlation comparison on the the PAM dataset

Methods	CU $\uparrow$	PQ $\uparrow$	CLAP $\uparrow$	AQAScore $\uparrow$
Resonate-PT	5.238	5.923	0.458	0.651
w. DPO ( $\beta=100$ )	5.237	5.924	0.462	0.676
w. SFT	5.114	5.764	0.461	0.675
w. SFT + GRPO	5.184	5.883	0.472	0.728
\rowcolor blue!10 w. GRPO	5.328	6.064	0.476	0.737

Table 3: Ablation study on different training paradigms.

Methods	CU $\uparrow$	PQ $\uparrow$	CLAP $\uparrow$	AQAScore $\uparrow$
Resonate-PT	5.238	5.923	0.458	0.651
w. CLAPScore	5.276	6.063	0.479	0.711
\rowcolor blue!10 w. AQAScore	5.328	6.064	0.476	0.737

Table 4: Ablation study on different reward models.

Figure 2: Ablation study on Flow-GRPO configurations.

Explore similar work

FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech
Jun 22, 2026Haoxu Wang, Biao Tian, Weiqin Li +3Flow Matching TTS Synthesis
Tongyi Lab, Alibaba Group, China
Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generation
Mar 2, 2026Yi Gu, Yanqing Liu, Chen Yang +1Text-to-Audio Generation RL for Diffusion Models
Tsinghua University · Microsoft
UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation
Sep 23, 2026Runwu Shi, Kai Li, Yujin Wang +8Text-to-Audio Generation Audio Representation Learning
Institute of Science Tokyo · Tsinghua University · Wuhan University +2

Jun 22, 2026J/K move · Enter open · S save

FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech

Haoxu Wang, Biao Tian, Weiqin Li, Xiang Lv+2

Tongyi Lab, Alibaba Group, China

Flow Matching TTS Synthesis RL Fine-Tuning Multi-Objective Optimization

Open paper AskPDF arXiv

Existing Reinforcement Learning (RL) research for Text-to-Speech (TTS) focuses on large language models (LLMs), leaving Flow-Matching (FM) under-explored. We present FlowTTS-GRPO, an online RL framework for FM-based TTS. By converting ordinary differential equation (ODE) trajectories into stochastic differential equation (SDE) paths, our method enables direct fine-tuning of open-source FM models without auxiliary models. We show that a weighted reward combination converges faster than a probabilistic scheme, and identify three practical optimizations: omitting classifier-free guidance (CFG) during training accelerates convergence; synthesizing hard cases improves robustness; and applying RL to the FM component enhances audio-detail metrics. Experiments on CosyVoice 3.0 and F5-TTS demonstrate objective and subjective preference gains in speaker similarity and perceptual quality, with F5-TTS also improving intelligibility.

Abstract

Explore similar work

FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech

Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generation

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation