eess.ASJun 24, 2026

Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTS

Authors: Runwu ShiYujin WangHongjin SongJiang WangYaozhong KangNabeela KhanWeiqiao ShanBenjamin Yen+3 more

Organizations: Institute of Science Tokyo · Wuhan University · Beijing Institute of Technology

Abstract

Classifier-free guidance (CFG) is widely used in flow-matching-based zero-shot text-to-speech (TTS), where generation is conditioned on text content and a speech prompt. Standard CFG uses a single guidance weight for their joint conditional effect, while branch-selective guidance emphasizes text or speaker conditioning and can introduce a trade-off between text accuracy and speaker similarity. In this paper, we revisit CFG under independently masked conditions and decompose the guidance field into text, speaker, and joint residuals. We show that condition-specific branch differences couple the joint residual with the corresponding text or speaker residual under a shared weight. Trajectory analysis further shows that the joint residual varies over flow time and contains information that cannot be represented by reweighting the text and speaker residuals alone. Based on these observations, we propose joint residual reweighting, which assigns independent weights to the three residuals. Experiments on F5-TTS, CosyVoice2, and GLM-TTS across three evaluation sets show overall improvements in speaker similarity and text accuracy over the default CFG settings without retraining.

Explore similar work

CardsList