Harmonizing Spectral Evolution in Conditional Flow Matching for TTS

Isha Pandey; Varad Deshpande; Abhijat Bharadwaj; Ganesh Ramakrishnan

cs.LGSep 28, 2026

Harmonizing Spectral Evolution in Conditional Flow Matching for TTS

Authors: Isha Pandey, Varad Deshpande, Abhijat Bharadwaj, Ganesh Ramakrishnan

Organizations: Indian Institute of Technology Bombay

Abstract

Conditional Flow Matching (CFM) models for text-to-speech (TTS) suffer from incoherent frequency evolution during inference. While similar spectral imbalances are addressed in diffusion models for other domains, those generic solutions fail to generalize to the inherently uncoordinated acoustic dynamics of CFM. We demonstrate that this issue can be effectively mitigated by introducing a novel training-free frequency-selective boosting strategy. Using the Discrete Wavelet Transform (DWT), our method dynamically modulates mel-spectrogram sub-bands during ODE integration, synchronizing spectral development by penalizing aggressive low-frequency growth and boosting lagging high-frequency details. Validated across diverse architectures (Matcha-TTS, F5-TTS, IndicF5), our approach reduces the required Number of Function Evaluations (NFE) from 32 to 26 and improves Frechet Audio Distance (FAD) by up to 61%, all without compromising mean opinion scores, speaker similarity, and speech intelligibility.

Figures & tables

Figure 1 : Evolution of DWT sub-band energy ( $\ell_{2}$ -norm) across F5-TTS inference steps.

Panel of Figure 2 — Figure 2 : DWT sub-band norm evolution: (a) reweighted F5-TTS, (b) baseline Matcha-TTS, and (c) reweighted Matcha-TTS.

Figure 3 : Frequency-selective reweighting process.

Trajectory	Method	WER $\downarrow$	CER $\downarrow$	FAD $\downarrow$
Full 32 NFE	Baseline	17.45	6.32	1.10
	Mel-EQ	17.31	6.28	1.08
	FSB (ours)	15.05	5.91	0.93
Full 26 NFE	Baseline	18.40	7.02	1.04
	Mel-EQ	17.34	6.33	0.98
	FSB (ours)	15.50	6.01	0.51

Table 1: Frequency-domain interventions on F5-TTS.

F5-TTS on LibriTTS-clean
Method	NFE	FAD $\downarrow$	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	SSS $\uparrow$
Baseline	32	0.99	6.63	3.76	3.93	0.76
Baseline	26	0.98	6.71	3.80	3.05	0.74
FSB (ours)	26	0.51	6.70	3.80	3.76	0.77
F5-TTS on LibriTTS-other
Baseline	32	1.48	6.61	3.28	3.64	0.72

Table 2 : Cross-model evaluation on full ODE trajectories.

Setting	Naturalness	Intelligibility	Perceptual quality
Baseline, full 32	3.81	4.43	4.34
Baseline, cutoff-26/32	3.51	4.09	3.69
FSB, full 26 (ours)	3.80	4.45	4.50

Table 3 : Matcha-TTS human evaluation.

Dataset	LL schedule	FAD $\downarrow$	WER $\downarrow$	CER $\downarrow$
LibriTTS-clean	Baseline	12.93	6.7	3.7
	Exponential	10.78	7.2	4.7
	Linear	13.15	8.1	5.3
	Polynomial	17.29	19.8	13.3
LibriTTS-other	Baseline	9.03	8.3	5.1
	Exponential	8.82	8.2	4.9

Table 4 : Ablation of LL-subband reweighting schedule (cutoff-26/32). Other subband coefficients held fixed.

Method	Mean $\\|\hat{a}\\|$	Max $\\|\hat{a}\\|$	$\boldsymbol{r\;(R^{2})}$
Baseline	$1.00{\times}$	$1.00{\times}$	–
FSB: LL+HL	$1.01{\times}$	$0.80{\times}$	$0.049\;(0.787)$
DWT: LL only	$0.93{\times}$	$0.77{\times}$	$0.089\;(0.698)$
Mel-EQ	$1.03{\times}$	$1.03{\times}$	$0.042\;(0.786)$
Pre-Emphasis	$6.62{\times}$	$6.35{\times}$	$0.087\;(0.971)$

Table 5 : Trajectory-stability diagnostics (100 LibriTTS-clean samples). Acceleration normalized by baseline;

r

is the log-linear divergence rate.

Explore similar work

Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech
Sep 28, 2026Yentl Collin, Evan Dufraisse, Amr Mohamed +3Flow-Matching Text-To-Speech Latent Flow
Institute of Foundation Models (IFM), MBZUAI
Mask, Sample, Revise: A Revisable CTMC Inference Stack for Guided Discrete Flow Matching Text-to-Speech
Jun 12, 2026Alef Iury Siqueira Ferreira, Lucas Rafael Stefanel Gris, Luiz Fernando de Araújo Vidal +4Flow-Matching Text-To-Speech Text-To-Audio
Federal University of Goi´as, Goiˆania, Brazil · Federal University of Mato Grosso, Cuiab´a, Brazil · Elsa Speak, S˜ao Paulo, Brazil
Fréchet Distance Loss on Speech Representations for Text-to-Speech Synthesis
Jul 7, 2026Ho-Lam Chung, Kuan-Po Huang, Bo-Ru Lu +1Flow-Matching Text-To-Speech Autoregressive Text-To-Speech
Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan · Graduate Institute of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan · Amazon

Sep 28, 2026, cs.SDJ/K move · Enter open · S save

Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech

Yentl Collin, Evan Dufraisse, Amr Mohamed, Amine Khelif Khelif+2

Institute of Foundation Models (IFM), MBZUAI

Flow-Matching Text-To-Speech Latent Flow Flow Map Few-Step Generation Invariant Synthesis Schedule

Open paper AskPDF arXiv

Flow-matching text-to-speech (TTS) models achieve high synthesis quality but require many neural function evaluations (NFEs) to integrate their generative trajectories. Recent few-step flow-map distillation approaches for TTS construct targets from numerically integrated teacher trajectories, creating a trade-off between target accuracy and training cost. We propose Local Flow-Map Distillation (LFMD), which adapts Eulerian Map Distillation to conditional TTS and avoids teacher trajectory integration during target construction. For inference, we derive a sampling schedule (TD-DP) from teacher dynamics and consistency of the learned maps, with a single cost graph supporting multiple NFE budgets without external audio-metric evaluation. Because scheduling offers no flexibility at one NFE, we refine this regime with alignment-aware temporal self-distillation using soft-DTW. Across Seed-TTS and LibriSpeech-PC, LFMD improves low-NFE synthesis over a matched integral-distillation baseline. On Seed-TTS, the refined student reaches 1.80% WER with 1-NFE, compared with 1.76% for its 32-NFE teacher.

Abstract

Explore similar work

Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech

Mask, Sample, Revise: A Revisable CTMC Inference Stack for Guided Discrete Flow Matching Text-to-Speech

Fréchet Distance Loss on Speech Representations for Text-to-Speech Synthesis