Guidance at Pure Noise Erodes Speaker Similarity in Few-Step Token-Conditioned Flow-Matching TTS
Organizations: Department of Aerospace Engineering, KAIST, Daejeon, Republic of Korea.
Abstract
Text-to-speech systems built from a language model and a flow-matching mel decoder now run the decoder with two to four Euler steps instead of the official ten or more. At two steps the word error rate (WER) is unchanged, but speaker similarity (SIM) drops by one to six points. This paper shows that most of this loss comes from classifier-free guidance on the first step. At pure noise the network can only return the conditional mean, so guidance shifts every trajectory by nearly the same vector. Later steps do not remove this shift, and its size grows with the length of the first step, so it is negligible with ten or more steps and accounts for most of the loss with two. Two training-free samplers are derived from this analysis. First-step unguided sampling (FSU) leaves the first step unguided and recovers -- of the two-step SIM loss on CosyVoice2/3 with no significant WER increase. FSU-Mid replaces the two Euler steps by one midpoint step with an unguided first evaluation, recovers -- on IndexTTS-2 and matches the ten-step teacher on CosyVoice2 and GLM-TTS. Both use three network function evaluations, one fewer than the official two-step sampler, and FSU-Mid gives the highest mean recovery among eight published guidance rules re-run at two steps on four decoders.
Figures & tables
| Decoder | Condition | Grid | Steps | Uncond. branch |
|---|---|---|---|---|
| CosyVoice 2 [ 11 ] | tokens | cosine | 10 | all-null |
| CosyVoice 3 [ 44 ] | tokens | cosine | 10 | all-null |
| IndexTTS-2 [ 12 ] | tokens | uniform | 25 | all-null |
| GLM-TTS [ 13 ] | tokens | cosine | 10 | speaker-null |
| F5-TTS [ 45 ] | text | sway | 32 | all-null |
| ZipVoice [ 21 ] | text | t-shift | 16 | text-null |
| Method | NFE | CosyVoice 2 | CosyVoice 3 | IndexTTS-2 | GLM-TTS | Mean | UTMOS |
| Official CFG | 4 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 | – |
| Guidance interval [ 24 ] ⋆ | 3 | 88 / 85 | 91 / 82 | 55 / 57 | 78 / 69 | 76 | +0.13 |
| Selective-CFG (default) [ 35 ] | 4 | 4 / 16 | 2 / 6 | 9 / 9 | 2 / 6 | 1 | 0.05 |
| Selective-CFG (input-text) [ 35 ] | 4 | 85 / 86 | 70 / 56 | 52 / 51 | 63 / 52 | 64 | 0.06 |
| CFG-Zero ∗ scale [ 27 ] | 4 | 9 / 5 | 3 / 2 | 8 / 2 | 5 / 3 | 2 | +0.04 |
| CFG-Zero ∗ zero-init [ 27 ] | 2 | coll. / coll. | coll. / coll. | coll. / coll. | coll. / coll. | coll. | 2.05 |
| Method (NFE) | SIM en / zh | Recovery en / zh | SIM hard | SIM LSPC | WER / CER | UTMOS en / zh |
|---|---|---|---|---|---|---|
| CosyVoice 2 | ||||||
| Official CFG (4) | 68.86 / 76.63 | 0% / 0% | 73.05 | 68.62 | 2.77 / 1.23 | 3.79 / 3.10 |
| TV-CFG (corrected) [ 25 ] (4) | 72.93 / 79.47 | 70% / 66% | 77.03 (83%) | 71.82 (62%) | 2.85 / 1.28 | 4.05 / 3.22 |
| Sel.-CFG (input-text) [ 35 ] (4) | 74.15 / 80.49 | 91% / 90% | 77.52 (94%) | 74.06 (105%) | 2.77 / 1.29 | 3.97 / 3.21 |
| MAMBO-G [ 28 ] (4) | 72.96 / 79.51 | 70% / 67% | 76.47 (72%) | 72.31 (71%) | 2.78 / 1.23 | 4.21 / 3.57 |
| FSU (ours) (3) | 73.95 / 80.37 | 87% / 87% | 77.39 (91%) | 73.23 (89%) | 2.94 / 1.21 | 4.17 / 3.44 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Config | NFE | First packet (ms) | Decoder share | Thr. | ||
|---|---|---|---|---|---|---|
| teacher, 10 steps | 20 | 37.7% | 1.29 | 1.64 | 1.35 | |
| baseline, 2 steps | 4 | 19.4% | 1.57 | 2.17 | 1.95 | |
| FSU (ours) , 2 steps | 3 | 19.4% | 1.57 | 2.13 | 1.94 | |
| FSU-Mid (ours) | 3 | 19.4% | 1.59 | 2.12 | 1.93 |
| Decoder | all-null | token-null | speaker-null | 2-step loss | ||
|---|---|---|---|---|---|---|
| CosyVoice 2 | 2.11 | 1.98 | 0.95 | 0.44 | 0.21 | 5.8 |
| CosyVoice 3 | 2.10 | 1.94 | 1.23 | 0.47 | 0.23 | 6.7 |
| IndexTTS-2 | 1.93 | 1.94 | 1.24 | 0.54 | 0.29 | 6.2 |
| GLM-TTS | 1.99 | 2.26 | 1.23 | 0.55 | 0.49 (0.28 † ) | 0.8 (3.4 † ) |
| F5-TTS (text) | 1.52 | 0.89 | 1.25 | 1.58 | 1.09 | WER |
| ZipVoice (text) | 1.58 | 0.85 | 1.29 | 1.58 | 1.97 | WER |
| Condition | Official | TV-CFG | Sel.-CFG | MAMBO-G | FSU (ours) | FSU-Mid (ours) | Teacher | Rec. | WER |
|---|---|---|---|---|---|---|---|---|---|
| CosyVoice 2 | |||||||||
| ko | 74.59 | 78.67 | 81.41 | 79.60 | 80.08 | 81.60 | 80.30 | 96% | 3.95 / 3.08 |
| ja | 79.81 | 83.58 | 85.60 | 84.26 | 84.81 | 85.60 | 84.81 | 100% | 10.53 / 10.58 |
| de | 67.75 | 71.26 | 74.13 | 71.18 | 71.56 | 73.34 | 73.62 | 65% | 63.4 / 81.6 ‡ |
| long en, offline | 72.54 | 77.14 | 77.70 | 76.91 | 77.90 | 79.45 | 79.09 | 82% | 2.54 / 2.75 |
| long zh, offline | 78.83 | 81.45 | 82.04 | 81.42 | 82.12 | 83.87 | 83.15 | 76% | 1.46 / 1.37 |
TTS Synthesis
Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.
See all 294 papers