Reinforcement learning can improve text-to-speech synthesis through feedback on generated speech. However, sentence-level rewards do not identify local errors, while signed regression updates can lead to excessive fitting or repulsion of sampled outputs. We propose LoCA (Local Credit Assignment) for autoregressive flow-matching text-to-speech. LoCA uses ASR-based text alignment to locate content errors in generated speech and redistribute sentence-level advantages across acoustic frames. To improve training stability, we introduce reference-relative positive and negative thresholds that limit continued fitting and repulsion. Experiments on VoxCPM2 show that LoCA reduces sample-weighted content error on a multilingual CV3 evaluation subset by 20.5% and 14.3% relative to the pretrained backbone and GROW, respectively, while improving speaker similarity. LoCA also improves content accuracy on the Seed-TTS-Eval hard subset. Ablation results support the contributions of local credit and threshold-based regression control.
Figures & tables
Figure 1: Local-credit construction in an illustrative 12-frame example. Six unflagged and four error frames give mean label 0.6; two unknown tail frames are excluded. With local weight 0.2 and sentence advantage 0.05, error frames receive final coefficient -0.07, unflagged frames +0.13, and unknown frames retain +0.05. The boxes represent generated latent frames, not discrete speech tokens.
CV3 selected (3,624)
CV3 regular
Seed hard
CV3 hard
Method
Error ↓
Cases ↓
SIM ↑
Error ↓
Error ↓
Cases ↓
Error ↓
Cases ↓
VoxCPM2 base
5.986
61
67.168
5.887
8.193
6
8.772
6
GROW
5.553
60
67.562
5.461
9.575
9
8.157
7
LoCA
4.758
51
69.137
4.673
6.884
5
7.163
6
Table 1: Main model comparison. Every metric in a row uses the same selected model. Cases are severe proxy counts and SIM is cosine similarity scaled by 100. Lower Error/Cases and higher SIM are preferred. CV3 regular covers the 3,500 ordinary-language items; the full selection also includes 124 hard items. CV3 hard weights 60 Chinese and 64 English items.
LoCA variant
Local
Thresholds
Error ↓
Cases ↓
SIM ↑
LoCA (full)
Yes
Yes
4.758
51
69.137
w/o local credit
No
Yes
5.508
54
67.525
w/o thresholds
Yes
No
5.488
44
67.722
Table 2: LoCA component ablations. Yes/No indicates component presence. Metrics use the full 3,624-item CV3 selection. The variant w/o thresholds removes the positive and negative regression thresholds.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Hard-set trajectories for the baselines and LoCA ablations. Columns show content error, severe proxy cases and speaker similarity. Chinese/English CV3 hard means use weights 60/124 and 64/124; case counts are summed. The dashed line is the base. Lines connect evaluated points without smoothing or extrapolation.
Method
Split
Step
n
Error ↓
Cases ↓
SIM ↑
VoxCPM2 base
EN
0
1088
1.710
1
66.789
VoxCPM2 base
ZH
0
2020
1.006
1
72.676
GROW
EN
800
1088
1.621
2
67.588
GROW
ZH
800
2020
0.911
3
73.361
LoCA w/o thresholds
EN
400
1088
1.544
0
67.336
LoCA w/o thresholds
ZH
600
2020
0.914
0
74.140
Appendix
Table 3: Best observed standard Seed results per method and language. Each row is selected by its own content error, with step shown. Error is CER for ZH and WER for EN; Cases is the severe proxy count and SIM is scaled by 100.
Method
Step
High err. ↓
Silent ↓
Tail ↓
Internal ↓
Cap ↓
Union ↓
VoxCPM2 base
0
5
0
0
0
3
6
GROW
400
8
0
0
0
7
9
LoCA w/o thresholds
600
5
0
0
0
3
6
LoCA
1200
5
0
0
0
1
5
LoCA w/o local
1000
8
0
0
1
4
10
Appendix
Table 4: Seed-hard severe-failure categories at each method’s reported checkpoint, n=400 . Union counts each flagged utterance once.
Figure 3: Full-subset CV3 and Russian trajectories, including speaker similarity. Russian is outside the six-language RL training pool. All plotted points are reported evaluations; the base is a horizontal reference.
Method
Step
CV3 SIM ↑
CV3 P808 ↑
Seed-h SIM ↑
Seed-h P808 ↑
VoxCPM2 base
0
67.168
3.570
67.632
3.691
GROW
400
67.562
3.580
67.346
3.664
GROW
800
67.558
3.556
67.135
3.626
GROW
1200
66.941
3.537
67.559
3.680
GROW
1600
66.363
3.551
65.767
3.694
GROW
2000
66.662
3.581
66.854
3.640
Appendix
Table 5: Auxiliary quality metrics at the selected checkpoints and representative training steps for GROW and LoCA. SIM and P808 are higher-is-better; SIM backends differ between CV3 and Seed.
Figure 4: Inverse error ratio (base error / method error), at each system’s reported checkpoint. Larger is better; 1 is the base. Asterisks mark Spanish, Italian and Russian, excluded from the RL training pool. Hard Chinese/English are separate tasks, not extra languages.
Method
Step
zh ↓
en ↓
de ↓
es ↓
fr ↓
it ↓
ru ↓
h-zh ↓
h-en ↓
VoxCPM2 base
0
3.996
5.193
5.551
4.571
10.657
5.023
6.219
9.952
7.665
GROW
400
3.398
4.997
4.846
4.491
10.314
4.275
5.904
10.402
6.053
LoCA w/o thresholds
600
3.396
5.017
4.500
5.306
10.638
3.917
5.300
8.501
5.333
LoCA
1200
3.238
4.507
3.878
3.608
9.027
3.731
4.723
7.186
7.143
LoCA w/o local
1000
3.402
4.806
4.562
4.284
10.502
4.803
5.617
8.120
7.593
Appendix
Table 6: Exact CV3 language errors at each method’s selected checkpoint. Chinese uses CER; other splits use WER. All values are percentages; h-zh/h-en are the hard subsets.