We propose Relativistic Adversarial Feedback (RAF), a novel training objective for GAN vocoders that improves in-domain fidelity and generalization to unseen scenarios. Although modern GAN vocoders employ advanced architectures, their training objectives often fail to promote generalizable representations. RAF addresses this problem by leveraging speech self-supervised learning models to assist discriminators in evaluating sample quality, encouraging the generator to learn richer representations. Furthermore, we utilize relativistic pairing for real and fake waveforms to improve the modeling of the training data distribution. Experiments across multiple datasets show consistent gains in both objective and subjective metrics on GAN-based vocoders. Importantly, the RAF-trained BigVGAN-base outperforms the LSGAN-trained BigVGAN in perceptual quality using only 12% of the parameters. Comparative studies further confirm the effectiveness of RAF as a training framework for GAN vocoders.
Figures & tables
Figure 1: Overall framework of the RAF. (a) represents the process of deriving the non-separable discriminator gap and quality gap. (b) represents the training objectives for the generator and the discriminator. ϕ denotes the mel transformation. i denotes the data index.
Figure 2: An illustration of the effect of segment size on quality estimation error.
Figure 3: Toy experiments following [ 31 ] . (a), (b), and (c) represent the training process of MetricGAN-RAF-v1, MetricGAN-RAF-v2, and RAF for two-cluster data, respectively. True data are red, and fake data are blue. RAF escapes mode collapse more quickly than the other training objectives.
Model
Method
Training time (days)
Training steps
Signal fidelity
Perceptual quality
# Params
xRT ↑
M-STFT ↓
PESQ ↑
Periodicity ↓
V/UV F1 ↑
UTMOS ↑
SCOREQ
Full-ref ↓
No-ref ↑
BigVGAN-base
LSGAN
5.9
1 M
0.874
3.452
0.134
0.924
3.450
0.195
3.515
14.0
72
RAF
4.8
0.5 M
0.866
3.619
0.119
0.940
3.617
0.161
3.635
RAF
9.4
1 M
0.841
3.767
0.111
0.944
3.651
0.148
3.667
BigVGAN
LSGAN
11.9
1 M
0.822
3.824
0.110
0.944
3.509
0.180
3.556
112.4
44
Table 1: Objective evaluation results for LibriTTS-dev set. For models with † , we used the official checkpoints for inference.
Dataset
Model
M-STFT ↓
PESQ ↑
Periodicity ↓
F1 ↑
UTMOS ↑
SCOREQ Full-ref ↓
SCOREQ No-ref ↑
Vanilla GAN / RAF
Vanilla GAN / RAF
Vanilla GAN / RAF
Vanilla GAN / RAF
Vanilla GAN / RAF
Vanilla GAN / RAF
Vanilla GAN / RAF
LJSPEECH
BigVGAN-base
0.899 / 0.879
3.738 / 3.892
0.122 / 0.108
0.947 / 0.954
4.054 / 4.210
0.149 / 0.119
4.365 / 4.445
HiFi-GAN (v1)
1.162 / 1.052
2.923 / 3.276
0.146 / 0.127
0.935 / 0.943
3.826 / 4.031
0.227 / 0.163
4.192 / 4.330
Vocos
0.921 / 0.891
3.500 / 3.680
0.122 / 0.115
0.951 / 0.953
3.756 / 4.017
0.217 / 0.147
4.127/ 4.312
Deeply Korean
BigVGAN-base
0.924 / 0.884
2.921 / 3.312
0.170 / 0.150
0.951 / 0.962
1.297 / 1.294
0.256 / 0.227
0.887 / 0.881
HiFi-GAN (v1)
1.310 / 1.145
2.100 / 2.368
0.190 / 0.187
0.955 / 0.948
1.292 / 1.295
0.317 / 0.265
0.920 / 0.957
Table 2: Objective evaluation results for unseen datasets, grouped by dataset and model. Bold values indicate the higher score between the Vanilla GAN and RAF. Vanilla GAN refers to LSGAN for BigVGAN-base and HiFi-GAN (v1), and to HingeGAN for Vocos.
SMOS
LibriTTS
Deeply Korean
GT
4.804 ± 0.027
4.847 ± 0.032
LSGAN
4.526 ± 0.048
3.824 ± 0.090
RAF (Ours)
4.592 ± 0.044
4.324 ± 0.074
Table 3: 5-scale Similarity Mean Opinion Score results with 95% Confidence Interval on LibriTTS-test and Deeply Korean dataset with BigVGAN-base as a backbone.
Methods
PESQ ↑
UTMOS ↑
SCOREQ
Full-ref ↓
No-ref ↑
RAF (Ours)
3.493
3.515
0.183
3.557
w/o softplus
3.593
3.430
0.199
3.510
w [αW,αH,αM]=[1,1,1]
3.478
3.452
0.203
3.500
w/o H,W
3.665
2.872
0.434
2.934
Table 4: Ablation studies on the components of RAF on the LibriTTS-dev set with BigVGAN-base as a backbone.
Training objectives
Long
GP
PESQ ↑
UTMOS ↑
SCOREQ ↑
Full-ref ↓
No-ref ↑
LSGAN
✗
✗
3.117
3.185
0.264
3.302
HingeGAN
✗
✗
2.880
3.101
0.305
3.232
RpGAN-GP
✗
✓
3.176
3.297
0.259
3.359
LSGAN + Q recon
✗
✗
3.219
3.200
0.261
3.312
✓
✗
3.345
3.375
0.220
3.472
Table 5: A comparison result of GAN training objectives on LibriTTS-dev set using BigVGAN-base. MetricGAN-RAF-v2 is a modified version of MetricGAN for RAF.
Training objectives
Dataset
Seen
Unseen
LibriTTS
LJSPEECH
Deeply
UR
LSGAN+ Q recon
3.376
3.988
1.29
2.668
MetricGAN-RAF-v2
3.197
3.655
1.28
2.435
MetricGAN-RAF-v1
2.122
2.981
1.274
1.782
RpGAN-GP
3.297
3.823
1.314
2.56
Table 6: A comparison result of training objectives on multiple datasets. All objectives, except RpGAN-GP, were trained on long segments and with gradient penalties. RpGAN-GP was trained with gradient penalties.