FineSpeaker: Reliable Speaker Representation Learning at Fine Temporal Resolution
Authors: Yongkang Yin, Wenhao Yang, Lichun Fan, Jian Luan, Hang Su, Yuexian Zou
Organizations: Guangdong Provincial Key Laboratory of Ultra High Definition lmmersive Media Technology, Shenzhen Graduate School, Peking University · ADSPLAB, School of ECE, Peking University, Shenzhen, China · MiML Plus, Xiaomi Inc., Beijing, China
Speaker representation learning is fundamental to speech processing tasks such as speaker verification and diarization. Although long-context speaker encoders can produce stable and discriminative embeddings, short-duration verification and diarization require finer-grained temporal representations. Short temporal contexts limit the available speaker evidence and make speaker identity difficult to disentangle from phonetic and acoustic variability. This creates a temporal resolution-representation reliability trade-off: speaker representations should maintain local within-speaker consistency, between-speaker discriminability and remain sensitive to speaker changes at fine temporal resolution. Accordingly, we propose a framework for reliable speaker representation learning at fine temporal resolution. FineSpeaker first constructs resolution-controllable frame-wise speaker representations through mean pooling over an adjustable n-frame window, enabling flexible control of representation granularity. It then introduces a speaker encoder with local temporal modeling that exploits short-range speaker context to improve the stability and discriminability of short-duration and frame-wise representations. Furthermore, we develop a generative disentanglement strategy for optimizing frame-level speaker representations, combining VAE-based probabilistic modeling with flow matching to characterize speaker-related variation and suppress non-speaker interference. Experiments demonstrate competitive full-utterance performance, improved fine-grained speaker verification on VoxCeleb and CN-Celeb, and strong zero-shot diarization results, highlighting the benefits of balancing temporal resolution and representation reliability.
Figures & tables
Fig. 1: Overview of the proposed fine-grained speaker representation learning method, including the speaker encoder architecture, the generative disentanglement training scheme, and multi-granularity representation extraction. And evaluated on downstream speaker verification and diarization tasks.
Fig. 2: Overview of the proposed speaker encoder architecture. The input is downsampled by a factor of two and processed by a four-stage Multi-Scale Convolution (MSC) backbone with TCN-based local temporal modeling. The resulting features are projected to 192 dimensions and then processed by sliding-window self-attention to produce frame-level speaker representations with T/2 temporal steps. Mean pooling over local windows or the entire sequence produces frame-wise or utterance-level representations, respectively.
Fig. 3: Illustration of the proposed local sliding-window self-attention module. Each frame attends only to the features within a local temporal window, which moves along the sequence with a stride of one.
Fig. 4: Overview of the proposed generative disentanglement training framework. Given an input log-Mel spectrogram, a frozen pretrained Whisper encoder extracts semantic features, while the speaker encoder extracts frame-level speaker features. A VAE-based module models the speaker representations and obtains latent speaker representations through reparameterization. After temporal alignment, both representations are concatenated to form the joint latent representation Z0 . The noisy representation Zt is then processed by DiT blocks to estimate Z0′ , which is decoded to reconstruct the input.
Model
Params
FLOPs
VoxCeleb1-O
VoxCeleb1-E
VoxCeleb1-H
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
ECAPA-TDNN
6.2M
1.0G
0.9839
0.0872
1.1169
0.1254
2.1191
0.2099
CAM++
7.0M
1.1G
0.8350
0.1046
0.8595
0.0990
1.7672
0.1810
ResNet
6.3M
4.5G
0.9995
0.1125
1.1141
0.1227
1.9872
0.1935
Res2Net
4.0M
1.8G
1.5424
0.1553
1.4521
0.1528
2.5355
0.2416
ERes2Net
6.6M
3.4G
1.0956
0.1183
1.0906
0.1230
2.0665
0.2029
TABLE I: Full-utterance speaker verification results on three VoxCeleb1 test sets. Bold and underlined values denote the best and second-best results.
Model
VoxCeleb1-O
VoxCeleb1-E
VoxCeleb1-H
0.5 s
1 s
2 s
0.5 s
1 s
2 s
0.5 s
1 s
2 s
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
ECAPA-TDNN
18.6114
0.9999
7.4124
0.6280
2.8777
0.3123
18.6634
1.0000
7.5910
0.6113
2.9587
0.3053
21.4904
1.0000
10.4887
0.7595
4.8889
0.4546
CAM++
15.6938
0.9998
5.8622
0.5601
2.2323
0.2780
15.5746
0.9998
5.8523
0.5215
2.1656
0.2428
18.0948
0.9999
8.3271
0.6756
3.8753
0.3852
ResNet
16.3902
1.0000
7.1910
0.7537
3.4817
0.4207
16.1213
1.0000
7.0106
0.7749
3.5052
0.4395
21.5845
1.0000
11.2249
0.9060
6.0413
0.6032
Res2Net
13.8496
0.9999
6.6693
0.6066
3.2889
0.3571
13.9450
1.0000
6.6692
0.5768
3.1752
0.3261
17.2292
1.0000
9.4023
0.7072
5.0585
0.4545
TABLE II: Short-duration speaker verification results on the three VoxCeleb1 test sets. Bold and underlined values denote the best and second-best results.
Model
VoxCeleb1-O
VoxCeleb1-E
VoxCeleb1-H
2 f
10 f
15 f
2 f
10 f
15 f
2 f
10 f
15 f
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
ECAPA-TDNN
47.8875
1.0000
44.9959
1.0000
38.2574
1.0000
48.0757
1.0000
44.9593
1.0000
38.2033
1.0000
48.8396
1.0000
46.0504
1.0000
39.9533
1.0000
CAM++
32.7798
0.9995
19.5082
0.9991
15.1904
0.9987
33.5502
0.9996
19.9441
0.9994
15.3838
0.9994
37.8243
0.9999
24.0232
0.9999
19.2660
0.9999
ResNet
31.0855
1.0000
9.1386
0.8964
6.1886
0.6246
30.6526
1.0000
8.7790
0.8867
5.9352
0.6862
35.1405
1.0000
13.6103
0.9692
9.7669
0.8474
Res2Net
27.0117
0.9998
8.7471
0.7260
6.0596
0.5728
26.8183
0.9999
8.8374
0.6999
5.9441
0.5384
30.5054
1.0000
11.8146
0.8073
8.5443
0.6681
TABLE III: Frame-wise speaker verification results on the VoxCeleb1 test sets at different temporal granularities. Here, f denotes one output frame; † and ‡ indicate 8× and 4× the reference frame rate, respectively. Bold and underline values denote the best and second-best results at each frame rate, while ITALIC values denote the best result at each aligned temporal span.
Model
Full utterance
Short duration
Frame-wise
All
0.5 s
1 s
2 s
2 f
10 f
15 f
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
EER ↓
minDCF ↓
ECAPA-TDNN
8.8175
0.4467
30.6153
1.0000
23.8785
0.9715
18.1484
0.7324
46.1216
1.0000
41.1612
1.0000
37.6522
1.0000
CAM++
7.7105
0.4354
27.6204
1.0000
21.2458
0.8688
16.0324
0.6951
36.2373
0.9968
29.9153
0.9764
28.2092
0.9630
ResNet
7.9020
0.4305
29.0041
1.0000
21.3901
0.9999
16.7583
0.8200
34.1524
1.0000
22.6106
0.9999
18.6879
0.9241
Res2Net
8.5610
0.4760
25.7131
0.9962
21.0476
0.8403
16.6229
0.7020
31.0691
0.9988
22.2623
0.8690
19.7029
0.8401
TABLE IV: Speaker verification results on the CN-Celeb test set at different granularities. Here, f denotes one output frame; notation and highlighting follow Tables I and II .
Model
VoxConverse
AMI-SDM
AliMeeting
AISHELL-4
Short chunk
Frame-wise
Short chunk
Frame-wise
Short chunk
Frame-wise
Short chunk
Frame-wise
0.3 s
0.5 s
2 f
10 f
15 f
0.3 s
0.5 s
2 f
10 f
15 f
0.3 s
0.5 s
2 f
10 f
15 f
0.3 s
0.5 s
2 f
10 f
15 f
ECAPA-TDNN
53.701
30.779
82.650
56.683
48.682
78.928
57.360
93.295
82.024
77.147
48.343
40.624
77.680
56.810
47.341
71.530
53.461
85.482
71.912
64.440
CAM++
51.316
22.245
60.884
46.914
31.848
82.707
48.914
79.634
68.572
58.350
47.024
36.898
62.687
51.765
44.930
66.431
51.941
73.387
66.403
52.069
ResNet
49.286
29.446
52.346
17.926
15.370
80.843
65.663
80.672
52.442
41.825
46.012
32.456
48.174
25.031
25.454
61.119
42.292
69.856
20.332
14.581
Res2Net
44.036
20.517
53.058
12.566
12.248
80.572
47.318
73.938
31.002
24.535
43.965
29.698
45.084
21.702
22.718
56.534
30.120
58.630
15.961
11.620
TABLE V: Zero-shot speaker diarization results under different sets and temporal settings. Results are reported as diarization error rate (DER, %), where f denotes one output frame. Notation and highlighting conventions follow Tables I and II .
Configuration
Utterance-level
Frame-level
All
0.5 s
1 s
2 s
2f
10f
15f
ours
0.6699
9.1226
3.8427
1.7345
4.7404
4.3751
4.0691
w/o Generator
0.6595
10.4581
4.3456
1.8426
4.9340
4.6025
4.4305
Short-Utt. Training
0.8935
8.7977
4.0441
2.0367
4.7039
4.3691
4.2554
Enc. + Frame Loss
1.2441
8.9668
4.3400
2.3998
7.0192
6.2570
6.0130
ERes2Netv2-based Enc. ⋆
0.7603
11.1286
5.0411
2.2295
14.9580
5.6168
3.3659
TABLE VI: Ablation results on VoxCeleb1-O. EER (%) is reported, bold indicates the best results; All denotes full-duration evaluation, f the number of pooled frames, and ⋆ denotes an 80-ms per-frame span instead of 20 ms.
VAE KL (βKL)
Utterance-level
Frame-level
All
0.5 s
1 s
2 s
2f
10f
15f
0.001
0.6699
9.1226
3.8427
1.7345
4.7404
4.3751
4.0691
0.005
0.7606
9.0343
4.0725
1.8223
5.0147
4.6926
4.4742
0.01
0.6965
9.3228
4.0287
1.8163
4.8982
4.5469
4.3210
TABLE VII: sensitivity to the VAE KL weight β on Vox1-O. EER (%) is reported, bold indicates the best results.
Fig. 5: UMAP visualization of speaker representations on VoxCeleb1. Ten speakers are randomly selected, with 20 utterances from each speaker. Representations at different temporal granularities are projected into a two-dimensional space using UMAP. Different colors indicate different speakers, and f denotes the number of output frames used for temporal pooling.
Fig. 6: UMAP visualization of speaker representations from the VoxConverse sample auzru . Representations at different temporal granularities are projected into a two-dimensional space. Colors denote different speakers, while red crosses and black triangles indicate non-speech and overlapping regions.
slide window size
Utterance-level
Frame-level
All
0.5 s
1 s
2 s
2f
10f
15f
25
0.7872
10.6840
4.6295
2.0217
9.9363
8.7606
8.0910
35
0.6646
10.5285
4.3782
1.7884
7.5289
6.7230
6.2751
45
0.6593
10.4028
4.4231
1.8718
6.0834
5.5315
5.2874
51
0.6595
10.4581
4.3456
1.8426
4.9340
4.6025
4.4305
55
0.6446
10.5139
4.4477
1.9210
5.1163
4.6974
4.3724
TABLE VIII: Effect of the sliding-attention window size on Vox1-O. EER (%) is reported, bold indicates the best results.
λ
Utterance-level
Frame-level
All
0.5 s
1 s
2 s
2f
10f
15f
0.8
0.6380
9.4044
3.9876
1.7890
4.7933
4.4033
4.1756
1.0
0.6699
9.1226
3.8427
1.7345
4.7404
4.3751
4.0691
1.2
0.6858
9.6585
3.9888
1.8628
4.7088
4.0537
4.0537
TABLE IX: Sensitivity to the loss weight λ on VoxCeleb1-O. EER (%) is reported, bold indicates the best results.
School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, U.K. · Marine Geosciences, National Oceanography Centre, SO14 3ZH Southampton, U.K.