Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification
Authors: Zezhong Jin, Xiaoyu Wang, Zhe Li, Chong-Xin Gan, Siqing Qin, Man-Wai Mak, Kong Aik Lee
Organizations: Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hong Kong SAR · Baidu Inc. · Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA
Large self-supervised speech models provide powerful representations for speaker verification, but full fine-tuning is expensive. Most conventional adapters insert small trainable modules to modify the output of each layer, while keeping the original residual connections unchanged. Building on our preliminary study of manifold-constrained Hyper-Connections in speaker encoders, we explore parameter-efficient adaptation of both residual paths and feature selection across layers. We first propose a Manifold-Constrained Residual Adapter (MCRA), which splits the hidden dimensions of each Transformer residual state into N groups and uses a small Sinkhorn-normalized matrix to exchange information among them. A learnable gate controls the balance between the original and mixed residual paths. The gate adds only one scalar parameter at each residual connection and initially favors the original path, allowing gradual adaptation. We further propose Frame-Adaptive Layer Aggregation (FALA), an attention-based aggregation method that uses a single learned query shared across all utterances to compute layer weights separately for each speech frame. Because the layer representations depend on the input, the resulting weights vary across both utterances and frames, unlike the fixed weights of a conventional layer-weighted sum. Compared with a frozen WavLM baseline using a fixed layer-weighted sum, our method achieves relative EER reductions of 22.4% on VoxCeleb1-O and 2.9% on CN-Celeb, with trainable adaptation parameters equivalent to only 0.0009% and 0.0016% of the WavLM-Large parameters, respectively.
Figures & tables
Fig. 1 : Architectural comparison between the standard residual shortcut and mHC: (a) standard residual connection, (b) high-level mHC overview, and (c) detailed mHC design, where Hpre aggregates streams before F(⋅) and Hpost splits the output back into streams.
Fig. 2 : Overall architecture of the proposed system. The pretrained WavLM network is frozen. In each Transformer layer, MCRA produces the mixed residual rc , which is fused with the original residual rc through a learnable gate to obtain rc , where c∈{SA,FFN} . FALA then computes frame-dependent layer weights before the speaker encoder. Frozen LayerNorm and dropout operations are omitted from the diagram for clarity.
Architecture
Params (M)
VoxCeleb1-O
VoxCeleb1-E
VoxCeleb1-H
VoxSRC21-val
EER(%)
minDCF
EER(%)
minDCF
EER(%)
minDCF
EER(%)
minDCF
Res2Net-48w2s [ 3 ]
5.50
1.75
0.183
1.70
0.182
2.98
0.283
–
–
Res2Net-MWA [ 37 , 38 ]
5.50
1.71
0.165
–
–
–
–
–
–
Res2Net
4.03
1.56
0.150
1.41
0.149
2.48
0.230
4.27
0.334
mHC-Res2Net [ 23 ]
4.03
1.41
0.154
1.40
0.145
2.43
0.228
4.13
0.332
ResNet34-FPM [ 39 , 38 ]
5.85
1.98
0.205
–
–
–
–
–
–
TABLE I : Comparison with representative Fbank-based speaker verification systems, including recent state-of-the-art methods. EER (%) and minDCF are reported on the standard VoxCeleb1 test sets and VoxSRC21-val. The best result within each architecture group is shown in bold.
Architecture
Params (M)
EER (%)
minDCF
ECAPA-TDNN-L (ours) [ 4 ]
20.76
8.65
0.498
ECAPA-TDNN+AAMSupCon [ 44 ]
–
8.49
0.500
D-TDNN [ 42 ]
2.85
8.41
0.468
SC-EcapaTdnn [ 45 ]
8.28
8.26
0.480
mHC-ECAPA-TDNN-L (ours), G=8
20.76
8.01
0.432
TABLE II : Results on the CN-Celeb evaluation set.
PTM
Encoder
Fine-tuning Method
AS-Norm
Per-Module #Params
Trainable #Params in PTM
VoxCeleb1-O
VoxCeleb1-E
VoxCeleb1-H
EER(%)
minDCF
EER(%)
minDCF
EER(%)
minDCF
WavLM-Base+
TDNN
LoRA [ 19 , 21 ]
✗
2.40M
2.40M
2.35
0.179
2.65
0.178
5.04
0.288
Houlsby Adapter [ 46 , 21 ]
✗
9.50M
9.50M
2.30
0.202
2.77
0.204
5.60
0.339
Inner+Inter Adapter [ 21 ]
✗
4.80M
4.80M
1.82
0.156
2.18
0.150
4.05
0.251
Deep Speaker Prompting [ 21 ]
✗
0.30M
0.30M
2.16
0.172
2.41
0.169
4.62
0.292
UniPET-SPK [ 21 ]
✗
5.10M
5.10M
1.59
0.130
1.74
0.138
3.41
0.206
TABLE III : Comparison with representative fine-tuning and parameter-efficient adaptation methods on VoxCeleb1. The AS-Norm setting follows each source paper. Full fine-tuning results are shown in italics, and the best PEFT results for each PTM and metric are highlighted in bold.
PTM
Encoder
Fine-tuning Method
Per-Module #Params
Trainable #Params in PTM
CN-Celeb1.Eval
EER(%)
minDCF
WavLM-Large
ECAPA-TDNN
None (Frozen PTM) [ 22 ]
0.00M
0.00M
8.72
0.448
Full fine-tuning [ 22 ]
–
316.00M
7.25
0.428
MCRA
0.065K
3.145K
9.05
0.454
FALA
2.048K
2.048K
8.90
0.459
MCRA + FALA
2.113K
5.168K
8.47
0.447
TABLE IV : Comparison of the proposed parameter-efficient adaptation methods on CN-Celeb1.Eval under the CN-Celeb1&2 training setting. All MCRA configurations use N=8 . Best results are shown in bold.
Fig. 3 : Effect of the number of residual streams in the original mHC systems on (a) the VoxCeleb test sets and (b) CN-Celeb.
Fig. 4 : Effect of the number of feature groups in the proposed MCRA–FALA system on (a) the VoxCeleb1 test sets and (b) CN-Celeb.
Residual mixing
Row sum
Column sum
EER (%)
minDCF
Identity
1
1
0.82
0.114
Unconstrained
–
–
0.92
0.117
Row softmax
1
–
0.88
0.114
Sinkhorn
1
1
0.66
0.085
TABLE V : Comparison of residual mixing constraints on VoxCeleb1-O.
Placement
EER (%)
minDCF
Self-attention residual only
0.83
0.116
FFN residual only
0.77
0.148
Both attention and FFN
0.66
0.085
TABLE VI : Effect of MCRA placement on VoxCeleb1-O. Sinkhorn normalization, a learnable gate, and N=4 are fixed for all configurations.
Gate design
Learnable
EER (%)
minDCF
No gate
✗
0.77
0.105
Fixed gate
✗
0.74
0.095
Learnable gate
✓
0.66
0.085
TABLE VII : Ablation of the MCRA gate design on VoxCeleb1-O.
Fig. 5 : Frame-dependent FALA layer weights under (a) clean speech, (b) additive noise at an SNR of 5 dB, and (c) reverberation. The bar plots on the right show the layer weights averaged over time.
Fig. 6 : Cosine-score distributions for the (left) frozen WavLM baseline and (right) proposed MCRA–FALA system. Blue and orange curves denote target and non-target trials, respectively; dashed vertical lines denote the decision thresholds. The insets enlarge the overlapping score regions.
Fig. 7 : t-SNE visualization of speaker embeddings extracted by (a) the frozen WavLM baseline and (b) the proposed MCRA–FALA system. The visualization contains eight speakers and 45 utterances per speaker. Different colors denote different speakers.
Fig. 8 : Comparison of trainable parameters in WavLM and EER on VoxCeleb1-O. The shaded region indicates methods with fewer than 10K trainable parameters.
Audio large language models (ALLMs) can understand speech content, yet their ability to use speaker identity for verification remains limited. We propose TS-SP (Two-Stage Speaker Preservation), a parameter-efficient framework for learning speaker-preserving representations and making them accessible to an ALLM's language-model component. We instantiate and evaluate TS-SP on Qwen2.5-Omni-7B. First, we adapt the audio encoder with speaker identity supervision. We then freeze the adapted encoder and train the language model to compare speakers. Both stages use low-rank adaptation (LoRA), keeping the pretrained base weights fixed. On Vox1-O, TS-SP reduces the equal error rate (EER) from 7.01% for the Paired Loss Adaptation Baseline to 4.37%. EER remains within 4.31--4.79% under unseen prompts. Cross-domain evaluation on CN-Celeb yields a similar EER to the baseline, but lower accuracy at the native decision threshold. These findings support two-stage adaptation for improving speaker verification on the evaluated backbone.
Junjie Li, Zheng Liang, Zhe Li +2
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR, China · National University of Singapore
Speaker verification at large scale remains an open challenge as fixed-margin losses treat all samples equally regardless of quality. We hypothesize that mislabeled or degraded samples introduce noisy gradients that disrupt compact speaker manifolds. We propose Curry (CURriculum Ranking), an adaptive loss that estimates sample difficulty online via Sub-center ArcFace: confidence scores from dominant sub-center cosine similarity rank samples into easy, medium, and hard tiers using running batch statistics, without auxiliary annotations. Learnable weights guide the model from stable identity foundations through manifold refinement to boundary sharpening. To our knowledge, this is the largest-scale speaker verification system trained to date. Evaluated on VoxCeleb1-O, and SITW, Curry reduces EER by 86.8% and 60.0% over the Sub-center ArcFace baseline, establishing a new paradigm for robust speaker verification on imperfect large-scale data.
Automatic speaker verification must remain reliable across devices, rooms, and compression pipelines. We present ReDimNet2+, which scales training of the compact ReDimNet2 backbone across seven public corpora (63,934 speakers, about 8,675 hours). Analysis of a VoxBlink2 subset reveals a shift in predicted spectral coloration, motivating codec and waveform augmentation alongside this multi-corpus training, large-margin fine-tuning (LMFT), and graph-based retrieval reranking. With random 4-second evaluation windows for all models, ReDimNet2+ LMFT reduces pooled VoxCeleb1 EER from 2.42% to 0.82% and a 26-condition robustness stress-test EER from 7.21% to 1.99%. Under this shared local protocol, it reaches 0.35% EER on VoxCeleb1-O versus 0.787% for the best evaluated WeSpeaker checkpoint. On a VoxBlink2 retrieval subset, reranking improves the final model's Pr@k from 0.7413 to 0.7687.