Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification

Zezhong Jin; Xiaoyu Wang; Zhe Li; Chong-Xin Gan; Siqing Qin; Man-Wai Mak; Kong Aik Lee

cs.SDOct 19, 2026

Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification

Authors: Zezhong Jin, Xiaoyu Wang, Zhe Li, Chong-Xin Gan, Siqing Qin, Man-Wai Mak, Kong Aik Lee

Organizations: Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hong Kong SAR · Baidu Inc. · Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA

Abstract

Large self-supervised speech models provide powerful representations for speaker verification, but full fine-tuning is expensive. Most conventional adapters insert small trainable modules to modify the output of each layer, while keeping the original residual connections unchanged. Building on our preliminary study of manifold-constrained Hyper-Connections in speaker encoders, we explore parameter-efficient adaptation of both residual paths and feature selection across layers. We first propose a Manifold-Constrained Residual Adapter (MCRA), which splits the hidden dimensions of each Transformer residual state into N groups and uses a small Sinkhorn-normalized matrix to exchange information among them. A learnable gate controls the balance between the original and mixed residual paths. The gate adds only one scalar parameter at each residual connection and initially favors the original path, allowing gradual adaptation. We further propose Frame-Adaptive Layer Aggregation (FALA), an attention-based aggregation method that uses a single learned query shared across all utterances to compute layer weights separately for each speech frame. Because the layer representations depend on the input, the resulting weights vary across both utterances and frames, unlike the fixed weights of a conventional layer-weighted sum. Compared with a frozen WavLM baseline using a fixed layer-weighted sum, our method achieves relative EER reductions of 22.4% on VoxCeleb1-O and 2.9% on CN-Celeb, with trainable adaptation parameters equivalent to only 0.0009% and 0.0016% of the WavLM-Large parameters, respectively.

Figures & tables

Fig. 1 : Architectural comparison between the standard residual shortcut and mHC: (a) standard residual connection, (b) high-level mHC overview, and (c) detailed mHC design, where

\mathbf{H}_{\mathrm{pre}}

aggregates streams before

\mathcal{F}(\cdot)

and

\mathbf{H}_{\mathrm{post}}

splits the output back into streams.

Fig. 2 : Overall architecture of the proposed system. The pretrained WavLM network is frozen. In each Transformer layer, MCRA produces the mixed residual $\widehat{\bm{r}}^{c}$ , which is fused with the original residual $\bm{r}^{c}$ through a learnable gate to obtain $\widetilde{\bm{r}}^{c}$ , where $c\in\{\mathrm{SA},\mathrm{FFN}\}$ . FALA then computes frame-dependent layer weights before the speaker encoder. Frozen LayerNorm and dropout operations are omitted from the diagram for clarity.

Architecture	Params (M)	VoxCeleb1-O		VoxCeleb1-E		VoxCeleb1-H		VoxSRC21-val
Architecture	Params (M)	EER(%)	minDCF	EER(%)	minDCF	EER(%)	minDCF	EER(%)	minDCF
Res2Net-48w2s [ 3 ]	5.50	1.75	0.183	1.70	0.182	2.98	0.283	–	–
Res2Net-MWA [ 37 , 38 ]	5.50	1.71	0.165	–	–	–	–	–	–
Res2Net	4.03	1.56	0.150	1.41	0.149	2.48	0.230	4.27	0.334
mHC-Res2Net [ 23 ]	4.03	1.41	0.154	1.40	0.145	2.43	0.228	4.13	0.332
ResNet34-FPM [ 39 , 38 ]	5.85	1.98	0.205	–	–	–	–	–	–

TABLE I : Comparison with representative Fbank-based speaker verification systems, including recent state-of-the-art methods. EER (%) and minDCF are reported on the standard VoxCeleb1 test sets and VoxSRC21-val. The best result within each architecture group is shown in bold.

Architecture	Params (M)	EER (%)	minDCF
ECAPA-TDNN-L (ours) [ 4 ]	20.76	8.65	0.498
ECAPA-TDNN+AAMSupCon [ 44 ]	–	8.49	0.500
D-TDNN [ 42 ]	2.85	8.41	0.468
SC-EcapaTdnn [ 45 ]	8.28	8.26	0.480
mHC-ECAPA-TDNN-L (ours), $G=8$	20.76	8.01	0.432

TABLE II : Results on the CN-Celeb evaluation set.

PTM	Encoder	Fine-tuning Method	AS-Norm	Per-Module #Params	Trainable #Params in PTM	VoxCeleb1-O		VoxCeleb1-E		VoxCeleb1-H
PTM	Encoder	Fine-tuning Method	AS-Norm	Per-Module #Params	Trainable #Params in PTM	EER(%)	minDCF	EER(%)	minDCF	EER(%)	minDCF
WavLM-Base+	TDNN	LoRA [ 19 , 21 ]	✗	2.40M	2.40M	2.35	0.179	2.65	0.178	5.04	0.288
		Houlsby Adapter [ 46 , 21 ]	✗	9.50M	9.50M	2.30	0.202	2.77	0.204	5.60	0.339
		Inner+Inter Adapter [ 21 ]	✗	4.80M	4.80M	1.82	0.156	2.18	0.150	4.05	0.251
		Deep Speaker Prompting [ 21 ]	✗	0.30M	0.30M	2.16	0.172	2.41	0.169	4.62	0.292
		UniPET-SPK [ 21 ]	✗	5.10M	5.10M	1.59	0.130	1.74	0.138	3.41	0.206

TABLE III : Comparison with representative fine-tuning and parameter-efficient adaptation methods on VoxCeleb1. The AS-Norm setting follows each source paper. Full fine-tuning results are shown in italics, and the best PEFT results for each PTM and metric are highlighted in bold.

PTM	Encoder	Fine-tuning Method	Per-Module #Params	Trainable #Params in PTM	CN-Celeb1.Eval
PTM	Encoder	Fine-tuning Method	Per-Module #Params	Trainable #Params in PTM	EER(%)	minDCF
WavLM-Large	ECAPA-TDNN	None (Frozen PTM) [ 22 ]	0.00M	0.00M	8.72	0.448
		Full fine-tuning [ 22 ]	–	316.00M	7.25	0.428
		MCRA	0.065K	3.145K	9.05	0.454
		FALA	2.048K	2.048K	8.90	0.459
		MCRA + FALA	2.113K	5.168K	8.47	0.447

TABLE IV : Comparison of the proposed parameter-efficient adaptation methods on CN-Celeb1.Eval under the CN-Celeb1&2 training setting. All MCRA configurations use

N=8

. Best results are shown in bold.

Fig. 3 : Effect of the number of residual streams in the original mHC systems on (a) the VoxCeleb test sets and (b) CN-Celeb.

Fig. 4 : Effect of the number of feature groups in the proposed MCRA–FALA system on (a) the VoxCeleb1 test sets and (b) CN-Celeb.

Residual mixing	Row sum	Column sum	EER (%)	minDCF
Identity	1	1	0.82	0.114
Unconstrained	–	–	0.92	0.117
Row softmax	1	–	0.88	0.114
Sinkhorn	1	1	0.66	0.085

TABLE V : Comparison of residual mixing constraints on VoxCeleb1-O.

Placement	EER (%)	minDCF
Self-attention residual only	0.83	0.116
FFN residual only	0.77	0.148
Both attention and FFN	0.66	0.085

TABLE VI : Effect of MCRA placement on VoxCeleb1-O. Sinkhorn normalization, a learnable gate, and

N=4

are fixed for all configurations.

Gate design	Learnable	EER (%)	minDCF
No gate	✗	0.77	0.105
Fixed gate	✗	0.74	0.095
Learnable gate	✓	0.66	0.085

TABLE VII : Ablation of the MCRA gate design on VoxCeleb1-O.

Fig. 5 : Frame-dependent FALA layer weights under (a) clean speech, (b) additive noise at an SNR of 5 dB, and (c) reverberation. The bar plots on the right show the layer weights averaged over time.

Fig. 6 : Cosine-score distributions for the (left) frozen WavLM baseline and (right) proposed MCRA–FALA system. Blue and orange curves denote target and non-target trials, respectively; dashed vertical lines denote the decision thresholds. The insets enlarge the overlapping score regions.

Panel of Figure 14 — Fig. 7 : t-SNE visualization of speaker embeddings extracted by (a) the frozen WavLM baseline and (b) the proposed MCRA–FALA system. The visualization contains eight speakers and 45 utterances per speaker. Different colors denote different speakers.

Fig. 8 : Comparison of trainable parameters in WavLM and EER on VoxCeleb1-O. The shaded region indicates methods with fewer than 10K trainable parameters.

Speaker Verification

Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.

See all 62 papers

TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models
Oct 4, 2026Junjie Li, Zheng Liang, Zhe Li +2Speaker Verification Audio-Language Models
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR, China · National University of Singapore
What and When to Learn: CURriculum Ranking Loss for Large-Scale Speaker Verification
Mar 25, 2026Massa Baali, Sarthak Bisht, Rita Singh +1Speaker Verification Curriculum Learning
Carnegie Mellon University, Pittsburgh, USA
ReDimNet2+: Multi-Corpus Data Scaling for Robust Speaker Verification
Sep 29, 2026Kirill Borodin, Vasilii Kudryavtsev, Maxim Maslov +1Speaker Verification
lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia

Oct 4, 2026J/K move · Enter open · S save

TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models

Junjie Li, Zheng Liang, Zhe Li, Tianchi Liu+1

Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR, China · National University of Singapore

Speaker Verification Audio-Language Models Audio Representation Learning

Open paper AskPDF arXiv

Audio large language models (ALLMs) can understand speech content, yet their ability to use speaker identity for verification remains limited. We propose TS-SP (Two-Stage Speaker Preservation), a parameter-efficient framework for learning speaker-preserving representations and making them accessible to an ALLM's language-model component. We instantiate and evaluate TS-SP on Qwen2.5-Omni-7B. First, we adapt the audio encoder with speaker identity supervision. We then freeze the adapted encoder and train the language model to compare speakers. Both stages use low-rank adaptation (LoRA), keeping the pretrained base weights fixed. On Vox1-O, TS-SP reduces the equal error rate (EER) from 7.01% for the Paired Loss Adaptation Baseline to 4.37%. EER remains within 4.31--4.79% under unseen prompts. Cross-domain evaluation on CN-Celeb yields a similar EER to the baseline, but lower accuracy at the native decision threshold. These findings support two-stage adaptation for improving speaker verification on the evaluated backbone.

Abstract

Speaker Verification

Explore similar work

TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models

What and When to Learn: CURriculum Ranking Loss for Large-Scale Speaker Verification

ReDimNet2+: Multi-Corpus Data Scaling for Robust Speaker Verification