cs.SDMar 25, 2026

What and When to Learn: CURriculum Ranking Loss for Large-Scale Speaker Verification

Authors: Massa Baali, Sarthak Bisht, Rita Singh, Bhiksha Raj

Organizations: Carnegie Mellon University, Pittsburgh, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Speaker verification at large scale remains an open challenge as fixed-margin losses treat all samples equally regardless of quality. We hypothesize that mislabeled or degraded samples introduce noisy gradients that disrupt compact speaker manifolds. We propose Curry (CURriculum Ranking), an adaptive loss that estimates sample difficulty online via Sub-center ArcFace: confidence scores from dominant sub-center cosine similarity rank samples into easy, medium, and hard tiers using running batch statistics, without auxiliary annotations. Learnable weights guide the model from stable identity foundations through manifold refinement to boundary sharpening. To our knowledge, this is the largest-scale speaker verification system trained to date. Evaluated on VoxCeleb1-O, and SITW, Curry reduces EER by 86.8% and 60.0% over the Sub-center ArcFace baseline, establishing a new paradigm for robust speaker verification on imperfect large-scale data.

Figures & tables

Figure 1: The adaptive curriculum learning pipeline. The encoder E\mathcal{E} maps raw waveforms to speaker embeddings projected into a sub-center angular distance space, where each speaker region contains KK sub-centers capturing acoustic variability. Per-sample confidence scores from the dominant sub-center cosine similarity are ranked against running batch statistics (μ^,σ^)(\hat{\mu},\hat{\sigma}) into Hard, Medium, and Easy tiers. Learnable weights WHW_{H} , WMW_{M} , WEW_{E} scale each sample’s gradient contribution before loss reduction.
DatasetLossEER (%)minDCF
VoxCeleb1-OBaseline2.870.45
VoxCeleb1-OCurry0.380.04
SITWBaseline4.000.27
SITWCurry1.600.07
Table 1: EER (%) and minDCF on VoxCeleb1-O and SITW. ( ↓\downarrow better)
CategorySubgroupBaselineCurry
GenderFemale (59 spks)6.231.09
Male (43 spks)9.951.84
AgeF (18–25), 13 spks7.191.58
F (26–35), 13 spks6.341.15
F (36–45), 7 spks4.410.40
F (46–55), 14 spks7.351.00
Table 2: EER (%) across demographic subgroups in EARS dataset on speaker verficiation (↓ better)
Figure 2: Cosine EER (%) evolution over training steps on a subset of the most difficult training instances. Curry (top) converges smoothly and stabilizes, while the Sub-center ArcFace baseline (bottom) plateaus above 4% and exhibits instability in later epochs.
More in this topic

Speaker Verification

Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.

See all 62 papers

Explore similar work

CardsList

  1. ReDimNet2+: Multi-Corpus Data Scaling for Robust Speaker Verification

    Sep 29, 2026Kirill Borodin, Vasilii Kudryavtsev, Maxim Maslov +1Speaker Verification

    lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia

  2. Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification

    Oct 19, 2026Zezhong Jin, Xiaoyu Wang, Zhe Li +4Speaker VerificationAdapter Tuning

    Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hong Kong SAR · Baidu Inc. · Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA

  3. TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models

    Oct 4, 2026Junjie Li, Zheng Liang, Zhe Li +2Speaker VerificationAudio-Language Models

    Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR, China · National University of Singapore