S2SND-v2: Unified Online and Offline Speaker Diarization with Speaker-Adapted w2v-BERT 2.0

Jincheng Wang; Li Li; Weixin Zhu; Yannan Wang; Tao Yu; Juan Liu; Ming Li

cs.SDOct 26, 2026

S2SND-v2: Unified Online and Offline Speaker Diarization with Speaker-Adapted w2v-BERT 2.0

Authors: Jincheng Wang, Li Li, Weixin Zhu, Yannan Wang, Tao Yu, Juan Liu, Ming Li

Organizations: School of Artificial Intelligence, Wuhan University, Wuhan, China · School of Artificial Intelligence, the Chinese University of Hong Kong, Shenzhen, China · Tencent Ethereal Audio Lab, Tencent, Shenzhen, China

Abstract

This paper presents S2SND-v2, a unified online and offline speaker diarization framework built on S2SND that improves acoustic feature extraction and the use of the resulting representations. For acoustic feature extraction, we introduce a speaker-adapted w2v-BERT 2.0 front end and compare full and pruned construction routes. To better use these features, overlap-aware activity-query masking improves speaker embedding extraction under overlap, while late-speaker training aligns unknown-speaker training with online inference conditions. Combining these improvements, the pruned system achieves competitive average diarization error rates of 14.52% online and 12.03% offline across eight test sets.

Figures & tables

Figure 2: Three w2v-BERT front end construction routes. Blue arrows denote training with speaker-verification data; red arrows denote training with diarization data.

Configuration	Param. (M)	FLOPs (G)	Mode	AMI-SDM	AIS-4	AliM	NSF	MSD	DH3	RAMC	VoxC	Avg.
Full w2v-BERT	580+33	475.37	Online	16.92	10.78	11.88	21.22	17.77	16.49	10.96	11.92	14.74
			Offline	13.27	9.66	9.90	15.47	13.58	15.09	10.43	9.80	12.15
Prune-after-diarization	91+33	75.63	Online	17.76	12.12	12.14	23.36	17.80	15.79	11.19	12.27	15.30
			Offline	13.52	10.54	10.02	17.66	13.62	14.48	10.56	10.48	12.61
Prune-before-diarization	91+33	75.63	Online	18.05	10.98	12.14	24.36	17.86	15.68	11.06	11.96	15.26
			Offline	14.41	9.93	10.17	18.62	13.73	14.06	10.50	10.14	12.70

Table 1: Front end routes and cumulative S2SND-v2 ablation: DER (%) with a 0s collar. All models use four simulation-source corpora and eight real-data training splits. Parameters are shown as w2v-BERT encoder+remaining modules (millions). FLOPs measure total computation of the full system when decoding an 8s window once. Additions build on prune-before-diarization. Bold indicates the best displayed result per dataset and mode, including ties; lower is better.

System	AMI-SDM	AIS-4	AliM	NSF	MSD	DH3	RAMC	VoxC	Avg.
Online inference
S2SND (retrained) [ 6 ]	19.48	12.69	13.71	26.87	20.34	19.21	11.31	13.62	17.15
S2SND-v2 (ReDimNetV2-B6)	18.92	11.32	14.01	24.88	19.98	18.25	11.54	11.70	16.33
S2SND-v2 (w2v-BERT 2.0)	16.95	11.21	12.20	20.23	17.91	15.70	10.92	11.04	14.52
Offline inference
Pyannote v3.1 [ 2 ]	22.40	12.20	24.40	–	25.30	21.70	22.20	11.30	–

Table 2: DER (%) comparison on eight test sets without oracle VAD and with a 0s collar. Online and offline systems are grouped separately; dashes denote unavailable results. Avg. is omitted for incomplete coverage. Both S2SND-v2 variants are trained in this work and include overlap-aware activity-query masking and late-speaker training. External results are from the cited publications or official repositories; evaluation protocols may differ. Bold values are the best within each mode based on displayed values, including ties; lower is better.

Speaker Diarization

Speaker diarization identifies who spoke when in audio, including speaker changes, overlapping speech, and speaker identity across conversations.

See all 59 papers

DiariZen Explained: A Tutorial for the Open Source State-of-the-Art Speaker Diarization Pipeline
Apr 23, 2026Nikhil RaghavSpeaker Embedding Speaker Diarization
Institute for Advancing Intelligence, TCG CREST, Kolkata – 700 091, India · Department of Computer Science, RKMVERI, Howrah – 711 202, India
FASTDIAR: Frame-level speaker encoder for Streaming Diarization
Oct 2, 2026Nikita Torgashov, Okan KöpüklüSpeaker Embedding Speaker Diarization
Department of Speech, Music and Hearing KTH Royal Institute of Technology Stockholm, Sweden · Microsoft Munich, Germany
DiarizationLM: Speaker Diarization Post-Processing with Large Language Models
Jan 7, 2024Quan Wang, Yiling Huang, Guanlong Zhao +3Speaker Diarization
Google LLC, USA

Apr 23, 2026J/K move · Enter open · S save

DiariZen Explained: A Tutorial for the Open Source State-of-the-Art Speaker Diarization Pipeline

Nikhil Raghav

Institute for Advancing Intelligence, TCG CREST, Kolkata – 700 091, India · Department of Computer Science, RKMVERI, Howrah – 711 202, India

Speaker Embedding Speaker Diarization

Open paper AskPDF arXiv

Speaker diarization (SD) is the task of answering "who spoke when" in a multi-speaker audio stream. Classically, an SD system clusters segments of speech belonging to an individual speaker's identity. Recent years have seen substantial progress in SD through end-to-end neural diarization (EEND) approaches. DiariZen, a hybrid SD pipeline built upon a structurally pruned WavLM-Large encoder, a Conformer backend with powerset classification, and VBx clustering, represents the leading open-source state of the art at the time of writing across multiple benchmarks. Despite its strong performance, the DiariZen architecture spans several repositories and frameworks, making it difficult for researchers and practitioners to understand, reproduce, or extend the system as a whole. This tutorial paper provides a self-contained, block-by-block explanation of the complete DiariZen pipeline, decomposing it into seven stages: (1) audio loading and sliding window segmentation, (2) WavLM feature extraction with learned layer weighting, (3) Conformer backend and powerset classification, (4) segmentation aggregation via overlap-add, (5) speaker embedding extraction with overlap exclusion, (6) VBx clustering with PLDA scoring, and (7) reconstruction and RTTM output. For each block, we provide the conceptual motivation, source code references, intermediate tensor shapes, and annotated visualizations of the actual outputs on a 30s excerpt from the AMI Meeting Corpus. The implementation is available at https://github.com/nikhilraghav29/diarizen-tutorial, which includes standalone executable scripts for each block and a Jupyter notebook that runs the complete pipeline end-to-end.

Abstract

Speaker Diarization

Explore similar work

DiariZen Explained: A Tutorial for the Open Source State-of-the-Art Speaker Diarization Pipeline

FASTDIAR: Frame-level speaker encoder for Streaming Diarization

DiarizationLM: Speaker Diarization Post-Processing with Large Language Models