Meetings last up to an hour, and nobody knows in advance how many people will speak. Neural diarizers with a fixed number of output slots, such as Streaming Sortformer and Nemotron-3, are accurate and run online, but they cannot name more speakers than they have slots. We remove this limit without retraining. The slot head keeps deciding who is active and when speech overlaps, and speaker embeddings are clustered inside a single slot, with the official e-means algorithm, only after every slot has been used. In the streaming system, re-clustering is confined to a bounded window, labels are revised for a few seconds, and identities are anchored to the slots of the head. The resulting system has no limit on the number of speakers and stays accurate when speakers outnumber the slots, where fixed-slot models break down. On meetings that grow to an hour it stays far more accurate than Streaming Sortformer, which accumulates errors, and its cost per update does not grow with the meeting. The same recipe turns the four-slot Streaming Sortformer into an open-speaker system. Across recordings, e-means makes it affordable to link the speakers of a large archive frame by frame, a task that arises when conversational corpora are prepared, and it reaches the clustering quality of Lloyd's k-means several times faster. The method needs little tuning, and e-means runs with its default recipe from a few hundred to more than a million embeddings.
Figures & tables
Figure 1: eDIAR at a glance. Streaming at 1.04 s. (a) DER against the number of speakers, from two to twenty, on LibriSpeech-OpenSpk conversations, compared with the two fixed-slot models; the dotted lines mark four and eight slots. (b) DER of six one-hour meetings with 12 to 20 speakers (LibriSpeech-OpenSpk utterances, speakers joining over time), scored on their first 15, 30 and 45 minutes. (c) Linking the speakers of 1,248 sessions with 1,251 VoxCeleb1 speakers across recordings: archive DER against clustering time at the level of 1.16 million frames; numbers next to the markers are passes over the data. (d) Processing time per update over the same long recordings, for our bounded 90 s window and for re-clustering the whole history, with the real-time budget of one 0.72 s chunk.
VoxConverse
AMI
ICSI
LibriSpeech
System
DER
SCA
DER
SCA
DER
SCA
DER
SCA
Streaming Sortformer 4spk, 30.4 s config
13.3
43.2
7.2
81.2
29.1
13.3
40.7
33.3
Nemotron-3 (8 slots, 30.4 s)
5.5
75.0
9.0
68.8
27.0
53.3
18.1
66.7
pyannote 3.1
5.6
45.5
27.4
37.5
33.2
26.7
32.1
33.3
Mini-batch k -means (Sculley)
7.6
31.8
13.5
12.5
36.1
0.0
14.7
72.2
Lloyd k -means++ (25 ep.)
13.8
59.1
9.5
37.5
36.1
26.7
25.3
77.8
Table 1: Non-streaming. Mean over all sessions of each corpus, DER and SCA in %. Best per column in bold, second best underlined. Streaming Sortformer is a four-speaker model and cannot output more than four speakers.
VoxConverse
AMI
ICSI
LibriSpeech
System
DER
SCA
DER
SCA
DER
SCA
DER
SCA
Streaming Sortformer 4spk (1.04 s)
14.1
38.6
8.3
75.0
27.7
13.3
41.4
27.8
Nemotron-3 head, 8 slots (1.04 s)
6.4
75.0
9.3
75.0
27.4
46.7
19.8
66.7
Ours, as first emitted (1.04 s)
5.9
75.0
9.2
75.0
27.5
53.3
18.5
55.6
Ours, after ≤ 10 s revision (1.04 s)
5.8
75.0
9.1
75.0
27.4
53.3
16.4
61.1
Nemotron-3 head, 8 slots (0.64 s)
6.8
72.7
9.1
75.0
27.5
46.7
20.1
66.7
Table 2: Streaming. Same protocol as Table 1 . “As first emitted”: labels as shown in real time; “after revision”: labels once they are 10 s old. Bold and underline mark the best and second best value within each latency block, 1.04 s and 0.64 s. Streaming Sortformer appears only in the 1.04 s block because its model card gives just two configurations, 1.04 s and 30.4 s, and we did not run it at 0.64 s. The Nemotron-3 head uses the same streaming inputs as our system. Clustering baselines are reported in Appendix B.3 .
Vox
AMI
ICSI
Libri
Streaming Sortformer 4spk
-8.3 [-12.3, -4.5] ∗
+0.8 [-0.9, +2.6]
-0.3 [-5.3, +4.9]
-25.0 [-32.4, -17.2] ∗
Nemotron-3 head (8 slots)
-0.6 [-1.5, -0.1] ∗
-0.1 [-0.4, +0.1]
-0.0 [-0.3, +0.3]
-3.3 [-5.8, -1.3] ∗
Table 3: DER difference, ours after revision minus competitor, with 95 % paired bootstrap intervals over sessions, in percentage points; negative means ours is better. ∗ : interval excludes zero.
Slot head and clustering
Vox
AMI
ICSI
Libri
Streaming Sortformer head (4 slots)
15.8
10.6
30.8
41.4
+ our clustering
6.6
12.1
30.8
16.2
Nemotron-3 head (8 slots)
6.4
9.7
26.5
18.1
+ our clustering
4.6
9.0
26.5
9.9
Table 4: DER (%) with two different slot heads on the same embeddings, non-streaming clustering of streaming-latency inputs. The Sortformer head is thresholded per frame, without its own post-processing.
VoxConverse
AMI
ICSI
Full-length, streaming (1.04 s)
DER
SCA
DER
SCA
DER
SCA
Streaming Sortformer 4spk (1.04 s)
18.1
29.5
8.9
93.8
36.3
6.7
Nemotron-3 head, 8 slots
8.5
59.1
9.7
81.2
28.5
33.3
Ours, as first emitted
8.0
59.1
9.6
87.5
28.7
40.0
Ours, after revision
7.9
59.1
9.6
87.5
28.6
40.0
Table 5: Full-length, uncropped sessions, streaming at 1.04 s. Mean session lengths are 8, 34 and 61 minutes.
Figure 2: Speaker linking across an archive of 1,248 sessions with 1,251 VoxCeleb1 speakers. Archive DER on the left and spherical clustering loss on the right against clustering time, for e-means and Lloyd’s k -means at frame level, solid, and at segment level, dashed. Numbers next to the markers are passes over the data. The dotted line is the standard recipe, agglomerative clustering of one centroid per local speaker. CPU, 16 threads.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Set
Source
Sessions
Speakers per session
Length per session
VoxConverse
YouTube clips, real
44
1–17 (full: 1–21)
480 s (full: up to 20 min)
AMI
design-scenario meetings, real
16
3–4
420 s (full: 14–50 min)
ICSI
research meetings, real
15
4–7 (full: 4–10)
420 s (full: 44–86 min)
LibriSpeech-OpenSpk
read speech, constructed
18
2–12
24–150 s
unseen sessions
read speech, constructed
60
2–12
about 90 s
many speakers
read speech, constructed
20
14–20
about 3.6 min
Appendix
Table 8: Evaluation sets. “Full” refers to the uncut recordings used in the full-length evaluation.
Figure 3: Figure 1 a and b restricted to up to 12 speakers and 60 minutes, with offline pyannote 3.1 added; hollow markers are the Nemotron-3 head.
Figure 4: (a) Five ICSI days of up to three hours (hollow markers: Nemotron-3 head, almost identical to ours). (b) One-hour meetings with 12 to 20 speakers, short turns, all speakers present from the start.
VoxConverse
AMI
ICSI
LibriSpeech
System
DER
SCA
DER
SCA
DER
SCA
DER
SCA
Online Leader-Follower (non-streaming)
6.2
54.5
9.9
12.5
30.2
0.0
10.4
72.2
Ours (non-streaming)
4.5
75.0
8.9
68.8
26.9
53.3
8.6
88.9
Online Leader-Follower (1.04 s)
5.9
50.0
10.7
0.0
31.6
6.7
11.6
61.1
Ours, after ≤ 10 s revision (1.04 s)
5.8
75.0
9.1
75.0
27.4
53.3
16.4
61.1
Online Leader-Follower (0.64 s)
6.6
38.6
10.8
0.0
31.5
6.7
12.1
61.1
Appendix
Table 9: Online leader–follower clustering and our system, non-streaming and streaming. Bold and underline compare the two systems within each setting.
DER (%), held-out half
VoxConverse
AMI
ICSI
LibriSpeech
streaming, 1.04 s input buffer
Streaming Sortformer 4spk
14.1
9.0
23.7
42.5
Nemotron-3 head, 8 slots
5.6
10.6
21.7
21.9
Online Leader-Follower
5.2
12.1
23.7
11.8
Ours, as first emitted
5.3
10.4
21.9
19.6
Ours, after revision
5.2
10.4
21.7
17.6
Appendix
Table 12: DER (%) on the held-out half of each corpus only, that is, the alternate sessions that were not used for setting any threshold (22 VoxConverse, 8 AMI, 7 ICSI and 9 LibriSpeech-OpenSpk sessions). Upper block: streaming systems with a 1.04 s input buffer, fed in 100 ms chunks; “as first emitted” are the labels shown in real time and “after revision” the labels once they are 10 s old. Lower block: non-streaming systems that see the whole session; Streaming Sortformer and the Nemotron-3 head run in their 30.4 s configuration. Bold and underline mark the best and second best value within each block.
DER (%) on 60 unseen sessions
K=2
4
6
8
10
12
all
Streaming Sortformer 4spk
12.0
16.8
39.8
56.2
64.2
68.8
43.0
Nemotron-3 head (8 slots)
8.7
14.3
9.8
11.3
36.2
46.7
21.2
Online Leader-Follower
7.9
11.9
8.7
10.7
18.5
22.7
13.4
Ours, as first emitted
8.7
14.3
9.6
13.7
36.5
40.3
20.5
Ours, after revision
8.2
13.8
9.2
13.2
29.4
32.7
17.8
Appendix
Table 15: Streaming systems on 60 new LibriSpeech-OpenSpk conversations (ten per K , unseen seed, 1.04 s).
Nemotron-3 head
Leader–follower
Ours
Corpus
overlap (% of speech)
DER
miss
DER
miss
DER
miss
VoxConverse
2.8
52.6
51.3
53.1
51.0
52.8
51.4
AMI
7.0
56.3
53.5
60.1
52.2
56.1
53.5
ICSI
18.9
60.7
58.1
62.2
57.6
60.5
58.1
LibriSpeech-OpenSpk
1.0
65.1
63.9
60.3
60.2
61.1
60.8
Appendix
Table 20: DER (%) and missed speech (%) inside reference-overlap regions only, collar 0, non-streaming inputs. “overlap” is the share of reference speech that is overlapped.
Clustering engine inside the splitting step
Vox
Libri
AMI
ICSI
ms / call
e-means (official, 5 ep, 3 restarts)
4.45
8.56
8.90
26.90
28.6
e-means (1 restart)
4.41
9.00
8.90
26.90
24.4
Lloyd k-means (25 it, 3 restarts)
4.43
8.92
8.90
26.90
7.2
Lloyd k-means (5 it, 3 restarts)
4.39
8.90
8.90
26.90
1.7
mini-batch k-means (Sculley, 5 ep)
4.37
8.82
8.90
26.90
13.9
online-birth core (no k selection)
4.43
9.71
8.90
26.90
0.0
Appendix
Table 22: DER (%) of the non-streaming system with different clustering engines in the splitting step, and mean engine time per call (ms).
Figure 5: Top: spherical clustering loss relative to 25 Lloyd iterations on the engine inputs of the benchmark. Bottom: DER of the non-streaming system on LibriSpeech-OpenSpk when the slot-splitting engine is limited to the given number of passes.
VoxConverse
AMI
ICSI
LibriSpeech
System
MAE
cpWER
MAE
cpWER
MAE
cpWER
MAE
cpWER
Streaming Sortformer 4spk, 30.4 s config
2.0
–
0.2
14.9
1.5
–
3.3
69.4
Nemotron-3 (8 slots, 30.4 s)
0.7
–
0.3
13.2
0.5
–
1.0
23.6
pyannote 3.1
1.0
–
0.9
–
1.3
–
2.2
–
Online Leader-Follower
2.5
–
3.9
15.7
6.2
–
0.9
9.8
Mini-batch k -means (Sculley)
1.7
–
4.9
25.2
6.2
–
3.1
19.4
Appendix
Table 29: Speaker-count MAE and oracle-transcript cpWER (%); upper block non-streaming, lower block streaming (rows as in Tables 1 and 2 ); cpWER is only defined for AMI and LibriSpeech.