eDIAR: Training-Free Open-Speaker Streaming Diarization for Long Meetings and Large Archives via e-means
Organizations: KAIST AI · Ewha W. University
Abstract
Meetings last up to an hour, and nobody knows in advance how many people will speak. Neural diarizers with a fixed number of output slots, such as Streaming Sortformer and Nemotron-3, are accurate and run online, but they cannot name more speakers than they have slots. We remove this limit without retraining. The slot head keeps deciding who is active and when speech overlaps, and speaker embeddings are clustered inside a single slot, with the official e-means algorithm, only after every slot has been used. In the streaming system, re-clustering is confined to a bounded window, labels are revised for a few seconds, and identities are anchored to the slots of the head. The resulting system has no limit on the number of speakers and stays accurate when speakers outnumber the slots, where fixed-slot models break down. On meetings that grow to an hour it stays far more accurate than Streaming Sortformer, which accumulates errors, and its cost per update does not grow with the meeting. The same recipe turns the four-slot Streaming Sortformer into an open-speaker system. Across recordings, e-means makes it affordable to link the speakers of a large archive frame by frame, a task that arises when conversational corpora are prepared, and it reaches the clustering quality of Lloyd's -means several times faster. The method needs little tuning, and e-means runs with its default recipe from a few hundred to more than a million embeddings.
Figures & tables
| VoxConverse | AMI | ICSI | LibriSpeech | |||||
| System | DER | SCA | DER | SCA | DER | SCA | DER | SCA |
| Streaming Sortformer 4spk, 30.4 s config | 13.3 | 43.2 | 7.2 | 81.2 | 29.1 | 13.3 | 40.7 | 33.3 |
| Nemotron-3 (8 slots, 30.4 s) | 5.5 | 75.0 | 9.0 | 68.8 | 27.0 | 53.3 | 18.1 | 66.7 |
| pyannote 3.1 | 5.6 | 45.5 | 27.4 | 37.5 | 33.2 | 26.7 | 32.1 | 33.3 |
| Mini-batch -means (Sculley) | 7.6 | 31.8 | 13.5 | 12.5 | 36.1 | 0.0 | 14.7 | 72.2 |
| Lloyd -means++ (25 ep.) | 13.8 | 59.1 | 9.5 | 37.5 | 36.1 | 26.7 | 25.3 | 77.8 |
| VoxConverse | AMI | ICSI | LibriSpeech | |||||
| System | DER | SCA | DER | SCA | DER | SCA | DER | SCA |
| Streaming Sortformer 4spk (1.04 s) | 14.1 | 38.6 | 8.3 | 75.0 | 27.7 | 13.3 | 41.4 | 27.8 |
| Nemotron-3 head, 8 slots (1.04 s) | 6.4 | 75.0 | 9.3 | 75.0 | 27.4 | 46.7 | 19.8 | 66.7 |
| Ours, as first emitted (1.04 s) | 5.9 | 75.0 | 9.2 | 75.0 | 27.5 | 53.3 | 18.5 | 55.6 |
| Ours, after 10 s revision (1.04 s) | 5.8 | 75.0 | 9.1 | 75.0 | 27.4 | 53.3 | 16.4 | 61.1 |
| Nemotron-3 head, 8 slots (0.64 s) | 6.8 | 72.7 | 9.1 | 75.0 | 27.5 | 46.7 | 20.1 | 66.7 |
| Vox | AMI | ICSI | Libri | |
| Streaming Sortformer 4spk | -8.3 [-12.3, -4.5] ∗ | +0.8 [-0.9, +2.6] | -0.3 [-5.3, +4.9] | -25.0 [-32.4, -17.2] ∗ |
| Nemotron-3 head (8 slots) | -0.6 [-1.5, -0.1] ∗ | -0.1 [-0.4, +0.1] | -0.0 [-0.3, +0.3] | -3.3 [-5.8, -1.3] ∗ |
| Slot head and clustering | Vox | AMI | ICSI | Libri |
| Streaming Sortformer head (4 slots) | 15.8 | 10.6 | 30.8 | 41.4 |
| + our clustering | 6.6 | 12.1 | 30.8 | 16.2 |
| Nemotron-3 head (8 slots) | 6.4 | 9.7 | 26.5 | 18.1 |
| + our clustering | 4.6 | 9.0 | 26.5 | 9.9 |
| VoxConverse | AMI | ICSI | ||||
| Full-length, streaming (1.04 s) | DER | SCA | DER | SCA | DER | SCA |
| Streaming Sortformer 4spk (1.04 s) | 18.1 | 29.5 | 8.9 | 93.8 | 36.3 | 6.7 |
| Nemotron-3 head, 8 slots | 8.5 | 59.1 | 9.7 | 81.2 | 28.5 | 33.3 |
| Ours, as first emitted | 8.0 | 59.1 | 9.6 | 87.5 | 28.7 | 40.0 |
| Ours, after revision | 7.9 | 59.1 | 9.6 | 87.5 | 28.6 | 40.0 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Set | Source | Sessions | Speakers per session | Length per session |
| VoxConverse | YouTube clips, real | 44 | 1–17 (full: 1–21) | 480 s (full: up to 20 min) |
| AMI | design-scenario meetings, real | 16 | 3–4 | 420 s (full: 14–50 min) |
| ICSI | research meetings, real | 15 | 4–7 (full: 4–10) | 420 s (full: 44–86 min) |
| LibriSpeech-OpenSpk | read speech, constructed | 18 | 2–12 | 24–150 s |
| unseen sessions | read speech, constructed | 60 | 2–12 | about 90 s |
| many speakers | read speech, constructed | 20 | 14–20 | about 3.6 min |
| VoxConverse | AMI | ICSI | LibriSpeech | |||||
| System | DER | SCA | DER | SCA | DER | SCA | DER | SCA |
| Online Leader-Follower (non-streaming) | 6.2 | 54.5 | 9.9 | 12.5 | 30.2 | 0.0 | 10.4 | 72.2 |
| Ours (non-streaming) | 4.5 | 75.0 | 8.9 | 68.8 | 26.9 | 53.3 | 8.6 | 88.9 |
| Online Leader-Follower (1.04 s) | 5.9 | 50.0 | 10.7 | 0.0 | 31.6 | 6.7 | 11.6 | 61.1 |
| Ours, after 10 s revision (1.04 s) | 5.8 | 75.0 | 9.1 | 75.0 | 27.4 | 53.3 | 16.4 | 61.1 |
| Online Leader-Follower (0.64 s) | 6.6 | 38.6 | 10.8 | 0.0 | 31.5 | 6.7 | 12.1 | 61.1 |
| DER (%), held-out half | VoxConverse | AMI | ICSI | LibriSpeech |
| streaming, 1.04 s input buffer | ||||
| Streaming Sortformer 4spk | 14.1 | 9.0 | 23.7 | 42.5 |
| Nemotron-3 head, 8 slots | 5.6 | 10.6 | 21.7 | 21.9 |
| Online Leader-Follower | 5.2 | 12.1 | 23.7 | 11.8 |
| Ours, as first emitted | 5.3 | 10.4 | 21.9 | 19.6 |
| Ours, after revision | 5.2 | 10.4 | 21.7 | 17.6 |
| DER (%) on 60 unseen sessions | 4 | 6 | 8 | 10 | 12 | all | |
| Streaming Sortformer 4spk | 12.0 | 16.8 | 39.8 | 56.2 | 64.2 | 68.8 | 43.0 |
| Nemotron-3 head (8 slots) | 8.7 | 14.3 | 9.8 | 11.3 | 36.2 | 46.7 | 21.2 |
| Online Leader-Follower | 7.9 | 11.9 | 8.7 | 10.7 | 18.5 | 22.7 | 13.4 |
| Ours, as first emitted | 8.7 | 14.3 | 9.6 | 13.7 | 36.5 | 40.3 | 20.5 |
| Ours, after revision | 8.2 | 13.8 | 9.2 | 13.2 | 29.4 | 32.7 | 17.8 |
| Nemotron-3 head | Leader–follower | Ours | |||||
| Corpus | overlap (% of speech) | DER | miss | DER | miss | DER | miss |
| VoxConverse | 2.8 | 52.6 | 51.3 | 53.1 | 51.0 | 52.8 | 51.4 |
| AMI | 7.0 | 56.3 | 53.5 | 60.1 | 52.2 | 56.1 | 53.5 |
| ICSI | 18.9 | 60.7 | 58.1 | 62.2 | 57.6 | 60.5 | 58.1 |
| LibriSpeech-OpenSpk | 1.0 | 65.1 | 63.9 | 60.3 | 60.2 | 61.1 | 60.8 |
| Clustering engine inside the splitting step | Vox | Libri | AMI | ICSI | ms / call |
| e-means (official, 5 ep, 3 restarts) | 4.45 | 8.56 | 8.90 | 26.90 | 28.6 |
| e-means (1 restart) | 4.41 | 9.00 | 8.90 | 26.90 | 24.4 |
| Lloyd k-means (25 it, 3 restarts) | 4.43 | 8.92 | 8.90 | 26.90 | 7.2 |
| Lloyd k-means (5 it, 3 restarts) | 4.39 | 8.90 | 8.90 | 26.90 | 1.7 |
| mini-batch k-means (Sculley, 5 ep) | 4.37 | 8.82 | 8.90 | 26.90 | 13.9 |
| online-birth core (no k selection) | 4.43 | 9.71 | 8.90 | 26.90 | 0.0 |
| VoxConverse | AMI | ICSI | LibriSpeech | |||||
| System | MAE | cpWER | MAE | cpWER | MAE | cpWER | MAE | cpWER |
| Streaming Sortformer 4spk, 30.4 s config | 2.0 | – | 0.2 | 14.9 | 1.5 | – | 3.3 | 69.4 |
| Nemotron-3 (8 slots, 30.4 s) | 0.7 | – | 0.3 | 13.2 | 0.5 | – | 1.0 | 23.6 |
| pyannote 3.1 | 1.0 | – | 0.9 | – | 1.3 | – | 2.2 | – |
| Online Leader-Follower | 2.5 | – | 3.9 | 15.7 | 6.2 | – | 0.9 | 9.8 |
| Mini-batch -means (Sculley) | 1.7 | – | 4.9 | 25.2 | 6.2 | – | 3.1 | 19.4 |
Speaker Diarization
Speaker diarization identifies who spoke when in audio, including speaker changes, overlapping speech, and speaker identity across conversations.
See all 66 papers