Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions
Authors: Alexander Polok, Dominik Klement, Ivan Medennikov, Antoine Laurent, Petr Pálka, Samuele Cornell, Shinji Watanabe, Sanjeev Khudanpur, Jagadeesh Balam, Jan Černocký, Lukáš Burget, Matthew Wiesner
Organizations: Brno University of Technology, Czech Republic · NVIDIA, USA · pyannoteAI, France · Carnegie Mellon University, USA · Johns Hopkins University, USA
Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently, operates on short (30 s) windows, and optimizes its diarization and TS-ASR components separately. We address the first two. We integrate the diarization conditioning into a longform Parakeet backbone (DiCoP) and add cross-speaker attention, which lets the model attend across the speaker dimension so that speakers are no longer decoded in isolation, and an STNO mask refiner that re-estimates the diarization-derived conditioning. As conversational data with genuine multi-speaker interaction is scarce, we scale training with a pseudo-labeling pipeline that mines 5.4k hours of in-the-wild multi-speaker audio from YODAS2. Combining these with synthetic and real data yields state-of-the-art results across several meeting benchmarks. We open-source the resulting models, DiCoP and Pandemonium, and the pseudo-labeled data.
Figures & tables
Figure 1: Overview of Pandemonium . A diarizer yields per-speaker masks that condition (FDDT) a longform Parakeet encoder processing the S speakers as a batch. Cross-speaker attention attends across speakers and a mask refiner sharpens the ambiguous STNO conditioning (blue boxes); a shared CTC head decodes each.
In-domain
Out-of-domain
System
NSF
AMI
ICSI
Mx6
SBC
DiPCo
Avg
Oracle diarization
DiCoW v3.3 [ 28 ]
15.5
14.5
14.1
11.4
15.0
27.1
16.3
SE-DiCoW [ 28 ]
15.3
14.4
14.3
11.7
15.2
26.2
16.2
DiCoP (no CSA)
17.3
14.5
14.4
11.9
16.0
26.1
16.7
Pandemonium
14.1
13.4
13.3
11.1
13.8
22.4
14.7
Table 1: Main results, cpWER [%] ↓ . DiCoP: longform Parakeet backbone (loc/TDT of Fig. 2 ); Pandemonium adds CSA and the STNO refiner, trained synthetic → pseudo → real. + MOVER: all distant microphones decoded separately, merged with MOVER [ 12 ] (SBC: stereo, unchanged). SOTA: best published single-microphone result (Sep 2026). Best single-microphone result per block in bold ; abbreviations in Section 3.1 .
Figure 2: Accuracy/speed trade-off per evaluation set (oracle diarization) against real-time-factor speedup (RTFx, greedy decoding). Line color: evaluation set; marker: model, connected in order of speed; all decode full conversations. All systems but Pandemonium train on the DiCoW-orig. data ( Section 3.1 ); DiCoP variants differ in attention ( loc / glob : local/global) and training loss, without CSA.
System / selection
Hrs
NSF
AMI
A4
Ali
Mx6
SBC
Avg
DiCoW v3.3 (orig.) [ 28 ]
1.2k
15.5
14.5
33.7
39.3
11.4
15.0
21.6
SE-DiCoW (orig.) [ 28 ]
1.2k
15.3
14.4
40.2
43.4
11.7
15.2
23.4
DiCoW (synthetic) [ 30 ]
4.0k
23.8
24.7
26.8
35.9
16.8
22.1
25.0
DiCoW — pseudo-labels only (no real data):
All pseudo-labels
3.4k
17.2
19.1
24.3
31.3
11.7
15.7
19.9
LID check (only)
3.1k
17.3
19.4
24.5
31.4
11.8
16.3
20.1
Table 2: Influence of pseudo-label quality on meeting benchmarks. Rows: DiCoW with each cleaning filter applied to the full pool; orig.: trained on the DiCoW-orig. data. cpWER [%] ↓ . Best per column in bold . Underlined : in-domain, i.e., the matched training set was used.
In-domain
Out-of-domain
System
NSF
AMI
Mx6
SBC
Avg
Pandemonium (full)
14.1
13.4
11.1
13.8
13.1
Training-data source:
w/o real (synth+pseudo)
14.4
18.1
17.7
14.0
16.1
pseudo-labels only
20.2
26.6
16.8
16.9
20.1
synthetic only
16.7
17.6
10.9
15.0
15.0
Table 3: Ablation of Pandemonium . Top: training data. Bottom: architecture, removing the STNO refiner, then also CSA (plain DiCoP), and the full model trained on 30 s windows. cpWER [%] ↓ .