Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions
Authors: Alexander Polok, Dominik Klement, Ivan Medennikov, Antoine Laurent, Petr Pálka, Samuele Cornell, Shinji Watanabe, Sanjeev Khudanpur, Jagadeesh Balam, Jan Černocký, Lukáš Burget, Matthew Wiesner
Organizations: Brno University of Technology, Czech Republic · NVIDIA, USA · pyannoteAI, France · Carnegie Mellon University, USA · Johns Hopkins University, USA
Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently, operates on short (30 s) windows, and optimizes its diarization and TS-ASR components separately. We address the first two. We integrate the diarization conditioning into a longform Parakeet backbone (DiCoP) and add cross-speaker attention, which lets the model attend across the speaker dimension so that speakers are no longer decoded in isolation, and an STNO mask refiner that re-estimates the diarization-derived conditioning. As conversational data with genuine multi-speaker interaction is scarce, we scale training with a pseudo-labeling pipeline that mines 5.4k hours of in-the-wild multi-speaker audio from YODAS2. Combining these with synthetic and real data yields state-of-the-art results across several meeting benchmarks. We open-source the resulting models, DiCoP and Pandemonium, and the pseudo-labeled data.
Figures & tables
Figure 1: Overview of Pandemonium . A diarizer yields per-speaker masks that condition (FDDT) a longform Parakeet encoder processing the S speakers as a batch. Cross-speaker attention attends across speakers and a mask refiner sharpens the ambiguous STNO conditioning (blue boxes); a shared CTC head decodes each.
In-domain
Out-of-domain
System
NSF
AMI
ICSI
Mx6
SBC
DiPCo
Avg
Oracle diarization
DiCoW v3.3 [ 28 ]
15.5
14.5
14.1
11.4
15.0
27.1
16.3
SE-DiCoW [ 28 ]
15.3
14.4
14.3
11.7
15.2
26.2
16.2
DiCoP (no CSA)
17.3
14.5
14.4
11.9
16.0
26.1
16.7
Pandemonium
14.1
13.4
13.3
11.1
13.8
22.4
14.7
Table 1: Main results, cpWER [%] ↓ . DiCoP: longform Parakeet backbone (loc/TDT of Fig. 2 ); Pandemonium adds CSA and the STNO refiner, trained synthetic → pseudo → real. + MOVER: all distant microphones decoded separately, merged with MOVER [ 12 ] (SBC: stereo, unchanged). SOTA: best published single-microphone result (Sep 2026). Best single-microphone result per block in bold ; abbreviations in Section 3.1 .
Figure 2: Accuracy/speed trade-off per evaluation set (oracle diarization) against real-time-factor speedup (RTFx, greedy decoding). Line color: evaluation set; marker: model, connected in order of speed; all decode full conversations. All systems but Pandemonium train on the DiCoW-orig. data ( Section 3.1 ); DiCoP variants differ in attention ( loc / glob : local/global) and training loss, without CSA.
System / selection
Hrs
NSF
AMI
A4
Ali
Mx6
SBC
Avg
DiCoW v3.3 (orig.) [ 28 ]
1.2k
15.5
14.5
33.7
39.3
11.4
15.0
21.6
SE-DiCoW (orig.) [ 28 ]
1.2k
15.3
14.4
40.2
43.4
11.7
15.2
23.4
DiCoW (synthetic) [ 30 ]
4.0k
23.8
24.7
26.8
35.9
16.8
22.1
25.0
DiCoW — pseudo-labels only (no real data):
All pseudo-labels
3.4k
17.2
19.1
24.3
31.3
11.7
15.7
19.9
LID check (only)
3.1k
17.3
19.4
24.5
31.4
11.8
16.3
20.1
Table 2: Influence of pseudo-label quality on meeting benchmarks. Rows: DiCoW with each cleaning filter applied to the full pool; orig.: trained on the DiCoW-orig. data. cpWER [%] ↓ . Best per column in bold . Underlined : in-domain, i.e., the matched training set was used.
In-domain
Out-of-domain
System
NSF
AMI
Mx6
SBC
Avg
Pandemonium (full)
14.1
13.4
11.1
13.8
13.1
Training-data source:
w/o real (synth+pseudo)
14.4
18.1
17.7
14.0
16.1
pseudo-labels only
20.2
26.6
16.8
16.9
20.1
synthetic only
16.7
17.6
10.9
15.0
15.0
Table 3: Ablation of Pandemonium . Top: training data. Bottom: architecture, removing the STNO refiner, then also CSA (plain DiCoP), and the full model trained on 30 s windows. cpWER [%] ↓ .
Multi-speaker automatic speech recognition (MSASR) aims to jointly predict content transcriptions, speaker identities, and timestamps, thereby addressing the key question of "who spoke what and when" and holds substantial practical value in real-world multi-speaker scenarios. However, MSASR still encounters considerable challenges in the presence of fast turn transitions, overlapping speech, and complex, diverse multi-speaker scenarios. In this work, we propose DiaScriber, an end-to-end multi-speaker diarization and transcription model built on a speech large language model. We first construct diverse data pipelines to cover a wide variety of multi-speaker scenarios and their complexities, including validation and refinement, turn-transition and overlapping-speech simulation, and multimodal annotation. Furthermore, DiaScriber is developed based on the pretrained version of Qwen3.5-Omni through a three-stage training strategy involving continual pretraining, supervised fine-tuning, and reinforcement learning. Experiments show that DiaScriber achieves superior performance over comparison methods across extensive multi-speaker scenario test sets and demonstrates outstanding generalization ability in unseen multi-speaker scenarios.
We present TagSpeech, a unified LLM-based framework that utilizes Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The framework is built on two key designs: (1) decoupled semantic and speaker streams fine-tuned via Serialized Output Training (SOT) to learn turn-taking dynamics; and (2) an interleaved time anchor mechanism that not only supports fine-grained timestamp prediction but also acts as a synchronization signal between semantic understanding and speaker tracking. Compared to previous works that primarily focus on speaker-attributed ASR or implicit diarization, TagSpeech addresses the challenge of fine-grained speaker-content alignment and explicitly models "who spoke what and when" in an end-to-end manner. Experiments on AMI and AliMeeting benchmarks demonstrate that our method achieves consistent improvements in Diarization Error Rate (DER) over strong end-to-end baselines, including Qwen-Omni and Gemini, particularly in handling complex speech overlaps. Moreover, TagSpeech employs a parameter-efficient training paradigm in which the LLM backbone is frozen and only lightweight projectors are trained, resulting in strong performance with low computational cost.
Mingyue Huo, Yiwen Shao, Yuheng Zhang
University of Illinois Urbana-Champaign · Johns Hopkins University
In long-form multi-party conversations, highly imbalanced speaker activity and frequent overlap make it difficult to identify "who spoke when and what". Sliding-window continuous speech separation (CSS) mitigates sparse supervision, but often suffers from cross-window speaker inconsistency and residual crosstalk, which in practice requires diarization for reliable speaker attribution. Motivated by the stability of speakers' directions of arrival (DOAs) in meetings, we propose PATSE, a multi-channel Position-Aware Target Speaker Extraction front-end that uses DOA as a spatial prior to directly extract the speech of each target speaker. PATSE combines a DOA-guided spatial encoder and conditioner to generate speaker-attributed streams, from which speaker activity can be inferred via simple post-processing (e.g., VAD) without explicit diarization. Experiments on both replayed and real conversations show consistent ASR gains outperforming CSS and diarization-based pipelines.
Yichi Wang, Junzhe Chen, Wangjin Zhou +1
Graduate School of Informatics, Kyoto University, Kyoto, Japan