Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics

Naohiro Tawara; Samuele Cornell; Alexander Polok; Marc Delcroix; Lukáš Burget; Shinji Watanabe

cs.CLMar 24, 2026

Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics

Authors: Naohiro Tawara, Samuele Cornell, Alexander Polok, Marc Delcroix, Lukáš Burget, Shinji Watanabe

Organizations: NTT, Inc., Japan · CMU, USA · BUT, Czechia

Abstract

Conversational automatic speech recognition remains challenging due to overlapping speech, far-field noise, and varying speaker counts. While recent LLM-based systems perform well on single-speaker benchmarks, their robustness in multi-speaker settings is unclear. We systematically compare LLM-based and modular pipeline approaches along four axes: overlap robustness, semantic fidelity, speaker count, and single- versus multi-channel input. To capture meaning-altering errors that conventional metrics miss, we introduce tcpSemER, which extends tcpWER by replacing Levenshtein distance with embedding-based semantic similarity. We further decompose tcpWER into overlapping and non-overlapping components for finer-grained analysis. Experiments across three datasets show that LLM-based systems are competitive in two-speaker settings but degrade as speaker count and overlap increase, whereas modular pipelines remain more robust.

Figures & tables

Figure 1: Relative change in tcpWER and tcpSemER when switching from CHiME-8 to CHiME-7 text normalization, averaged across the top-4 CHiME-8 DASR challenge systems per dataset. Error bars denote standard deviation.

Category	Ref	Hyp	WER	SemER	$\Delta$
Negation	it is lovely	it is not	0.33	0.67	-0.34
Entity	yeah that is a mat	yeah that is a lot	0.20	0.83	-0.63
Filler	that is a great idea chris	that is a great idea chris yeah yeah yeah	0.50	0.05	0.45
Paraphrase	or maybe like a slogan	that could be like a slogan	0.60	0.12	0.48

Table 1: Representative utterance-level divergences between WER and SemER from top-3 NSF1 challenge systems.

\Delta=\mathrm{WER}-\mathrm{SemER}

;

\Delta>0

: WER penalizes more;

\Delta<0

: SemER penalizes more. Red : substitutions; blue : insertions.

				WER		SemER	DER
	System	LLM	Ch.	cp	tcp	tcp
MX6 (2 spk, 14% ovl, distributed, $\sim$ 15 min)
1	DiCoW	–	1ch	$14.48$	$14.71$	$7.84$	$21.05$
2	VibeVoice	✓	1ch	$16.03$	$16.32$	$7.51$	$9.89$
3	Voxtral MTv2	✓	1ch	$19.37$	$19.59$	$10.46$	$21.97$
4	Gemini 3.0	✓	1ch	$58.34$	$115.54$	$80.79$	$78.49$

Table 2: Results on all datasets. Collars are 5 s (tcpWER) and 0.25 s (DER). All values are in %. For each dataset, we report the number of speakers, the overlap ratio, the microphone setup, and the average meeting duration.

			tcpWER (%)
	System	Ch.	all	ov	1spk	ov norm	1spk norm
MX6 ( $N_{\mathrm{ref}}^{\mathrm{ov}}=37628$ , $N_{\mathrm{ref}}^{\mathrm{1spk}}=32990$ )
1	DiCoW	1ch	$14.71$	$8.50$	$6.21$	$15.96$	$13.29$
2	VibeVoice	1ch	$16.32$	$10.73$	$5.59$	$20.13$	$11.97$
3	CH8 DASR NTT (S)	mch	$15.32$	$7.56$	$7.76$	$14.19$	$16.61$
4	VibeVoice + MOVER	mch	$10.79$	$7.05$	$3.74$	$13.22$	$8.01$

Table 3: Overlap-aware tcpWER decomposition (%).

\mathrm{ov}

\mathrm{1spk}

: contribution to total error;

\mathrm{ov}^{\rm norm}

\mathrm{1spk}^{\rm norm}

: normalized by region-specific word count.

Figure 2: Decomposition of tcpWER into deletion, insertion, and substitution errors for VibeVoice (Vib), DiCoW (DiC), and CH8 DASR NTT (NTT) on NSF1, shown by speaker count. The proportion of overlapped speech (ovl) is shown in parentheses.

Conversational Speech Recognition

See all 9 papers

Soft Posterior Speaker Injection for Multi-Talker Speech Recognition
Sep 1, 2026Jian Zhu, Jun Sun, Jiang Yang +6Multi-Speaker ASR Automatic Speech Recognition
Zhejiang Lab, Hangzhou, China · Zhejiang International Studies University, Hangzhou, China · University of Science and Technology of China, Hefei, China
Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation
Mar 11, 2026Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez +2Speaker Embedding Speaker Verification
Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA
Are you speaking my languages? On spoken language adherence in multimodal LLMs
Jun 15, 2026Hyungwon Kim, Kandarp Joshi, Lillian Zhou +2Audio-Language Model Evaluation LLM Prompting
Google DeepMind

Sep 1, 2026J/K move · Enter open · S save

Soft Posterior Speaker Injection for Multi-Talker Speech Recognition

Jian Zhu, Jun Sun, Jiang Yang, Ying Zhou+5

Zhejiang Lab, Hangzhou, China · Zhejiang International Studies University, Hangzhou, China · University of Science and Technology of China, Hefei, China

Multi-Speaker ASR Automatic Speech Recognition

Open paper AskPDF arXiv

Multi-talker automatic speech recognition (MT-ASR) remains challenging in the presence of overlapping speech. Hard segmentation introduces irreversible errors, whereas serialized output training (SOT) avoids explicit segmentation but does not condition a pretrained encoder on speaker activity. We propose Soft Posterior Speaker Injection (SPSI). A Soft Posterior Head predicts per-frame speaker posteriors

\hat{\mathbf{P}}

and injects them into Whisper through Multi-layer Feature-wise Linear Modulation (MFLM) and Speaker Memory Prompts (SMP). The benefit of SPSI is largest where overlap is heaviest and under domain transfer. On controlled two-speaker LibriSpeech overlap, SPSI reduces concatenated minimum-permutation word error rate (cpWER) from

61.5\%

60.0\%

in the high-overlap bin, and from

51.9\%

51.0\%

on the full set, relative to SOT. By contrast, Speaker CE, SD-CTC, SA-DiCoW, and Pipeline (oracle/est.\ VAD) do not outperform SOT. Freeze-posterior overlap-heavy adaptation reduces held-out LibriCSS cpWER from

42.3\%

36.8\%

on sessions

8

9

, a

5.5

-point gain over SOT. The source code is available at https://github.com/HackerHyper/SPSI.

Abstract

Conversational Speech Recognition

Explore similar work

Soft Posterior Speaker Injection for Multi-Talker Speech Recognition

Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

Are you speaking my languages? On spoken language adherence in multimodal LLMs