Modeling Overlapped Speech with Shuffles

Matthew Wiesner; Samuele Cornell; Alexander Polok; Lucas Ondel Yang; Lukáš Burget; Sanjeev Khudanpur

cs.SDMar 18, 2026

Modeling Overlapped Speech with Shuffles

Authors: Matthew Wiesner, Samuele Cornell, Alexander Polok, Lucas Ondel Yang, Lukáš Burget, Sanjeev Khudanpur

Organizations: Johns Hopkins University, USA · LISN, CNRS, France · Carnegie Mellon University, USA · Brno University of Technology, Czechia

Abstract

We propose to model parallel streams of data, such as overlapped speech, using shuffles. Specifically, this paper shows how the shuffle product and partial order finite-state automata (FSAs) can be used for alignment and speaker-attributed transcription of overlapped speech. We train using the total score on these FSAs as a loss function, marginalizing over all possible serializations of overlapping sequences at subword, word, and phrase levels. To reduce graph size, we impose temporal constraints by constructing partial order FSAs. We address speaker attribution by modeling (token, speaker) tuples directly. Viterbi alignment through the shuffle product FSA directly enables one-pass alignment. We evaluate performance on synthetic LibriSpeech overlaps. To our knowledge, this is the first algorithm that enables single-pass alignment of multi-talker recordings. All algorithms are implemented using k2 / Icefall.

Figures & tables

Panel of Figure 1 — Figure 1: FSAs representing different utterance group serializations. Tuples on states represent indices into the two serialized sequences. Colors represent unique sources. (a) The shuffle product FSA between two sequences $y_{1}=abc,y_{2}=\color[rgb]{1,0,0}xy$ . (b) The pruned shuffle FSA obtainable from application of partial order constraints (see Section 3.2 ). (c) The subgraph corresponding to utterance-level serialized output training (SOT). (d) The subgraph corresponding to token-level SOT, assuming token order $a\leq\color[rgb]{1,0,0}x\color[rgb]{0,0,0}\leq b\leq\color[rgb]{1,0,0}y\color[rgb]{0,0,0}\leq c$ .

Panel of Figure 2 — Figure 2: The compact selfless label topology from [ 33 ] . Note that, different from the traditional CTC-topology, it forces tokens to appear on exactly one frame, and forces $\oslash$ between every token in the transcript, i.e., between “a” and a subsequent “b”.

Statistics of the Avg. Utterance Group by Datset
Dataset	# Segs	# Spks	Dur (s)	Spk / s	% Ovlp
2spk Ovlp	8	2	39.1	1.4	42.9
3spk Ovlp	8	3	35.4	1.7	55.0
Libri2Mix	2	2	8.4	1.6	64.2
Libri3Mix	3	3	9.0	2.2	72.2
Synth	2	2	9.2	1.5	52.7

Table 1: Statistics of evaluation datasets. Spk / s denotes the speaker density, or average number of active speakers per frame; overlap percentage denotes the fraction of time with more than one active speaker.

Token Order	Synth	LibriMix Test			LibriSpeech		Avg
Token Order	Synth	3 Clean	2 Clean	2 Both	Clean	Other	Avg
SOT	43.7	71.9	50.5	65.8	6.4	12.0	41.7
$\kappa$ = 0s	36.6	69.6	39.3	52.8	7.0	12.2	36.3
$\kappa$ = 1s	20.7	61.4	24.1	41.5	6.0	10.6	27.4
$\kappa$ = 2s	19.2	60.5	22.3	38.7	5.8	10.5	26.2
$\kappa$ = 4s	18.5	60.6	21.1	39.7	5.7	10.4	26.0

Table 2: Test set results with tcpWER collar = 5s for WavLM Base at 25 Hz using the factored speaker attribution model

p(\sigma_{t}\mid\mathbf{x},\bar{\oslash})\times p(\upsilon_{t}\mid\mathbf{x})

. This table compares serialization strategies (Token Order). All values are tcpWER (%).

System
WavLM	Loss	Spk Order	Synth	LibriMix Test			LibriSpeech Test		Average
WavLM	Loss	Spk Order	Synth	3 Clean	2 Clean	2 Both	Clean	Other	Average
Large	shuffle	start	12.7	60.9	17.6	30.2	5.4	7.9	22.5
	sd-ctc	start	13.3	58.5	16.8	32.3	5.2	8.2	22.4
	shuffle	length	11.8	44.3	8.1	25.3	5.3	8.2	17.2
	sd-ctc	length	11.1	43.8	7.8	28.2	5.3	8.2	17.4

Table 3: Test set results with tcpWER (collar = 5s). All models use the factored speaker attribution model

p(\sigma_{t}|\mathbf{x},\bar{\oslash})\times p(\upsilon_{t}|\mathbf{x})

and a 50 Hz frame rate. The table compares speaker ordering strategies and training losses. All values are WER (%).

Align Graph	Collar ( $\kappa$ )	Model	2spk overlap			3spk overlap
Align Graph	Collar ( $\kappa$ )	Model	IoU $\uparrow$	BE $\downarrow$	$\mathbf{\tau}$ % $\downarrow$	IoU $\uparrow$	BE $\downarrow$	$\mathbf{\tau}$ % $\downarrow$
$G\left(\mathcal{Y^{\prime}}\right)$	2s	CTC	56.2	89	13.8	45.0	138	41.2
$G\left(\mathcal{Y^{\prime}}\right)$	2s	Shuffle	61.8	69	6.3	58.2	81	16.9
	32s	Shuffle	61.7	70	6.2	-	-	-
	2s	SD-CTC	63.4	66	6.2	59.5	78	16.9
	32s	SD-CTC	63.2	66	6.2	-	-	-

Table 4: Alignment results on synthetic LibriSpeech mixtures. Performance in terms of the IoU, BE in milliseconds, and the Kendall-

\tau

distance, (

\tau

). Alignment error is measured against MFA word-level alignments on the corresponding single-speaker recordings.

Decoding	Synth	LibriMix Test			LibriSpeech		Avg
Decoding	Synth	3 Clean	2 Clean	2 Both	Clean	Other	Avg
Greedy	11.8	44.3	8.1	25.3	5.3	8.2	17.2
tgt spk HLG	9.6	42.5	6.2	24.2	3.6	6.1	14.8
+ oracle # spks	9.6	42.5	6.2	24.2	3.6	6.1	14.8
UME [ 53 ]	-	15.9	6.4	19.6	-	-	-
Yang MTASR [ 54 ]	-	-	5.1	-	-	-	-

Table 5: Test set word error rates (%). All results are tcpWER (collar=5s).

Speaker-Attributed ASR

See all 11 papers

Scaling Forced Alignment to End-User Devices
Sep 17, 2026Lawry Sorenson, Michael Crandall, Eric K. Ringger +1Forced Alignment Audio-Text Alignment
Brigham Young University Computer Science Department Provo, UT, United States
FuseAlign: Forced Alignment in the Wild
Sep 27, 2026Mithilesh Vaidya, Stephen Bailey, Sumukh Badam +2Forced Alignment Audio-Text Alignment
Descript, Inc.
G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition
Mar 11, 2026Jing Peng, Ziyi Chen, Haoyu Li +7Speaker-Attributed ASR Automatic Speech Recognition
Shanghai Jiao Tong University · Shenzhen Research Institute of Big Data · Nanjing University +2

Sep 17, 2026J/K move · Enter open · S save

Scaling Forced Alignment to End-User Devices

Lawry Sorenson, Michael Crandall, Eric K. Ringger, Stephen D. Richardson

Brigham Young University Computer Science Department Provo, UT, United States

Forced Alignment Audio-Text Alignment Speech Processing Dynamic Programming

Open paper AskPDF arXiv

The Viterbi algorithm has been previously used to perform forced alignment of audio to text to mine training data from online resources. However, many existing implementations have quadratic time and space complexity, scaling poorly to long input sequences. We propose two optimizations to address this issue. First, we apply the Hirschberg algorithm to perform the alignment in place using linear memory. Second, we model the alignment between speech and text as a constrained random walk, allowing us to prune the search space with arbitrary confidence while accounting for transcription errors. The Hirschberg optimization reduces memory usage from 140 GB to 5 MB for three-hour inputs while producing identical alignments in one-third the time of torchaudio when both run on a CPU. We achieve an additional 2x speedup with pruning on inputs longer than 20 minutes while preserving alignment accuracy in more than 98% of tested cases.

Abstract

Speaker-Attributed ASR

Explore similar work

Scaling Forced Alignment to End-User Devices

FuseAlign: Forced Alignment in the Wild

G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition