A Systematic Comparison of Search Space Complexity and Efficiency for Time-Synchronous End-to-End ASR Models

Simon Berger; Larissa Kleppel; Ralf Schlüter

cs.CLOct 22, 2026

A Systematic Comparison of Search Space Complexity and Efficiency for Time-Synchronous End-to-End ASR Models

Authors: Simon Berger, Larissa Kleppel, Ralf Schlüter

Organizations: Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Germany · AppTek GmbH, Germany

Abstract

Search is a central component of Automatic Speech Recognition (ASR) systems and has been deeply investigated for classical ASR modeling based on Hidden Markov Models (HMMs). However, in work on current sequence-to-sequence ASR architectures it is often simplified to a small fixed-size beam search without external Language Model (LM). In this work, we present a comparative analysis of search space complexity and efficiency for Connectionist Temporal Classification (CTC) and neural transducer models under different pruning strategies and LM configurations. We show that these end-to-end (E2E) models work with a small search space most of the time but exhibit large temporary peaks in hypothesis count during periods of uncertainty. Score-based pruning dynamically adapts the search to model uncertainty, leading to a significantly lower Real-Time Factor (RTF). Our results highlight the limitations of search with low fixed-size beam pruning and provide practical guidelines for efficient search in E2E ASR.

Figures & tables

Model	Lex.	LM		WER [%]		RTF $\cdot 10^{3}$
Model	Lex.	Subword	Word	dev-o	test-o	RTF $\cdot 10^{3}$
BPE CTC	No	-	-	7.0	7.1	0.62
		LSTM	-	5.1	5.1	22
		TraFo		4.6	4.7	3357
	Yes	-	-	6.5	6.6	0.93
		-	4-gram	5.2	5.6	1.6

TABLE I: WER and RTF results on LibriSpeech for different models given various search conditions and LM combinations.

Model	Lex.	Word LM	WER [%]		RTF $\cdot 10^{3}$
Model	Lex.	Word LM	dev.all	test.all	RTF $\cdot 10^{3}$
BPE CTC	No	-	11.7	12.8	0.62
	Yes	-	11.0	12.0	0.73
	Yes	4-gram	9.5	10.4	0.89
BPE Transd.	No	-	9.0	10.2	7.7
	Yes	-	9.0	10.1	8.6

TABLE II: WER and RTF results on Loquacious for different models given various search conditions and LM combinations.

Score Thresh.	Beam Size	WER [%]	Search Err. [%]	#Hypotheses
Score Thresh.	Beam Size	WER [%]	Search Err. [%]	p50	p90	p99
0.0	1	10.4	30.4	1	1	1
4.0	16	9.4	14.7	1	2	7
8.0	128	6.7	4.9	2	6	41
12.0	256	4.8	1.3	5	28	242
16.0	256	4.6	0.6	35	167	254

TABLE III: WER and hypothesis count statistics for different pruning parameters with BPE CTC lexicon-constrained search + word-level Transformer LM on LibriSpeech dev-other.

Score Thresh.	Beam Size	WER [%]	Search Err. [%]	#Hypotheses
Score Thresh.	Beam Size	WER [%]	Search Err. [%]	p50	p90	p99
0.0	1	10.0	24.1	1	1	1
4.0	16	11.0	20.3	1	2	6
8.0	128	6.9	7.2	2	7	51
12.0	256	4.2	1.7	7	46	250
16.0	1024	3.8	0.4	55	433	1012

TABLE IV: WER and hypothesis count statistics for different pruning parameters with BPE transducer lexicon-constrained search + word-level Transformer LM on LibriSpeech dev-other.

Score Thresh.	Beam Size	WER [%]	Search Err. [%]	#Hypotheses
Score Thresh.	Beam Size	WER [%]	Search Err. [%]	p50	p90	p99
0.0	1	11.9	16.8	1	1	1
2.0	16	10.9	10.9	1	2	4
4.0	32	8.6	2.6	1	3	11
6.0	32	8.0	0.7	1	5	30
8.0	32	7.9	0.5	2	11	31

TABLE V: WER and hypothesis count statistics for different pruning parameters with BPE transducer lexicon-constrained search + word-level 4-gram LM on Loquacious dev.all.

Score Thresh.	Beam Size	WER [%]	#Hypotheses
Score Thresh.	Beam Size	WER [%]	Avg.	p50	p90	p99
0.0	1	46.6	1.0	1	1	1
4.0	128	17.9	6.1	3	11	79
6.0	512	15.5	88.3	55	180	512
8.0	1024	15.0	251.0	148	675	1024
10.0	4096	14.9	932.2	451	2926	4096

TABLE VI: WER and hypothesis count statistics for different pruning parameters with phoneme CTC lexicon-constrained search + word-level 4-gram LM on Spanish telephony.

Fig. 1: Illustrative number of active within-word hypotheses at each search step for a selected LibriSpeech dev-other utterance with BPE CTC /transducer using lexicon-constrained search including a word-level 4-gram LM .

Fig. 2: Search RTF vs. WER for fixed and dynamic pruning on LibriSpeech dev-other with BPE transducer + subword-level LSTM LM .

Fig. 3: Search RTF vs. WER for fixed and dynamic pruning on LibriSpeech dev-other with BPE transducer + word-level Transformer LM .

Fig. 4: Search RTF vs. WER for fixed and dynamic pruning on Spanish telephony with phoneme CTC + word-level 4-gram LM .

$B_{\mathrm{ASR}}$	WER [%]	Search RTF $\cdot 10^{3}$		RTF reduction [%]
$B_{\mathrm{ASR}}$	WER [%]	Fixed	Dynamic	RTF reduction [%]
4	5.3	19	18	5.3
16	4.7	27	21	22.2
64	4.6	49	24	51.0
128	4.5	74	25	66.2

TABLE VII: WER and Search RTF breakdown for fixed and dynamic pruning with different max beam sizes

B_{\mathrm{ASR}}

on LibriSpeech dev-other with BPE Transducer + subword-level LSTM LM.

$B_{\mathrm{ASR}}$	WER [%]	Search RTF $\cdot 10^{3}$		RTF reduction [%]
$B_{\mathrm{ASR}}$	WER [%]	Fixed	Dynamic	RTF reduction [%]
4	10.4	72	60	16.7
16	5.9	261	107	59.0
64	4.5	955	186	80.5
256	4.0	3063	276	91.0
1024	3.8	5781	310	94.6

TABLE VIII: WER and Search RTF breakdown for fixed and dynamic pruning with max beam sizes

B_{\mathrm{ASR}}

on LibriSpeech dev-other with BPE Transducer + word-level Transformer LM.

Corpus	Model	LM	WER [%]	Fixed Pruning		Dynamic Pruning		RTF
Corpus	Model	LM	WER [%]	Avg. Hyps	RTF $\cdot 10^{3}$	Avg. Hyps	RTF $\cdot 10^{3}$	red. [%]
LibriSpeech	BPE CTC	-	7.0	1	0.62	1.0	0.62	0.0
		Word 4-gram	5.2	256	326	13.5	1.6	99.5
		Subword LSTM	5.1	128	140	27.7	22	84.3
		Word Trafo	4.6	256	1203	61.2	154	87.2
	Phoneme CTC	Word 4-gram	5.5	256	128	13.2	1.5	98.8

TABLE IX: RTF reduction through dynamic pruning for different setups. Measured on dev-other for LibriSpeech, dev.all for Loquacious and an in-house test-set for Spanish. Avg. Hyps counts active subword-level hypotheses after pruning.

Corpus	Model	LM	LM Scale	ILM / Prior Scale	$B_{\text{ASR}}$	$B_{\text{LM}}$	$\Delta_{\text{ASR}}$	$\Delta_{\text{LM}}$
LibriSpeech	BPE CTC	-	-	-	1	-	0.0	-
		Word 4-gram	0.6	0.2	256	16	12.0	6.0
		Subword LSTM	0.8	0.2	1024	128	14.0	12.0
		Word Trafo	0.8	0.2	256	16	16.0	8.0
	Phoneme CTC	Word 4-gram	0.6	0.2	256	8	12.0	6.0
	Phoneme CTC	Word Trafo	0.8	0.2	256	16	16.0	8.0

TABLE X: Tuned decoding parameters for dynamic pruning setups.

Inference-Time Search

Inference-time search uses extra computation at deployment to explore, score, and select candidate reasoning paths or actions without retraining a model.

See all 105 papers

Design of the IBM Granite 5.0 TurboCTC ASR Model
Sep 17, 2026Brian Kingsbury, George Saon, Masayuki Suzuki +5Transformer Connectionist Temporal Classification
IBM Research
Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR
Jul 6, 2026Ho Lam Chung, Yiming Chen, Dau-Cheng Lyu +2Test-Time Scaling Automatic Speech Recognition
National Taiwan University, Taiwan · ASUS, Taiwan · National University of Singapore, Singapore
Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASR
Nov 11, 2025Julian Irigoyen, Arthur Söhler, Andreas Søeborg KirkedalAutomatic Speech Recognition Implicit Regularization
Copenhagen Business School, Copenhagen, Denmark · Jabra (GN Group), Copenhagen, Denmark

Sep 17, 2026J/K move · Enter open · S save

Design of the IBM Granite 5.0 TurboCTC ASR Model

Brian Kingsbury, George Saon, Masayuki Suzuki, Hong-Kwang J. Kuo+4

IBM Research

Transformer Connectionist Temporal Classification Automatic Speech Recognition Conformer

Open paper AskPDF arXiv

We describe the architecture, training methodology and inference speedups of Granite 5.0 Turbo CTC, a 470 million parameter encoder-only model with an excellent speed-accuracy tradeoff. The architecture uses pyramidal temporal subsampling within Conformer blocks using strided depthwise convolutions, block-diagonal (chunk-wise) self-attention, and conditioning on intermediate predictions from the middle layer. Training highlights are the use of only publicly available data, the novel use of a Muon optimizer, and balanced data sampling. Inference speedups include replacing 1 x 1 convolutions with linear layers and optimizing the attention computation in the Conformer blocks. Collectively, these result in a model that is on the speed-accuracy Pareto frontier of the Open ASR leaderboard for English short-form ASR while being twice as fast as the fastest competitor. The model can be used under a permissive license and downloaded from https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc.

Abstract

Inference-Time Search

Explore similar work

Design of the IBM Granite 5.0 TurboCTC ASR Model

Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR

Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASR