Depth through Recurrence: Towards Ultra-Efficient On-Device ASR

Chen Feng; Tianyi Xu; Ramchalam Kinattinkara Ramakrishnan; Yicheng Lin; Shaojie Zhuo; Zhaocong Yuan; Chenzheng Su; Xiaopeng Zhang

cs.LGOct 27, 2026

Depth through Recurrence: Towards Ultra-Efficient On-Device ASR

Authors: Chen Feng, Tianyi Xu, Ramchalam Kinattinkara Ramakrishnan, Yicheng Lin, Shaojie Zhuo, Zhaocong Yuan, Chenzheng Su, Xiaopeng Zhang

Organizations: Qualcomm AI Research † · McGill University

Abstract

Modern automatic speech recognition (ASR) systems have achieved remarkable accuracy by scaling model depth and capacity, but at the cost of increased memory, energy, and computational demands. On edge devices where ASR is often most needed, such as watches and glasses, deploying such large models is infeasible due to extremely constrained resources. This raises a fundamental question: can we achieve high representational power in deep ASR models without scaling up parameterization? In this work, we revisit the role of depth and identify layer wise representational dynamics, in which most layers learn functionally similar transformations. Motivated by this insight, we propose a block-recurrent ASR architecture that replaces parameterized depth with a small set of recurrent blocks, each consisting of weight-shared layers, thereby preserving effective depth while drastically reducing model size. Through representation-guided grouping and knowledge distillation, block-recurrent models retain the accuracy of large reference models while using an order of magnitude fewer parameters. Across Open ASR Leaderboard benchmarks, a model with only two shared blocks recovers 97.4% of the accuracy of large models on average. The compact backbone further enables efficient on-device personalization through lightweight MoE-LoRA adaptation, allowing user-specific models to match or surpass the accuracy of general-purpose ASR systems with significantly lower inference cost. By reusing weights across recurrent steps, the proposed architecture reduces memory-access overhead and inference energy consumption. Together, these results show that high quality ASR can be achieved without large parameterization, and that structured recurrence provides a principled path towards ultra-compressed, energy-efficient, and personalized speech recognition on the edge.

Figures & tables

Figure 1: Overview of the proposed block-recurrent ASR framework. A large-scale ASR composed of $L$ encoder blocks is analyzed using CKA to identify representational similar layers. Guided by this analysis, an edge-scale ASR model is constructed with block-recurrence, and trained through knowledge distillation. The resulting compact backbone further enables efficient on-device personalization through MoE-LoRA adapters conditioned on user embeddings.

Figure 2: Layer-wise CKA similarity for STT-480ms, Parakeet-TDT-0.6B-v2, Canary-1B-Flash, and Granite-4.0-1B-Speech encoder models.

Figure 3: Representational grouping vs. random grouping (five runs with error bars).

	Configuration		Individual Datasets WER % $\downarrow$								Overall WER % $\downarrow$
Model	K	Params. (M)	AMI	Earn	Giga	LS-c	LS-o	SPGI	TED	VoxP	Avg.	$\Delta$
STT-480ms	–	108.9	26.23	20.56	14.15	2.54	5.97	5.80	5.58	7.60	11.05	–
Edge-STT	2	14.2	22.43	18.76	18.42	5.98	14.29	6.36	8.83	12.10	13.40	+2.35
	4	26.8	18.16	15.24	15.60	4.29	10.55	4.72	7.34	9.89	10.72	-0.33
	6	39.5	16.72	13.96	14.34	3.65	9.10	4.19	6.44	8.79	9.65	-1.40
	8	52.1	14.93	13.17	13.36	3.25	8.08	3.76	6.10	8.38	8.88	-2.17

Table 1: Open ASR Leaderboard results for block-recurrent student models distilled from a single large-scale teacher. The teacher performance is reported in the first row in each section, followed by student models with varying numbers of shared blocks

K

. Encoder parameter counts are reported alongside the corresponding teacher encoder size to highlight the degree of parameter reduction.

Table 2: Performance comparison of teacher models (first row), block-recurrent, and personalized models with varying MoE-LoRA ranks on the VoxCeleb2, averaged over

106

test speakers. Zero-shot denotes ASR without MoE-LoRA personalization.

Figure 4: Per-speaker WER comparison for 106 test speakers, among the STT-480ms reference model, the block-recurrent baseline ( $K=8$ ), and the personalized MoE-LoRA model (rank 32). Each grouped bar represent an individual speaker. Dashed horizontal lines, shown in the corresponding colors, indicate the overall pooled WER for each model.

		GPU (Blackwell RTX Pro 6000)		CPU (fp32)
Model	$K$	Scalar (fp16)	Dot (fp16)	Threadripper 7995WX	Apple M4 Pro
STT-480ms	–	$6.22\pm.01\ (1.00\times)$	$3.88\ (1.00\times)$	$55.7\pm.43\ (1.00\times)$	$43.44\pm.34\ (1.00\times)$
Edge-STT	1	$5.13\pm.02\ (\mathbf{1.21\times})$	$3.67\ (\mathbf{1.06\times})$	$52.0\pm.44\ (\mathbf{1.07\times})$	$42.26\pm.20\ (\mathbf{1.03\times})$
	2	$5.21\pm.02\ (1.19\times)$	$3.68\ (\mathbf{1.06\times})$	$52.0\pm.39\ (\mathbf{1.07\times})$	$42.35\pm.25\ (\mathbf{1.03\times})$
	4	$5.27\pm.01\ (1.18\times)$	$3.70\ (1.05\times)$	$52.8\pm.35\ (1.05\times)$	$42.52\pm.15\ (1.02\times)$
	6	$5.59\pm.01\ (1.11\times)$	$3.73\ (1.04\times)$	$52.9\pm.33\ (1.05\times)$	$42.75\pm.33\ (1.02\times)$

Table 3: Inference latency (ms) and speedup for block-recurrent STT model under various block counts

K

. GPU inference uses custom Triton kernels in two variants: a scalar kernel that computes per-token outer products and operates primarily in a memory-bound regime, and a dot kernel based on tl.dot , which uses tensor-core-accelerated tiled matmul.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Encoder	Encoder Params.	Decoder	Decoder Params.	Enc/Dec Param Ratio
STT-480ms	17 FastConformer blocks	108.9M	RNNT/CTC head	5.9M	18.5x
Parakeet-v2	24 Conformer blocks	608.9M	TDT head	8.9M	68.4x
Canary-1B-Flash	32 FastConformer blocks	883.6M	4 Transformer blocks	72.6M	12.1x
Granite-4.0-1B-Speech	16 Conformer blocks	440.2M	LLM decoder	1.8B	0.24x

Appendix

Table 4: ASR models: Enc/Dec breakdown

Model	STT-480ms	Parakeet-TDT-0.6B-v2
encoder layers	17	24
hidden dim.	512	1024
encoder type	FastConformer with casual chunked attention	Conformer
att_context_size	[70, 6]	–
decoder type	hybrid CTC/RNN-T	TDT
tokenizer	BPE	BPE

Appendix

Table 5: Large-scale teacher models

Table 6: Scaling block-recurrent student models

Model	Block count K	Grouping (start layer index, end layer index)
STT-480ms	2	[(1, 16), (17, 17)]
	4	[(1, 13), (14, 15), (16, 16), (17, 17)]
	6	[(1, 11), (12, 12), (13, 13), (14, 15), (16, 16), (17, 17)]
	8	[(1, 8), (9, 11), (12, 12), (13, 13), (14, 14), (15, 15), (16, 16), (17, 17)]
	10	[(1, 4), (5, 7), (8, 9), (10, 11), (12, 12), (13, 13), (14, 14), (15, 15), (16, 16), (17, 17)]
Parakeet-TDT-0.6B-v2	4	[(1, 5), (6, 8), (9, 23), (24, 24)]

Appendix

Table 7: Representational grouping details

Dataset	Domain	Utterance Length	Samples for STT (train/val.)	Samples for Parakeet (train/val.)
LibriSpeech	Read speech	0.8-30 s	281242 / 5551	278978 / 5417
Tedlium v3	Talks / lectures	0.5-15 s	265274 / 500	249141 / 486
VoxPopuli	Parliamentary speech	0.5-30 s	181050 / 1742	173958 / 1647
AMI	Meetings	0.5-30 s	73141 / 9196	56627 / 7090
Earnings22	Financial calls	1-30 s	47564 / 2456	33906 / 1849
GigaSpeech	Read + spontaneous	0.5–20 s	910140 / 5688	856002 / 4656

Appendix

Table 8: Training datasets used for distilling block-recurrent models.

Split	# of Speakers	Total duration (min)
train	2507	70452
dev	279	8037
test	106	3986

Appendix

Table 9: Dataset split of Voxceleb2 for personalized ASR.

Hyperparameter	Value
batch size	8
learning rate (LR)	5e-4/5e-5
LR scheduler	constant
optimizer	AdamW
$\beta_{1}$	0.9
$\beta_{2}$	0.98

Appendix

Table 10: Training hyperparameter

Hyperparameter	Value
batch size	8
learning rate (LR)	5e-4
LR scheduler	constant
optimizer	AdamW
$\beta_{1}$	0.9
$\beta_{2}$	0.98

Appendix

Table 11: Training hyperparameter for MoE-LoRA

Model	LS vs TED	LS vs VoxP	TED vs VoxP
STT-480ms	0.9989	0.9977	0.9992
Parakeet-v2	0.9969	0.9943	0.9945

Appendix

Table 12: Cross-dataset Pearson correlations of CKA similarity matrices

Figure 5: Angular updates across successive layers for STT-480ms, Parakeet-TDT-0.6B-v2, Canary-1B-Flash, and Granite-4.0-1B-Speech models. Each gray line represents the angle $\phi(h_{l},h_{l+1})$ as a function of layer $l$ for a single audio sample, while the dark line shows the averaged angle across a set of calibration speech samples.

Panel of Figure 6 — Figure 6: Teacher–student projected representation trajectories under different block counts $K$ . Blue curves denote the teacher encoder trajectory, while red curves show the block-recurrent student trajectory obtained via recurrent unrolling. Numerical annotations indicate layer indices for the teacher and block indices for the student.

Model	Params (M)	Avg. WER
Moonshine-tiny	30	12.54%
Moonshine-streaming-tiny	30	12.03%
Edge-STT K=4 (ours)	26.8	10.72%
Moonshine-base	60	9.67%
Edge-STT K=6 (ours)	39.5	9.65%
Whisper-large-v3-turbo	800	7.80%

Appendix

Table 13: Cross-model-family positioning comparison

Model	Effective memory (MB)	AMI	Earn	Giga	LS-c	LS-o	SPGI	TED	VoxP	Avg. WER
STT Teacher	217.8	26.23	20.56	14.15	2.54	5.97	5.80	5.58	7.60	11.05
STT Teacher (W4)	54.5	28.97	21.85	15.54	2.94	7.37	6.77	6.33	8.32	12.26
Edge-STT K=4	53.6	18.16	15.24	15.60	4.29	10.55	4.72	7.34	9.89	10.72
Parakeet Teacher	1217.8	11.16	11.09	9.73	1.70	3.19	2.18	3.37	5.96	6.05
Parakeet Teacher (W4)	304.5	12.53	11.55	9.97	1.76	3.67	2.5	3.45	6.34	6.47
Edge-Parakeet K=6	320.0	12.60	11.52	11.96	2.57	5.96	2.56	4.47	7.08	7.34

Appendix

Table 14: Memory-budget-matched quantization baseline

Training Objective	LS-c	LS-o	Avg. WER
Decoder KL only	4.65	11.65	8.15
Task loss + decoder KL	3.87	10.17	7.02
Task loss + encoder KD + decoder KL (ours)	3.84	9.79	6.81

Appendix

Table 15: KL divergence baseline

Model	Avg. WER (%)
LoRA baseline	8.69
MoE-LoRA (ours)	8.03

Appendix

Table 16: Conventional LoRA fine-tuning baseline

		GPU Scalar	CPU (fp32)
Model	$K$	Blackwell RTX Pro 6000	Threadripper 7995WX
STT-480ms	–	$2.019\ (1.00\times)$	$19.45\ (1.00\times)$
Edge-STT	1	$1.574\ \mathbf{(1.28\times)}$	$18.17\ \mathbf{(1.07\times)}$
	2	$1.608\ (1.26\times)$	$18.18\ (1.07\times)$
	4	$1.635\ (1.23\times)$	$18.46\ (1.05\times)$
	6	$1.623\ (1.24\times)$	$18.49\ (1.05\times)$

Appendix

Table 17: Energy per inference under weight sharing (

L{=}17

B{=}1

T{=}77

frames). GPU energy via nvmlDeviceGetTotalEnergyConsumption (hardware counter, fp16); CPU energy via Linux RAPL (fp32). All values in Joules. Savings ratio = Full Model energy / K-group energy (

>{1}

means less energy). Single run; energy std not independently measured (estimated

{<}2\%

). GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (128 MB L2). CPU: AMD Ryzen Threadripper PRO 7995WX (96 cores, 384 MB L3).

Model	$K$	Apple M4 Pro
STT-480ms	–	$32.11\ (1.00\times)$
Edge-STT	1	$21.98\ \mathbf{(1.46\times)}$
	2	$23.44\ (1.37\times)$
	4	$24.38\ (1.31\times)$
	6	$25.82\ (1.24\times)$
	8	$27.34\ (1.17\times)$

Appendix

Table 18: DRAM energy per inference under weight sharing (

L{=}17

B{=}1

T{=}77

frames, fp32) on Apple M4 Pro. All values in mili-Joules. Savings ratio = Full Model energy / K-group energy (

>{1}

means less energy).

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 484 papers

Jun 3, 2026

Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers

End-to-end ASR systems typically use fixed-depth acoustic encoders at inference, making it difficult to trade additional test-time computation for improved recognition without training a larger model. A natural approach is to reuse a shared Transformer block recurrently, but we find that naive looping does not fully exploit additional recurrent compute. We introduce LARM, a depth-conditioned looped Transformer that turns recurrent encoder depth into a controllable test-time compute axis. LARM combines sparse CTC checkpoints, supervision-clock embeddings, FiLM depth conditioning, and delayed soft-posterior feedback. These components structure the loop into recognition checkpoints separated by latent refinement phases and allow shared weights to specialize across recurrent steps. On LibriSpeech, LARM improves WER as the number of inference loops increases and achieves performance competitive with deeper unshared-parameter baselines. Our results show that test-time compute scaling can extend beyond autoregressive language-model reasoning to continuous non-autoregressive speech recognition.

Yacouba Kaloga, Shashi Kumar, Shakeel A. Sheikh +3

Idiap Research Institute, Switzerland · EPFL, Switzerland · BUT, Czech Republic

Inference-Time Scaling Test-Time Scaling Automatic Speech Recognition Recurrent Transformers

Ask PDF

Apr 16, 2026

Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference

Deploying high-quality automatic speech recognition (ASR) on edge devices requires models that jointly optimize accuracy, latency, and memory footprint while operating entirely on CPU without GPU acceleration. We conduct a systematic empirical study of state-of-the-art ASR architectures, encompassing encoder-decoder, transducer, and LLM-based paradigms, evaluated across batch, chunked, and streaming inference modes. Through a comprehensive benchmark of over 50 configurations spanning OpenAI Whisper, NVIDIA Nemotron, Parakeet TDT, Canary, Conformer Transducer, and Qwen3-ASR, we identify NVIDIA's Nemotron Speech Streaming as the strongest candidate for real-time English streaming on resource-constrained hardware. We then re-implement the complete streaming inference pipeline in ONNX Runtime and conduct a controlled evaluation of multiple post-training quantization strategies, including importance-weighted k-quant, mixed-precision schemes, and round-to-nearest quantization, combined with graph-level operator fusion. These optimizations reduce the model from 2.47 GB to as little as 0.67 GB while maintaining word error rate (WER) within 1% absolute of the full-precision PyTorch baseline. Our recommended configuration, the int4 k-quant variant, achieves 8.20% average streaming WER across eight standard benchmarks, running comfortably faster than real-time on CPU with 0.56 s algorithmic latency, establishing a new quality-efficiency Pareto point for on-device streaming ASR.

Nenad Banfic, David Fan, Kunal Vaishnavi +5

CoreAI, Microsoft

Efficient Neural Network Inference Streaming ASR Post-Training Quantization On-Device Inference

Ask PDF

Sep 17, 2026

Design of the IBM Granite 5.0 TurboCTC ASR Model

We describe the architecture, training methodology and inference speedups of Granite 5.0 Turbo CTC, a 470 million parameter encoder-only model with an excellent speed-accuracy tradeoff. The architecture uses pyramidal temporal subsampling within Conformer blocks using strided depthwise convolutions, block-diagonal (chunk-wise) self-attention, and conditioning on intermediate predictions from the middle layer. Training highlights are the use of only publicly available data, the novel use of a Muon optimizer, and balanced data sampling. Inference speedups include replacing 1 x 1 convolutions with linear layers and optimizing the attention computation in the Conformer blocks. Collectively, these result in a model that is on the speed-accuracy Pareto frontier of the Open ASR leaderboard for English short-form ASR while being twice as fast as the fastest competitor. The model can be used under a permissive license and downloaded from https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc.

Brian Kingsbury, George Saon, Masayuki Suzuki +5

IBM Research

Transformer Connectionist Temporal Classification Automatic Speech Recognition Conformer

Ask PDF

Abstract

Automatic Speech Recognition

Explore similar work

Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers

Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference

Design of the IBM Granite 5.0 TurboCTC ASR Model