Automatic speech recognition (ASR) has advanced rapidly in recent years, driven by large-scale pretrained models and end-to-end architectures such as SLAM-ASR. A key component of SLAM-ASR systems is the Whisper speech encoder, which provides robust acoustic representations. While model pruning has been explored for the full Whisper encoder-decoder architecture, its impact within the SLAM-ASR setting remains under-investigated. In this work, we analyze the effects of layer pruning in the Whisper encoder when used as the acoustic backbone of SLAM-ASR. We further examine the extent to which LoRA-based fine-tuning can recover performance degradation caused by pruning. Experiments conducted across three Whisper variants (Small, Medium, Large-v2), three languages representing distinct resource levels (Danish, Dutch, English), and over 200 training runs demonstrate that pruning two encoder layers causes only 2-4% WER degradation, and that combining this pruning with LoRA adaptation consistently outperforms the unpruned baseline while reducing total parameters by 7-14%. Moreover, our error analysis reveals that LoRA primarily compensates through the language model's linguistic priors, reducing total word errors by 11-21% for Dutch and English, with substitutions and deletions showing the largest reductions. However, for low-resource Danish, the reduction is smaller (4-7%), and LoRA introduces increased insertion errors, indicating that compensation effectiveness depends on the LLM's pre-existing language proficiency and available training data.
Figures & tables
Figure 1: Overview of the SLAM-ASR architecture.
Language
Split
Samples
Hours
Danish
Train
3,592
4.18
Dev
2,511
3.21
Test
2,684
3.45
Dutch
Train
43,458
54.15
Dev
12,032
16.03
Test
12,033
16.38
Table 1: Dataset statistics for each language from Common Voice 22.
Danish
Dutch
English
Layers
Red.%
Base
+LoRA
Base
+LoRA
Base
+LoRA
Whisper-Small (88.15M base params)
12 (full)
0%
47.41
41.94
17.98
15.40
21.84
17.87
10
16.1%
50.49
47.23
21.16
16.98
25.81
20.64
8
32.2%
71.22
51.20
27.25
21.04
33.15
24.40
6
48.2%
115.68
67.02
32.14
24.45
40.81
31.85
Table 2: Test set WER (%) across encoder layer pruning configurations for three Whisper scales. Red.% : reduction of parameters in percentage. Base : projector-only training; +LoRA : LoRA additionally applied to the LLM. Blue : full baseline WER. Green : 2 layer pruned+LoRA that outperforms the full baseline.
Encoder
Configuration
Params
Net Δ
DA
NL
EN
Small
12L Baseline
88.15M
–
47.41
17.98
21.84
12L + LoRA
89.65M
+1.5M
41.94
15.40
17.87
10L + LoRA
75.5M
–12.7M
47.23
16.98
20.64
Medium
24L Baseline
307.24M
–
39.00
12.74
16.33
24L + LoRA
309.0M
+1.8M
38.88
10.92
14.13
22L + LoRA
284.0M
–23.2M
37.91
11.84
15.82
Table 3: Sweet-spot configurations (bold) per encoder scale, showing WER (%) on the test set. Net Δ is relative to the full baseline. Reported params use the r=16 LoRA config (Dutch/English); Danish uses r=8 ( ∼ 0.8M vs. ∼ 1.5M overhead).
% Utterances Degraded
Encoder
Lang
Base − 2L
+LoRA
Δ Recov.
Small
DA
42.7
38.1
− 4.6
NL
33.3
23.5
− 9.8
EN
35.9
23.9
− 12.0
DA
41.9
35.7
− 6.2
Medium
NL
24.6
24.3
− 0.3
Table 4: Utterance-level degradation after removing two layers, with and without LoRA. Δ Recov.: reduction in percentage of degraded utterances after applying LoRA.
WER Reduction
Encoder
Lang
Full+LoRA
Pruned+LoRA
Small
DA
5.47
3.26
NL
2.58
4.18
EN
3.97
5.17
Medium
DA
0.12
1.82
NL
1.82
2.38
Table 5: WER reduction from applying LoRA to full baseline and two-layer pruned models, measured in percentage points. Bold indicates the larger reduction. In seven of nine conditions, LoRA provides a larger reduction for the pruned model.
Pattern
Lang
Reference
Pruned Hypothesis
+LoRA Hypothesis
Hallucination
DA
der kogte gryden over
da kom gud og gik op
der kogte gryden over
Named entity
DA
charlot var altid inde hos fru simonin nu
chrisel indelte altså inden for sin sinde
charlot var altid inde hos fru simonin nu
Semantic drift
DA
lille soldat du skal være vor konge…
livet skal du have og du skal have den dig selv…
lille soldat du skal være vor konge…
Repetition
NL
…heel interessant en hebben ons zeer geholpen…
zeer grotendeels [ × 28]
…heel interessant en hebben ons zeer geholpen…
Named entity
NL
de veiligheidssituatie is sindsdien rampzalig verslechterd
de veiligheid ziet de waarde van een dienst…
de veiligheidssituatie is sindsdien rampzalig verslechterd
Compound word
NL
de luchtvaartmaatschappijen
de luchtvaart maatschappijen
de luchtvaartmaatschappijen
Table 6: Representative error patterns where the pruned model produces severe errors ( WER>0.8 ) but LoRA achieves full recovery ( WER=0.0 ). Examples sampled across all three encoder variants.
Encoder
Lang
Sub
Ins
Del
Tot
Small
DA
–7.7
+11.9
–10.7
–5.7
NL
–17.5
–37.4
–15.3
–20.4
EN
–19.8
–9.8
–37.4
–20.6
DA
–4.9
+18.4
–22.9
–4.4
Medium
NL
–13.7
–34.5
–11.4
–16.7
EN
–10.6
+0.5
–30.3
–11.6
Table 7: Word-level error change (%) after applying LoRA to the two-layer pruned configurations. Sub : substitutions; Ins : insertions; Del : deletions; Tot : total word errors. Bold : error increase.
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to 18.5% of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to 20.1% after distillation, compared to 21.9% zero-shot, going from a baseline of 18.2%. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).
We challenge the conventional view of neural network pruning as solely a compression technique, demonstrating that one-shot magnitude pruning serves as a powerful implicit regularizer for ASR. Using Whisper-small, we combine gradient- and Fisher-based sensitivity diagnostics with targeted, component-wise pruning. This reveals architectural asymmetries: decoder FFNs are pruning-fragile, whereas decoder self-attention and the last encoder layers contain redundancy that, when removed, improves generalization. Without fine-tuning, pruning 50% of decoder self-attention reduces WER by 2.38% absolute (20.44% relative) on LibriSpeech test-other; pruning the last four encoder layers at 50% instead yields a 1.72% absolute (14.8% relative) improvement. Gains persisted on Common Voice and TED-LIUM datasets. Beyond regularization benefits, our sensitivity-aware approach enables more aggressive one-shot compression. At 40% sparsity, where established global pruning approaches catastrophically fail, our method preserves near-baseline accuracy. This positions pruning as a first-class architectural design tool: knowing where to prune is as important as how much to prune.
Julian Irigoyen, Arthur Söhler, Andreas Søeborg Kirkedal
Speech-LLMs are expensive to run, making compression important for real-world deployment. However, compressed models are usually selected using aggregate word error rate (WER), which can hide how pruning affects different demographic groups. In this work, we systematically study the effect of audio encoder pruning on SLAM-ASR for different demographic groups. Using the Fair-Speech and Common Voice datasets, we found that the pruning does not affect all demographic groups equally; the gap between best- and worst-performing groups increases in fold. These disparities appear across all three encoder scales, but only the largest model initially hides them behind aggregate WER. LoRA adaptation improves WER for every group, but benefits groups already performing well more strongly and widens for certain groups. On Common Voice English, Danish, and Dutch, accent gaps persist but do not clearly widen, showing that the fairness effects of pruning vary across datasets and must be measured directly. Our findings suggest that for pruned models, deployment decisions should include per-group WER, with the worst-performing group's error rate as an explicit criterion.
Ganesh Pavan Kartikeya Bharadwaj Kolluri, Michael Kampouridis, Ravi Shekhar
School of Computer Science and Electronic Engineering, University of Essex, UK