cs.AIOct 12, 2026

Understanding Personalization Gains in Whisper ASR

Authors: Sergei Katkov, Antonio Liotta

Organizations: Free University of Bozen-Bolzano, Bolzano, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Whisper-style ASR models provide strong zero-shot transcription, but their residual errors are not evenly distributed across datasets, recording conditions, and metadata-based groups. We study when lightweight low-rank adaptation (LoRA) improves Whisper and whether observed gains require group-specific adapters. Across Common Voice, LibriSpeech, and VoxPopuli, we compare zero-shot Whisper, pooled LoRA, and group-specific LoRA under matched within-group splits. We find that adaptation is not uniformly beneficial: gains are concentrated in difficult Common Voice groups, are consistent on VoxPopuli, and largely disappear on low-error LibriSpeech. Crucially, pooled LoRA often matches, and sometimes clearly outperforms, group-specific LoRA. Gains over zero-shot should therefore be tested against matched pooled adaptation before being interpreted as evidence of personalization.

Figures & tables

DatasetGroupsUtterancesApprox. audio hoursMean duration (s)
Common Voice9,1391,376,2961,932.75.06
LibriSpeech1,949249,787852.912.29
VoxPopuli260106,779295.89.97
Table 1: Dataset statistics under the within-group protocol. Audio hours are approximate totals; repeated adaptation subsets are sampled separately.
DatasetMean WERMedian WERP90 WERP99 WERWER >> 10%WER >> 15%WER >> 20%
Common Voice7.896.4215.2831.5825.510.44.4
LibriSpeech5.014.458.4313.854.80.60.3
VoxPopuli11.3710.8215.2322.7563.810.42.3
Table 2: Per-group zero-shot WER distributions with Whisper small. WER values and tail masses are expressed as percentages; P90 and P99 denote the 90th and 99th percentiles across groups.
Figure 1: Per-group zero-shot WER distributions for Whisper small. The y-axis shows WER as a fraction. The violin shapes summarize the distribution across speaker proxies, and the box plots indicate the central tendency and spread. The long upper tails highlight difficult-group effects, especially for Common Voice.
DatasetModelWER%
Common Voicetiny18.71
Common Voicebase13.76
Common Voicesmall9.02
Common Voicemedium6.89
LibriSpeechtiny9.65
LibriSpeechbase6.92
Table 3: Overall zero-shot Whisper performance under the within-group protocol. WER values are percentages; these population-level numbers provide context for the matched adaptation subsets in Table 4 .
DatasetModelMatched zero-shot WERPooled LoRA WERPersonalized LoRA WER
Common Voicetiny19.66 ±\pm 2.8519.31 ±\pm 0.9518.69 ±\pm 1.12
Common Voicebase14.29 ±\pm 2.1114.13 ±\pm 0.7714.10 ±\pm 0.62
Common Voicesmall9.38 ±\pm 1.649.12 ±\pm 0.539.47 ±\pm 0.50
Common Voicemedium7.11 ±\pm 1.406.75 ±\pm 0.506.94 ±\pm 0.55
LibriSpeechtiny11.62 ±\pm 1.2011.61 ±\pm 1.0911.65 ±\pm 1.24
LibriSpeechbase8.14 ±\pm 0.588.05 ±\pm 0.708.09 ±\pm 0.68
Table 4: Adaptation across datasets. WER values are percentages; bold marks the best adapted condition in each row.
ModelDifficulty bandZero-shot WERPooled WERGain (pp)
tinyHardest 10%48.8138.94+9.88
baseHardest 10%38.0031.30+6.70
smallHardest 10%26.2922.21+4.09
mediumHardest 10%19.7117.49+2.22
tinyHardest 20%38.1332.44+5.69
baseHardest 20%29.0224.93+4.09
Table 5: Pooled LoRA on Common Voice difficulty bands. WER columns are percentages; gain is the absolute percentage-point improvement over zero-shot.
ModelMethodHardest 10%Middle 40–60%Easiest 20%
tinyPooled+12.61 ±\pm 0.44+1.47 ±\pm 0.11+0.35 ±\pm 0.06
tinyPersonalized+11.36 ±\pm 0.22+1.43 ±\pm 0.05+0.37 ±\pm 0.04
basePooled+10.91 ±\pm 0.19+0.87 ±\pm 0.02+0.43 ±\pm 0.02
basePersonalized+8.77 ±\pm 0.15+0.59 ±\pm 0.07+0.15 ±\pm 0.03
smallPooled+8.08 ±\pm 0.24+0.62 ±\pm 0.05+0.51 ±\pm 0.03
smallPersonalized+6.22 ±\pm 0.24+0.17 ±\pm 0.03+0.08 ±\pm 0.02
Table 6: Common Voice difficulty-band comparison of pooled and personalized LoRA. Values are absolute WER gains in percentage points.
More in this topic

Personalized Speech Recognition

See all 9 papers

Explore similar work

CardsList

  1. ASR Personalization in Pathological Voice: Accuracy, Efficiency, and Duration Effects

    Oct 17, 2026Giuseppe Giaimis, Sergei Katkov, Muhammad Azfar Yaqub +1Automatic Speech RecognitionPersonalized Speech Recognition

    Free University of Bolzano, Bolzano, Italy

  2. WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers

    Sep 12, 2025Akshat Pandey, Karun Kumar, Raphael TangDomain Adaptation for ASRDomain Adaptation

    Comcast · Speech AI · Washington, D.C., USA +3

  3. Whisper: Courtside Edition Enhancing ASR Performance Through LLM-Driven Context Generation

    Feb 21, 2026Yonathan Ron, Shiri Gilboa, Tammuz DubnovDomain Adaptation for ASRAutomatic Speech Recognition

    Reichman University