Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis
Authors: Souranil Kahali, Rituparna Bose, Abner Hernandez, Tomas Arias-Vergara, Andreas Maier, Ning Ma, Paula Andrea Perez-Toro
Organizations: Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany · School of Computer Science, University of Sheffield, Sheffield, UK · Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM), Munich, Germany
Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong generalisation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN->EN+DE continuation, and direct EN+DE fine-tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper-Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two-stage Whisper-Small trajectory shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves.
Figures & tables
Fig. 1: Overview of the multilingual MedASR adaptation and analysis pipeline. English and German medical speech are used for monolingual fine-tuning, two-stage EN→EN+DE continuation, and direct EN+DE fine-tuning. Adapted models are evaluated with WER, and Whisper-Small encoder representations are analysed through drift and probing tasks.
Model
Adapt
English ↓
German ↓
Combined ↓
Zero-shot
Whisper B
None
21.30
70.42
44.65
Whisper S
None
17.57
62.98
39.15
Whisper M
None
16.77
71.99 †
43.02
Whisper L
None
16.50
56.51
35.52
Monolingual (Mono) medical fine-tuning
TABLE I: ASR WER (%) on English and German medical speech test sets. Base (B), Small (S), Medium (M) and Large (L) Whisper versions where adapted.
Type
Utterances
EN symptom term
REF: stomach pain and bloating ; ZS: stomach pain and rotting ; S2: stomach pain and bloating .
EN clinical term
REF: could I have a concussion ; ZS: could I have a competition ; S2: could I have a concussion .
REF: die Prostater drüben; ZS/S2: Brotzeit verdrücken/verdrüben; DE-Large: die Prostata drüben.
TABLE II: Representative transcription errors. ZS-Small / S2-Small = Whisper-Small zero-shot / two-stage EN → EN+DE; DE-Large = German-only Whisper-Large-v3. Italics mark the contested term.
Fig. 2: Bootstrap-averaged representation drift across encoder layers (L0–L12) over five seeds; shading denotes ± 1 std. Cosine similarity is shown on the left and Linear CKA on the right. Pretrained → EN-FT produces the largest upper-layer shift, whereas EN-FT → ML-FT remains close to the adapted encoder. ML-FT EN vs. DE shows high cosine similarity but low Linear CKA, indicating centroid alignment with language-sensitive geometry.
Drift pair
Cosine range
Lin. CKA range
Pretrained → EN-FT
0.906–1.000
0.946–1.000
EN-FT → ML-FT
0.989–1.000
0.988–1.000
ML-FT: EN vs. DE
0.980–0.994
0.040–0.107
TABLE III: Representation drift summary over five bootstrap seeds. Values report the per-layer range across L0–L12.
Fig. 3: Layer-wise probe macro-F1 across encoder layers (L0–L12) for the three probing tasks, bootstrap-aggregated over five seeds (shading: ± 1 std). (a) Domain probe (medical English vs. LibriSpeech) and (b) Language probe (English medical vs. German medical) remain near ceiling across all checkpoints and layers, so peak-layer claims are not meaningful here. (c) Error-predictability probe uses rank-based low-/high-WER tertile labels. The pretrained encoder carries the strongest linearly recoverable error signal at the best layer; this signal weakens after English fine-tuning and multilingual continuation while decoder WER improves on the same 100 clips (19.87% → 13.61% → 12.48%).
Checkpoint
Domain F1
Lang. F1
WER F1
Dec. WER
min
min
best layer
(%)
Pretrained
≥0.984
≥0.990
0.721±0.028 (L2)
19.87
EN-FT
≥0.984
≥0.993
0.619±0.039 (L6)
13.61
ML-FT
≥0.985
≥0.993
0.556±0.033 (L11)
12.48
Dec. WER is computed on the same 100 English utterances used to label the WER probe (greedy decoding) and is not layer-wise.
TABLE IV: Probe summary over five bootstrap seeds. Domain and Language report the minimum macro-F1 across layers; WER F1 reports the best-layer low-/high-WER probe result under a rank-based tertile split ( n=33 per class).
National Electronics and Computer Technology Center (NECTEC), Thailand · Language Understanding Laboratory, Myanmar · King Mongkut’s University of Technology Thonburi, Thailand +1