CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition

Tianchi Liu; Nicholas Moratelli; Khurram Azeem Hashmi; Mohammadreza Zolfaghari; Rishabh Jain; Louis Lu; Changdae Park; Amril Nazir

cs.CLOct 13, 2026

CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition

Authors: Tianchi Liu, Nicholas Moratelli, Khurram Azeem Hashmi, Mohammadreza Zolfaghari, Rishabh Jain, Louis Lu, Changdae Park, Amril Nazir

Organizations: Tether AI Research

Abstract

Feature distillation is usually expected to add to logit distillation. We find that this expectation does not hold for two common feature losses in compact multilingual automatic speech recognition (ASR), and analyze why through their gradients. The gradient of a projection-based loss is orthogonal to the task gradient, while that of Centered Kernel Alignment (CKA), a similarity-based loss, is weakly aligned with it. CKA lowers the word error rate (WER) on non-English languages, but at the expense of data-rich English. Three loss-level remedies (one trained, two probed) all fall short, and the CKA gradient overlaps with the subspace English uses in the shared encoder by 97.5% to 99.5% (about 81% by chance). Guided by this diagnosis, we propose CAFE (Confined-Adapter Feature distillation), which confines the CKA gradient to adapters that English bypasses, addressing the trade-off in the parameters rather than in the loss. On the Open ASR Leaderboard, CAFE lowers the average WER from 8.55% to 8.02%, beats an adapter-only capacity control at 8.51%, and makes English more stable across checkpoints. At 254M parameters, it trails the 0.6B teacher by 2 WER points and is competitive with much larger models across 48 test sets in 24 languages. We use 564k hours of training data and state-of-the-art models to keep the findings relevant to practice.

Figures & tables

Figure 1: Averaged English and Multilingual WER vs. model size on the Open ASR Leaderboard.

Figure 2: Feature-only runs: the same supervised recipe plus one feature distillation loss, with logit distillation off. (a) Feature loss (log scale). (b) The same loss divided by its initial value. (c) Wall-clock time per training step. VkD w/o norm is the projection loss without LayerNorm.

feature term	$\\|g_{\text{feat}}\\|/\\|g_{\text{sup}}\\|$	$\cos(g_{\text{feat}},g_{\text{sup}})$
VkD variant	$7.3\times 10^{-4}$	$+0.009$
VkD w/o norm	$0.15$	$-0.005$
CKA	$1.1\times 10^{-3}$	$+0.087$

Table 1: Median ratio of gradient norms, and cosine similarity between the feature and supervised gradients.

System	EN-7	ML-9	Avg
Baseline	9.85	7.24	8.55
VkD variant	10.18	7.22	8.70
CKA	11.25	6.83	9.04
Lang-gated CKA	11.21	7.18	9.19

Table 2: Feature losses on top of the Baseline (WER after CA). Lang-gated CKA applies CKA to non-English frames only.

Figure 3: CAFE: feature distillation confined to adapters. There is one adapter at each distilled layer $\ell\in\{3,7,11,15,19,23\}$ , and only two are drawn. $A(\cdot)$ denotes the residual update added by the adapter.

				English (7 sets)		Multilingual (9 sets)		Avg (EN, ML)
system	Lang.	Adapter	feature-KD	mean $\pm$ std	CA	mean $\pm$ std	CA	mean $\pm$ std	CA
Baseline	25	–	–	10.33 $\pm$ 0.82	9.85	7.59 $\pm$ 0.20	7.24	8.96 $\pm$ 0.39	8.55
CKA	25	–	CKA	11.42 $\pm$ 0.67	11.25	7.22 $\pm$ 0.12	6.83	9.32 $\pm$ 0.33	9.04
Capacity	25	✓	–	9.98 $\pm$ 0.53	9.67	7.77 $\pm$ 0.11	7.34	8.88 $\pm$ 0.27	8.51
CAFE w/o sg	25	✓	CKA (no sg)	10.75 $\pm$ 0.69	10.44	7.37 $\pm$ 0.12	6.95	9.06 $\pm$ 0.32	8.70
CAFE (ours)	25	✓	CKA (confined)	9.32 $\pm$ 0.32	9.03	7.43 $\pm$ 0.13	7.00	8.38 $\pm$ 0.19	8.02

Table 3: Main comparison: WER (lower is better) on the Open ASR Leaderboard sets ( Srivastav et al., 2025 ) . Lang. is the number of training languages. Each system differs from the Baseline (supervised loss plus logit distillation) in two factors: Adapter (whether it adds the language-routed adapters) and feature-KD (the feature loss: – for none, and CKA (confined) for CKA confined to the adapters). CAFE w/o sg is CAFE without the stop-gradient of Eq. ( 8 ). For each system we give the mean

\pm

std over the last 7 checkpoints and the WER of their checkpoint average (CA).

	CAFE	Baseline
Params	254.45M	253.95M
RTFx bs1 $\uparrow$	212.9 $\pm$ 6.9	221.3 $\pm$ 5.2
RTFx bs32 $\uparrow$	2602.3 $\pm$ 68.9	2656.6 $\pm$ 79.8
P50 lat. (s) $\downarrow$	0.089 $\pm$ 0.001	0.088 $\pm$ 0.000
Peak mem (MB) $\downarrow$	2593.4	2592.5

Table 4: Adapter overhead: CAFE versus the same model without adapters (Baseline). RTFx is the inverse real-time factor, P50 is the median latency at batch size 1, and peak memory is measured at batch size 32.

	Parakeet (teacher)	Whisper medium	omniASR-LLM 300M v2	OWSM- CTC v4	Qwen3-ASR 0.6B	omniASR- CTC-1B v2	Whisper small	omniASR-CTC 300M v2	OWSM v4 small	CAFE (Ours)	CAFE- 8lang
Params (M)	627	764	1,632	1,011	782	976	242	326	370	254	254
Number of languages	25	$\sim$ 100	$>$ 1600	75	30	$>$ 1600	$\sim$ 100	$>$ 1600	75	25	8
RTFx bs1 $\uparrow$	198	21	10	590	18	513	35	848	39	213	213
RTFx bs32 $\uparrow$	2,520	386	117	1,282	310	2,142	546	3,405	37	2,602	2,602
P50 lat. bs1 (s) $\downarrow$	0.095	0.704	1.990	0.033	1.070	0.037	0.417	0.024	0.501	0.089	0.089
Peak mem (MB) $\downarrow$	3,310	6,932	17,310	25,803	6,266	16,033	2,617	14,802	2,505	2,593	2,593

Table 5: Comparison with external systems: mean WER per test suite (

\downarrow

), model size, and inference cost. Parakeet is the teacher. Systems are grouped by size. Within each of the two middle groups, bold marks the best result per row and underline the second best. CAFE-8lang (last column) uses the same method, trained on the 8 MLS languages only, and is excluded from the group comparison. Per-dataset results are in Appendix B , Table 6 .

Figure 4: English and multilingual test-set WER during training for Baseline-8lang and proposed CAFE-8lang.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Parakeet (teacher)	Whisper medium	omniASR-LLM 300M v2	OWSM- CTC v4	Qwen3-ASR 0.6B	omniASR- CTC-1B v2	Whisper small	omniASR-CTC 300M v2	OWSM v4 small	CAFE (ours)	CAFE- 8lang
Number of languages	25	$\sim$ 100	$>$ 1600	75	30	$>$ 1600	$\sim$ 100	$>$ 1600	75	25	8
Params (M) $\downarrow$	627	764	1,632	1,011	782	976	242	326	370	254	254
RTFx bs1 $\uparrow$	198	21	10	590	18	513	35	848	39	213	213
RTFx bs32 $\uparrow$	2,520	386	117	1,282	310	2,142	546	3,405	37	2,602	2,602
P50 lat. bs1 (s) $\downarrow$	0.095	0.704	1.990	0.033	1.070	0.037	0.417	0.024	0.501	0.089	0.089
Peak mem (MB) $\downarrow$	3,310	6,932	17,310	25,803	6,266	16,033	2,617	14,802	2,505	2,593	2,593

Appendix

Table 6: Comparison with external systems on all test sets: per-dataset WER and inference efficiency (Appendix A ). Parakeet is the teacher (reference). Systems are grouped by size, and bold and underline mark the best and second-best mean per row within the large group (Whisper-medium, omniASR-LLM-300M v2, OWSM-CTC v4, Qwen3-ASR-0.6B, omniASR-CTC-1B v2) and within the compact group (Whisper-small, omniASR-CTC-300M v2, OWSM v4 small, and CAFE). CAFE-8lang (last column) uses the same method, trained on the 8 MLS languages only to replicate the result at a smaller data scale; it is not included in the group comparison, has the same inference cost as CAFE, and is N/A for languages outside its set. Long-form results for omniASR and Qwen3-ASR-0.6B use 40 s non-overlapping windows (the maximum input length of omniASR). Qwen3-ASR-0.6B was also run with its native 1200 s windows, but degenerated (repetition loops or early stopping) with WER of roughly 90–115%, so it is reported with the same 40 s windows as omniASR.

Lang.	Utterances	Hours	%
en	62,165,596	200,020.7	35.46%
de	4,702,308	26,909.2	4.77%
fr	4,511,551	25,603.1	4.54%
es	4,354,218	23,614.3	4.19%
it	3,183,254	22,482.3	3.99%
pl	2,774,073	20,770.6	3.68%

Appendix

Table 7: Training data per language, sorted by hours. Blue bars show each language’s share of the total.

Lang. Samples Hours % VoxPopuli (23 langs, 373,686 h, 66.24%) en 3,053,532 24,103.2 6.45% de 2,979,597 23,198.5 6.21% fr 2,955,798 22,777.4 6.10% it 2,814,827 21,788.2 5.83% es 2,788,598 21,348.5 5.71% pl 2,652,542 20,556.1 5.50% nl 2,474,233 18,882.2 5.05% cs 2,333,712 18,125.7 4.85% ro 2,296,650 17,717.2 4.74% el 2,303,443 17,563.8 4.70% hu 2,278,164 17,484.2 4.68% pt 2,282,071 17,334.2 4.64% bg 2,234,897 17,095.7 4.57% sv 2,054,815 15,741.4 4.21% fi 1,799,765 13,894.5 3.72% da 1,763,822 13,356.2 3.57% lt 1,754,809 13,331.6 3.57% lv 1,595,156 12,292.1 3.29% sk 1,476,106 11,398.3 3.05% sl 1,433,486 10,979.5 2.94% et 1,281,814 9,788.0 2.62% hr 1,004,410 7,817.1 2.09% mt 919,343 7,112.8 1.90% YODAS (23 langs, 104,306 h, 18.49%) en 39,801,853 100,102.2 95.97% ru 1,873,248 1,732.4 1.66% es 947,615 779.4 0.75% fr 557,519 715.5 0.69% de 415,050 496.2 0.48% pt 727,788 262.1 0.25% it 86,580 120.2 0.12% nl 71,477 37.8 0.04% pl 28,678 21.1 0.02% uk 9,479 10.2 0.01% hu 6,530 8.5 0.01% ro 2,303 5.8 0.01% fi 4,637 3.8 0.00% cs 4,185 3.8 0.00% sv 3,189 1.9 0.00% bg 1,533 1.8 0.00% el 514 1.0 0.00% lt 390 0.6 0.00% da 656 0.6 0.00% et 513 0.5 0.00% sk 287 0.5 0.00% hr 261 0.2 0.00% lv 75 0.1 0.00%

Lang. Samples Hours % MLS (8 langs, 50,686 h, 8.99%) en 10,808,037 44,659.7 88.11% de 469,940 1,966.5 3.88% nl 374,287 1,554.2 3.07% fr 258,213 1,076.6 2.12% es 220,701 917.7 1.81% it 59,623 247.4 0.49% pt 37,533 161.0 0.32% pl 25,043 103.6 0.20% Common Voice (24 langs, 6,492 h, 1.15%) en 1,500,870 2,225.7 34.28% de 837,721 1,248.0 19.22% fr 740,021 1,033.6 15.92% es 397,304 568.7 8.76% it 222,224 326.5 5.03% ru 135,348 193.1 2.97% lv 135,129 176.4 2.72% pt 121,417 136.0 2.09% hu 72,866 108.4 1.67% pl 67,810 89.8 1.38% nl 66,017 81.8 1.26% uk 53,832 67.5 1.04% cs 40,203 50.6 0.78% et 18,482 33.8 0.52% sv 29,351 31.0 0.48% sk 25,398 30.3 0.47% lt 14,258 20.0 0.31% ro 12,363 13.6 0.21% el 11,317 12.9 0.20% bg 7,950 11.8 0.18% sl 10,168 10.8 0.17% fi 7,316 9.1 0.14% da 6,219 7.5 0.12% mt 4,368 5.5 0.08% Monolingual English corpora Libri-Light (en, 23,735 h, 4.21%) NSC (en, 3,098 h, 0.55%) Europarl-ASR (en, 1,016 h, 0.18%) LibriSpeech (en, 961 h, 0.17%) AMI (en, 77 h, 0.01%) VCTK (en, 41 h, 0.01%)

Appendix

Table 8: Training data by corpus and language. Each header gives a corpus with its number of languages, total hours, and share of all training data; the rows below give utterances, hours, and share within the corpus (blue bars) for each language. Left: VoxPopuli and YODAS. Right: MLS and Common Voice, then the six English-only corpora (Libri-Light, NSC, Europarl-ASR, LibriSpeech, AMI, VCTK). The green row is the total.

Test set / Language	Samples	Hours	Test set	Samples	Hours
From Open ASR Leaderboard			From the official dataset’s test set
English short-form (7)			Multilingual LibriSpeech (8)
ami	11,653	8.54	en	3,769	15.55
earnings22	2,737	5.43	de	3,394	14.29
gigaspeech	19,898	35.35	nl	628	2.51
LibriSpeech.clean	2,620	5.40	pl	284	1.17

Appendix

Table 9: Per-subset evaluation statistics for all 48 unique test subsets (samples and audio hours), grouped by source; blue subtotal rows give per-group totals. The four Open ASR Leaderboard Multilingual LibriSpeech languages reuse the same audio as the corresponding Multilingual LibriSpeech (8) languages.

system	$W$	EN-7	ML-9	Avg
CAFE	5	9.10	7.02	8.06
	7	9.03	7.00	8.02
	10	9.05	7.03	8.04
Baseline	5	9.85	7.25	8.55
	7	9.85	7.24	8.55
	10	9.99	7.17	8.58

Appendix

Table 10: Robustness to the CA window size: EN-7 / ML-9 / Avg for the last

W\in\{5,7,10\}

checkpoints.

Overall	English	Non-English
98.9%	97.6%	99.9%

Appendix

Table 11: Gate accuracy on 7,692 utterances: 500 randomly sampled per test set from the Open ASR Leaderboard EN-7 and ML-9 test sets (all 192 for MLS pt, which has fewer than 500).

	Baseline	CAFE (w/ label)	CAFE + gate (w/o label)	Adapters always on	Adapters never on
EN-7	9.85	9.03	9.11	15.95	9.03
FLEURS (5)	7.22	7.00	7.04	7.00	8.46
MLS (4)	7.27	7.01	7.01	7.01	7.62
ML-9	7.24	7.00	7.03	7.00	8.04
Avg (EN, ML)	8.55	8.02	8.07	11.48	8.54

Appendix

Table 12: WER (last-7 CA) of CAFE when the EN/non-EN decision is taken from the test set (w/ label) or made by the gate (w/o label), with the Baseline for reference. The last two columns apply the adapters to every utterance or to none.

EN-7	$\Delta$ WER	95% CI	Perm. $p$	Ckpt	Wilcoxon $p$
CKA vs Baseline	$+1.40$	$[+1.25,+1.56]$	$10^{-4}$	0/7	0.016
CAFE vs Baseline	$-0.82$	$[-0.94,-0.71]$	$10^{-4}$	7/7	0.016
CAFE vs Capacity	$-0.64$	$[-0.72,-0.56]$	$10^{-4}$	6/7	0.047
ML-9
CKA vs Baseline	$-0.42$	$[-0.54,-0.29]$	$10^{-4}$	7/7	0.016
CAFE vs Baseline	$-0.24$	$[-0.39,-0.08]$	0.0028	5/7	0.156

Appendix

Table 13: Significance on EN-7 and ML-9.

\Delta

WER is the difference between the averaged models (negative means the first system is better), with its 95% bootstrap CI and permutation

p

(

10^{-4}

is the smallest value 10,000 draws can give). Ckpt is the number of the 7 checkpoint pairs on which the first system is better; Wilcoxon is over the same pairs.

Feature-Based Knowledge Distillation

Feature-based knowledge distillation transfers internal representations from large teacher models to smaller students for efficient learning.

See all 49 papers

Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition
Sep 24, 2026Asmee Mishra, Mengjie Qian, Brechtje Post +1Speech Foundation Models Low-Resource Speech Recognition
Department of Engineering, University of Cambridge, United Kingdom · Phonetics Laboratory, University of Cambridge, United Kingdom
Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision
Mar 7, 2026Shreyas Gopal, Donghang Wu, Ashutosh Anshul +5Speech Language Models Language Model Distillation
College of Computing and Data Science, Nanyang Technological University, Singapore · AI Singapore, National University of Singapore · Institute for Infocomm Research (I2R), A∗STAR, Singapore
Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
Jul 14, 2026Harsha Vardhan Khurdula, Abhinav Kumar Singh, Yoeven D Khemlani +1Automatic Speech Recognition Diffusion Language Models
Interfaze AI · San Francisco, CA, USA

Sep 24, 2026J/K move · Enter open · S save

Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition

Asmee Mishra, Mengjie Qian, Brechtje Post, Kate Knill

Department of Engineering, University of Cambridge, United Kingdom · Phonetics Laboratory, University of Cambridge, United Kingdom

Speech Foundation Models Low-Resource Speech Recognition Parameter-Efficient Fine-Tuning Multilingual ASR

Open paper AskPDF arXiv

Adapting multilingual speech foundation models to low-resource languages remains difficult, especially for languages that are poorly represented during pre-training. While parameter-efficient fine-tuning (PEFT) reduces the cost of adapting large models, conventional approaches such as LoRA rely on generic low-rank parameterizations and do not explicitly use downstream task information to define the adaptation subspace. To investigate whether task-informed PEFT can better support low-resource ASR, we apply Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR, and introduce two complementary extensions: Asymmetric-Coupled FCCA (AC-FCCA), which exploits structured cross-layer sharing, and Adaptive-Rank FCCA (AR-FCCA), which reallocates adaptation capacity across projection matrices under a fixed parameter budget. Under controlled multilingual experiments, we evaluate these approaches on languages that are poorly represented or unsupported during pre-training alongside well-represented languages. Standard FCCA is competitive with, and usually outperforms, trainable-parameter-budget-matched LoRA. AR-FCCA provides the most consistent improvement over standard FCCA across both model architectures, with statistically significant gains in several evaluation settings, while retaining the same number of trainable parameters. These results show that task-informed subspace construction can be effective for low-resource speech adaptation, and that adaptive rank allocation provides a robust way to improve parameter efficiency without increasing model capacity.

Abstract

Feature-Based Knowledge Distillation

Explore similar work

Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition

Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model