Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition

Asmee Mishra; Mengjie Qian; Brechtje Post; Kate Knill

cs.CLSep 24, 2026

Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition

Authors: Asmee Mishra, Mengjie Qian, Brechtje Post, Kate Knill

Organizations: Department of Engineering, University of Cambridge, United Kingdom · Phonetics Laboratory, University of Cambridge, United Kingdom

Abstract

Adapting multilingual speech foundation models to low-resource languages remains difficult, especially for languages that are poorly represented during pre-training. While parameter-efficient fine-tuning (PEFT) reduces the cost of adapting large models, conventional approaches such as LoRA rely on generic low-rank parameterizations and do not explicitly use downstream task information to define the adaptation subspace. To investigate whether task-informed PEFT can better support low-resource ASR, we apply Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR, and introduce two complementary extensions: Asymmetric-Coupled FCCA (AC-FCCA), which exploits structured cross-layer sharing, and Adaptive-Rank FCCA (AR-FCCA), which reallocates adaptation capacity across projection matrices under a fixed parameter budget. Under controlled multilingual experiments, we evaluate these approaches on languages that are poorly represented or unsupported during pre-training alongside well-represented languages. Standard FCCA is competitive with, and usually outperforms, trainable-parameter-budget-matched LoRA. AR-FCCA provides the most consistent improvement over standard FCCA across both model architectures, with statistically significant gains in several evaluation settings, while retaining the same number of trainable parameters. These results show that task-informed subspace construction can be effective for low-resource speech adaptation, and that adaptive rank allocation provides a robust way to improve parameter efficiency without increasing model capacity.

Figures & tables

Language	Lang. Token	Train	Val	Test
Asturian	$\langle\|\mathrm{es}\|\rangle$	7.5	0.9	2.4
Sorani Kurdish	$\langle\|\mathrm{fa}\|\rangle$	10.5	1.2	3.0
Kyrgyz	$\langle\|\mathrm{kk}\|\rangle$	9.3	1.3	3.2
Mandarin	$\langle\|\mathrm{zh}\|\rangle$	9.7	1.3	3.1
Persian	$\langle\|\mathrm{fa}\|\rangle$	10.0	1.5	3.7

Table 1 : FLEURS dataset statistics (hours) for target languages.

		Unsupported			Supported
Method	%Para	Ast.	Sor.	Kyr.	Man.	Per.
Vanilla	0	51.11	114.03	90.46	8.88	47.30
FFT	100	15.60	35.53	18.39	9.97	14.20
LoRA-128	9.82	15.96	37.83	19.75	10.10	14.92
LoRA-8	0.61	19.17	44.29	25.63	10.66	16.16
FCCA	0.61	17.07 ∗	44.97	22.25 ∗	10.46	16.20

Table 2 : WER (%) on FLEURS test sets using Whisper medium; CER (%) for Mandarin.

Method	%Para	Ast.	Sor.
Vanilla	0	48.94	105.30
FFT	100	16.06	37.93
LoRA-128	4.50	16.05	40.78
LoRA-6	0.21	19.78	44.99
FCCA	0.20	19.61	44.22
AC-FCCA-H		19.63	44.36

Table 3 : WER (%) on FLEURS test sets using Qwen3-ASR-1.7B.

Figure 1 : Adaptive FCCA rank allocation across Whisper model layers and attention projections for Asturian. Darker cells indicate larger allocated rank.

Explore similar work

Sep 14, 2026cs.CL

Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR

Extending large-scale multilingual automatic speech recognition (ASR) models to low-resource languages remains challenging. Model performance is skewed toward high-resource languages and degrades sharply for languages with limited labeled data and pre-training exposure. To address this, we investigate parameter-efficient approaches for transferring knowledge from resource-rich source languages to low-resource target languages on Whisper. Alongside warm initialization and attention-based fusion, we propose Sequential Adapter Stacking, which places a trainable target-language adapter on top of a frozen source-language adapter. Under controlled experiments, these approaches are evaluated on three target languages unsupported by Whisper -- Asturian, Assamese, and Xhosa -- using source languages with varying degrees of relatedness. Sequential Adapter Stacking with the closest related source consistently and significantly outperforms full fine-tuning across the three targets, with 5--8% relative WER reductions. These gains largely persist with only one hour of target training data.

Thai Thi Thanh Thao Dang, Mengjie Qian, Kate Knill

Department of Engineering, University of Cambridge, UK

Multilingual Automatic Speech Recognition Automatic Speech Recognition Low-Resource Languages Large Language Model Adaptation

Ask PDF

Sep 1, 2026cs.LG

Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through frozen-core adaptation: a calibration pass fixes left and right bases for each weight matrix, and fine-tuning optimizes only an

r\times r

core. This removes the ability of trainable factors to repair a poor initial span and makes subspace quality directly observable. We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates. Under a matched

r^2

budget, we compare eight basis constructors on 11 tasks, four model settings, and three seeds. On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks. It ranks first at all three Qwen scales and finishes within 0.13 points of the best method on Llama-3.2-1B. Controlled ablations show gains of 2.7--17.2 points from whitening and identify QR as necessary for stable core optimization in the tested regime. Finally, FCCA comes within 0.32 and 0.23 average points of LoRA and DoRA while optimizing 36.9K rather than roughly 7.4M parameters. These results show that a carefully selected fixed span can recover most of the benefit of movable low-rank factors at a much smaller trainable and optimizer-state cost.

Wentao Ye, Zhanming Shen, Zhiqing Xiao +3

Zhejiang University · Hunyuan, Tencent

Large Language Model Adaptation Frozen Subspace Covariance Fedavg Token Budget Allocation

Ask PDF

Jun 9, 2026eess.AS

Gumbel-BEARD: Automatic Layer Selection for Self-Supervised Adaptation of Whisper in Low-Resource Domains

Speech foundation models often struggle in low-resource domains due to domain mismatch and data scarcity. We propose Gumbel-BEARD, a domain adaptation framework that automates Whisper encoder layer selection via an end-to-end trainable hard Gumbel-Softmax selector. It enables self-supervised adaptation with a BEST-RQ objective that dynamically adapts to target acoustic characteristics without manual tuning. Experiments on the MyST child speech corpus demonstrate efficiency and scalability: with 10 h of labeled data for fine-tuning, our method matches a fully supervised baseline trained on the complete 133 h labeled set. We establish new state-of-the-art word error rates (WERs) of 8.21% using Whisper-medium on MyST and 11.06% using Whisper-small on the OGI Spontaneous dataset. Evaluation on CORAAL further confirms robustness to adult dialectal domain shifts, with up to 6% relative WER reduction, highlighting the generalizability of our approach to diverse low-resource conditions.

Zilai Wang, Natarajan Balaji Shankar, Mohan Shi +2

University of California, Los Angeles, USA

Speech Encoder Domain Adaptation Parameter-Efficient Adaptation Text Corpora Gumbel-Softmax Relaxation

Ask PDF