Speech foundation models struggle with low-resource Pacific Indigenous languages because of severe data scarcity. Furthermore, full fine-tuning risks catastrophic forgetting. To address this gap, we present an empirical study adapting models to real-world Pacific datasets. We investigate the impact of data volume, adaptation strategies, and representational drift on speech foundation models for various Pacific languages. Additionally, we analyze a continual learning framework for sequential language acquisition. Empirical results across three distinct Pacific Indigenous languages demonstrate that adapting to these linguistically distant languages induces severe internal representational drift. Consequently, these models face a strict plasticity and stability dilemma. While LoRA adapts well initially, it suffers from catastrophic forgetting during sequential learning. Ultimately, this study highlights the urgent need for robust adaptation strategies tailored to underrepresented languages.
Figures & tables
Locale
Samples
Total Duration (H)
Shortest (S)
Longest (S)
Average (S)
Bislama
11,576
13.75
0.17
34.65
4.28
Nafsan
9,245
14.83
0.48
67.55
5.78
Lelepa
3,022
3.55
0.31
24.06
4.23
Total
23,843
32.13
0.17
67.55
4.85
Table 1: Sample duration statistics across different locales. Specifically, the terms shortest, longest, and average indicate the sample durations in seconds.
Language
Hours
Setting
Test CER
Test WER
Bislama
0.5
Full FT
18.88
41.97
0.5
LoRA
20.49
45.88
1.0
Full FT
14.57
32.95
1.0
LoRA
17.60
39.51
2.0
Full FT
15.05
30.92
2.0
LoRA
15.70
33.94
Table 2: Comparison of Fine-tuning Performance: Full FT vs. LoRA (Encoder-Decoder) Across Training Hours
Figure 1: Relative representational drift across the twelve encoder layers.
Figure 2: Relative representational drift across the twelve decoder layers.
Model
Adaptation Strategy
Updated Params
Chinese Test CER
English Test WER
French Test WER
whisper-small
None (Before Fine-Tuning)
0
34.62
15.68
24.27
whisper-small
Low-Rank Adaptation (LoRA)
20.1M
36.71
18.89
27.82
whisper-small
Full Fine-Tuning (FullFT)
244M
45.02
26.24
40.26
Table 3: Evaluation of Catastrophic Forgetting on the English LibriSpeech Test Set. The table illustrates the degradation in English transcription accuracy after the Whisper-Small model is adapted to the extreme low-resource Lelepa dataset.
Adaptation
Updated
Lelepa CER ( ↓ )
English WER ( ↓ )
Baseline
None
-
15.68
FullFT
Encoder + Decoder
27.27
26.24
LoRA
Encoder + Decoder
28.41
21.24
Decoder-only LoRA
Decoder Only
34.68
18.27
Encoder-only LoRA
Encoder Only
29.87
31.26
Table 4: Impact of Component-Specific Adaptation on Target Accuracy and Catastrophic Forgetting. We evaluate Whisper-Small fine-tuned on Lelepa and test its knowledge retention on English. The results reveal a strict trade-off between target acoustic adaptation and source language retention.
Strategy
Training Stage
Test WER (%)
Avg. WER
Nafsan
Lelepa
Full FT
After Nafsan
47.02
N/A
64.70%
After Lelepa
45.67
83.72
LoRA [ 28 ]
After Nafsan
53.23
N/A
76.52%
After Lelepa
84.42
68.62
DoRA [ 32 ]
After Nafsan
53.57
N/A
77.81%
Table 5: Sequential Continual Learning Performance: Comparison of Full FT and LoRA on Nafsan → Lelepa Sequence (Whisper-Small)