Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages

Kuan-Tang Huang; Cheng-Yeh Yang; Chien-Chun Wang; Hung-Shin Lee; Hsin-Min Wang; Berlin Chen

cs.CLAug 29, 2026

Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages

Authors: Kuan-Tang Huang, Cheng-Yeh Yang, Chien-Chun Wang, Hung-Shin Lee, Hsin-Min Wang, Berlin Chen

Organizations: National Taiwan Normal University, Taiwan · EZAI, Taiwan · Academia Sinica, Taiwan

Abstract

Low-resource ASR remains difficult because scarce transcripts provide limited supervised evidence for target-side generation. To address this gap, we propose SAMA-ASR, a lightweight adapter mechanism that augments the decoder with semantic anchors from auxiliary translations and an acoustic anchor from speech; in principle, the mechanism can be applied to similar encoder--decoder multitask speech models. Through cross-modal adaptation, SAMA-ASR conditions decoder states on translation-derived semantic embeddings and a speech embedding, combining utterance-level meaning with speech-grounded evidence before token prediction. At evaluation time, these semantic anchors can be generated automatically by an upstream speech-to-text translator rather than supplied as oracle translations. Experiments on two 30-hour datasets covering the low-resource Sinitic varieties Taiwanese Hokkien and Hakka show that SAMA-ASR improves over acoustic, prior prompt-based, and semantic-only translation-guided baselines and remains effective in practical automatic semantic-anchor settings; translator-capacity analyses show that useful semantic anchors can be produced by a compact ST model.

Figures & tables

Figure 1: Overview of the SAMA-ASR framework. Snowflakes mark frozen modules and flames mark trainable modules. Automatic translations are produced by ST, encoded by mBERT, and fused with the frozen Whisper speech embedding through gated cross-attention. MT is used only in the optional multilingual diagnostic setting.

Baseline
Method	Train	Eval	Aud.	YT-THDC			HAT
Method	Train	Eval	Aud.	CER $\downarrow$	Rel. Impr. $\uparrow$	Rel. Impr. $\uparrow$	CER $\downarrow$	Rel. Impr. $\uparrow$	Rel. Impr. $\uparrow$
					vs. Base	vs. Audio		vs. Base	vs. Audio
Self Attn.	–	–	No	33.49	–	–	27.78	–	–
Cross Attn.	–	–	Yes	32.89	1.79	–	26.58	4.32	–
Prior Text-Guided Baselines + SAMA-ASR (Practical Evaluation)

Table 1: Main ASR results on YT-THDC and HAT. CER values are percentages; lower is better. Relative-improvement columns report CER reduction rates (%) against the context-only Self Attn. baseline (“vs. Base”) and the audio-only Cross Attn. baseline (“vs. Audio”); higher is better. “Oracle”/“Auto” denote the auxiliary-translation source, and “Aud.” indicates whether an explicit acoustic anchor is used. Bold practical SAMA-ASR results are significant at

p<0.05

by Bisani and Ney (2004) : standalone SAMA-ASR is compared with all non-LoRA baselines above it, and LoRA+SAMA-ASR is compared with LoRA.

Acoustic-Anchor Condition	YT-THDC	HAT
Self Attn. baseline	33.49	27.78
Cross Attn. baseline	32.89	26.58
Random Noise (matched mean/std)	42.62	30.91
Shuffle Across Utterances	45.49	35.59
SAMA-ASR	23.48	22.48

Table 2: Controlled acoustic-anchor ablation. CERs (%; lower is better). The controlled variants retain the complete SAMA-ASR architecture and trainable parameter count; the assigned corruption is used during both training and evaluation.

Figure 2: Position-wise CER analysis of autoregressive cold-start fragility. Average CER is reported over the first ten target-token positions on the test sets. TG-ASR is the strong semantic-only translation-guided baseline, while SAMA-ASR adds the acoustic-anchor path.

Figure 3: Evaluation-time translator-capacity analysis. Values are SAMA-ASR CERs using automatic Mandarin translations from translators of different sizes. Dashed lines denote context-only CER baselines; red values report translator BLEU, and green values report average cosine similarity between mBERT embeddings of automatic and oracle Mandarin translations.

Method	Data	YT-THDC	HAT
Self Attention	10 min	61.32	96.44
Cross Attention	10 min	66.18	78.27
SAMA-ASR	10 min	65.63	72.67
Self Attention	1 hr	55.92	68.37
Cross Attention	1 hr	54.97	57.20
SAMA-ASR	1 hr	53.45	54.68

Table 3: Paired-data budget analysis. CERs (%) when the same subset trains both ASR adaptation and the translator; SAMA-ASR uses automatic Mandarin translations at evaluation.

Text Encoder	Train	Eval	Aud.	YT-THDC	HAT
Random Init.	Oracle	Auto	Yes	27.39	27.00
Whisper Emb.	Oracle	Auto	Yes	33.91	28.13
mBERT	Oracle	Auto	Yes	23.48	22.48

Table 4: Text-encoder ablation. CERs (%) in audio-enhanced SAMA-ASR with oracle training anchors and automatic evaluation anchors.

Panel of Figure 4 — Figure 4: Auxiliary-language oracle-translation diagnostic. CER reductions are relative to the context-only baseline; bars below zero are better. Non-Mandarin auxiliary translations are produced by MT from oracle Mandarin translations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Split	YT-THDC		HAT
Split	Duration	# Utts.	Duration	# Utts.
Train	27.51	50,984	27.50	19,120
Test	2.79	4,859	2.50	1,725
Total	30.30	55,843	30.00	20,845

Appendix

Table 5: Low-resource dataset statistics. Durations are in hours.

Train Set	Train Translation	YT-THDC	HAT
30 hr	Pseudo	23.93	21.51
30 hr	Oracle	23.48	22.48
10 hr	Pseudo	50.50	39.53
10 hr	Oracle	30.04	30.93
1 hr	Pseudo	56.29	65.97
1 hr	Oracle	53.45	54.68

Appendix

Table 6: Training-translation source ablation. All rows evaluate with automatic Mandarin auxiliary translations; values are CERs (%). Pseudo-translation training uses translator-generated auxiliary translations, while oracle-translation training uses paired Mandarin translations.

System / ST Source	BLEU $\uparrow$	CER $\downarrow$
Cross Attn. baseline	–	32.89
SAMA-ASR + Tiny	32.37	32.56
SAMA-ASR + Base	39.03	31.90
SAMA-ASR + Small	52.39	26.80
SAMA-ASR + Medium	54.75	24.74
SAMA-ASR + Large	59.57	24.14

Appendix

Table 7: Robustness to noisy translation resources on YT-THDC. Both training and evaluation use pseudo Mandarin translations generated by the indicated ST model; values are BLEU and CER (%).

Figure 5: Semantic-neighborhood visualization of automatic translations. UMAP shows mBERT embeddings for five sampled test utterances per dataset. Colors denote utterances, marker shapes denote translation sources, and axes are for visualization only.

System	Transcription and English Translation (Gloss)
REF	干焦會當靠到銀行抑是公家機關看診
	(Can only rely on arriving at banks or government agencies for medical consultations.)
Self Attn.	干焦會當靠教囡仔抑是公家機關看證
	(Can only rely on teaching children or government agencies for verification.)
Cross Attn.	就按呢咱靠教議員學抑是公家機關看證
	(Just like this, we rely on teaching councilors to learn or government agencies for verification.)

Appendix

Table 8: YT-THDC error correction case study. Red highlights mark the target phrase and corresponding substitutions. Baselines produce acoustically or semantically plausible substitutions, whereas SAMA-ASR recovers 「到銀行」 ( kàu gîn-hâng , arriving at banks) by combining semantic guidance with acoustic grounding.

Distances from Taiwanese Hokkien
Language	Syntactic	Phonological	Inventory
Sinitic Family
Mandarin	0.58	0.53	0.65
Indo-European Family
English	0.62	0.64	0.55
Hindi	0.70	0.62	0.59

Appendix

Table 9: URIEL linguistic distances. Cosine distances are measured from Taiwanese Hokkien and Hakka; lower is closer.

Figure 6: Multilingual translation-composition diagnostic. Best-first and worst-first orders add auxiliary translations by oracle single-language performance;

L

denotes the number of auxiliary translations.

Method	Sem.	Aud.	CER $\downarrow$	RTF $\downarrow$	Params (B)	VRAM (GB)
Self Attn.	No	No	33.49	0.216	0.942	3.66
TG-ASR (Oracle)	Yes	No	20.15	0.307	1.244	4.86
SAMA-ASR (Oracle)	Yes	Yes	16.79	0.391	1.345	5.22

Appendix

Table 10: Inference cost with provided translations on YT-THDC. Oracle translations are available at inference time in this controlled setting, isolating the overhead of semantic and acoustic anchoring from upstream ST generation. “Sem.” and “Aud.” indicate the explicit semantic- and acoustic-anchor branches.

ST Model	CER $\downarrow$	RTF $\downarrow$	Params (B)	VRAM (GB)
Tiny	32.56	0.447	1.383	5.35
Base	31.90	0.466	1.418	5.53
Small	26.80	0.489	1.586	6.13
Medium	23.48	0.603	2.109	8.17
Large-v2	22.68	0.674	2.889	11.29

Appendix

Table 11: End-to-end deployment cost on YT-THDC. Each pipeline generates one automatic Mandarin semantic anchor with the indicated upstream ST model, encodes it with mBERT, and performs SAMA-ASR decoding. MT-based multilingual-anchor expansion is not included.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR
Sep 14, 2026Thai Thi Thanh Thao Dang, Mengjie Qian, Kate KnillLow-Resource Speech Recognition Parameter-Efficient Fine-Tuning
Department of Engineering, University of Cambridge, UK
Automatic Speech Recognition for Low-Resource Sinhala: A Critical Review of Methods, Challenges, and Future Directions
Oct 5, 2026Chanuka Dinuwan, Sanath Jayasena, Buddhika KarunarathneASR Evaluation Automatic Speech Recognition
MSc in Data Science and Artificial Intelligence Department of Computer Science & Engineering University of Moratuwa, Sri Lanka · University of Moratuwa, Sri Lanka
Low resource cross-modal alignment using HGNN to enhance speech representation
Sep 19, 2026Yannick Yomie Nzeuhang, Marie Tahon, Paulin Melatagia YontaCross-Modal Alignment Speech Processing
Department of computer sciences, University of Yaounde I, Street, Yaounde, 812, Cameroon. · IRD, UMMISCO, Street, Bondy, F-93143, France. · LIUM, Le Mans Universit´e, Avenue Olivier Messiaen, Le Mans, 72085, France.

Sep 14, 2026J/K move · Enter open · S save

Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR

Thai Thi Thanh Thao Dang, Mengjie Qian, Kate Knill

Department of Engineering, University of Cambridge, UK

Low-Resource Speech Recognition Parameter-Efficient Fine-Tuning Cross-Lingual Transfer Multilingual ASR

Open paper AskPDF arXiv

Extending large-scale multilingual automatic speech recognition (ASR) models to low-resource languages remains challenging. Model performance is skewed toward high-resource languages and degrades sharply for languages with limited labeled data and pre-training exposure. To address this, we investigate parameter-efficient approaches for transferring knowledge from resource-rich source languages to low-resource target languages on Whisper. Alongside warm initialization and attention-based fusion, we propose Sequential Adapter Stacking, which places a trainable target-language adapter on top of a frozen source-language adapter. Under controlled experiments, these approaches are evaluated on three target languages unsupported by Whisper -- Asturian, Assamese, and Xhosa -- using source languages with varying degrees of relatedness. Sequential Adapter Stacking with the closest related source consistently and significantly outperforms full fine-tuning across the three targets, with 5--8% relative WER reductions. These gains largely persist with only one hour of target training data.

Abstract

Automatic Speech Recognition

Explore similar work

Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR

Automatic Speech Recognition for Low-Resource Sinhala: A Critical Review of Methods, Challenges, and Future Directions

Low resource cross-modal alignment using HGNN to enhance speech representation