Beyond Aggregate WER: In-domain Adaptation, Learner Speech, Code-Switching, and Speaker Attribution in Online Spanish Learning Classrooms

Olga Kellert; Samhitha Harish; Johnatan E Bonilla; Nemika Tyagi; Phanos Kareen; Nelvin Licona-Guevara

cs.CLOct 27, 2026

Beyond Aggregate WER: In-domain Adaptation, Learner Speech, Code-Switching, and Speaker Attribution in Online Spanish Learning Classrooms

Authors: Olga Kellert, Samhitha Harish, Johnatan E Bonilla, Nemika Tyagi, Phanos Kareen, Nelvin Licona-Guevara

Organizations: Arizona State University · Freie Universität Berlin

Abstract

Automatic speech recognition (ASR) increasingly supplies transcripts for educational NLP, but aggregate word error rate (WER) can conceal errors concentrated in learner speech, multilingual turns, and speaker attribution. We compare platform-integrated, open-weight, adapted, and commercial ASR configurations on held-out sessions of online L2 Spanish interaction between L1 English learners and their instructors and tutors. Because English is the learner's shared first language, the interactions contain Spanish-only, English-only, and code-switched turns. Evaluation combines overall WER and character error rate (CER), WER disaggregated by speaker role and language context, and diarization error rate (DER). We also adapt Whisper large-v3 and MOSS-Transcribe-Diarize with LoRA using human-corrected in-domain classroom data. MOSS adaptation produced consistent improvements across speaker roles and language contexts, whereas Whisper showed smaller gains. Domain match proved more important than data volume: Larger out-of-domain bilingual and learner corpora did not outperform the smaller classroom corpus, where learner speech and code-switching are embedded in instructor--learner interaction. The open-weight MOSS adapter was competitive with commercial Muse Voice Transcribe on WER and CER and achieved lower system-provided DER on the recordings eligible for each system. Learner, English-only, and code-switched speech remained more difficult than instructor and Spanish-only speech. The findings demonstrate the value of condition-aware evaluation and interactionally matched data for bilingual educational ASR.

Figures & tables

Figure 1: Study pipeline. Whisper and MOSS are adapted with LoRA on 38 human-corrected Zoom sessions, with the recipe fixed by five-fold cross-validation. Six systems are scored on seven held-out sessions against human-corrected references.

Split	Total	Classroom	Language-Tutoring	Course levels
Adaptation	38	25	13	31 beginner; 7 intermediate
Evaluation	7	5	2	4 beginner; 2 intermediate; 1 advanced

Table 1: Number of sessions in the adaptation and evaluation sets. Splits were defined at the session level.

System	WER	95% CI	CER
Zoom captions	36.00	[32.67, 40.17]	31.62
Whisper large-v3	34.50	[31.97, 37.89]	28.72
Whisper large-v3 adapter	32.30	[29.24, 35.74]	27.00
MOSS base	26.52	[21.48, 31.85]	19.08
MOSS adapter	16.39	[13.46, 20.41]	12.71
Muse	15.59	[13.07, 19.01]	10.43

Table 2: Overall WER and CER across the seven evaluation sessions. WER is followed by its 95% session-bootstrap confidence interval. All values are percentages; lower is better.

System	Instructor	Learner	Gap (pp)
Zoom captions	29.00	54.27	25.27
Whisper large-v3	28.20	50.94	22.74
Whisper large-v3 adapter	26.50	47.44	20.94
MOSS base	21.80	38.84	17.04
MOSS adapter	13.46	24.04	10.58
Muse	12.80	22.87	10.07

Table 3: WER by speaker role. Values are percentages; the learner–instructor gap is reported in percentage points.

System	CS	Spanish	English
Zoom captions	47.20	32.34	54.55
Whisper large-v3	44.83	31.03	53.34
Whisper large-v3 adapter	41.67	29.09	50.58
MOSS base	33.54	24.36	35.67
MOSS adapter	21.44	14.77	24.14
Muse	20.41	14.05	22.93

Table 4: WER by language context. Values are percentages; CS denotes code-switched speech.

System	DER (%)
MOSS base	16.27
MOSS adapter	10.66
Muse	19.12

Table 5: Pooled system-provided DER for long-form ASR systems, weighted by reference speech time. Values are percentages; lower is better.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	WER		CER
Subset	Base	Adapter	Base	Adapter
All 38 sessions	20.99	17.85	12.48	10.91
Instructor speech	16.04	13.93	9.99	8.64
Learner speech	50.50	34.58	35.04	22.91
Classroom sessions	19.48	17.38	11.92	10.76
Language-Tutoring	25.70	19.31	14.27	11.39

Appendix

Table 6: Cross-validated adaptation of MOSS over the 38 adaptation sessions, comparing the base model with the adapter. Values are percentages, except that the learner–instructor gap is in percentage points; lower is better. The overall WER reduction is 3.14 points, 95% paired bootstrap CI

[-4.19,-2.15]

System	Window	Complete
Whisper large-v3	23.22	23.72
Whisper large-v3 adapter	18.68	24.29
Whisper adapter, LID loss	19.77	34.29

Appendix

Table 7: The same Whisper models under two decoding regimes, 38 sessions. Window: each 30-second window decoded independently. Complete: sequential long-form decoding. Values are WER percentages.

		WER
Training data	Ext.	Overall	Learner
Classroom only (control)	—	22.59	26.59
+ SPLLOC, all, long	34%	23.25	30.82
+ SPLLOC, UG, long	34%	23.25	28.21
+ Bangor Miami	38%	23.55	31.13
+ SPLLOC, UG, short	49%	23.74	33.05

Appendix

Table 8: External training data added to the first fold, scored on its two development sessions and sorted by overall WER (%); lower is better. Ext.: external share of the training data. All: all course levels; UG: undergraduate learners only. Long: SPLLOC sessions concatenated into 15–28-minute examples; short: as recorded. Learner turns only: instructor speech removed. A dash means no external data was used.

Figure 2: WER (%) jointly disaggregated by speaker role and language context for all six systems. CS denotes code-switched speech. Darker cells indicate higher error, and the lowest value in each column is in bold. The outlined row is Muse Voice Transcribe. Lower is better.

Diarizer	DER (%)
pyannote-3.1	15.82
pyannote-community-1	15.44
DiariZen	12.31
NeMo Sortformer	18.91
Nemotron-3-Diarization	12.04

Appendix

Table 9: Pooled DER for external acoustic diarizers, weighted by reference speech time. Values are percentages; lower is better.

ASR Evaluation

Automatic speech recognition (ASR) evaluation measures transcript accuracy, robustness, fairness, and usefulness across languages, speakers, and real-world conditions.

See all 145 papers

Jun 19, 2026

OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics

Advances in deep learning and end-to-end Automatic Speech Recognition (ASR) have enabled robust multilingual models, but evaluation metrics remain limited in assessing accuracy. Efforts to improve or replace the common metric Word Error Rate (WER) often focus on English, leaving evaluations for low-resource languages under-explored and hindering fair cross-lingual comparisons. We present OpenWER, an open-source implementation that improves WER robustness through language-specific normalisation and compound word detection. A token-based Levenshtein alignment preserves complementary metrics and allows metadata embedding for granular accuracy scores. Our analysis of 52 languages shows absolute WER reductions of up to 25% compared to common libraries. OpenWER contributes to fairness in ASR research by increasing the reliability of WER across diverse languages and enabling more comprehensive accuracy evaluations.

Korbinian Kuhn, Gottfried Zimmermann

Stuttgart Media University, Germany · University of Tübingen, Germany

ASR Evaluation Low-Resource Language Processing Multilingual ASR

Ask PDF

Oct 7, 2026

Itgan at NADI 2026 shared task: Parameter-Efficient Whisper Adaptation for Robust, Mixed-Dialect and Code-Switched Arabic ASR

We describe the Itgan systems for the three ASR subtasks of NADI 2026, namely robust country-level ASR (1.1), mixed-dialect ASR (1.2), and Tunisian code-switched ASR (1.3). All three share one recipe, Whisper adapted with LoRA on consumer GPUs, and each was carried by a different addition to it. On 1.1, where the dialect label is given at test time, per-dialect specialists continued from a pooled adapter gave the largest gain, and the submitted system reached 57.1% country-average WER. A post-evaluation linear probe on frozen encoder features routes utterances without the label and recovers 44% of what oracle routing gives. On 1.2 the choice of base model mattered more than adapter capacity, and system combination helped only once we added a decorrelated member, reaching 46.7% WER. On 1.3 our system placed second at 14.49% WER with the lowest CER among the leading submissions, 5.38%. Its last 0.60 WER points came without further training, mostly from an exact weight-space average of independently trained runs, with ROVER voting adding the remainder. Every comparison carries a paired-bootstrap test, and we report eight directions that did not work.

Ibrahim Almajai

Independent Researcher

Robust ASR Automatic Speech Recognition Code-Switching Speech Recognition Domain Adaptation for ASR Parameter-Efficient Fine-Tuning

Ask PDF

Sep 10, 2026

Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavior on code switched speech in low resource, diacritic rich languages remains poorly characterized. We present a switch aware evaluation of eleven modern systems (six ASR models and five audio LMs) on English Yoruba code-switched speech, using a deterministic 2000 utterance evaluation set and a shared scoring pipeline. Beyond word error rate (WER), we report switch localized diagnostics: a switch entry token error rate (SETER), windowed switch point error rates, language specific error rates, and a diacritic insensitive WER. Our central finding is that aggregate WER hides code switching behavior. The best system by WER (an ASR model) is statistically indistinguishable from a leading audio LM on WER, yet the audio LM is significantly better on every switch localized metric. Across faithful systems, Yoruba token recognition collapses (error 0.97 for almost all systems) while English tokens are recognized far better, and errors concentrate sharply at switches into Yoruba. Several generative audio LMs fail as exact transcribers, producing translation, verbosity, and prompt leakage that are strongly prompt dependent. We release manifests, metric implementations, and evaluation scripts to support reproducible, switch aware benchmarking for African code switched speech.

Chibuzor Okocha, Christan Earl Grant

Department of Computer & Information Science & Engineering University of Florida, Gainesville, FL, USA

Audio-Language Model Evaluation Code-Switching Speech Recognition ASR Evaluation Low-Resource Speech Recognition

Ask PDF

Abstract

ASR Evaluation

Explore similar work

OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics

Itgan at NADI 2026 shared task: Parameter-Efficient Whisper Adaptation for Robust, Mixed-Dialect and Code-Switched Arabic ASR

Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech