Automatic speech recognition (ASR) increasingly supplies transcripts for educational NLP, but aggregate word error rate (WER) can conceal errors concentrated in learner speech, multilingual turns, and speaker attribution. We compare platform-integrated, open-weight, adapted, and commercial ASR configurations on held-out sessions of online L2 Spanish interaction between L1 English learners and their instructors and tutors. Because English is the learner's shared first language, the interactions contain Spanish-only, English-only, and code-switched turns. Evaluation combines overall WER and character error rate (CER), WER disaggregated by speaker role and language context, and diarization error rate (DER). We also adapt Whisper large-v3 and MOSS-Transcribe-Diarize with LoRA using human-corrected in-domain classroom data. MOSS adaptation produced consistent improvements across speaker roles and language contexts, whereas Whisper showed smaller gains. Domain match proved more important than data volume: Larger out-of-domain bilingual and learner corpora did not outperform the smaller classroom corpus, where learner speech and code-switching are embedded in instructor--learner interaction. The open-weight MOSS adapter was competitive with commercial Muse Voice Transcribe on WER and CER and achieved lower system-provided DER on the recordings eligible for each system. Learner, English-only, and code-switched speech remained more difficult than instructor and Spanish-only speech. The findings demonstrate the value of condition-aware evaluation and interactionally matched data for bilingual educational ASR.
Figures & tables
Figure 1: Study pipeline. Whisper and MOSS are adapted with LoRA on 38 human-corrected Zoom sessions, with the recipe fixed by five-fold cross-validation. Six systems are scored on seven held-out sessions against human-corrected references.
Split
Total
Classroom
Language-Tutoring
Course levels
Adaptation
38
25
13
31 beginner; 7 intermediate
Evaluation
7
5
2
4 beginner; 2 intermediate; 1 advanced
Table 1: Number of sessions in the adaptation and evaluation sets. Splits were defined at the session level.
System
WER
95% CI
CER
Zoom captions
36.00
[32.67, 40.17]
31.62
Whisper large-v3
34.50
[31.97, 37.89]
28.72
Whisper large-v3 adapter
32.30
[29.24, 35.74]
27.00
MOSS base
26.52
[21.48, 31.85]
19.08
MOSS adapter
16.39
[13.46, 20.41]
12.71
Muse
15.59
[13.07, 19.01]
10.43
Table 2: Overall WER and CER across the seven evaluation sessions. WER is followed by its 95% session-bootstrap confidence interval. All values are percentages; lower is better.
System
Instructor
Learner
Gap (pp)
Zoom captions
29.00
54.27
25.27
Whisper large-v3
28.20
50.94
22.74
Whisper large-v3 adapter
26.50
47.44
20.94
MOSS base
21.80
38.84
17.04
MOSS adapter
13.46
24.04
10.58
Muse
12.80
22.87
10.07
Table 3: WER by speaker role. Values are percentages; the learner–instructor gap is reported in percentage points.
System
CS
Spanish
English
Zoom captions
47.20
32.34
54.55
Whisper large-v3
44.83
31.03
53.34
Whisper large-v3 adapter
41.67
29.09
50.58
MOSS base
33.54
24.36
35.67
MOSS adapter
21.44
14.77
24.14
Muse
20.41
14.05
22.93
Table 4: WER by language context. Values are percentages; CS denotes code-switched speech.
System
DER (%)
MOSS base
16.27
MOSS adapter
10.66
Muse
19.12
Table 5: Pooled system-provided DER for long-form ASR systems, weighted by reference speech time. Values are percentages; lower is better.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
WER
CER
Subset
Base
Adapter
Base
Adapter
All 38 sessions
20.99
17.85
12.48
10.91
Instructor speech
16.04
13.93
9.99
8.64
Learner speech
50.50
34.58
35.04
22.91
Classroom sessions
19.48
17.38
11.92
10.76
Language-Tutoring
25.70
19.31
14.27
11.39
Appendix
Table 6: Cross-validated adaptation of MOSS over the 38 adaptation sessions, comparing the base model with the adapter. Values are percentages, except that the learner–instructor gap is in percentage points; lower is better. The overall WER reduction is 3.14 points, 95% paired bootstrap CI [−4.19,−2.15] .
System
Window
Complete
Whisper large-v3
23.22
23.72
Whisper large-v3 adapter
18.68
24.29
Whisper adapter, LID loss
19.77
34.29
Appendix
Table 7: The same Whisper models under two decoding regimes, 38 sessions. Window: each 30-second window decoded independently. Complete: sequential long-form decoding. Values are WER percentages.
WER
Training data
Ext.
Overall
Learner
Classroom only (control)
—
22.59
26.59
+ SPLLOC, all, long
34%
23.25
30.82
+ SPLLOC, UG, long
34%
23.25
28.21
+ Bangor Miami
38%
23.55
31.13
+ SPLLOC, UG, short
49%
23.74
33.05
Appendix
Table 8: External training data added to the first fold, scored on its two development sessions and sorted by overall WER (%); lower is better. Ext.: external share of the training data. All: all course levels; UG: undergraduate learners only. Long: SPLLOC sessions concatenated into 15–28-minute examples; short: as recorded. Learner turns only: instructor speech removed. A dash means no external data was used.
Figure 2: WER (%) jointly disaggregated by speaker role and language context for all six systems. CS denotes code-switched speech. Darker cells indicate higher error, and the lowest value in each column is in bold. The outlined row is Muse Voice Transcribe. Lower is better.
Diarizer
DER (%)
pyannote-3.1
15.82
pyannote-community-1
15.44
DiariZen
12.31
NeMo Sortformer
18.91
Nemotron-3-Diarization
12.04
Appendix
Table 9: Pooled DER for external acoustic diarizers, weighted by reference speech time. Values are percentages; lower is better.