Spoken native language (L1) identification for second language learners has traditionally relied on acoustic attributes of speech. Comparatively little attention has been paid to analysing speech transcripts for L1 traces. In this paper, we present a study of L1 identification using both speech and corresponding transcripts from a spoken learner English corpus. We compare a WavLM-based audio L1 classifier with a zero-shot GPT-4o classifier operating on transcripts and analyse their predictions across proficiency levels using a CEFR-balanced evaluation set. Our experiments show that the audio classifier outperforms the transcript-based approach overall when full transcripts are available, although their relative strengths vary with learner proficiency. Audio performs better at A1, whereas text performs better from A2 to C1 with full transcripts. Transcript-based accuracy generally increases with proficiency, while acoustic performance remains comparatively stable. Restricting transcript length substantially reduces text-based accuracy, whereas audio is less sensitive to input length and performs relatively well with limited input. These findings provide insights into L1 identification from acoustic and textual representations of learner speech and how performance varies with proficiency and the amount of available language material.
Figures & tables
CEFR level
n
Avg. Length
A1
54
190 ±88
A2
54
286 ±88
B1
54
390 ±94
B2
54
464 ±80
C1
54
555 ±101
Overall
270
377 ±157
Table 1: BULATS-L1 dataset composition by CEFR level. Each level includes nine submissions from each of six L1 backgrounds. Transcript lengths are measured in words using the ASR generated transcripts. The values are reported as mean ± standard deviation.
Accuracy
Macro-F1
CEFR
Audio
Text
Audio
Text
A1
81.48
48.15
81.03
45.31
A2
83.33
87.04
81.31
87.06
B1
88.89
90.74
88.45
90.58
B2
92.59
94.44
92.46
94.30
C1
81.48
92.59
81.25
92.67
Table 2: Submission-level NLI accuracy and macro-F1 (%) on BULATS-L1 by CEFR level. Text results are unrestricted transcripts.
Figure 1: Effect of available input on audio- and text-based NLI
Len.
A1
A2
B1
B2
C1
All
100
42.59
61.11
62.96
77.78
55.56
60.00
200
51.85
81.48
85.19
90.74
81.48
78.15
300
48.15
83.33
92.59
94.44
90.74
81.85
Full
48.15
87.04
90.74
94.44
92.59
82.59
Table 3: Transcript-based NLI accuracy (%) by CEFR level and maximum transcript length in words.
Parts
Len
A1
A2
B1
B2
C1
All
P1
91
70.37
74.07
85.19
88.89
77.78
79.26
P1,3
166
72.22
81.48
85.19
90.74
81.48
82.22
P1,3,4
245
81.48
81.48
85.19
92.59
77.78
83.70
P1,3,4,5
377
81.48
83.33
88.89
92.59
81.48
85.56
Table 4: Audio NLI accuracy (%) on restricted audio parts by CEFR level and average length in words.
Figure 2: NLI accuracy on BULATS-L1 for linear fusion of audio and text predictive distributions
Language identification determines which language or languages appear in text or speech, including dialects, code-switching, and low-resource varieties.
Native language identification (NLI) infers a non-native speaker's first language (L1) from their second language (L2) production. We investigate a token probability approach that scores each text sample as a continuation of a prompt prefix specifying a candidate L1, then predicts the candidate with the highest score. We evaluate our approach using Llama 3.1 8B and 70B on three datasets: TOEFL9, Write and Improve 2024, and BULATS-L1. We compare its performance with previous prompting-based classification approaches. Token-probability scoring substantially improves on direct classification, particularly with the 8B model, while the 70B model achieves state-of-the-art performance across all datasets. Our results indicate that pretrained LLMs' token probabilities contain information useful for distinguishing L1 backgrounds from L2 English production.
Ahmet Yavuz Uluslu, Mark Gales, Kate Knill +1
ALTA Institute, Department of Engineering, University of Cambridge · Department of Computational Linguistics, University of Zurich · PRODAFT, Yverdon-les-Bains, Switzerland
Spoken language identification (LID) aims to recognize the target language regardless of accent. In practice, however, LID models fine-tuned from self-supervised speech representations frequently confuse accents with languages, misclassifying non-native (L2) speech as the speaker's first language (L1). We show that non-native speech representations lie between native target-language and native L1 poles, causing systematic misclassification. To address this, we introduce a geometric projection that estimates an L1-bias direction solely from native speech and removes it before the frozen LID head. Across five MMS-LID models and non-native corpora, this projection substantially improves target language identification for L2-accented speech while preserving predictions for native speech. These results show that accent-induced L1 bias can be corrected directly within the representation space without L2 training data or model adaptation.
Minu Kim, Jihwan Lee, David R. Mortensen +1
Signal Analysis and Interpretation Lab (SAIL), University of Southern California, USA · Language Technologies Institute, Carnegie Mellon University, USA
Increasingly, speech and language processing tasks take either audio or text directly rather than extracting features from these as the input to the classifier or regressor. Often these systems make use of complex, for example transformer-based, processes that have the ability to derive highly non-linear mappings between the input and the output. Unfortunately these systems can also learn ''shortcuts'' where the classifier is overly reliant on particular aspects of the input to yield the output. For the task of language proficiency assessment, this over-reliance can enable learners to increase their score by exploiting the shortcut rather than improving their ability. This paper introduces a novel training criterion that is able to reduce the classifier's reliance on shortcuts, thus for example limiting this option for malpractice in language assessment. This process is illustrated on two forms of assessment system, one based on the audio the other on the speech recognition text. The results show that, for both systems, there is higher correlations with features that could be exploited for malpractice than expected from the human reference, indicating an over-reliance on these features. By introducing the modified training criterion, this correlation can be reduced to be closer to the reference correlation.
Shilin Gao, Mark J. F. Gales, Kate M. Knill
ALTA Institute, Machine Intelligence Laboratory, Cambridge University Engineering Department Cambridge, UK