Inferring L1 Backgrounds from L2 English with LLM Token Probabilities

Ahmet Yavuz Uluslu; Mark Gales; Kate Knill; Gerold Schneider

cs.CLOct 16, 2026

Inferring L1 Backgrounds from L2 English with LLM Token Probabilities

Authors: Ahmet Yavuz Uluslu, Mark Gales, Kate Knill, Gerold Schneider

Organizations: ALTA Institute, Department of Engineering, University of Cambridge · Department of Computational Linguistics, University of Zurich · PRODAFT, Yverdon-les-Bains, Switzerland

Abstract

Native language identification (NLI) infers a non-native speaker's first language (L1) from their second language (L2) production. We investigate a token probability approach that scores each text sample as a continuation of a prompt prefix specifying a candidate L1, then predicts the candidate with the highest score. We evaluate our approach using Llama 3.1 8B and 70B on three datasets: TOEFL9, Write and Improve 2024, and BULATS-L1. We compare its performance with previous prompting-based classification approaches. Token-probability scoring substantially improves on direct classification, particularly with the 8B model, while the 70B model achieves state-of-the-art performance across all datasets. Our results indicate that pretrained LLMs' token probabilities contain information useful for distinguishing L1 backgrounds from L2 English production.

Figures & tables

Figure 1: Accuracy as a function of input length for Llama 3.1 8B and 70B using language-prefix likelihood scoring (solid lines) and direct classification (dashed lines). Panels show results on Write & Improve 2024, TOEFL9, and BULATS-L1.

Dataset	Previous SoTA	Our approach
TOEFL9	95.1% ( Zhang and Salle, 2023 )	96.0%
W&I 2024	88.9% ( Uluslu et al., 2026a )	92.0%
BULATS-L1	83.3% ( Uluslu et al., 2026b )	84.1%

Table 1: Full-text accuracy compared with previously reported results. Our approach uses L1-prefix probability scoring with Llama 3.1 70B.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Sentence-level ranking and word-level log-probability deviations from the nine-L1 mean (in nats). The upper panel sorts candidate L1s by total sentence deviation ( $D_{l}$ ), and the lower panel shows word-level deviations ( $D_{l,j}$ ).

Can We Still Trace L1 Signals? Investigating the Resilience of Native Language Signals in the LLM Era
Mar 20, 2026Nabelanita Utami, Ryohei Sasano
Graduate School of Informatics, Nagoya University
Acoustic and Lexical Signals for Native Language Identification of Language Learners
Oct 23, 2026Ahmet Yavuz Uluslu, Mengjie Qian, Kate Knill +1Language Identification Speech Processing
ALTA Institute, Department of Engineering, University of Cambridge · Department of Computational Linguistics, University of Zurich
Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models
Sep 28, 2026Yusheng Zhou, Eleanor Lin, David JurgensPrompt Sensitivity Language Model Generation Evaluation
Electrical Engineering and Computer Science Department University of Michigan Ann Arbor, MI 48105, USA

Mar 20, 2026J/K move · Enter open · S save

Can We Still Trace L1 Signals? Investigating the Resilience of Native Language Signals in the LLM Era

Nabelanita Utami, Ryohei Sasano

Graduate School of Informatics, Nagoya University

Open paper AskPDF arXiv

The widespread use of LLM-based writing assistance has raised an interesting question about the homogenization of English. As LLMs tend to revise texts toward mainstream English conventions reflected in their training data, the subtle fingerprints that reflect an author's native language (L1) may be gradually disappearing. This study investigates this phenomenon by analyzing native language identification (NLI) performance on academic abstracts. To this end, we construct two NLI datasets of academic abstracts extracted from arXiv and the ACL Anthology that covers eight native language groups across three time periods: pre-neural network (NN), pre-LLM, and post-LLM. We then evaluate NLI performance for each era using NLI classifiers obtained by fine-tuning LLMs. The results reveal a consistent decline in NLI performance over time. Notably, however, the decline is more pronounced from the pre-NN era to the pre-LLM era than from the pre-LLM era to the post-LLM era. This suggests that although academic English appears to have become increasingly homogenized in the LLM era, this homogenization did not suddenly emerge with the advent of LLMs; rather, it has progressed gradually since the emergence of neural approaches to language processing. Furthermore, a rewriting experiment using recent LLMs shows a larger NLI performance drop than the progression across eras alone, suggesting that increased LLM use may lead to further homogenization in the future.

Abstract

Explore similar work

Can We Still Trace L1 Signals? Investigating the Resilience of Native Language Signals in the LLM Era

Acoustic and Lexical Signals for Native Language Identification of Language Learners

Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models