Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models

Yusheng Zhou; Eleanor Lin; David Jurgens

cs.CLSep 28, 2026

Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models

Authors: Yusheng Zhou, Eleanor Lin, David Jurgens

Organizations: Electrical Engineering and Computer Science Department University of Michigan Ann Arbor, MI 48105, USA

Abstract

Large language models (LLMs) are increasingly used by people whose first language is not English, yet these users have been shown to receive systematically lower-quality responses than fluent speakers. Which specific features of non-native English drive this gap remains unclear, because fluency is itself a composite of mechanical accuracy, vocabulary use, organization, and discourse coherence. Here, we introduce FABLE, a controlled dataset of 190,911 English prompt variants derived from 174K real user prompts for writing-related tasks. Evaluating responses from 34 open-weight LLMs, we find a clear asymmetry; while models do not propagate surface errors such as misspellings into their outputs, models do mirror higher-level rhetorical and lexical qualities present in the user's prompt. Further, the overall quality of responses differs substantially between the least- and most-fluent prompts. These results highlight a key LLM performance disparity for non-native English LLM users, resulting in both lower-quality and less-fluent answers.

Figures & tables

Figure 1: The top-10 task categories for Chinese prompts in WildChat and ShareLM, categorized using the taxonomy proposed by Chatterji et al. (2025) , shows that many real-world user prompts involve writing and information-seeking tasks rather than other questions. Bold denotes categories of prompts we use.

Dimension	Description
Content	Topic relevance and depth of idea development
Organization	Logical sequencing, coherence, and paragraph structure
Vocabulary	Lexical richness, word choice, and idiomatic usage
Language Use	Grammar and sentence construction
Mechanics	Spelling, punctuation, capitalization, and formatting

Table 1: The five dimensions in the adapted ESL Composition Profile ( Jacobs et al., 1981 ) .

Figure 2: Prompt rhetorical and lexical qualities do not exhibit a systematic impact on model responses’ surface form features. This figure presents the regression results between different linguistic properties of responses and different fluency metrics of prompts.

Figure 3: The prompt’s qualities significantly influence the LLM’s response qualities, shown by regression coefficients of the prompt’s rhetorical/lexical quality levels (cols) on the response’s (row).

Figure 4: Higher prompt fluency leads to better response quality, with Content and Mechanics showing larger positive correlation than Vocabulary and Language Use. Points represent regression coefficients, and error bars indicate 95% confidence intervals.

Figure 5: Different prompt fluency dimensions exhibit different impact on response quality in semantically matched comparisons. Points represent regression coefficients relating within-pair differences in each fluency dimension to differences in overall response quality.

Figure 6: Prompt fluency shows broadly consistent positive associations with response quality across task types, with variation in individual dimensions. Colors indicate coefficients from separate regressions within each task.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Prompt
1	Your foreign friend want to introduce Chinese food to classmate, send email ask about information. Please write a reply email, content include: 1. intoduction to Chinese food; 2. recommend a food and explain reason. Word count about 120.
2	Your foreign friend wants to introduce Chinese food to the class and sent an email asking for some information. Please write a reply email, including: 1. an introduction to Chinese food; 2. recommend a dish and explain the reason. Word count around 120.
4	Your foreign friend is going to introduce Chinese cuisine to classmates and has sent an email inquiring about relevant information. Please write a reply email to him, including: 1. An introduction to Chinese cuisine; 2. Recommend a dish and explain the reasons. Word count: around 120 words.

Appendix

Table 2: Example translations generated from the same Chinese prompt under different instructed levels. While the semantic intent remains largely preserved, the fluency shows a notable gap across different levels.

Figure 7: Standardized regression coefficients relating instructed proficiency levels to surface form features of translated prompts. Negative coefficients indicate fewer errors for error metrics; positive coefficients indicate greater reading ease for readability. Stars denote statistical significance: * $p<0.1$ , ** $p<0.05$ , *** $p<0.01$ .

Figure 8: Standardized regression coefficients relating the five instructed proficiency levels to judged proficiency scores of translated prompts. Results show both targeted and cross-dimensional associations. Stars denote statistical significance: * $p<0.1$ , ** $p<0.05$ , *** $p<0.01$ .

Instructed dimension	Correlation
Content	$-0.112$
Organization	$0.073$
Vocabulary	$-0.063$
Language Use	$-0.010$
Mechanics	$-0.015$

Appendix

Table 3: Pearson correlations between instructed proficiency levels and the same-intent score.

Model family	Model name
Nemotron	NVIDIA Nemotron Nano 9B v2
	Llama 3.3 Nemotron Super 49B v1.5
OLMo	OLMo 2 13B Instruct (1124)
	Olmo 3 7B Instruct
	Olmo 3.1 32B Instruct
Phi	Phi-4

Appendix

Table 4: The 34 evaluated models, grouped by model family. Version and release identifiers are retained where applicable. IT denotes instruction-tuned models.

Prompt Sensitivity

Prompt sensitivity is the extent to which changes in wording, format, or context alter an AI model’s outputs and measured performance.

See all 115 papers

Individual and Combined Effects of English as a Second Language and Typos on LLM Performance
Apr 6, 2026Serena Liu, Yutong Yang, Prisha Sheth +9Language Model Robustness LLM Evaluation
Harvard University, Boston, MA 02138, US
Multilingual Large Language Models do not comprehend all natural languages to equal degrees
Feb 23, 2026Natalia Moskvina, Raquel Montero, Masaya Yoshida +8Multilingual Language Models Multilingual Language Model Evaluation
Universitat Autònoma de Barcelona · Institució Catalana de Recerca i Estudis Avançats (ICREA) · Radboud University Nijmegen +1
English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training
Apr 14, 2026Mehak Dhaliwal, Shashwat Chaurasia, Yao Qin +2Cross-Lingual Reasoning in Language Models Low-Resource Language Processing
UC Santa Barbara · Amazon

Apr 6, 2026J/K move · Enter open · S save

Individual and Combined Effects of English as a Second Language and Typos on LLM Performance

Serena Liu, Yutong Yang, Prisha Sheth, Weixuan Dong+8

Harvard University, Boston, MA 02138, US

Language Model Robustness LLM Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) are used globally, and because much of their training data is in English, they typically perform best on English inputs. As a result, many non-native English speakers interact with them in English as a second language (ESL), and these inputs often contain typographical errors. Prior work has largely studied the effects of ESL variation and typographical errors separately, even though they often co-occur in real-world use. In this study, we use the Trans-EnV framework to transform standard English inputs into eight ESL variants and apply MulTypo to inject typos at three levels: low, moderate, and severe. We find that combining ESL variation and typos generally leads to larger performance drops than either factor alone, though the combined effect is not simply additive. This pattern is clearest on closed-ended tasks, where performance degradation can be characterized more consistently across ESL variants and typo levels, while results on open-ended tasks are more mixed. Overall, these findings suggest that evaluations on clean standard English may overestimate real-world model performance, and that evaluating ESL variation and typographical errors in isolation does not fully capture model behavior in realistic settings.

Abstract

Prompt Sensitivity

Explore similar work

Individual and Combined Effects of English as a Second Language and Typos on LLM Performance

Multilingual Large Language Models do not comprehend all natural languages to equal degrees

English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training