Text which has been translated from another language tends to carry with it evidence of translation\unicodex2014hence, it is often referred to as translationese. Multilingual large language models (MLLMs) generate text in a variety of languages. However, it is still unclear if MLLMs' generations resemble internal translation (from English or, potentially, other languages) and, thus, result in translationese. Here, we ask the following research questions: (1) Does text generated by MLLMs resemble translationese? (2) How does translationese produced by MLLMs differ from translationese produced through direct translation? We leverage established indicators of translated text to evaluate text generated by state-of-the-art MLLMs in five languages, comparing to both non-translated and human-written baselines in order to isolate translationese from other kinds of interference. Through the use of high-accuracy classification models, analyses of variance on individual linguistic features, and the collection of human annotations in a subset of two languages (German and Spanish), we assess the translationese content of MLLM generations and examine the key features that distinguish MLLM-generated text from typical translation-related interference.
Figures & tables
Figure 1: An example of translationese interference. This example shows a German-to-English translation with interference in part-of-speech n-gram distribution, which can be automatically detected with various machine learning models.
Language
Accuracy (%)
German
99.79
Spanish
99.81
Greek
99.95
Pashto
87.50
English
99.36
Table 1: SVM classification accuracy by language, in percentages.
Figure 3
Figure 4: Full results for Spanish. See Figure 3 for the complete dictionary of abbreviations.
Figure 5: Full results for Greek. See Figure 3 for the complete dictionary of abbreviations.
Figure 6: Full results for Pashto. See Figure 3 for the complete dictionary of abbreviations.
Figure 7: Distribution of the German function word auch . See Figure 3 for the complete dictionary of abbreviations.
Figure 8: Distribution of the German function word äußerst . See Figure 3 for the complete dictionary of abbreviations.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Language
Generator
Translator
German
Spanish
Greek
Pashto
English
Human*
N/A
3200
3200
2800
400
3200
Human
Human
3200
3200
2800
400
3200
Appendix
Table 2: Final distribution of each language’s training set, showing number of samples generated for each configuration/language pairing. Human* refers to a native speaker of the target language.
Configuration
Language
Generator
Translator
German
Spanish
Greek
Pashto
English
Human*
N/A
800
800
700
100
800
Human
Human
800
800
700
100
800
Human
Machine
800
800
700
100
800
Human
Gemini
800
800
700
100
800
Human
Llama
800
800
700
100
800
Appendix
Table 3: Final distribution of each language’s test set, showing number of samples generated for each configuration/language pairing. Human* refers to a native speaker of the target language.
Translator
German
Spanish
Greek
Pashto
English
Gemini
100
99.6
100
83.0
92.8
Llama
100
95.3
92.2
78.0
93.1
Appendix
Table 4: Results for each language of the LLM translation settings, with the values indicating percentage of samples classified as translated by the SVM. Gemini refers to text samples which were written by native human speakers and translated by the Gemini LLM, and Llama refers to text samples which were written by native human speakers and translated by the Llama LLM.
Configuration
Language
Generator
Translator
German
Spanish
English
Human*
N/A
1244
1276
3200
Human
Human
1244
1276
3200
Appendix
Table 5: Final distribution of each language’s training set for UdS experiments, showing number of samples generated for each configuration/language pairing. Human* refers to a native speaker of the target language.
Configuration
Language
Generator
Translator
German
Spanish
English
Human*
N/A
311
319
800
Human
Human
311
319
800
Human
Machine
311
319
0
Human
Gemini
311
319
0
Human
Llama
311
319
0
Appendix
Table 6: Final distribution of each language’s test set for UdS experiments, showing number of samples generated for each configuration/language pairing. Human* refers to a native speaker of the target language.
Language
Accuracy (%)
German
96.66
Spanish
88.52
English
94.16
Appendix
Table 7: The SVM’s classification accuracy by language in percentages, trained on the Europarl-UdS data.
Figure 9: Full results for English on the Europarl-UdS data. See Figure 3 for the complete dictionary of abbreviations.
Figure 10: Full results for German on the Europarl-UdS data. See Figure 3 for the complete dictionary of abbreviations.
Figure 11: Full results for Spanish on the Europarl-UdS data. See Figure 3 for the complete dictionary of abbreviations.
Translator
German
Spanish
Gemini
76.0
89.7
Llama
60.6
87.8
Appendix
Table 8: Results for each language of the LLM translation settings on the UdS dataset, with the values indicating percentage of samples classified as translated by the SVM. Gemini refers to text samples which were written by native human speakers and translated by the Gemini LLM, and Llama refers to text samples which were written by native human speakers and translated by the Llama LLM.
Figure 12: Distribution of the Spanish article el across configurations. See Figure 3 for the complete dictionary of abbreviations.
Figure 13: Distribution of the Spanish article la across configurations. See Figure 3 for the complete dictionary of abbreviations.
Figure 14: Distribution of the Spanish article las across configurations. See Figure 3 for the complete dictionary of abbreviations.
Figure 15: Distribution of the Spanish article los across configurations. See Figure 3 for the complete dictionary of abbreviations.
Machine-translated data is widely used in multilingual NLP, particularly where native text is scarce. However, translated text differs systematically from native text. This phenomenon is known as translationese, and it reflects both traces of the source language and characteristic properties of translation itself. In this paper, we study how training on machine-translated data affects small English language models, focusing on how translationese from different source languages shapes linguistic acceptability judgments and language modeling for different domains. We train models on English text translated from 24 typologically and resource-diverse source languages, enabling a systematic analysis of how source language and corpus properties influence what models learn. Our results show that the source language has a clear impact on model behavior: general perplexity is more driven by the lexical diversity of the translated corpus, but grammatical performance is strongly correlated to typological similarity to English if trained on enough data. Even translation quality is a strong predictor of language modeling performance.
Jenny Kunz
Department of Computer and Information Science Linköping University
Multilingual large language models (mLLMs) achieve strong performance in machine translation, yet our understanding of the mechanisms by which they transform representations from one language to another remains incomplete. Prior work suggests that translation decomposes into separable processes within an mLLM, where conceptual content is first represented independently, followed by a production into language-specific form. In this work, we show that translation is even more modular than previously assumed and that the output language production in translation processes is actually further separable into a syntax and a surface language process. We construct controlled multilingual datasets that isolate cross-linguistic differences in word-order and use causal interventions and probing to track how representations are transformed during translation. We find that models first construct target-side word-order before realizing the target language surface form. We identify individual attention heads that are selectively sensitive to syntactic transformations while remaining largely invariant to language identity. These results establish the commitment to a syntactic structure as an independent stage in translation, extending prior decompositions and showing how translation is implemented by functionally different components within mLLMs.
Mikhail Sonkin, Tanja Baeumel, Daniil Gurgurov +2
Saarland University · University of Göttingen · German Research Center for Artificial Intelligence (DFKI) +1
Translations often carry traces of the source language, a phenomenon known as translationese. We introduce the first freely available English-to-Swedish dataset contrasting translationese sentences with idiomatic alternatives, designed to probe intrinsic preferences of language models. It includes error tags and descriptions of the problems in the original translations. In experiments evaluating smaller Swedish and multilingual LLMs with our dataset, we find that they often favor the translationese phrasing. Human alternatives are chosen more often when the English source sentence is omitted, indicating that exposure to the source biases models toward literal translations, although even without context models often prefer the translationese variant. Our dataset and findings provide a resource and benchmark for developing models that produce more natural, idiomatic output in non-English languages.
Jenny Kunz, Anja Jarochenko, Marcel Bollmann
Department of Computer and Information Science, Linköping University