Text which has been translated from another language tends to carry with it evidence of translation\unicodex2014hence, it is often referred to as translationese. Multilingual large language models (MLLMs) generate text in a variety of languages. However, it is still unclear if MLLMs' generations resemble internal translation (from English or, potentially, other languages) and, thus, result in translationese. Here, we ask the following research questions: (1) Does text generated by MLLMs resemble translationese? (2) How does translationese produced by MLLMs differ from translationese produced through direct translation? We leverage established indicators of translated text to evaluate text generated by state-of-the-art MLLMs in five languages, comparing to both non-translated and human-written baselines in order to isolate translationese from other kinds of interference. Through the use of high-accuracy classification models, analyses of variance on individual linguistic features, and the collection of human annotations in a subset of two languages (German and Spanish), we assess the translationese content of MLLM generations and examine the key features that distinguish MLLM-generated text from typical translation-related interference.
Figures & tables
Figure 1: An example of translationese interference. This example shows a German-to-English translation with interference in part-of-speech n-gram distribution, which can be automatically detected with various machine learning models.
Language
Accuracy (%)
German
99.79
Spanish
99.81
Greek
99.95
Pashto
87.50
English
99.36
Table 1: SVM classification accuracy by language, in percentages.
Figure 3
Figure 4: Full results for Spanish. See Figure 3 for the complete dictionary of abbreviations.
Figure 5: Full results for Greek. See Figure 3 for the complete dictionary of abbreviations.
Figure 6: Full results for Pashto. See Figure 3 for the complete dictionary of abbreviations.
Figure 7: Distribution of the German function word auch . See Figure 3 for the complete dictionary of abbreviations.
Figure 8: Distribution of the German function word äußerst . See Figure 3 for the complete dictionary of abbreviations.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Language
Generator
Translator
German
Spanish
Greek
Pashto
English
Human*
N/A
3200
3200
2800
400
3200
Human
Human
3200
3200
2800
400
3200
Appendix
Table 2: Final distribution of each language’s training set, showing number of samples generated for each configuration/language pairing. Human* refers to a native speaker of the target language.
Configuration
Language
Generator
Translator
German
Spanish
Greek
Pashto
English
Human*
N/A
800
800
700
100
800
Human
Human
800
800
700
100
800
Human
Machine
800
800
700
100
800
Human
Gemini
800
800
700
100
800
Human
Llama
800
800
700
100
800
Appendix
Table 3: Final distribution of each language’s test set, showing number of samples generated for each configuration/language pairing. Human* refers to a native speaker of the target language.
Translator
German
Spanish
Greek
Pashto
English
Gemini
100
99.6
100
83.0
92.8
Llama
100
95.3
92.2
78.0
93.1
Appendix
Table 4: Results for each language of the LLM translation settings, with the values indicating percentage of samples classified as translated by the SVM. Gemini refers to text samples which were written by native human speakers and translated by the Gemini LLM, and Llama refers to text samples which were written by native human speakers and translated by the Llama LLM.
Configuration
Language
Generator
Translator
German
Spanish
English
Human*
N/A
1244
1276
3200
Human
Human
1244
1276
3200
Appendix
Table 5: Final distribution of each language’s training set for UdS experiments, showing number of samples generated for each configuration/language pairing. Human* refers to a native speaker of the target language.
Configuration
Language
Generator
Translator
German
Spanish
English
Human*
N/A
311
319
800
Human
Human
311
319
800
Human
Machine
311
319
0
Human
Gemini
311
319
0
Human
Llama
311
319
0
Appendix
Table 6: Final distribution of each language’s test set for UdS experiments, showing number of samples generated for each configuration/language pairing. Human* refers to a native speaker of the target language.
Language
Accuracy (%)
German
96.66
Spanish
88.52
English
94.16
Appendix
Table 7: The SVM’s classification accuracy by language in percentages, trained on the Europarl-UdS data.
Figure 9: Full results for English on the Europarl-UdS data. See Figure 3 for the complete dictionary of abbreviations.
Figure 10: Full results for German on the Europarl-UdS data. See Figure 3 for the complete dictionary of abbreviations.
Figure 11: Full results for Spanish on the Europarl-UdS data. See Figure 3 for the complete dictionary of abbreviations.
Translator
German
Spanish
Gemini
76.0
89.7
Llama
60.6
87.8
Appendix
Table 8: Results for each language of the LLM translation settings on the UdS dataset, with the values indicating percentage of samples classified as translated by the SVM. Gemini refers to text samples which were written by native human speakers and translated by the Gemini LLM, and Llama refers to text samples which were written by native human speakers and translated by the Llama LLM.
Figure 12: Distribution of the Spanish article el across configurations. See Figure 3 for the complete dictionary of abbreviations.
Figure 13: Distribution of the Spanish article la across configurations. See Figure 3 for the complete dictionary of abbreviations.
Figure 14: Distribution of the Spanish article las across configurations. See Figure 3 for the complete dictionary of abbreviations.
Figure 15: Distribution of the Spanish article los across configurations. See Figure 3 for the complete dictionary of abbreviations.