Translations often carry traces of the source language, a phenomenon known as translationese. We introduce the first freely available English-to-Swedish dataset contrasting translationese sentences with idiomatic alternatives, designed to probe intrinsic preferences of language models. It includes error tags and descriptions of the problems in the original translations. In experiments evaluating smaller Swedish and multilingual LLMs with our dataset, we find that they often favor the translationese phrasing. Human alternatives are chosen more often when the English source sentence is omitted, indicating that exposure to the source biases models toward literal translations, although even without context models often prefer the translationese variant. Our dataset and findings provide a resource and benchmark for developing models that produce more natural, idiomatic output in non-English languages.
Figures & tables
Figure 1: Error tag distribution for the Swedish translations; cf. Sec. 3.2 for an explanation of tags.
Figure 2: Prompting setups. Each box shows a minimal pair : Sentence 1 is a translationese variant, sentence 2 is an idiomatic variant of a translation of the same English sentence. We compute the perplexity of each variant to determine which one the model prefers. Translations of text in the figure: * Translate the following sentence to Swedish. † Translate the following sentence to Swedish, considering the context.
Model
Human > OPUS
Human > GPT-5
SGB all
SGB filtered
Acc.
Δ LP
Acc.
Δ LP
Acc.
Δ LP
Acc.
Δ LP
LLaMA-3-8B
47.83
1.83
40.33
0.91
38.57
-3.01
56.71
7.13
LLaMA-3-8B-it
49.83
2.01
42.50
1.17
40.33
-2.65
58.95
8.47
ai.se-LLaMA-8B
56.67
5.17
44.50
2.36
49.01
3.67
81.34
16.56
ai.se-LLaMA-8B-it
52.83
3.36
43.50
1.47
34.12
-5.80
58.95
5.60
EuroLLM-1.7B
48.33
1.09
41.17
0.83
41.46
-2.01
58.06
4.70
Table 2: Preferences for prompts without translation context for human alternatives over OPUS or GPT translations. We compare to the datasets by Kunz (2026) derived from Katourgi (2022) ( SGB ).
Table 3: Preferences for machine translations vs. human alternatives for prompts with translation context, with 0–10 preceding sentences from the source document. Highest scores for each model are bold.
Figure 3: Percentage of samples where models prefer the OPUS-translated sentence over the human alternative, by error tag. See Section 3.2 for explanations of error tags.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Machine-translated data is widely used in multilingual NLP, particularly where native text is scarce. However, translated text differs systematically from native text. This phenomenon is known as translationese, and it reflects both traces of the source language and characteristic properties of translation itself. In this paper, we study how training on machine-translated data affects small English language models, focusing on how translationese from different source languages shapes linguistic acceptability judgments and language modeling for different domains. We train models on English text translated from 24 typologically and resource-diverse source languages, enabling a systematic analysis of how source language and corpus properties influence what models learn. Our results show that the source language has a clear impact on model behavior: general perplexity is more driven by the lexical diversity of the translated corpus, but grammatical performance is strongly correlated to typological similarity to English if trained on enough data. Even translation quality is a strong predictor of language modeling performance.
Jenny Kunz
Department of Computer and Information Science Linköping University
Translated texts exhibit systematic differences from comparable texts originally written in the target language. Explaining this phenomenon, commonly known as translationese, remains an open challenge. Translationese has been attributed to production tendencies (e.g. interference, simplification), socio-cultural variables, and language-pair effects, yet a unified explanatory account is lacking. We investigate the hypothesis that translationese is a response to the cognitive load inherent in the translation task. We test whether observable translationese can be predicted from quantifiable measures of translation task difficulty. Translationese is measured as a segment-level probability of being a translation produced by an automatic classifier (translatedness score). Translation task difficulty includes source-text and cross-lingual transfer components. They are captured by information-theoretic metrics based on LLM surprisal and by established syntactic and semantic alternatives. We use a bidirectional English-German corpus comprising written and spoken subcorpora. Results indicate that translationese can only be explained in part by translation task difficulty, especially in the English-to-German direction. For most experiments, cross-lingual transfer difficulty contributes more than source-text complexity. Information-theoretic indicators match or outperform traditional features in written mode, but offer no advantage in spoken mode. Source-text syntactic complexity and translation-solution entropy emerged as the strongest predictors of translationese across language pairs and modes.
Maria Kunilovskaya
Department of Language Science and Technology, Saarland University
We present a crowdsourced dataset for Piedmontese, an endangered Romance language of northwestern Italy. The dataset comprises 145 Italian-Piedmontese parallel sentences derived from Flores+, with translations produced by speakers writing in their natural orthographic style rather than adhering to standardized conventions, along with manual word alignment. We use this resource to benchmark several large language models on tokenization parity, topic classification, and machine translation. Our analysis reveals that Piedmontese incurs a tokenization penalty relative to higher-resource Romance languages, yet LLMs achieve classification performance approaching that of Italian, French, and English. Machine translation results are asymmetric: models translate adequately from Piedmontese into high-resource languages, but generation into Piedmontese remains challenging. The dataset and code are publicly released.
Gianluca Vico, Jindřich Libovický
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Praha, Czech Republic