Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction

Jackson Petty; Jaulie Goe; Tal Linzen

cs.CLApr 8, 2026

Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction

Authors: Jackson Petty, Jaulie Goe, Tal Linzen

Organizations: Department of Linguisticsλ New York University · Center for Data Scienceδ New York University

Abstract

Low-resource languages pose a challenge for machine translation with large language models (LLMs), which require large amounts of training data. One potential way to circumvent this data dependence is to rely on LLMs' ability to use in-context descriptions of languages, like textbooks and dictionaries. To do so, LLMs must be able to infer the link between the languages' grammatical descriptions and the sentences in question. Here we isolate this skill using a formal analogue of the task: string transduction based on a formal grammar provided in-context. We construct synchronous context-free grammars which define pairs of formal languages designed to model particular aspects of natural language grammar, morphology, and written representation. Using these grammars, we measure how well LLMs can translate sentences from one formal language into another when given both the grammar and the source-language sentence. We vary the size of the grammar, the lengths of the sentences, the syntactic and morphological properties of the languages, and their written script. We note three key findings. First, LLMs' translation accuracy decreases markedly as a function of grammar size and sentence length. Second, differences in morphology and written representation between the source and target languages can strongly diminish model performance. Third, we examine the types of errors committed by models and find they are most prone to recall the wrong words from the target language vocabulary, hallucinate new words, or leave source-language words untranslated.

Figures & tables

\begin{Bmatrix}[l]\mathrm{S}\to\langle\textrm{NP}\,\textrm{VP},\textrm{NP}\,\textrm{VP}\rangle\\ \textrm{VP}\to\langle\textrm{VB}\,\textrm{NP},\textrm{NP}\,\textrm{VB}\rangle\\ \textrm{NP}\to\langle\texttt{I},\texttt{watashi wa}\rangle\\ \textrm{NP}\to\langle\texttt{the box},\texttt{hako wo}\rangle\\ \textrm{VB}\to\langle\texttt{open},\texttt{akemasu}\rangle\\ \end{Bmatrix}\quad\sim\quad\begin{matrix}[l]\textrm{EN:}&\texttt{I open the box}\\ \textrm{JP:}&\texttt{watashi wa hako wo akemasu}\end{matrix}

Figure 1: ( left ) A small SCFG defining fragments of English and Japanese using production rules, from which is sampled ( right ) a corresponding pair of English and Japanese sentences. Adapted from Chiang (2021) .

Language	Word Order	Example Sentence
English	S V O	the jaguar ate the man
Basque	S O V	jaguarrak gizona jan du Personal communication
Hixkaryana	O V S	toto yonoye kamara Derbyshire 1977, p. 593

Table 1: Languages can differ in the ordering between subjects , verbs , and objects .

	NoAgr $\to$ NoAgr	Agr $\to$ NoAgr	Agr $\to$ Agr	NoAgr $\to$ Agr
1sg	na lam $\to$ ni tor	na lam mi $\to$ ni tor	na lam mi $\to$ ni tor ik	na lam $\to$ ni tor ik
3sg	sa lam $\to$ su tor	sa lam su $\to$ su tor	sa lam su $\to$ su tor o	sa lam $\to$ su tor o
3pl	ran lam $\to$ ren tor	ran lam sar $\to$ ren tor	ran lam sar $\to$ ren tor on	ran lam $\to$ ren tor on

Table 2: Different person/number agreement paradigms between the source and target languages; verbal suffixes in orange optionally reflect the person (1st or 3rd) and number (singular or plural) of the pronoun .

Table 3: Examples of target sentence fragments in the various scripts.

Figure 3: Mean accuracy on size (all models) and word order, morphology, and orthography ( gpt-5 only) experiments; error bars show 95% confidence intervals. We find that grammar size and sentence length, a language’s morphology type, and its orthography impact translation accuracy, while its basic word order does not.

Figure 4: Error distributions for gpt-5 by experiment. Error categories are not mutually exclusive. The most frequent error types are source vocabulary words leakage, recall errors , omission of words, and in the case of the orthography experiment, hallucinating vocabulary words and translating words in the wrong orthography .

Table 4: Examples of the various error types; words of note are highlighted in red, with specific characters (Devanagari GA U+0917 ; Abkhazian HA U+04A9 ) bolded.

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Full results for grammar size & sentence length experiment. Error bars show 95% confidence interval.

Model	Metric	57	77	117	837	4,037	6,037	8,037
gpt-5	Exact Match	$0.967$	$0.967$	$0.979$	$0.912$	$0.750$	$0.688$	$0.487$
	Bag of Words	$0.967$	$0.967$	$0.983$	$0.954$	$0.842$	$0.762$	$0.558$
	$\BLEU$	$0.994$	$0.995$	$0.995$	$0.970$	$0.910$	$0.878$	$0.780$
	$\chrFPP$	$0.998$	$0.998$	$0.998$	$0.992$	$0.970$	$0.952$	$0.909$
gpt-5-mini	Exact Match	$0.975$	$0.925$	$0.946$	$0.575$	$0.083$	$0.029$	$0.013$
	Bag of Words	$0.975$	$0.929$	$0.958$	$0.804$	$0.133$	$0.042$	$0.021$

Appendix

Table 5: Mean results by grammar size for all models in the size experiment.

Model	Metric	5.5	10.5	15.5	20.5	36.5
gpt-5	Exact Match	$0.950$	$0.859$	$0.801$	$0.753$	$0.734$
	Bag of Words	$0.955$	$0.899$	$0.846$	$0.808$	$0.793$
	$\BLEU$	$0.977$	$0.941$	$0.932$	$0.910$	$0.894$
	$\chrFPP$	$0.990$	$0.978$	$0.972$	$0.967$	$0.962$
gpt-5-mini	Exact Match	$0.615$	$0.495$	$0.488$	$0.459$	$0.467$
	Bag of Words	$0.640$	$0.557$	$0.548$	$0.506$	$0.502$

Appendix

Table 6: Mean results by input string length for all models in the size experiment.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
SVO → SVO	Exact Match	$0.996$	$1.000$	$0.988$	$0.858$	$0.471$	$0.338$	$0.246$
	Bag of Words	$0.996$	$1.000$	$0.988$	$0.858$	$0.471$	$0.338$	$0.246$
	$\BLEU$	$0.999$	$1.000$	$0.997$	$0.972$	$0.877$	$0.783$	$0.714$
	$\chrFPP$	$1.000$	$1.000$	$0.999$	$0.986$	$0.925$	$0.875$	$0.839$
SVO → SOV	Exact Match	$0.963$	$0.971$	$0.946$	$0.879$	$0.529$	$0.338$	$0.212$
	Bag of Words	$0.967$	$0.971$	$0.946$	$0.912$	$0.537$	$0.350$	$0.225$

Appendix

Table 7: Mean results for gpt-5 in the word order experiment, grouped by target word-order condition and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
SVO → SVO	Exact Match	$1.000$	$1.000$	$0.971$	$0.700$	$0.071$	$0.017$	$0.008$
	Bag of Words	$1.000$	$1.000$	$0.971$	$0.700$	$0.071$	$0.017$	$0.008$
	$\BLEU$	$1.000$	$1.000$	$0.996$	$0.933$	$0.566$	$0.382$	$0.261$
	$\chrFPP$	$1.000$	$1.000$	$0.998$	$0.966$	$0.732$	$0.614$	$0.510$
SVO → SOV	Exact Match	$0.988$	$0.983$	$0.967$	$0.550$	$0.058$	$0.021$	$0.017$
	Bag of Words	$0.988$	$0.988$	$0.971$	$0.733$	$0.058$	$0.025$	$0.017$

Appendix

Table 8: Mean results for gpt-5-mini in the word order experiment, grouped by target word-order condition and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
SVO → SVO	Exact Match	$0.892$	$0.696$	$0.650$	$0.125$	$0.004$	$0.000$	$0.000$
	Bag of Words	$0.896$	$0.696$	$0.650$	$0.125$	$0.004$	$0.000$	$0.000$
	$\BLEU$	$0.969$	$0.923$	$0.878$	$0.446$	$0.130$	$0.069$	$0.049$
	$\chrFPP$	$0.981$	$0.961$	$0.937$	$0.602$	$0.288$	$0.210$	$0.170$
SVO → SOV	Exact Match	$0.121$	$0.121$	$0.121$	$0.013$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.804$	$0.654$	$0.571$	$0.133$	$0.000$	$0.004$	$0.004$

Appendix

Table 9: Mean results for gpt-5-nano in the word order experiment, grouped by target word-order condition and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
SVO → SVO	Exact Match	$0.371$	$0.354$	$0.158$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.392$	$0.362$	$0.171$	$0.000$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.761$	$0.756$	$0.566$	$0.066$	$0.033$	$0.024$	$0.026$
	$\chrFPP$	$0.856$	$0.853$	$0.735$	$0.283$	$0.145$	$0.155$	$0.137$
SVO → SOV	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.412$	$0.400$	$0.163$	$0.000$	$0.000$	$0.000$	$0.000$

Appendix

Table 10: Mean results for gemma-3-12b-it in the word order experiment, grouped by target word-order condition and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
SVO → SVO	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.006$	$0.013$	$0.002$	$0.001$	$0.002$	$0.001$	$0.001$
	$\chrFPP$	$0.116$	$0.106$	$0.103$	$0.095$	$0.100$	$0.094$	$0.083$
SVO → SOV	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.000$	$0.004$	$0.000$	$0.000$	$0.000$	$0.000$

Appendix

Table 11: Mean results for gemma-3-4b-it in the word order experiment, grouped by target word-order condition and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
SVO → SVO	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	—	—	—
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	—	—	—
	$\BLEU$	$0.000$	$0.001$	$0.000$	$0.001$	—	—	—
	$\chrFPP$	$0.101$	$0.091$	$0.102$	$0.101$	—	—	—
SVO → SOV	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	—	—	—
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	—	—	—

Appendix

Table 12: Mean results for gemma-3-1b-it in the word order experiment, grouped by target word-order condition and grammar size.

Condition	Metric	6	10.5	15.5	20.5	32.5
SVO → SVO	Exact Match	$0.822$	$0.752$	$0.663$	$0.722$	$0.538$
	Bag of Words	$0.822$	$0.752$	$0.663$	$0.722$	$0.538$
	$\BLEU$	$0.922$	$0.916$	$0.894$	$0.927$	$0.871$
	$\chrFPP$	$0.952$	$0.954$	$0.939$	$0.960$	$0.926$
SVO → SOV	Exact Match	$0.827$	$0.733$	$0.670$	$0.633$	$0.565$
	Bag of Words	$0.832$	$0.733$	$0.679$	$0.655$	$0.579$

Appendix

Table 13: Mean results for gpt-5 in the word order experiment, grouped by target word-order condition and input string length.

Condition	Metric	6	10.5	15.5	20.5	32.5
SVO → SVO	Exact Match	$0.611$	$0.578$	$0.500$	$0.557$	$0.447$
	Bag of Words	$0.611$	$0.578$	$0.500$	$0.557$	$0.447$
	$\BLEU$	$0.755$	$0.774$	$0.708$	$0.771$	$0.664$
	$\chrFPP$	$0.832$	$0.859$	$0.820$	$0.861$	$0.784$
SVO → SOV	Exact Match	$0.578$	$0.539$	$0.497$	$0.482$	$0.450$
	Bag of Words	$0.601$	$0.570$	$0.527$	$0.507$	$0.483$

Appendix

Table 14: Mean results for gpt-5-mini in the word order experiment, grouped by target word-order condition and input string length.

Condition	Metric	6	10.5	15.5	20.5	32.5
SVO → SVO	Exact Match	$0.413$	$0.382$	$0.313$	$0.332$	$0.252$
	Bag of Words	$0.413$	$0.385$	$0.313$	$0.332$	$0.252$
	$\BLEU$	$0.583$	$0.534$	$0.467$	$0.496$	$0.395$
	$\chrFPP$	$0.688$	$0.643$	$0.567$	$0.581$	$0.485$
SVO → SOV	Exact Match	$0.223$	$0.027$	$0.003$	$0.000$	$0.000$
	Bag of Words	$0.425$	$0.367$	$0.275$	$0.255$	$0.210$

Appendix

Table 15: Mean results for gpt-5-nano in the word order experiment, grouped by target word-order condition and input string length.

Condition	Metric	6	10.5	15.5	20.5	32.5
SVO → SVO	Exact Match	$0.184$	$0.137$	$0.115$	$0.129$	$0.066$
	Bag of Words	$0.193$	$0.146$	$0.120$	$0.132$	$0.069$
	$\BLEU$	$0.335$	$0.332$	$0.310$	$0.339$	$0.278$
	$\chrFPP$	$0.464$	$0.470$	$0.437$	$0.473$	$0.417$
SVO → SOV	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.201$	$0.164$	$0.129$	$0.109$	$0.081$

Appendix

Table 16: Mean results for gemma-3-12b-it in the word order experiment, grouped by target word-order condition and input string length.

Condition	Metric	6	10.5	15.5	20.5	32.5
SVO → SVO	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.004$	$0.003$	$0.006$	$0.002$	$0.002$
	$\chrFPP$	$0.080$	$0.097$	$0.106$	$0.105$	$0.108$
SVO → SOV	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.003$	$0.000$	$0.000$	$0.000$	$0.000$

Appendix

Table 17: Mean results for gemma-3-4b-it in the word order experiment, grouped by target word-order condition and input string length.

Condition	Metric	6	10.5	15.5	20.5	32.5
SVO → SVO	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.000$	$0.001$	$0.001$	$0.001$	$0.001$
	$\chrFPP$	$0.077$	$0.095$	$0.104$	$0.107$	$0.111$
SVO → SOV	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$

Appendix

Table 18: Mean results for gemma-3-1b-it in the word order experiment, grouped by target word-order condition and input string length.

Figure 6: Full results for gpt-5 on the word order experiment. Error bars show 95% confidence interval.

Figure 7: Full results for gpt-5-mini on the word order experiment. Error bars show 95% confidence interval.

Figure 8: Full results for gpt-5-nano on the word order experiment. Error bars show 95% confidence interval.

Figure 9: Full results for gemma-3-12b-it on the word order experiment. Error bars show 95% confidence interval.

Figure 10: Full results for gemma-3-4b-it on the word order experiment. Error bars show 95% confidence interval.

Figure 11: Full results for gemma-3-1b-it on the word order experiment. Error bars show 95% confidence interval.

Condition	Metric	25	50	100	1,000	5,000	7,500	10,000
NoAgr → NoAgr	Exact Match	$0.996$	$1.000$	$1.000$	$0.967$	$0.796$	$0.796$	$0.729$
	Bag of Words	$0.996$	$1.000$	$1.000$	$0.967$	$0.796$	$0.796$	$0.729$
	$\BLEU$	$0.999$	$1.000$	$1.000$	$0.993$	$0.962$	$0.958$	$0.953$
	$\chrFPP$	$1.000$	$1.000$	$1.000$	$0.996$	$0.979$	$0.977$	$0.973$
Agr → NoAgr	Exact Match	$0.400$	$0.421$	$0.421$	$0.312$	$0.346$	$0.371$	$0.358$
	Bag of Words	$0.400$	$0.421$	$0.421$	$0.312$	$0.346$	$0.371$	$0.358$

Appendix

Table 19: Mean results for gpt-5 in the morphology experiment, grouped by agreement condition and grammar size.

Condition	Metric	25	50	100	1,000	5,000	7,500	10,000
NoAgr → NoAgr	Exact Match	$0.996$	$1.000$	$0.996$	$0.975$	$0.804$	$0.787$	$0.683$
	Bag of Words	$0.996$	$1.000$	$0.996$	$0.975$	$0.804$	$0.787$	$0.683$
	$\BLEU$	$0.999$	$1.000$	$1.000$	$0.995$	$0.961$	$0.956$	$0.936$
	$\chrFPP$	$1.000$	$1.000$	$1.000$	$0.997$	$0.979$	$0.977$	$0.963$
Agr → NoAgr	Exact Match	$0.400$	$0.421$	$0.421$	$0.312$	$0.325$	$0.367$	$0.354$
	Bag of Words	$0.400$	$0.421$	$0.421$	$0.312$	$0.325$	$0.367$	$0.354$

Appendix

Table 20: Mean results for gpt-5-mini in the morphology experiment, grouped by agreement condition and grammar size.

Condition	Metric	25	50	100	1,000	5,000	7,500	10,000
NoAgr → NoAgr	Exact Match	$0.887$	$0.904$	$0.871$	$0.829$	$0.346$	$0.450$	$0.300$
	Bag of Words	$0.887$	$0.904$	$0.871$	$0.829$	$0.346$	$0.450$	$0.300$
	$\BLEU$	$0.970$	$0.980$	$0.974$	$0.964$	$0.787$	$0.849$	$0.747$
	$\chrFPP$	$0.984$	$0.988$	$0.986$	$0.981$	$0.895$	$0.921$	$0.857$
Agr → NoAgr	Exact Match	$0.346$	$0.346$	$0.346$	$0.271$	$0.246$	$0.188$	$0.229$
	Bag of Words	$0.346$	$0.350$	$0.346$	$0.271$	$0.246$	$0.188$	$0.229$

Appendix

Table 21: Mean results for gpt-5-nano in the morphology experiment, grouped by agreement condition and grammar size.

Condition	Metric	5.5	10.5	15	19.5	33
NoAgr → NoAgr	Exact Match	$0.948$	$0.914$	$0.909$	$0.846$	$0.872$
	Bag of Words	$0.948$	$0.914$	$0.909$	$0.846$	$0.872$
	$\BLEU$	$0.981$	$0.979$	$0.980$	$0.979$	$0.985$
	$\chrFPP$	$0.989$	$0.988$	$0.990$	$0.988$	$0.991$
Agr → NoAgr	Exact Match	$0.573$	$0.442$	$0.363$	$0.267$	$0.177$
	Bag of Words	$0.573$	$0.442$	$0.363$	$0.267$	$0.177$

Appendix

Table 22: Mean results for gpt-5 in the morphology experiment, grouped by agreement condition and input string length.

Condition	Metric	5.5	10.5	15	19.5	33
NoAgr → NoAgr	Exact Match	$0.951$	$0.906$	$0.905$	$0.830$	$0.869$
	Bag of Words	$0.951$	$0.906$	$0.905$	$0.830$	$0.869$
	$\BLEU$	$0.981$	$0.976$	$0.980$	$0.973$	$0.982$
	$\chrFPP$	$0.990$	$0.986$	$0.990$	$0.985$	$0.990$
Agr → NoAgr	Exact Match	$0.573$	$0.436$	$0.363$	$0.259$	$0.170$
	Bag of Words	$0.573$	$0.436$	$0.363$	$0.259$	$0.170$

Appendix

Table 23: Mean results for gpt-5-mini in the morphology experiment, grouped by agreement condition and input string length.

Condition	Metric	5.5	10.5	15	19.5	33
NoAgr → NoAgr	Exact Match	$0.858$	$0.678$	$0.614$	$0.528$	$0.583$
	Bag of Words	$0.858$	$0.678$	$0.614$	$0.528$	$0.583$
	$\BLEU$	$0.933$	$0.891$	$0.883$	$0.873$	$0.896$
	$\chrFPP$	$0.963$	$0.942$	$0.941$	$0.933$	$0.944$
Agr → NoAgr	Exact Match	$0.533$	$0.336$	$0.216$	$0.162$	$0.092$
	Bag of Words	$0.533$	$0.339$	$0.216$	$0.162$	$0.092$

Appendix

Table 24: Mean results for gpt-5-nano in the morphology experiment, grouped by agreement condition and input string length.

Figure 12: Full results for gpt-5 on the morphology experiment. Error bars show 95% confidence interval.

Figure 13: Full results for gpt-5-mini on the morphology experiment. Error bars show 95% confidence interval.

Figure 14: Full results for gpt-5-nano on the morphology experiment. Error bars show 95% confidence interval.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
Latin	Exact Match	$0.992$	$0.983$	$1.000$	$0.863$	$0.546$	$0.267$	$0.208$
	Bag of Words	$0.992$	$0.983$	$1.000$	$0.863$	$0.546$	$0.267$	$0.208$
	$\BLEU$	$0.999$	$0.994$	$1.000$	$0.973$	$0.893$	$0.758$	$0.718$
	$\chrFPP$	$1.000$	$0.997$	$1.000$	$0.983$	$0.937$	$0.866$	$0.837$
Latin + diac.	Exact Match	$0.983$	$0.983$	$0.979$	$0.900$	$0.450$	$0.325$	$0.229$
	Bag of Words	$0.983$	$0.983$	$0.979$	$0.900$	$0.450$	$0.325$	$0.229$

Appendix

Table 25: Mean results for gpt-5 in the orthography experiment, grouped by target orthography and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
Latin	Exact Match	$0.800$	$0.992$	$0.979$	$0.800$	$0.067$	$0.037$	$0.017$
	Bag of Words	$0.800$	$0.992$	$0.979$	$0.800$	$0.067$	$0.037$	$0.017$
	$\BLEU$	$0.955$	$0.999$	$0.996$	$0.963$	$0.570$	$0.352$	$0.279$
	$\chrFPP$	$0.987$	$0.999$	$0.998$	$0.978$	$0.729$	$0.565$	$0.510$
Latin + diac.	Exact Match	$0.917$	$0.879$	$0.917$	$0.808$	$0.158$	$0.050$	$0.004$
	Bag of Words	$0.917$	$0.879$	$0.917$	$0.808$	$0.158$	$0.050$	$0.004$

Appendix

Table 26: Mean results for gpt-5-mini in the orthography experiment, grouped by target orthography and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
Latin	Exact Match	—	—	—	—	—	$0.008$	$0.000$
	Bag of Words	—	—	—	—	—	$0.008$	$0.000$
	$\BLEU$	—	—	—	—	—	$0.077$	$0.062$
	$\chrFPP$	—	—	—	—	—	$0.211$	$0.195$
Latin + diac.	Exact Match	$0.608$	$0.579$	$0.558$	—	—	—	—
	Bag of Words	$0.608$	$0.579$	$0.558$	—	—	—	—

Appendix

Table 27: Mean results for gpt-5-nano in the orthography experiment, grouped by target orthography and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
Latin	Exact Match	$0.325$	$0.258$	$0.150$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.325$	$0.263$	$0.150$	$0.000$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.705$	$0.676$	$0.535$	$0.089$	$0.034$	$0.027$	$0.015$
	$\chrFPP$	$0.818$	$0.799$	$0.685$	$0.263$	$0.161$	$0.138$	$0.106$
Latin + diac.	Exact Match	$0.312$	$0.296$	$0.108$	$0.000$	$0.000$	$0.000$	—
	Bag of Words	$0.325$	$0.308$	$0.121$	$0.000$	$0.000$	$0.000$	—

Appendix

Table 28: Mean results for gemma-3-12b-it in the orthography experiment, grouped by target orthography and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
Latin	Exact Match	$0.004$	$0.000$	$0.004$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.004$	$0.000$	$0.004$	$0.000$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.016$	$0.017$	$0.026$	$0.003$	$0.002$	$0.001$	$0.002$
	$\chrFPP$	$0.128$	$0.123$	$0.137$	$0.093$	$0.089$	$0.089$	$0.084$
Latin + diac.	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	—
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$	—

Appendix

Table 29: Mean results for gemma-3-4b-it in the orthography experiment, grouped by target orthography and grammar size.

Condition	Metric	46	66	106	826	4,026	6,026	8,026
Latin	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	—	—	—
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	—	—	—
	$\BLEU$	$0.000$	$0.002$	$0.001$	$0.001$	—	—	—
	$\chrFPP$	$0.101$	$0.105$	$0.104$	$0.098$	—	—	—
Latin + diac.	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	—	—	—
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	—	—	—

Appendix

Table 30: Mean results for gemma-3-1b-it in the orthography experiment, grouped by target orthography and grammar size.

Condition	Metric	6.5	11	15.5	20.5	34
Latin	Exact Match	$0.796$	$0.706$	$0.685$	$0.635$	$0.631$
	Bag of Words	$0.796$	$0.706$	$0.685$	$0.635$	$0.631$
	$\BLEU$	$0.907$	$0.899$	$0.912$	$0.898$	$0.907$
	$\chrFPP$	$0.944$	$0.943$	$0.951$	$0.942$	$0.948$
Latin + diac.	Exact Match	$0.806$	$0.696$	$0.696$	$0.653$	$0.564$
	Bag of Words	$0.806$	$0.696$	$0.696$	$0.653$	$0.564$

Appendix

Table 31: Mean results for gpt-5 in the orthography experiment, grouped by target orthography and input string length.

Condition	Metric	6.5	11	15.5	20.5	34
Latin	Exact Match	$0.580$	$0.555$	$0.500$	$0.472$	$0.521$
	Bag of Words	$0.580$	$0.555$	$0.500$	$0.472$	$0.521$
	$\BLEU$	$0.746$	$0.748$	$0.716$	$0.712$	$0.730$
	$\chrFPP$	$0.823$	$0.842$	$0.821$	$0.816$	$0.821$
Latin + diac.	Exact Match	$0.633$	$0.529$	$0.513$	$0.532$	$0.412$
	Bag of Words	$0.633$	$0.529$	$0.513$	$0.532$	$0.412$

Appendix

Table 32: Mean results for gpt-5-mini in the orthography experiment, grouped by target orthography and input string length.

Condition	Metric	6.5	11	15.5	20.5	34
Latin	Exact Match	$0.016$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.016$	$0.000$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.168$	$0.069$	$0.040$	$0.020$	$0.011$
	$\chrFPP$	$0.327$	$0.223$	$0.165$	$0.140$	$0.110$
Latin + diac.	Exact Match	$0.786$	$0.603$	$0.595$	$0.435$	$0.429$
	Bag of Words	$0.786$	$0.603$	$0.595$	$0.435$	$0.429$

Appendix

Table 33: Mean results for gpt-5-nano in the orthography experiment, grouped by target orthography and input string length.

Condition	Metric	6.5	11	15.5	20.5	34
Latin	Exact Match	$0.168$	$0.136$	$0.062$	$0.078$	$0.074$
	Bag of Words	$0.168$	$0.136$	$0.062$	$0.081$	$0.074$
	$\BLEU$	$0.318$	$0.331$	$0.270$	$0.291$	$0.280$
	$\chrFPP$	$0.426$	$0.452$	$0.405$	$0.428$	$0.417$
Latin + diac.	Exact Match	$0.190$	$0.127$	$0.104$	$0.077$	$0.081$
	Bag of Words	$0.199$	$0.139$	$0.104$	$0.083$	$0.086$

Appendix

Table 34: Mean results for gemma-3-12b-it in the orthography experiment, grouped by target orthography and input string length.

Condition	Metric	6.5	11	15.5	20.5	34
Latin	Exact Match	$0.000$	$0.007$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.007$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.010$	$0.021$	$0.007$	$0.009$	$0.004$
	$\chrFPP$	$0.092$	$0.118$	$0.105$	$0.112$	$0.107$
Latin + diac.	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$

Appendix

Table 35: Mean results for gemma-3-4b-it in the orthography experiment, grouped by target orthography and input string length.

Condition	Metric	6.5	11	15.5	20.5	34
Latin	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	$\BLEU$	$0.001$	$0.001$	$0.000$	$0.001$	$0.001$
	$\chrFPP$	$0.084$	$0.100$	$0.104$	$0.110$	$0.116$
Latin + diac.	Exact Match	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
	Bag of Words	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$

Appendix

Table 36: Mean results for gemma-3-1b-it in the orthography experiment, grouped by target orthography and input string length.

Figure 15: Full results for gpt-5 on the orthography experiment. Error bars show 95% confidence interval.

Figure 16: Full results for gpt-5-mini on the orthography experiment. Error bars show 95% confidence interval.

Figure 17: Full results for gpt-5-nano on the orthography experiment. Error bars show 95% confidence interval.

Figure 18: Full results for gemma-3-12b-it on the orthography experiment. Error bars show 95% confidence interval.

Figure 19: Full results for gemma-3-4b-it on the orthography experiment. Error bars show 95% confidence interval.

Figure 20: Full results for gemma-3-1b-it on the orthography experiment. Error bars show 95% confidence interval.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?
Jun 2, 2026Renhao Pei, Yihong Liu, Sampo Pyysalo +2CoT Reasoning Low-Resource MT
ELLIS Institute Finland · University of Turku · Center for Information and Language Processing, LMU Munich +1
Current LLMs still cannot 'talk much' about grammar modules: Evidence from syntax
Mar 20, 2026Mohammed Q. Shormani, Yehia A. AlSohbaniMachine Translation
Ibb University, Yemen · Arab Open University, Saudia Arabia
Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?
Mar 17, 2026Aishwarya Ramasethu, Niyathi Allu, Rohin Garg +2Machine Translation Low-Resource MT
Prediction Guard · Independent · Scale AI +1

Jun 2, 2026J/K move · Enter open · S save

Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?

Renhao Pei, Yihong Liu, Sampo Pyysalo, Hinrich Schütze+1

ELLIS Institute Finland · University of Turku · Center for Information and Language Processing, LMU Munich+1

CoT Reasoning Low-Resource MT Machine Translation

Open paper AskPDF arXiv

Large language models (LLMs) offer a promising approach to machine translation (MT) for extremely low-resource languages by incorporating linguistic resources through in-context learning. However, LLMs often struggle to apply grammatical information effectively during translation. Inspired by recent progress in chain-of-thought reasoning, we investigate whether low-resource MT can benefit from structured intermediate steps of linguistic analysis and grammatical reasoning. We propose a pipeline for automatically generating step-by-step linguistic reasoning traces from Universal Dependencies treebanks, dictionaries, and grammar-rule banks. We evaluate these traces in three settings: in-context learning (ICL), supervised fine-tuning (SFT), and reinforcement fine-tuning (RFT), on Xibe and Chintang as test cases. Our results show that linguistic reasoning traces are most effective as inference-time guidance: in ICL, reliable sentence-specific traces substantially improve translation performance across most models, languages, and metrics. In contrast, using the linguistic reasoning traces as training data yields smaller and less consistent gains, as models learn the trace format but often generate erroneous content. These findings suggest that LLMs can leverage grammatical information for low-resource MT when given reliable linguistic analyses, while learning to generate such analyses remains a major bottleneck.

Abstract

Machine Translation

Explore similar work

Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?

Current LLMs still cannot 'talk much' about grammar modules: Evidence from syntax

Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?