Low-resource languages pose a challenge for machine translation with large language models (LLMs), which require large amounts of training data. One potential way to circumvent this data dependence is to rely on LLMs' ability to use in-context descriptions of languages, like textbooks and dictionaries. To do so, LLMs must be able to infer the link between the languages' grammatical descriptions and the sentences in question. Here we isolate this skill using a formal analogue of the task: string transduction based on a formal grammar provided in-context. We construct synchronous context-free grammars which define pairs of formal languages designed to model particular aspects of natural language grammar, morphology, and written representation. Using these grammars, we measure how well LLMs can translate sentences from one formal language into another when given both the grammar and the source-language sentence. We vary the size of the grammar, the lengths of the sentences, the syntactic and morphological properties of the languages, and their written script. We note three key findings. First, LLMs' translation accuracy decreases markedly as a function of grammar size and sentence length. Second, differences in morphology and written representation between the source and target languages can strongly diminish model performance. Third, we examine the types of errors committed by models and find they are most prone to recall the wrong words from the target language vocabulary, hallucinate new words, or leave source-language words untranslated.
Figures & tables
⎩⎨⎧[l]S→⟨NPVP,NPVP⟩VP→⟨VBNP,NPVB⟩NP→⟨I,watashi wa⟩NP→⟨the box,hako wo⟩VB→⟨open,akemasu⟩⎭⎬⎫∼[l]EN:JP:I open the boxwatashi wa hako wo akemasu
Figure 1: ( left ) A small SCFG defining fragments of English and Japanese using production rules, from which is sampled ( right ) a corresponding pair of English and Japanese sentences. Adapted from Chiang (2021) .
Language
Word Order
Example Sentence
English
S V O
the jaguar ate the man
Basque
S O V
jaguarrak gizona jan du Personal communication
Hixkaryana
O V S
toto yonoye kamara Derbyshire 1977, p. 593
Table 1: Languages can differ in the ordering between subjects , verbs , and objects .
NoAgr → NoAgr
Agr → NoAgr
Agr → Agr
NoAgr → Agr
1sg
na lam → ni tor
na lam mi → ni tor
na lam mi → ni tor ik
na lam → ni tor ik
3sg
sa lam → su tor
sa lam su → su tor
sa lam su → su tor o
sa lam → su tor o
3pl
ran lam → ren tor
ran lam sar → ren tor
ran lam sar → ren tor on
ran lam → ren tor on
Table 2: Different person/number agreement paradigms between the source and target languages; verbal suffixes in orange optionally reflect the person (1st or 3rd) and number (singular or plural) of the pronoun .
Table 3: Examples of target sentence fragments in the various scripts.
Figure 3: Mean accuracy on size (all models) and word order, morphology, and orthography ( gpt-5 only) experiments; error bars show 95% confidence intervals. We find that grammar size and sentence length, a language’s morphology type, and its orthography impact translation accuracy, while its basic word order does not.
Figure 4: Error distributions for gpt-5 by experiment. Error categories are not mutually exclusive. The most frequent error types are source vocabulary words leakage, recall errors , omission of words, and in the case of the orthography experiment, hallucinating vocabulary words and translating words in the wrong orthography .
Table 4: Examples of the various error types; words of note are highlighted in red, with specific characters (Devanagari GA U+0917 ; Abkhazian HA U+04A9 ) bolded.
Appendix figures & tables48 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Full results for grammar size & sentence length experiment. Error bars show 95% confidence interval.
Model
Metric
57
77
117
837
4,037
6,037
8,037
gpt-5
Exact Match
0.967
0.967
0.979
0.912
0.750
0.688
0.487
Bag of Words
0.967
0.967
0.983
0.954
0.842
0.762
0.558
\BLEU
0.994
0.995
0.995
0.970
0.910
0.878
0.780
\chrFPP
0.998
0.998
0.998
0.992
0.970
0.952
0.909
gpt-5-mini
Exact Match
0.975
0.925
0.946
0.575
0.083
0.029
0.013
Bag of Words
0.975
0.929
0.958
0.804
0.133
0.042
0.021
Appendix
Table 5: Mean results by grammar size for all models in the size experiment.
Model
Metric
5.5
10.5
15.5
20.5
36.5
gpt-5
Exact Match
0.950
0.859
0.801
0.753
0.734
Bag of Words
0.955
0.899
0.846
0.808
0.793
\BLEU
0.977
0.941
0.932
0.910
0.894
\chrFPP
0.990
0.978
0.972
0.967
0.962
gpt-5-mini
Exact Match
0.615
0.495
0.488
0.459
0.467
Bag of Words
0.640
0.557
0.548
0.506
0.502
Appendix
Table 6: Mean results by input string length for all models in the size experiment.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
SVO → SVO
Exact Match
0.996
1.000
0.988
0.858
0.471
0.338
0.246
Bag of Words
0.996
1.000
0.988
0.858
0.471
0.338
0.246
\BLEU
0.999
1.000
0.997
0.972
0.877
0.783
0.714
\chrFPP
1.000
1.000
0.999
0.986
0.925
0.875
0.839
SVO → SOV
Exact Match
0.963
0.971
0.946
0.879
0.529
0.338
0.212
Bag of Words
0.967
0.971
0.946
0.912
0.537
0.350
0.225
Appendix
Table 7: Mean results for gpt-5 in the word order experiment, grouped by target word-order condition and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
SVO → SVO
Exact Match
1.000
1.000
0.971
0.700
0.071
0.017
0.008
Bag of Words
1.000
1.000
0.971
0.700
0.071
0.017
0.008
\BLEU
1.000
1.000
0.996
0.933
0.566
0.382
0.261
\chrFPP
1.000
1.000
0.998
0.966
0.732
0.614
0.510
SVO → SOV
Exact Match
0.988
0.983
0.967
0.550
0.058
0.021
0.017
Bag of Words
0.988
0.988
0.971
0.733
0.058
0.025
0.017
Appendix
Table 8: Mean results for gpt-5-mini in the word order experiment, grouped by target word-order condition and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
SVO → SVO
Exact Match
0.892
0.696
0.650
0.125
0.004
0.000
0.000
Bag of Words
0.896
0.696
0.650
0.125
0.004
0.000
0.000
\BLEU
0.969
0.923
0.878
0.446
0.130
0.069
0.049
\chrFPP
0.981
0.961
0.937
0.602
0.288
0.210
0.170
SVO → SOV
Exact Match
0.121
0.121
0.121
0.013
0.000
0.000
0.000
Bag of Words
0.804
0.654
0.571
0.133
0.000
0.004
0.004
Appendix
Table 9: Mean results for gpt-5-nano in the word order experiment, grouped by target word-order condition and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
SVO → SVO
Exact Match
0.371
0.354
0.158
0.000
0.000
0.000
0.000
Bag of Words
0.392
0.362
0.171
0.000
0.000
0.000
0.000
\BLEU
0.761
0.756
0.566
0.066
0.033
0.024
0.026
\chrFPP
0.856
0.853
0.735
0.283
0.145
0.155
0.137
SVO → SOV
Exact Match
0.000
0.000
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.412
0.400
0.163
0.000
0.000
0.000
0.000
Appendix
Table 10: Mean results for gemma-3-12b-it in the word order experiment, grouped by target word-order condition and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
SVO → SVO
Exact Match
0.000
0.000
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.000
0.000
0.000
0.000
0.000
0.000
0.000
\BLEU
0.006
0.013
0.002
0.001
0.002
0.001
0.001
\chrFPP
0.116
0.106
0.103
0.095
0.100
0.094
0.083
SVO → SOV
Exact Match
0.000
0.000
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.000
0.000
0.004
0.000
0.000
0.000
0.000
Appendix
Table 11: Mean results for gemma-3-4b-it in the word order experiment, grouped by target word-order condition and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
SVO → SVO
Exact Match
0.000
0.000
0.000
0.000
—
—
—
Bag of Words
0.000
0.000
0.000
0.000
—
—
—
\BLEU
0.000
0.001
0.000
0.001
—
—
—
\chrFPP
0.101
0.091
0.102
0.101
—
—
—
SVO → SOV
Exact Match
0.000
0.000
0.000
0.000
—
—
—
Bag of Words
0.000
0.000
0.000
0.000
—
—
—
Appendix
Table 12: Mean results for gemma-3-1b-it in the word order experiment, grouped by target word-order condition and grammar size.
Condition
Metric
6
10.5
15.5
20.5
32.5
SVO → SVO
Exact Match
0.822
0.752
0.663
0.722
0.538
Bag of Words
0.822
0.752
0.663
0.722
0.538
\BLEU
0.922
0.916
0.894
0.927
0.871
\chrFPP
0.952
0.954
0.939
0.960
0.926
SVO → SOV
Exact Match
0.827
0.733
0.670
0.633
0.565
Bag of Words
0.832
0.733
0.679
0.655
0.579
Appendix
Table 13: Mean results for gpt-5 in the word order experiment, grouped by target word-order condition and input string length.
Condition
Metric
6
10.5
15.5
20.5
32.5
SVO → SVO
Exact Match
0.611
0.578
0.500
0.557
0.447
Bag of Words
0.611
0.578
0.500
0.557
0.447
\BLEU
0.755
0.774
0.708
0.771
0.664
\chrFPP
0.832
0.859
0.820
0.861
0.784
SVO → SOV
Exact Match
0.578
0.539
0.497
0.482
0.450
Bag of Words
0.601
0.570
0.527
0.507
0.483
Appendix
Table 14: Mean results for gpt-5-mini in the word order experiment, grouped by target word-order condition and input string length.
Condition
Metric
6
10.5
15.5
20.5
32.5
SVO → SVO
Exact Match
0.413
0.382
0.313
0.332
0.252
Bag of Words
0.413
0.385
0.313
0.332
0.252
\BLEU
0.583
0.534
0.467
0.496
0.395
\chrFPP
0.688
0.643
0.567
0.581
0.485
SVO → SOV
Exact Match
0.223
0.027
0.003
0.000
0.000
Bag of Words
0.425
0.367
0.275
0.255
0.210
Appendix
Table 15: Mean results for gpt-5-nano in the word order experiment, grouped by target word-order condition and input string length.
Condition
Metric
6
10.5
15.5
20.5
32.5
SVO → SVO
Exact Match
0.184
0.137
0.115
0.129
0.066
Bag of Words
0.193
0.146
0.120
0.132
0.069
\BLEU
0.335
0.332
0.310
0.339
0.278
\chrFPP
0.464
0.470
0.437
0.473
0.417
SVO → SOV
Exact Match
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.201
0.164
0.129
0.109
0.081
Appendix
Table 16: Mean results for gemma-3-12b-it in the word order experiment, grouped by target word-order condition and input string length.
Condition
Metric
6
10.5
15.5
20.5
32.5
SVO → SVO
Exact Match
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.000
0.000
0.000
0.000
0.000
\BLEU
0.004
0.003
0.006
0.002
0.002
\chrFPP
0.080
0.097
0.106
0.105
0.108
SVO → SOV
Exact Match
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.003
0.000
0.000
0.000
0.000
Appendix
Table 17: Mean results for gemma-3-4b-it in the word order experiment, grouped by target word-order condition and input string length.
Condition
Metric
6
10.5
15.5
20.5
32.5
SVO → SVO
Exact Match
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.000
0.000
0.000
0.000
0.000
\BLEU
0.000
0.001
0.001
0.001
0.001
\chrFPP
0.077
0.095
0.104
0.107
0.111
SVO → SOV
Exact Match
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.000
0.000
0.000
0.000
0.000
Appendix
Table 18: Mean results for gemma-3-1b-it in the word order experiment, grouped by target word-order condition and input string length.
Figure 6: Full results for gpt-5 on the word order experiment. Error bars show 95% confidence interval.
Figure 7: Full results for gpt-5-mini on the word order experiment. Error bars show 95% confidence interval.
Figure 8: Full results for gpt-5-nano on the word order experiment. Error bars show 95% confidence interval.
Figure 9: Full results for gemma-3-12b-it on the word order experiment. Error bars show 95% confidence interval.
Figure 10: Full results for gemma-3-4b-it on the word order experiment. Error bars show 95% confidence interval.
Figure 11: Full results for gemma-3-1b-it on the word order experiment. Error bars show 95% confidence interval.
Condition
Metric
25
50
100
1,000
5,000
7,500
10,000
NoAgr → NoAgr
Exact Match
0.996
1.000
1.000
0.967
0.796
0.796
0.729
Bag of Words
0.996
1.000
1.000
0.967
0.796
0.796
0.729
\BLEU
0.999
1.000
1.000
0.993
0.962
0.958
0.953
\chrFPP
1.000
1.000
1.000
0.996
0.979
0.977
0.973
Agr → NoAgr
Exact Match
0.400
0.421
0.421
0.312
0.346
0.371
0.358
Bag of Words
0.400
0.421
0.421
0.312
0.346
0.371
0.358
Appendix
Table 19: Mean results for gpt-5 in the morphology experiment, grouped by agreement condition and grammar size.
Condition
Metric
25
50
100
1,000
5,000
7,500
10,000
NoAgr → NoAgr
Exact Match
0.996
1.000
0.996
0.975
0.804
0.787
0.683
Bag of Words
0.996
1.000
0.996
0.975
0.804
0.787
0.683
\BLEU
0.999
1.000
1.000
0.995
0.961
0.956
0.936
\chrFPP
1.000
1.000
1.000
0.997
0.979
0.977
0.963
Agr → NoAgr
Exact Match
0.400
0.421
0.421
0.312
0.325
0.367
0.354
Bag of Words
0.400
0.421
0.421
0.312
0.325
0.367
0.354
Appendix
Table 20: Mean results for gpt-5-mini in the morphology experiment, grouped by agreement condition and grammar size.
Condition
Metric
25
50
100
1,000
5,000
7,500
10,000
NoAgr → NoAgr
Exact Match
0.887
0.904
0.871
0.829
0.346
0.450
0.300
Bag of Words
0.887
0.904
0.871
0.829
0.346
0.450
0.300
\BLEU
0.970
0.980
0.974
0.964
0.787
0.849
0.747
\chrFPP
0.984
0.988
0.986
0.981
0.895
0.921
0.857
Agr → NoAgr
Exact Match
0.346
0.346
0.346
0.271
0.246
0.188
0.229
Bag of Words
0.346
0.350
0.346
0.271
0.246
0.188
0.229
Appendix
Table 21: Mean results for gpt-5-nano in the morphology experiment, grouped by agreement condition and grammar size.
Condition
Metric
5.5
10.5
15
19.5
33
NoAgr → NoAgr
Exact Match
0.948
0.914
0.909
0.846
0.872
Bag of Words
0.948
0.914
0.909
0.846
0.872
\BLEU
0.981
0.979
0.980
0.979
0.985
\chrFPP
0.989
0.988
0.990
0.988
0.991
Agr → NoAgr
Exact Match
0.573
0.442
0.363
0.267
0.177
Bag of Words
0.573
0.442
0.363
0.267
0.177
Appendix
Table 22: Mean results for gpt-5 in the morphology experiment, grouped by agreement condition and input string length.
Condition
Metric
5.5
10.5
15
19.5
33
NoAgr → NoAgr
Exact Match
0.951
0.906
0.905
0.830
0.869
Bag of Words
0.951
0.906
0.905
0.830
0.869
\BLEU
0.981
0.976
0.980
0.973
0.982
\chrFPP
0.990
0.986
0.990
0.985
0.990
Agr → NoAgr
Exact Match
0.573
0.436
0.363
0.259
0.170
Bag of Words
0.573
0.436
0.363
0.259
0.170
Appendix
Table 23: Mean results for gpt-5-mini in the morphology experiment, grouped by agreement condition and input string length.
Condition
Metric
5.5
10.5
15
19.5
33
NoAgr → NoAgr
Exact Match
0.858
0.678
0.614
0.528
0.583
Bag of Words
0.858
0.678
0.614
0.528
0.583
\BLEU
0.933
0.891
0.883
0.873
0.896
\chrFPP
0.963
0.942
0.941
0.933
0.944
Agr → NoAgr
Exact Match
0.533
0.336
0.216
0.162
0.092
Bag of Words
0.533
0.339
0.216
0.162
0.092
Appendix
Table 24: Mean results for gpt-5-nano in the morphology experiment, grouped by agreement condition and input string length.
Figure 12: Full results for gpt-5 on the morphology experiment. Error bars show 95% confidence interval.
Figure 13: Full results for gpt-5-mini on the morphology experiment. Error bars show 95% confidence interval.
Figure 14: Full results for gpt-5-nano on the morphology experiment. Error bars show 95% confidence interval.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
Latin
Exact Match
0.992
0.983
1.000
0.863
0.546
0.267
0.208
Bag of Words
0.992
0.983
1.000
0.863
0.546
0.267
0.208
\BLEU
0.999
0.994
1.000
0.973
0.893
0.758
0.718
\chrFPP
1.000
0.997
1.000
0.983
0.937
0.866
0.837
Latin + diac.
Exact Match
0.983
0.983
0.979
0.900
0.450
0.325
0.229
Bag of Words
0.983
0.983
0.979
0.900
0.450
0.325
0.229
Appendix
Table 25: Mean results for gpt-5 in the orthography experiment, grouped by target orthography and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
Latin
Exact Match
0.800
0.992
0.979
0.800
0.067
0.037
0.017
Bag of Words
0.800
0.992
0.979
0.800
0.067
0.037
0.017
\BLEU
0.955
0.999
0.996
0.963
0.570
0.352
0.279
\chrFPP
0.987
0.999
0.998
0.978
0.729
0.565
0.510
Latin + diac.
Exact Match
0.917
0.879
0.917
0.808
0.158
0.050
0.004
Bag of Words
0.917
0.879
0.917
0.808
0.158
0.050
0.004
Appendix
Table 26: Mean results for gpt-5-mini in the orthography experiment, grouped by target orthography and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
Latin
Exact Match
—
—
—
—
—
0.008
0.000
Bag of Words
—
—
—
—
—
0.008
0.000
\BLEU
—
—
—
—
—
0.077
0.062
\chrFPP
—
—
—
—
—
0.211
0.195
Latin + diac.
Exact Match
0.608
0.579
0.558
—
—
—
—
Bag of Words
0.608
0.579
0.558
—
—
—
—
Appendix
Table 27: Mean results for gpt-5-nano in the orthography experiment, grouped by target orthography and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
Latin
Exact Match
0.325
0.258
0.150
0.000
0.000
0.000
0.000
Bag of Words
0.325
0.263
0.150
0.000
0.000
0.000
0.000
\BLEU
0.705
0.676
0.535
0.089
0.034
0.027
0.015
\chrFPP
0.818
0.799
0.685
0.263
0.161
0.138
0.106
Latin + diac.
Exact Match
0.312
0.296
0.108
0.000
0.000
0.000
—
Bag of Words
0.325
0.308
0.121
0.000
0.000
0.000
—
Appendix
Table 28: Mean results for gemma-3-12b-it in the orthography experiment, grouped by target orthography and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
Latin
Exact Match
0.004
0.000
0.004
0.000
0.000
0.000
0.000
Bag of Words
0.004
0.000
0.004
0.000
0.000
0.000
0.000
\BLEU
0.016
0.017
0.026
0.003
0.002
0.001
0.002
\chrFPP
0.128
0.123
0.137
0.093
0.089
0.089
0.084
Latin + diac.
Exact Match
0.000
0.000
0.000
0.000
0.000
0.000
—
Bag of Words
0.000
0.000
0.000
0.000
0.000
0.000
—
Appendix
Table 29: Mean results for gemma-3-4b-it in the orthography experiment, grouped by target orthography and grammar size.
Condition
Metric
46
66
106
826
4,026
6,026
8,026
Latin
Exact Match
0.000
0.000
0.000
0.000
—
—
—
Bag of Words
0.000
0.000
0.000
0.000
—
—
—
\BLEU
0.000
0.002
0.001
0.001
—
—
—
\chrFPP
0.101
0.105
0.104
0.098
—
—
—
Latin + diac.
Exact Match
0.000
0.000
0.000
0.000
—
—
—
Bag of Words
0.000
0.000
0.000
0.000
—
—
—
Appendix
Table 30: Mean results for gemma-3-1b-it in the orthography experiment, grouped by target orthography and grammar size.
Condition
Metric
6.5
11
15.5
20.5
34
Latin
Exact Match
0.796
0.706
0.685
0.635
0.631
Bag of Words
0.796
0.706
0.685
0.635
0.631
\BLEU
0.907
0.899
0.912
0.898
0.907
\chrFPP
0.944
0.943
0.951
0.942
0.948
Latin + diac.
Exact Match
0.806
0.696
0.696
0.653
0.564
Bag of Words
0.806
0.696
0.696
0.653
0.564
Appendix
Table 31: Mean results for gpt-5 in the orthography experiment, grouped by target orthography and input string length.
Condition
Metric
6.5
11
15.5
20.5
34
Latin
Exact Match
0.580
0.555
0.500
0.472
0.521
Bag of Words
0.580
0.555
0.500
0.472
0.521
\BLEU
0.746
0.748
0.716
0.712
0.730
\chrFPP
0.823
0.842
0.821
0.816
0.821
Latin + diac.
Exact Match
0.633
0.529
0.513
0.532
0.412
Bag of Words
0.633
0.529
0.513
0.532
0.412
Appendix
Table 32: Mean results for gpt-5-mini in the orthography experiment, grouped by target orthography and input string length.
Condition
Metric
6.5
11
15.5
20.5
34
Latin
Exact Match
0.016
0.000
0.000
0.000
0.000
Bag of Words
0.016
0.000
0.000
0.000
0.000
\BLEU
0.168
0.069
0.040
0.020
0.011
\chrFPP
0.327
0.223
0.165
0.140
0.110
Latin + diac.
Exact Match
0.786
0.603
0.595
0.435
0.429
Bag of Words
0.786
0.603
0.595
0.435
0.429
Appendix
Table 33: Mean results for gpt-5-nano in the orthography experiment, grouped by target orthography and input string length.
Condition
Metric
6.5
11
15.5
20.5
34
Latin
Exact Match
0.168
0.136
0.062
0.078
0.074
Bag of Words
0.168
0.136
0.062
0.081
0.074
\BLEU
0.318
0.331
0.270
0.291
0.280
\chrFPP
0.426
0.452
0.405
0.428
0.417
Latin + diac.
Exact Match
0.190
0.127
0.104
0.077
0.081
Bag of Words
0.199
0.139
0.104
0.083
0.086
Appendix
Table 34: Mean results for gemma-3-12b-it in the orthography experiment, grouped by target orthography and input string length.
Condition
Metric
6.5
11
15.5
20.5
34
Latin
Exact Match
0.000
0.007
0.000
0.000
0.000
Bag of Words
0.000
0.007
0.000
0.000
0.000
\BLEU
0.010
0.021
0.007
0.009
0.004
\chrFPP
0.092
0.118
0.105
0.112
0.107
Latin + diac.
Exact Match
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.000
0.000
0.000
0.000
0.000
Appendix
Table 35: Mean results for gemma-3-4b-it in the orthography experiment, grouped by target orthography and input string length.
Condition
Metric
6.5
11
15.5
20.5
34
Latin
Exact Match
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.000
0.000
0.000
0.000
0.000
\BLEU
0.001
0.001
0.000
0.001
0.001
\chrFPP
0.084
0.100
0.104
0.110
0.116
Latin + diac.
Exact Match
0.000
0.000
0.000
0.000
0.000
Bag of Words
0.000
0.000
0.000
0.000
0.000
Appendix
Table 36: Mean results for gemma-3-1b-it in the orthography experiment, grouped by target orthography and input string length.
Figure 15: Full results for gpt-5 on the orthography experiment. Error bars show 95% confidence interval.
Figure 16: Full results for gpt-5-mini on the orthography experiment. Error bars show 95% confidence interval.
Figure 17: Full results for gpt-5-nano on the orthography experiment. Error bars show 95% confidence interval.
Figure 18: Full results for gemma-3-12b-it on the orthography experiment. Error bars show 95% confidence interval.
Figure 19: Full results for gemma-3-4b-it on the orthography experiment. Error bars show 95% confidence interval.
Figure 20: Full results for gemma-3-1b-it on the orthography experiment. Error bars show 95% confidence interval.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Large language models (LLMs) offer a promising approach to machine translation (MT) for extremely low-resource languages by incorporating linguistic resources through in-context learning. However, LLMs often struggle to apply grammatical information effectively during translation. Inspired by recent progress in chain-of-thought reasoning, we investigate whether low-resource MT can benefit from structured intermediate steps of linguistic analysis and grammatical reasoning. We propose a pipeline for automatically generating step-by-step linguistic reasoning traces from Universal Dependencies treebanks, dictionaries, and grammar-rule banks. We evaluate these traces in three settings: in-context learning (ICL), supervised fine-tuning (SFT), and reinforcement fine-tuning (RFT), on Xibe and Chintang as test cases. Our results show that linguistic reasoning traces are most effective as inference-time guidance: in ICL, reliable sentence-specific traces substantially improve translation performance across most models, languages, and metrics. In contrast, using the linguistic reasoning traces as training data yields smaller and less consistent gains, as models learn the trace format but often generate erroneous content. These findings suggest that LLMs can leverage grammatical information for low-resource MT when given reliable linguistic analyses, while learning to generate such analyses remains a major bottleneck.
Renhao Pei, Yihong Liu, Sampo Pyysalo +2
ELLIS Institute Finland · University of Turku · Center for Information and Language Processing, LMU Munich +1
We aim to examine the extent to which Large Language Models (LLMs) can 'talk much' about grammar modules, providing evidence from syntax core properties translated by ChatGPT into Arabic. We collected 44 terms from generative syntax previous works, including books and journal articles, as well as from our experience in the field. These terms were translated by humans, and then by ChatGPT-5. We then analyzed and compared both translations. We used an analytical and comparative approach in our analysis. Findings unveil that LLMs still cannot 'talk much' about the core syntax properties embedded in the terms under study involving several syntactic and semantic challenges: only 25% of ChatGPT translations were accurate, while 38.6% were inaccurate, and 36.4.% were partially correct, which we consider appropriate. Based on these findings, a set of actionable strategies were proposed, the most notable of which is a close collaboration between AI specialists and linguists to better LLMs' working mechanism for accurate or at least appropriate translation.
Mohammed Q. Shormani, Yehia A. AlSohbani
Ibb University, Yemen · Arab Open University, Saudia Arabia
Large Language Models (LLMs) have achieved strong performance across many downstream tasks, yet their effectiveness in extremely low-resource machine translation remains limited. Standard adaptation techniques typically rely on large-scale parallel data or extensive fine-tuning, which are infeasible for the long tail of underrepresented languages. In this work, we investigate a more constrained question: in data-scarce settings, to what extent can linguistically similar pivot languages and few-shot demonstrations provide useful guidance for on-the-fly adaptation in LLMs? We study a data-efficient experimental setup that combines linguistically related pivot languages with few-shot in-context examples, without any parameter updates, and evaluate translation behavior under controlled conditions. Our analysis shows that while pivot-based prompting can yield improvements in certain configurations, particularly in settings where the target language is less well represented in the model's vocabulary, the gains are often modest and sensitive to few shot example construction. For closely related or better represented varieties, we observe diminishing or inconsistent gains. Our findings provide empirical guidance on how and when inference-time prompting and pivot-based examples can be used as a lightweight alternative to fine-tuning in low-resource translation settings.