Recent work advancing machine translation quality using LLMs for very low resource languages has relied on the injection of lexicon-based prompt components to expose vocabulary to models. However, these approaches run into a ceiling when models must guess sentence meaning purely from plausible combinations of possible word senses mapped to strings. In this paper we test UD-PrMPT, an approach using syntactic analyses in the Universal Dependencies formalism as prompt components to improve MT in five diverse low resource languages: Beja, Chintang, Chukchi, Coptic, and Xibe. Our results show additive improvements which stack on top of the lexicon based approach which are stronger for higher quality manual analyses but consistent for predicted parses as well.
Figures & tables
Figure 1: Chintang example: DiPMT prevents LLM translation hallucinations by proposing word senses, but these can be combined incorrectly. Our UD-PrMPT method integrates morphology tags and UD parses, showing ‘has to’ is the main predicate and the noun cannot be the subject, and constraining agreement to produce better translations.
Language
Kind
Size
Beja
Fieldwork Compilation
12k
Chukchi
Thematic Glossary
11k
Coptic
Linked Online Dictionary
1 9k
Chintang
Dictionaria Publication
10k
Xibe
Recorded Dictionary
1 4k
Table 1: The lexicon data for each language, a kind classification for reference, and the number of first-level entries in the target machine-readable dictionary format.
bej
ckt
cop
ctn
sjo
MarMoT
UFeats
87.98
–
96.19
85.30
89.93
UPOS
93.28
81.86
95.92
91.42
93.13
MaltParser
UAS
79.26
67.54
85.65
82.87
76.33
LAS
65.99
43.94
77.28
57.36
69.00
Table 2: Evaluation metrics on test from predicted parses (% accuracy). UD Chukchi (ckt) does not use UFeats.
Figure 2: BERTScore F1 for translations generated using gold versus predicted linguistic annotations across two language models and five prompting settings. Results for Qwen are omitted for visual clarity because its trends closely resemble those of Gemma.
Lang+Model
Reference
LEX
LEX+SYN
Chintang Gemma
I live up there, on top of the ravine.
That is where the ravine is.
Oh, I live up there in the ravine.
Chintang Qwen
Kalpana pulls out people’s hair!
Pull out Kalpana’s hair!
Kalpana is pulling out my hair!
Chintang GPT
It’s said they didn’t speak for a while.
Please speak for a moment.
They reportedly did not speak for a while.
Beja Gemma
They wanted to eat him.
Those are the big ones.
They wanted to kill him.
Coptic Gemma
They are like the light of day.
And they likened it to the light of the day.
And they were like the light of the day.
Xibe Gemma
Let us go sightseeing to the Ili River!
The yellow flowers bloom in the spring, and the wind blows gently!
Let us go to the valley of the Elbi river!
Table 3: Selected dev outputs under LEX and LEX+SYN . Chintang is the focus language. Full examples including baseline outputs and Δ F1, are in Appendix A.7 .
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: A sample of prompt elements for Chintang (ctn) with LEX , DEP , and CoNLLU using predicted syntax trees.
Model
System Card or Report
GPT-5.6 Terra
OpenAI System Card
Gemma-4-31B (-it)
Team et al. (2026)
Qwen3.5-27B
Huggingface System Card
Appendix
Table 4: Models used for our experiments.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.3406
14.5752
0.8553
Baseline
GPT-5.6-terra
0.3705
14.0662
0.8584
Baseline
Qwen3.5-27B
0.1342
12.1829
0.8468
Lexicon
Gemma-4-31b
2.0692
21.2349
0.8705
Lexicon
GPT-5.6-terra
1.1704
20.4120
0.8732
Lexicon
Qwen3.5-27B
1.5215
20.6745
0.8645
Appendix
Table 5: Beja (bej) dev results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.6002
14.3879
0.8622
Baseline
GPT-5.6-terra
1.6016
13.8054
0.8665
Baseline
Qwen3.5-27B
0.6902
12.7648
0.8556
Lexicon
Gemma-4-31b
0.6907
15.2981
0.8639
Lexicon
GPT-5.6-terra
1.7808
14.8351
0.8681
Lexicon
Qwen3.5-27B
0.7879
13.4402
0.8589
Appendix
Table 6: Chukchi (ckt) dev results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
6.6394
26.4302
0.8713
Baseline
GPT-5.6-terra
28.9172
49.6033
0.9286
Baseline
Qwen3.5-27B
1.9830
19.4095
0.8493
Lexicon
Gemma-4-31b
12.6869
36.1852
0.8981
Lexicon
GPT-5.6-terra
28.8503
50.3020
0.9317
Lexicon
Qwen3.5-27B
10.9625
33.2071
0.8889
Appendix
Table 7: Coptic (cop) dev results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.6085
12.8178
0.8639
Baseline
GPT-5.6-terra
1.3617
12.7415
0.8627
Baseline
Qwen3.5-27B
0.9082
12.3902
0.8612
Lexicon
Gemma-4-31b
4.6436
21.9314
0.8872
Lexicon
GPT-5.6-terra
4.3630
21.8108
0.8851
Lexicon
Qwen3.5-27B
3.4893
20.8321
0.8808
Appendix
Table 8: Chintang (ctn) dev results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
1.2094
19.4997
0.8525
Baseline
GPT-5.6-terra
5.2956
26.1174
0.8802
Baseline
Qwen3.5-27B
1.2042
19.4464
0.8455
Lexicon
Gemma-4-31b
1.0802
20.2488
0.8567
Lexicon
GPT-5.6-terra
5.8598
27.4408
0.8818
Lexicon
Qwen3.5-27B
1.2875
19.8759
0.8502
Appendix
Table 9: Xibe (sjo) dev results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.3469
14.4410
0.8555
Baseline
GPT-5.6-terra
0.3347
14.1161
0.8588
Baseline
Qwen3.5-27B
0.2469
13.9814
0.8490
Lexicon
Gemma-4-31b
2.2952
21.2402
0.8704
Lexicon
GPT-5.6-terra
1.5764
20.7902
0.8723
Lexicon
Qwen3.5-27B
1.7327
20.6883
0.8655
Appendix
Table 10: Beja (bej) dev results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.5015
14.3073
0.8612
Baseline
GPT-5.6-terra
1.4337
14.0032
0.8652
Baseline
Qwen3.5-27B
0.6922
12.8250
0.8556
Lexicon
Gemma-4-31b
0.6723
15.1761
0.8640
Lexicon
GPT-5.6-terra
1.8557
15.0947
0.8677
Lexicon
Qwen3.5-27B
1.0494
14.1055
0.8591
Appendix
Table 11: Chukchi (ckt) dev results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
6.5727
26.3763
0.8706
Baseline
GPT-5.6-terra
28.8392
49.6635
0.9284
Baseline
Qwen3.5-27B
2.1839
19.9679
0.8496
Lexicon
Gemma-4-31b
13.3913
37.0209
0.8990
Lexicon
GPT-5.6-terra
28.9393
50.5585
0.9321
Lexicon
Qwen3.5-27B
10.5821
33.3743
0.8885
Appendix
Table 12: Coptic (cop) dev results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.9482
12.8595
0.8640
Baseline
GPT-5.6-terra
1.5441
12.8494
0.8622
Baseline
Qwen3.5-27B
0.8586
12.4159
0.8617
Lexicon
Gemma-4-31b
4.4488
21.5798
0.8856
Lexicon
GPT-5.6-terra
4.3653
21.4010
0.8834
Lexicon
Qwen3.5-27B
3.7827
20.6767
0.8804
Appendix
Table 13: Chintang (ctn) dev results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.9776
19.1242
0.8525
Baseline
GPT-5.6-terra
5.9578
26.6827
0.8809
Baseline
Qwen3.5-27B
1.1914
19.1487
0.8449
Lexicon
Gemma-4-31b
1.1382
20.5770
0.8561
Lexicon
GPT-5.6-terra
5.8671
27.6324
0.8837
Lexicon
Qwen3.5-27B
1.6406
20.6398
0.8521
Appendix
Table 14: Xibe (sjo) dev results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.3166
13.5255
0.8554
Baseline
GPT-5.6-terra
0.4430
13.5159
0.8590
Baseline
Qwen3.5-27B
0.3664
13.5176
0.8494
Lexicon
Gemma-4-31b
1.3116
20.0209
0.8702
Lexicon
GPT-5.6-terra
1.2960
19.1972
0.8716
Lexicon
Qwen3.5-27B
1.4384
20.2500
0.8654
Appendix
Table 15: Beja (bej) test results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.3560
13.4491
0.8612
Baseline
GPT-5.6-terra
0.4862
13.4033
0.8659
Baseline
Qwen3.5-27B
0.7019
13.4046
0.8572
Lexicon
Gemma-4-31b
0.9594
15.3322
0.8640
Lexicon
GPT-5.6-terra
0.6871
14.2985
0.8676
Lexicon
Qwen3.5-27B
0.8670
14.1371
0.8599
Appendix
Table 16: Chukchi (ckt) test results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
6.4323
26.6546
0.8718
Baseline
GPT-5.6-terra
31.2051
52.4880
0.9314
Baseline
Qwen3.5-27B
2.4084
19.8988
0.8515
Lexicon
Gemma-4-31b
13.3405
37.3074
0.8972
Lexicon
GPT-5.6-terra
31.3266
53.1647
0.9338
Lexicon
Qwen3.5-27B
10.8785
33.1268
0.8857
Appendix
Table 17: Coptic (cop) texttttest results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.7070
13.1166
0.8664
Baseline
GPT-5.6-terra
1.4518
13.0424
0.8657
Baseline
Qwen3.5-27B
1.0067
12.5322
0.8630
Lexicon
Gemma-4-31b
4.6571
21.8816
0.8900
Lexicon
GPT-5.6-terra
4.8112
22.4292
0.8895
Lexicon
Qwen3.5-27B
3.8300
21.3008
0.8856
Appendix
Table 18: Chintang (ctn) test results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
1.0558
19.3262
0.8570
Baseline
GPT-5.6-terra
4.9633
26.2467
0.8833
Baseline
Qwen3.5-27B
1.3038
19.5234
0.8488
Lexicon
Gemma-4-31b
1.3428
19.5705
0.8605
Lexicon
GPT-5.6-terra
5.2909
26.8043
0.8863
Lexicon
Qwen3.5-27B
1.0025
19.8530
0.8538
Appendix
Table 19: Xibe (sjo) test results using the gold treebank.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.3869
13.2931
0.8541
Baseline
GPT-5.6-terra
0.3629
13.3222
0.8576
Baseline
Qwen3.5-27B
0.2762
12.8097
0.8493
Lexicon
Gemma-4-31b
1.3062
20.3689
0.8696
Lexicon
GPT-5.6-terra
1.3955
19.6067
0.8708
Lexicon
Qwen3.5-27B
1.4367
20.2212
0.8651
Appendix
Table 20: Beja (bej) test results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.4125
14.2330
0.8608
Baseline
GPT-5.6-terra
0.5480
13.2287
0.8643
Baseline
Qwen3.5-27B
0.6886
13.6135
0.8573
Lexicon
Gemma-4-31b
0.5775
14.9631
0.8626
Lexicon
GPT-5.6-terra
0.9376
14.5231
0.8668
Lexicon
Qwen3.5-27B
1.1107
14.2552
0.8597
Appendix
Table 21: Chukchi (ckt) test results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
6.3062
26.6146
0.8708
Baseline
GPT-5.6-terra
30.8087
52.3420
0.9316
Baseline
Qwen3.5-27B
2.7540
19.9402
0.8514
Lexicon
Gemma-4-31b
13.9570
37.5141
0.8978
Lexicon
GPT-5.6-terra
30.7845
52.9620
0.9335
Lexicon
Qwen3.5-27B
10.5563
32.5239
0.8848
Appendix
Table 22: Coptic (cop) test results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
0.6314
12.0142
0.8654
Baseline
GPT-5.6-terra
1.2025
13.1445
0.8652
Baseline
Qwen3.5-27B
1.0251
12.6076
0.8633
Lexicon
Gemma-4-31b
4.3077
21.7345
0.8892
Lexicon
GPT-5.6-terra
4.7300
21.9831
0.8886
Lexicon
Qwen3.5-27B
3.9613
21.3352
0.8864
Appendix
Table 23: Chintang (ctn) test results using predicted NLP annotations.
Setting
Model
BLEU
ChrF++
BERTScore F1
Baseline
Gemma-4-31b
1.1051
19.0809
0.8552
Baseline
GPT-5.6-terra
4.6174
26.1733
0.8824
Baseline
Qwen3.5-27B
0.8305
19.0194
0.8497
Lexicon
Gemma-4-31b
1.4615
20.3558
0.8606
Lexicon
GPT-5.6-terra
5.6454
27.1608
0.8867
Lexicon
Qwen3.5-27B
1.1249
19.5336
0.8538
Appendix
Table 24: Xibe (sjo) test results using predicted NLP annotations.
Figure 4: BERTScore F1 plotted against source length, measured in characters. Rows are models and columns are languages. Sentences are divided into three length bins within each language. Dotted vertical lines mark the bin cutoffs; colored lines show prompt settings; shaded bands show 95% confidence intervals.
Figure 5: BERTScore F1 plotted against source length, measured in tokens. Rows are models and columns are languages. Sentences are divided into three length bins within each language. Dotted vertical lines mark the bin cutoffs; colored lines show prompt settings; shaded bands show 95% confidence intervals.
Setting
Translation, Chintang to English
Δ BLEU
Δ chrF++
Δ F1
Gemma-4-31B
Reference
I live up there, on top of the ravine.
–
–
–
Baseline
I have a sister, Khojabhandu.
0
0
0
LEX
That is where the ravine is.
+3.3277
+17.8306
+0.0587
LEX+SYN
Oh, I live up there in the ravine.
+30.6625
+52.4039
+0.1023
Reference
Anita introduced Lokendra and Lakhman to each other.
–
–
–
Appendix
Table 25: Sample dev outputs for Chintang with Gemma-4-31B, Qwen3.5-27B, and GPT-5.6 Terra under each setting. Metric improvements Δ are reported relative to the baseline for BLEU, chrF++, and BERTScore F1.
Setting
Translation, Source to English
Δ BLEU
Δ chrF++
Δ F1
Beja
Reference
They wanted to eat him.
–
–
–
Baseline
The big house is here.
0
0
0
LEX
Those are the big ones.
0.0000
+4.0334
− 0.0018
LEX+SYN
They wanted to kill him.
+14.8408
+49.9049
+0.0967
Reference
His children and himself were really happy.
–
–
–
Appendix
Table 26: Sample dev outputs for Beja, Coptic, Xibe, and Chukchi with Gemma-4-31B under each setting. Metric improvements Δ are reported relative to the baseline for BLEU, chrF++, and BERTScore F1.
Model
Lang
LEX − base
LEX + CoNLLU − LEX
LEX + DEP − LEX
LEX + SYN − LEX
LEX + CoNLLU − LEX + DEP
LEX + SYN − LEX + CoNLLU
LEX + SYN − LEX + DEP
Gemma
bej
+1.49 ∗
+0.83 ∗
+0.41 ∗
+0.97 ∗
+0.42 ∗
+0.14
+0.56 ∗
Gemma
ckt
+0.29
+0.50 ∗
+0.31
+0.50 ∗
+0.19
+0.00
+0.19
Gemma
cop
+2.56 ∗
+1.13 ∗
+0.73 ∗
+1.19 ∗
+0.40 ∗
+0.06
+0.46 ∗
Gemma
ctn
+2.37 ∗
+1.36 ∗
+0.53 ∗
+1.49 ∗
+0.82 ∗
+0.14
+0.96 ∗
Gemma
sjo
+0.34 ∗
+1.48 ∗
+0.54 ∗
+1.46 ∗
+0.94 ∗
− 0.02
+0.92 ∗
GPT
bej
+1.26 ∗
+0.87 ∗
+0.64 ∗
+0.99 ∗
+0.23
+0.12
+0.36
Appendix
Table 27: Mean paired difference in sentence-level BERTScore F1, in points on a 0–100 scale (F1 × 100). Each column is the score from first setting minus the score from the second setting; a positive entry means the first setting scored higher. ∗ marks differences significant by paired t -test after Holm correction across all 150 comparisons ( α=0.05 ).
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Large language models (LLMs) offer a promising approach to machine translation (MT) for extremely low-resource languages by incorporating linguistic resources through in-context learning. However, LLMs often struggle to apply grammatical information effectively during translation. Inspired by recent progress in chain-of-thought reasoning, we investigate whether low-resource MT can benefit from structured intermediate steps of linguistic analysis and grammatical reasoning. We propose a pipeline for automatically generating step-by-step linguistic reasoning traces from Universal Dependencies treebanks, dictionaries, and grammar-rule banks. We evaluate these traces in three settings: in-context learning (ICL), supervised fine-tuning (SFT), and reinforcement fine-tuning (RFT), on Xibe and Chintang as test cases. Our results show that linguistic reasoning traces are most effective as inference-time guidance: in ICL, reliable sentence-specific traces substantially improve translation performance across most models, languages, and metrics. In contrast, using the linguistic reasoning traces as training data yields smaller and less consistent gains, as models learn the trace format but often generate erroneous content. These findings suggest that LLMs can leverage grammatical information for low-resource MT when given reliable linguistic analyses, while learning to generate such analyses remains a major bottleneck.
Renhao Pei, Yihong Liu, Sampo Pyysalo +2
ELLIS Institute Finland · University of Turku · Center for Information and Language Processing, LMU Munich +1
Low-resource machine translation requires methods that differ from those used for high-resource languages. This paper proposes a novel in-context learning approach to support low-resource machine translation of the Coptic language to English, with syntactic augmentation from Universal Dependencies parses of input sentences. Building on existing work using bilingual dictionaries to support inference for vocabulary items, we add several representations of syntactic analyses to our inputs , specifically exploring the inclusion of raw parser outputs, verbalizations of parses in plain English, and targeted instructions of difficult constructions identified in sub-trees and how they can be translated. Our results show that while syntactic information alone is not as useful as dictionary-based glosses, combining retrieved dictionary items with syntactic information achieves significant gains across model sizes, achieving new state-of-the-art translation results for Coptic.
Abhishek Purushothama, Emma Thronson, Alexia Guo +1
Transformer-based models achieve state-of-the-art dependency parsing for high-resource languages, yet their advantage over simpler architectures in low-resource settings remains poorly understood. We evaluate four parsers---the Biaffine LSTM, Stack-Pointer Network, AfroXLMR-large, and RemBERT---across twelve typologically diverse languages, with a focus on low-resource African languages. We find that the Biaffine LSTM consistently outperforms transformer models in low-resource regimes, with transformers recovering their advantage as training data increases. The crossover falls within a resource range typical of treebanks for under-resourced languages. Morphological complexity (measured via MATTR) emerges as a significant secondary predictor of transformers' relative disadvantage after controlling for corpus size. These results indicate that the Biaffine LSTM may be better suited for syntactic tool development in low-resource regimes until sufficient annotated data is available to leverage the representational capacity of pre-trained transformers.