Out of Parameters, Saved by Syntax: UD-PrMPT for Parser Augmented Low Resource Machine Translation
Organizations: Corpling Lab, Georgetown University
Abstract
Recent work advancing machine translation quality using LLMs for very low resource languages has relied on the injection of lexicon-based prompt components to expose vocabulary to models. However, these approaches run into a ceiling when models must guess sentence meaning purely from plausible combinations of possible word senses mapped to strings. In this paper we test UD-PrMPT, an approach using syntactic analyses in the Universal Dependencies formalism as prompt components to improve MT in five diverse low resource languages: Beja, Chintang, Chukchi, Coptic, and Xibe. Our results show additive improvements which stack on top of the lexicon based approach which are stronger for higher quality manual analyses but consistent for predicted parses as well.
Figures & tables
| Language | Kind | Size |
|---|---|---|
| Beja | Fieldwork Compilation | 12k |
| Chukchi | Thematic Glossary | 11k |
| Coptic | Linked Online Dictionary | 1 9k |
| Chintang | Dictionaria Publication | 10k |
| Xibe | Recorded Dictionary | 1 4k |
| bej | ckt | cop | ctn | sjo | |
| MarMoT | |||||
| UFeats | 87.98 | – | 96.19 | 85.30 | 89.93 |
| UPOS | 93.28 | 81.86 | 95.92 | 91.42 | 93.13 |
| MaltParser | |||||
| UAS | 79.26 | 67.54 | 85.65 | 82.87 | 76.33 |
| LAS | 65.99 | 43.94 | 77.28 | 57.36 | 69.00 |
| Lang+Model | Reference | LEX | LEX+SYN |
| Chintang Gemma | I live up there, on top of the ravine. | That is where the ravine is. | Oh, I live up there in the ravine. |
| Chintang Qwen | Kalpana pulls out people’s hair! | Pull out Kalpana’s hair! | Kalpana is pulling out my hair! |
| Chintang GPT | It’s said they didn’t speak for a while. | Please speak for a moment. | They reportedly did not speak for a while. |
| Beja Gemma | They wanted to eat him. | Those are the big ones. | They wanted to kill him. |
| Coptic Gemma | They are like the light of day. | And they likened it to the light of the day. | And they were like the light of the day. |
| Xibe Gemma | Let us go sightseeing to the Ili River! | The yellow flowers bloom in the spring, and the wind blows gently! | Let us go to the valley of the Elbi river! |
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | System Card or Report |
|---|---|
| GPT-5.6 Terra | OpenAI System Card |
| Gemma-4-31B (-it) | Team et al. (2026) |
| Qwen3.5-27B | Huggingface System Card |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.3406 | 14.5752 | 0.8553 |
| Baseline | GPT-5.6-terra | 0.3705 | 14.0662 | 0.8584 |
| Baseline | Qwen3.5-27B | 0.1342 | 12.1829 | 0.8468 |
| Lexicon | Gemma-4-31b | 2.0692 | 21.2349 | 0.8705 |
| Lexicon | GPT-5.6-terra | 1.1704 | 20.4120 | 0.8732 |
| Lexicon | Qwen3.5-27B | 1.5215 | 20.6745 | 0.8645 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.6002 | 14.3879 | 0.8622 |
| Baseline | GPT-5.6-terra | 1.6016 | 13.8054 | 0.8665 |
| Baseline | Qwen3.5-27B | 0.6902 | 12.7648 | 0.8556 |
| Lexicon | Gemma-4-31b | 0.6907 | 15.2981 | 0.8639 |
| Lexicon | GPT-5.6-terra | 1.7808 | 14.8351 | 0.8681 |
| Lexicon | Qwen3.5-27B | 0.7879 | 13.4402 | 0.8589 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 6.6394 | 26.4302 | 0.8713 |
| Baseline | GPT-5.6-terra | 28.9172 | 49.6033 | 0.9286 |
| Baseline | Qwen3.5-27B | 1.9830 | 19.4095 | 0.8493 |
| Lexicon | Gemma-4-31b | 12.6869 | 36.1852 | 0.8981 |
| Lexicon | GPT-5.6-terra | 28.8503 | 50.3020 | 0.9317 |
| Lexicon | Qwen3.5-27B | 10.9625 | 33.2071 | 0.8889 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.6085 | 12.8178 | 0.8639 |
| Baseline | GPT-5.6-terra | 1.3617 | 12.7415 | 0.8627 |
| Baseline | Qwen3.5-27B | 0.9082 | 12.3902 | 0.8612 |
| Lexicon | Gemma-4-31b | 4.6436 | 21.9314 | 0.8872 |
| Lexicon | GPT-5.6-terra | 4.3630 | 21.8108 | 0.8851 |
| Lexicon | Qwen3.5-27B | 3.4893 | 20.8321 | 0.8808 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 1.2094 | 19.4997 | 0.8525 |
| Baseline | GPT-5.6-terra | 5.2956 | 26.1174 | 0.8802 |
| Baseline | Qwen3.5-27B | 1.2042 | 19.4464 | 0.8455 |
| Lexicon | Gemma-4-31b | 1.0802 | 20.2488 | 0.8567 |
| Lexicon | GPT-5.6-terra | 5.8598 | 27.4408 | 0.8818 |
| Lexicon | Qwen3.5-27B | 1.2875 | 19.8759 | 0.8502 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.3469 | 14.4410 | 0.8555 |
| Baseline | GPT-5.6-terra | 0.3347 | 14.1161 | 0.8588 |
| Baseline | Qwen3.5-27B | 0.2469 | 13.9814 | 0.8490 |
| Lexicon | Gemma-4-31b | 2.2952 | 21.2402 | 0.8704 |
| Lexicon | GPT-5.6-terra | 1.5764 | 20.7902 | 0.8723 |
| Lexicon | Qwen3.5-27B | 1.7327 | 20.6883 | 0.8655 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.5015 | 14.3073 | 0.8612 |
| Baseline | GPT-5.6-terra | 1.4337 | 14.0032 | 0.8652 |
| Baseline | Qwen3.5-27B | 0.6922 | 12.8250 | 0.8556 |
| Lexicon | Gemma-4-31b | 0.6723 | 15.1761 | 0.8640 |
| Lexicon | GPT-5.6-terra | 1.8557 | 15.0947 | 0.8677 |
| Lexicon | Qwen3.5-27B | 1.0494 | 14.1055 | 0.8591 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 6.5727 | 26.3763 | 0.8706 |
| Baseline | GPT-5.6-terra | 28.8392 | 49.6635 | 0.9284 |
| Baseline | Qwen3.5-27B | 2.1839 | 19.9679 | 0.8496 |
| Lexicon | Gemma-4-31b | 13.3913 | 37.0209 | 0.8990 |
| Lexicon | GPT-5.6-terra | 28.9393 | 50.5585 | 0.9321 |
| Lexicon | Qwen3.5-27B | 10.5821 | 33.3743 | 0.8885 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.9482 | 12.8595 | 0.8640 |
| Baseline | GPT-5.6-terra | 1.5441 | 12.8494 | 0.8622 |
| Baseline | Qwen3.5-27B | 0.8586 | 12.4159 | 0.8617 |
| Lexicon | Gemma-4-31b | 4.4488 | 21.5798 | 0.8856 |
| Lexicon | GPT-5.6-terra | 4.3653 | 21.4010 | 0.8834 |
| Lexicon | Qwen3.5-27B | 3.7827 | 20.6767 | 0.8804 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.9776 | 19.1242 | 0.8525 |
| Baseline | GPT-5.6-terra | 5.9578 | 26.6827 | 0.8809 |
| Baseline | Qwen3.5-27B | 1.1914 | 19.1487 | 0.8449 |
| Lexicon | Gemma-4-31b | 1.1382 | 20.5770 | 0.8561 |
| Lexicon | GPT-5.6-terra | 5.8671 | 27.6324 | 0.8837 |
| Lexicon | Qwen3.5-27B | 1.6406 | 20.6398 | 0.8521 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.3166 | 13.5255 | 0.8554 |
| Baseline | GPT-5.6-terra | 0.4430 | 13.5159 | 0.8590 |
| Baseline | Qwen3.5-27B | 0.3664 | 13.5176 | 0.8494 |
| Lexicon | Gemma-4-31b | 1.3116 | 20.0209 | 0.8702 |
| Lexicon | GPT-5.6-terra | 1.2960 | 19.1972 | 0.8716 |
| Lexicon | Qwen3.5-27B | 1.4384 | 20.2500 | 0.8654 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.3560 | 13.4491 | 0.8612 |
| Baseline | GPT-5.6-terra | 0.4862 | 13.4033 | 0.8659 |
| Baseline | Qwen3.5-27B | 0.7019 | 13.4046 | 0.8572 |
| Lexicon | Gemma-4-31b | 0.9594 | 15.3322 | 0.8640 |
| Lexicon | GPT-5.6-terra | 0.6871 | 14.2985 | 0.8676 |
| Lexicon | Qwen3.5-27B | 0.8670 | 14.1371 | 0.8599 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 6.4323 | 26.6546 | 0.8718 |
| Baseline | GPT-5.6-terra | 31.2051 | 52.4880 | 0.9314 |
| Baseline | Qwen3.5-27B | 2.4084 | 19.8988 | 0.8515 |
| Lexicon | Gemma-4-31b | 13.3405 | 37.3074 | 0.8972 |
| Lexicon | GPT-5.6-terra | 31.3266 | 53.1647 | 0.9338 |
| Lexicon | Qwen3.5-27B | 10.8785 | 33.1268 | 0.8857 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.7070 | 13.1166 | 0.8664 |
| Baseline | GPT-5.6-terra | 1.4518 | 13.0424 | 0.8657 |
| Baseline | Qwen3.5-27B | 1.0067 | 12.5322 | 0.8630 |
| Lexicon | Gemma-4-31b | 4.6571 | 21.8816 | 0.8900 |
| Lexicon | GPT-5.6-terra | 4.8112 | 22.4292 | 0.8895 |
| Lexicon | Qwen3.5-27B | 3.8300 | 21.3008 | 0.8856 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 1.0558 | 19.3262 | 0.8570 |
| Baseline | GPT-5.6-terra | 4.9633 | 26.2467 | 0.8833 |
| Baseline | Qwen3.5-27B | 1.3038 | 19.5234 | 0.8488 |
| Lexicon | Gemma-4-31b | 1.3428 | 19.5705 | 0.8605 |
| Lexicon | GPT-5.6-terra | 5.2909 | 26.8043 | 0.8863 |
| Lexicon | Qwen3.5-27B | 1.0025 | 19.8530 | 0.8538 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.3869 | 13.2931 | 0.8541 |
| Baseline | GPT-5.6-terra | 0.3629 | 13.3222 | 0.8576 |
| Baseline | Qwen3.5-27B | 0.2762 | 12.8097 | 0.8493 |
| Lexicon | Gemma-4-31b | 1.3062 | 20.3689 | 0.8696 |
| Lexicon | GPT-5.6-terra | 1.3955 | 19.6067 | 0.8708 |
| Lexicon | Qwen3.5-27B | 1.4367 | 20.2212 | 0.8651 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.4125 | 14.2330 | 0.8608 |
| Baseline | GPT-5.6-terra | 0.5480 | 13.2287 | 0.8643 |
| Baseline | Qwen3.5-27B | 0.6886 | 13.6135 | 0.8573 |
| Lexicon | Gemma-4-31b | 0.5775 | 14.9631 | 0.8626 |
| Lexicon | GPT-5.6-terra | 0.9376 | 14.5231 | 0.8668 |
| Lexicon | Qwen3.5-27B | 1.1107 | 14.2552 | 0.8597 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 6.3062 | 26.6146 | 0.8708 |
| Baseline | GPT-5.6-terra | 30.8087 | 52.3420 | 0.9316 |
| Baseline | Qwen3.5-27B | 2.7540 | 19.9402 | 0.8514 |
| Lexicon | Gemma-4-31b | 13.9570 | 37.5141 | 0.8978 |
| Lexicon | GPT-5.6-terra | 30.7845 | 52.9620 | 0.9335 |
| Lexicon | Qwen3.5-27B | 10.5563 | 32.5239 | 0.8848 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 0.6314 | 12.0142 | 0.8654 |
| Baseline | GPT-5.6-terra | 1.2025 | 13.1445 | 0.8652 |
| Baseline | Qwen3.5-27B | 1.0251 | 12.6076 | 0.8633 |
| Lexicon | Gemma-4-31b | 4.3077 | 21.7345 | 0.8892 |
| Lexicon | GPT-5.6-terra | 4.7300 | 21.9831 | 0.8886 |
| Lexicon | Qwen3.5-27B | 3.9613 | 21.3352 | 0.8864 |
| Setting | Model | BLEU | ChrF++ | BERTScore F1 |
|---|---|---|---|---|
| Baseline | Gemma-4-31b | 1.1051 | 19.0809 | 0.8552 |
| Baseline | GPT-5.6-terra | 4.6174 | 26.1733 | 0.8824 |
| Baseline | Qwen3.5-27B | 0.8305 | 19.0194 | 0.8497 |
| Lexicon | Gemma-4-31b | 1.4615 | 20.3558 | 0.8606 |
| Lexicon | GPT-5.6-terra | 5.6454 | 27.1608 | 0.8867 |
| Lexicon | Qwen3.5-27B | 1.1249 | 19.5336 | 0.8538 |
| Setting | Translation, Chintang to English | BLEU | chrF++ | F1 |
| Gemma-4-31B | ||||
| Reference | I live up there, on top of the ravine. | – | – | – |
| Baseline | I have a sister, Khojabhandu. | 0 | 0 | 0 |
| LEX | That is where the ravine is. | +3.3277 | +17.8306 | +0.0587 |
| LEX+SYN | Oh, I live up there in the ravine. | +30.6625 | +52.4039 | +0.1023 |
| Reference | Anita introduced Lokendra and Lakhman to each other. | – | – | – |
| Setting | Translation, Source to English | BLEU | chrF++ | F1 |
| Beja | ||||
| Reference | They wanted to eat him. | – | – | – |
| Baseline | The big house is here. | 0 | 0 | 0 |
| LEX | Those are the big ones. | 0.0000 | +4.0334 | 0.0018 |
| LEX+SYN | They wanted to kill him. | +14.8408 | +49.9049 | +0.0967 |
| Reference | His children and himself were really happy. | – | – | – |
| Model | Lang | LEX base | LEX + CoNLLU LEX | LEX + DEP LEX | LEX + SYN LEX | LEX + CoNLLU LEX + DEP | LEX + SYN LEX + CoNLLU | LEX + SYN LEX + DEP |
|---|---|---|---|---|---|---|---|---|
| Gemma | bej | +1.49 ∗ | +0.83 ∗ | +0.41 ∗ | +0.97 ∗ | +0.42 ∗ | +0.14 | +0.56 ∗ |
| Gemma | ckt | +0.29 | +0.50 ∗ | +0.31 | +0.50 ∗ | +0.19 | +0.00 | +0.19 |
| Gemma | cop | +2.56 ∗ | +1.13 ∗ | +0.73 ∗ | +1.19 ∗ | +0.40 ∗ | +0.06 | +0.46 ∗ |
| Gemma | ctn | +2.37 ∗ | +1.36 ∗ | +0.53 ∗ | +1.49 ∗ | +0.82 ∗ | +0.14 | +0.96 ∗ |
| Gemma | sjo | +0.34 ∗ | +1.48 ∗ | +0.54 ∗ | +1.46 ∗ | +0.94 ∗ | 0.02 | +0.92 ∗ |
| GPT | bej | +1.26 ∗ | +0.87 ∗ | +0.64 ∗ | +0.99 ∗ | +0.23 | +0.12 | +0.36 |
Machine Translation
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
See all 283 papers