Do Rule-Based Tools Help LLMs Translate Endangered Uralic Languages in the Prompt or over MCP?
Organizations: Metropolia University of Applied Sciences, Helsinki, Finland · Neapolis University Pafos, Paphos, Cyprus
Abstract
Komi-Zyrian, Moksha, Erzya and Udmurt have mature rule-based resources, among them finite-state analysers, constraint grammars and bilingual dictionaries, which language models do not use. We ask whether giving a model access to these resources improves machine translation from the four languages into English, and whether it matters how the access is provided: as analyses computed in advance and written into the prompt, or as tools that the model calls itself through the Model Context Protocol (MCP). We evaluate manually: an expert linguist read every English output of the strongest model under three conditions, 816 sentences in all, accepted or rejected each one, and named the main error in every rejected translation. With the analyses in the prompt, the number of accepted translations rises from 13 to 43 out of 272; with the tools attached it rises to 55. The gain comes from the bilingual dictionary. For Udmurt, whose dictionary has no English side, the prompt-based condition brings no gain at all, while for the other three languages an English gloss reaches the prompt for 74 to 80% of the source words. Tool calling is also much harder to get working than prompt injection. Of 31 open-weight models, 14 never call a tool, 11 of them because they cannot produce a usable call, and half of the calls that are made name a language the toolkit does not support. We release the data set, every model output and all tool-call logs.
Figures & tables
| Condition | What is added to the request | Tools | Models |
|---|---|---|---|
| zeroshot | nothing | – | 31 |
| mcp_prompt | the tool-use system prompt, with no tools attached (control) | – | 31 |
| inject | lemma, reading and dictionary entry of every source word, computed in advance, in the user message | 6 | 31 |
| inject_rich | the same, computed with the thirteen-tool surface | 13 | 14 |
| mcp | the tool-use system prompt, with the tool schemas attached; the model decides whether to call | 6 | 31 |
| mcp_coarse | as mcp , with one sentence-level tool | 1 | 31 |
| Language | zeroshot | inject | mcp |
|---|---|---|---|
| Komi-Zyrian | 8 (11.8%) | 17 (25.0%) | 23 (33.8%) |
| Moksha | 1 (1.5%) | 11 (16.2%) | 11 (16.2%) |
| Erzya | 0 (0.0%) | 11 (16.2%) | 11 (16.2%) |
| Udmurt | 4 (5.9%) | 4 (5.9%) | 10 (14.7%) |
| Total | 13 (4.8%) | 43 (15.8%) | 55 (20.2%) |
| Primary error | Definition | Komi-Zyrian | Moksha | Erzya | Udmurt |
|---|---|---|---|---|---|
| Wholesale replacement | the situation of the reference is replaced, or several independent major changes are made to it | 39/10/9 | 48/21/26 | 54/24/25 | 42/41/23 |
| Lexical substitution | an action, entity, property or idiomatic meaning is replaced while part of the sentence structure survives | 10/16/15 | 6/12/9 | 9/11/11 | 11/14/13 |
| Participant, referent | who acts or who experiences the event changes: person, gender, kinship or the direction of an interaction | 3/9/9 | 2/8/13 | 1/10/11 | 3/2/7 |
| Polarity, modality, act | negation, propositional force or the status as statement, question or command changes; incompatible event states included | 3/4/3 | 2/5/2 | 0/2/4 | 1/2/4 |
| Omission | an important proposition or semantic component of the reference is missing | 2/7/3 | 8/8/5 | 4/9/5 | 5/4/6 |
| Unsupported addition | a substantive detail that the English reference does not establish is introduced | 2/5/4 | 0/2/2 | 0/1/1 | 1/0/4 |
| Model | Size | Calls | Model | Size | Calls |
|---|---|---|---|---|---|
| Qwen3-0.6B | 0.6 | 0% | Mistral-7B-Instruct-v0.3 | 7.0 | 0% |
| gemma-3-1b | 1.0 | 0% | OLMo-2-1124-7B | 7.0 | 0% |
| xLAM-2-1b | 1.0 | 23% | Qwen2.5-7B | 7.0 | 3% |
| Llama-3.2-1B | 1.2 | 71% | Hermes-3-Llama-3.1-8B | 8.0 | 7% |
| Hammer2.1-1.5b | 1.5 | 0% | Llama-3.1-8B | 8.0 | 98% |
| Qwen2.5-1.5B | 1.5 | 50% | Ministral-8B | 8.0 | 100% |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Direction | no tools | tool prompt | inject | mcp | mcp coarse |
|---|---|---|---|---|---|
| Komi-Zyrian English | 19.8 | 15.6 (-4.3) | 25.3 (+5.5) | 18.5 (-1.3) | 18.9 (-1.0) |
| Moksha English | 17.4 | 16.0 (-1.4) | 22.1 (+4.7) | 15.2 (-2.2) | 15.2 (-2.2) |
| Erzya English | 15.1 | 11.9 (-3.2) | 20.0 (+4.9) | 14.5 (-0.6) | 14.5 (-0.6) |
| Udmurt English | 16.3 | 14.0 (-2.2) | 16.4 (+0.2) | 14.8 (-1.4) | 15.6 (-0.7) |
| English Komi-Zyrian | 4.4 | 3.3 (-1.1) | 3.7 (-0.7) | 4.5 (+0.1) | 2.6 (-1.8) |
| English Moksha | 3.4 | 2.3 (-1.1) | 2.1 (-1.4) | 3.5 (+0.1) | 2.0 (-1.5) |
| zeroshot | inject | mcp | ||||
|---|---|---|---|---|---|---|
| Language | acc. | chrF | acc. | chrF | acc. | chrF |
| Komi-Zyrian | 8 | 29.9 | 17 | 37.8 | 23 | 40.5 |
| Moksha | 1 | 22.4 | 11 | 30.2 | 11 | 31.6 |
| Erzya | 0 | 20.2 | 11 | 29.7 | 11 | 30.9 |
| Udmurt | 4 | 24.5 | 4 | 26.2 | 10 | 29.7 |
| Model | Size | Tool calling | Evidence |
|---|---|---|---|
| Qwen3-0.6B | 0.6 | works | probe |
| xLAM-2-1b-fc-r | 1.0 | works | probe |
| gemma-3-1b-it | 1.0 | none | probe, parser search, runs |
| Llama-3.2-1B-Instruct | 1.2 | works | 3899 calls in the runs |
| Hammer2.1-1.5b | 1.5 | unreadable | emits calls no parser reads |
| Qwen2.5-1.5B-Instruct | 1.5 | works | probe |
| Model | Codes | Correct | Unsupported |
|---|---|---|---|
| Llama-3.1-8B-Instruct | 53513 | 48.4% | 50.3% |
| Llama-3.2-3B-Instruct | 40906 | 35.0% | 36.4% |
| Qwen3-1.7B | 40682 | 41.9% | 33.7% |
| Ministral-8B-Instruct-2410 | 36292 | 38.9% | 61.1% |
| Llama-3.2-1B-Instruct | 30530 | 21.8% | 78.2% |
| Qwen3-8B | 30207 | 45.1% | 54.9% |
| Task | Condition | Well-formed | Copied | Wrong lang. | Degenerate | Empty |
|---|---|---|---|---|---|---|
| Into the endangered languages | no tools | 72.0 | 3.8 | 11.0 | 13.1 | 0.1 |
| tool prompt | 50.1 | 7.8 | 24.0 | 9.5 | 8.5 | |
| inject | 67.1 | 5.9 | 10.4 | 15.8 | 0.8 | |
| inject+ | 66.3 | 6.8 | 3.9 | 23.0 | 0.0 | |
| mcp | 42.5 | 18.5 | 22.9 | 9.2 | 6.8 | |
| mcp coarse | 43.3 | 11.7 | 25.0 | 11.3 | 8.7 |
| Tokenizer | eng | kpv | mdf | myv | udm |
|---|---|---|---|---|---|
| Qwen3-0.6B | 1.51 | 3.96 | 3.65 | 3.77 | 3.52 |
| Qwen3-1.7B | 1.51 | 3.96 | 3.65 | 3.77 | 3.52 |
| Qwen3-4B | 1.51 | 3.96 | 3.65 | 3.77 | 3.52 |
| Qwen3-8B | 1.51 | 3.96 | 3.65 | 3.77 | 3.52 |
| Llama-3.2-1B-Instruct | 1.51 | 4.24 | 3.44 | 3.62 | 3.48 |
| Llama-3.2-3B-Instruct | 1.51 | 4.24 | 3.44 | 3.62 | 3.48 |
| Model | zs | inj | mcp | zs | inj | mcp |
|---|---|---|---|---|---|---|
| kpv–eng | ||||||
| granite-3.3-8b-instruct | 18.8 | 22.7 | 17.1 | 1.6 | 1.9 | 1.6 |
| Hermes-3-Llama-3.1-8B | 23.0 | 27.8 | 22.5 | 5.2 | 6.2 | 4.9 |
| Llama-3.1-8B-Instruct | 21.7 | 29.5 | 21.2 | 3.7 | 9.5 | 3.8 |
| Mistral-7B-Instruct-v0.3 | 17.3 | 24.8 | 17.7 | 1.8 | 3.7 | 2.2 |
| Mistral-Nemo-Instruct-2407 | 16.9 | 22.5 | 13.7 | 2.2 | 3.5 | 1.6 |
| Model | zs | inj | mcp | crs | zs | inj | mcp | crs |
| eng–kpv | ||||||||
| granite-3.3-8b-instruct | 5.9 | 3.3 | 1.8 | 2.2 | 6.2 | 3.4 | 1.9 | 2.3 |
| Hermes-3-Llama-3.1-8B | 7.0 | 7.8 | 6.6 | 5.7 | 7.2 | 8.2 | 6.8 | 6.2 |
| Llama-3.1-8B-Instruct | 2.4 | 2.1 | 2.9 | 2.0 | 2.4 | 2.3 | 3.2 | 2.1 |
| Mistral-7B-Instruct-v0.3 | 3.0 | 3.6 | 2.9 | 1.5 | 3.2 | 3.5 | 3.1 | 1.6 |
| Mistral-Nemo-Instruct-2407 | 3.9 | 4.3 | 1.8 | 2.2 | 4.0 | 4.5 | 2.0 | 2.7 |
Machine Translation
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
See all 287 papers