Komi-Zyrian, Moksha, Erzya and Udmurt have mature rule-based resources, among them finite-state analysers, constraint grammars and bilingual dictionaries, which language models do not use. We ask whether giving a model access to these resources improves machine translation from the four languages into English, and whether it matters how the access is provided: as analyses computed in advance and written into the prompt, or as tools that the model calls itself through the Model Context Protocol (MCP). We evaluate manually: an expert linguist read every English output of the strongest model under three conditions, 816 sentences in all, accepted or rejected each one, and named the main error in every rejected translation. With the analyses in the prompt, the number of accepted translations rises from 13 to 43 out of 272; with the tools attached it rises to 55. The gain comes from the bilingual dictionary. For Udmurt, whose dictionary has no English side, the prompt-based condition brings no gain at all, while for the other three languages an English gloss reaches the prompt for 74 to 80% of the source words. Tool calling is also much harder to get working than prompt injection. Of 31 open-weight models, 14 never call a tool, 11 of them because they cannot produce a usable call, and half of the calls that are made name a language the toolkit does not support. We release the data set, every model output and all tool-call logs.
Figures & tables
Figure 1: How conditions are built.
Condition
What is added to the request
Tools
Models
zeroshot
nothing
–
31
mcp_prompt
the tool-use system prompt, with no tools attached (control)
–
31
inject
lemma, reading and dictionary entry of every source word, computed in advance, in the user message
6
31
inject_rich
the same, computed with the thirteen-tool surface
13
14
mcp
the tool-use system prompt, with the tool schemas attached; the model decides whether to call
6
31
mcp_coarse
as mcp , with one sentence-level tool
1
31
Table 1: The seven conditions. Tools is the surface that supplies the linguistic material; Models is how many of the 31 models ran the condition for Komi-Zyrian. The other four languages ran zeroshot , mcp_prompt , inject , mcp and mcp_coarse on ten models.
Language
zeroshot
inject
mcp
Komi-Zyrian
8 (11.8%)
17 (25.0%)
23 (33.8%)
Moksha
1 (1.5%)
11 (16.2%)
11 (16.2%)
Erzya
0 (0.0%)
11 (16.2%)
11 (16.2%)
Udmurt
4 (5.9%)
4 (5.9%)
10 (14.7%)
Total
13 (4.8%)
43 (15.8%)
55 (20.2%)
Table 2: Outputs accepted as conveying the content of the English reference, by language and condition. Every output not accepted carries exactly one primary error category, defined in § 5.1 and counted in Table 3 .
Primary error
Definition
Komi-Zyrian
Moksha
Erzya
Udmurt
Wholesale replacement
the situation of the reference is replaced, or several independent major changes are made to it
39/10/9
48/21/26
54/24/25
42/41/23
Lexical substitution
an action, entity, property or idiomatic meaning is replaced while part of the sentence structure survives
10/16/15
6/12/9
9/11/11
11/14/13
Participant, referent
who acts or who experiences the event changes: person, gender, kinship or the direction of an interaction
3/9/9
2/8/13
1/10/11
3/2/7
Polarity, modality, act
negation, propositional force or the status as statement, question or command changes; incompatible event states included
3/4/3
2/5/2
0/2/4
1/2/4
Omission
an important proposition or semantic component of the reference is missing
2/7/3
8/8/5
4/9/5
5/4/6
Unsupported addition
a substantive detail that the English reference does not establish is introduced
2/5/4
0/2/2
0/1/1
1/0/4
Table 3: The error taxonomy, and the primary error category of every rejected output of Qwen3.8-27B as zeroshot / inject / mcp counts out of 68 sentences per cell. With the accepted outputs of Table 2 every column sums to 68.
Model
Size
Calls
Model
Size
Calls
Qwen3-0.6B
0.6
0%
Mistral-7B-Instruct-v0.3
7.0
0%
gemma-3-1b
1.0
0%
OLMo-2-1124-7B
7.0
0%
xLAM-2-1b
1.0
23%
Qwen2.5-7B
7.0
3%
Llama-3.2-1B
1.2
71%
Hermes-3-Llama-3.1-8B
8.0
7%
Hammer2.1-1.5b
1.5
0%
Llama-3.1-8B
8.0
98%
Qwen2.5-1.5B
1.5
50%
Ministral-8B
8.0
100%
Table 4: Share of sentences on which each model emitted at least one tool call under mcp , mcp_coarse and mcp_rich . Fourteen models never call, and one more calls on fewer than one sentence in two hundred. Size is in billions of parameters.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Direction
no tools
tool prompt
inject
mcp
mcp coarse
Komi-Zyrian → English
19.8
15.6 (-4.3)
25.3 (+5.5)
18.5 (-1.3)
18.9 (-1.0)
Moksha → English
17.4
16.0 (-1.4)
22.1 (+4.7)
15.2 (-2.2)
15.2 (-2.2)
Erzya → English
15.1
11.9 (-3.2)
20.0 (+4.9)
14.5 (-0.6)
14.5 (-0.6)
Udmurt → English
16.3
14.0 (-2.2)
16.4 (+0.2)
14.8 (-1.4)
15.6 (-0.7)
English → Komi-Zyrian
4.4
3.3 (-1.1)
3.7 (-0.7)
4.5 (+0.1)
2.6 (-1.8)
English → Moksha
3.4
2.3 (-1.1)
2.1 (-1.4)
3.5 (+0.1)
2.0 (-1.5)
Appendix
Table 5: chrF++ averaged over the ten models that ran every direction, 68 sentences each. Parentheses give the change from the zero-shot baseline. The floor is 11.5 for English targets and 9.3 to 10.8 for the Uralic ones.
zeroshot
inject
mcp
Language
acc.
chrF
acc.
chrF
acc.
chrF
Komi-Zyrian
8
29.9
17
37.8
23
40.5
Moksha
1
22.4
11
30.2
11
31.6
Erzya
0
20.2
11
29.7
11
30.9
Udmurt
4
24.5
4
26.2
10
29.7
Appendix
Table 6: The twelve cells where both kinds of evidence exist: Qwen3.8-27B, four languages, three conditions. acc. is how many of the 68 outputs an expert linguist accepted; chrF is chrF++ on the same outputs. The two agree at r=0.98 (Pearson) and ρ=0.90 (Spearman).
Model
Size
Tool calling
Evidence
Qwen3-0.6B
0.6
works
probe
xLAM-2-1b-fc-r
1.0
works
probe
gemma-3-1b-it
1.0
none
probe, parser search, runs
Llama-3.2-1B-Instruct
1.2
works
3899 calls in the runs
Hammer2.1-1.5b
1.5
unreadable
emits calls no parser reads
Qwen2.5-1.5B-Instruct
1.5
works
probe
Appendix
Table 7: Whether each model can emit a tool call that succeeds. A model counts as capable if the positive control passes, or if an exhaustive parser search finds a parser under which it passes, or if it made successful calls during the sweep – the control alone gives false negatives, since whether a model calls a tool depends on the prompt.
Model
Codes
Correct
Unsupported
Llama-3.1-8B-Instruct
53513
48.4%
50.3%
Llama-3.2-3B-Instruct
40906
35.0%
36.4%
Qwen3-1.7B
40682
41.9%
33.7%
Ministral-8B-Instruct-2410
36292
38.9%
61.1%
Llama-3.2-1B-Instruct
30530
21.8%
78.2%
Qwen3-8B
30207
45.1%
54.9%
Appendix
Table 8: Language codes the models sent to the tools, counted per call. Correct is a code UralicNLP supports for the language actually being processed; unsupported names nothing the library knows. The two do not sum to 100: the remainder, 8.4% over the whole study, is a code the library supports for the wrong language, which returns an answer about a language nobody asked for.
Task
Condition
Well-formed
Copied
Wrong lang.
Degenerate
Empty
Into the endangered languages
no tools
72.0
3.8
11.0
13.1
0.1
tool prompt
50.1
7.8
24.0
9.5
8.5
inject
67.1
5.9
10.4
15.8
0.8
inject+
66.3
6.8
3.9
23.0
0.0
mcp
42.5
18.5
22.9
9.2
6.8
mcp coarse
43.3
11.7
25.0
11.3
8.7
Appendix
Table 9: Share of outputs (%) by failure mode, over every model and sentence in each group of directions. Well-formed is none of the other four, a test of form and not of adequacy: such an output can still be a wrong translation. Copied reproduces the input, wrong lang. is in the source language, degenerate loops or runs past four times the reference length, empty returned nothing. Rows are conditions; the five columns sum to 100.
Tokenizer
eng
kpv
mdf
myv
udm
Qwen3-0.6B
1.51
3.96
3.65
3.77
3.52
Qwen3-1.7B
1.51
3.96
3.65
3.77
3.52
Qwen3-4B
1.51
3.96
3.65
3.77
3.52
Qwen3-8B
1.51
3.96
3.65
3.77
3.52
Llama-3.2-1B-Instruct
1.51
4.24
3.44
3.62
3.48
Llama-3.2-3B-Instruct
1.51
4.24
3.44
3.62
3.48
Appendix
Table 10: Subword pieces per word, measured on the parallel set itself, so the six columns describe the same 68 sentences in six languages. A value of 1.0 would mean one token per word; English costs 1.36 to 1.59 and the endangered languages 2.0 to 3.3 times that.
Model
zs
inj
mcp
zs
inj
mcp
kpv–eng
granite-3.3-8b-instruct
18.8
22.7
17.1
1.6
1.9
1.6
Hermes-3-Llama-3.1-8B
23.0
27.8
22.5
5.2
6.2
4.9
Llama-3.1-8B-Instruct
21.7
29.5
21.2
3.7
9.5
3.8
Mistral-7B-Instruct-v0.3
17.3
24.8
17.7
1.8
3.7
2.2
Mistral-Nemo-Instruct-2407
16.9
22.5
13.7
2.2
3.5
1.6
Appendix
Table 11: Every model and every direction into English, scored on the model extraction. The first three columns are chrF++ and the last three BLEU. The chrF++ floor is 11.5.
Model
zs
inj
mcp
crs
zs
inj
mcp
crs
eng–kpv
granite-3.3-8b-instruct
5.9
3.3
1.8
2.2
6.2
3.4
1.9
2.3
Hermes-3-Llama-3.1-8B
7.0
7.8
6.6
5.7
7.2
8.2
6.8
6.2
Llama-3.1-8B-Instruct
2.4
2.1
2.9
2.0
2.4
2.3
3.2
2.1
Mistral-7B-Instruct-v0.3
3.0
3.6
2.9
1.5
3.2
3.5
3.1
1.6
Mistral-Nemo-Instruct-2407
3.9
4.3
1.8
2.2
4.0
4.5
2.0
2.7
Appendix
Table 12: Every model and every direction into a Uralic language, scored on the model extraction. The first four columns are chrF++ and the last four the same measure on lemmatised text. Floors are 9.3 to 10.8 chrF++ and 9.9 to 11.5 lemmatised.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan