INVAR-Agri: Detecting Discrepancies in Multilingual Operator Manuals of Agricultural Implements with a Fine-Tuned LLM

Timo Oksanen

cs.CLOct 26, 2026

INVAR-Agri: Detecting Discrepancies in Multilingual Operator Manuals of Agricultural Implements with a Fine-Tuned LLM

Authors: Timo Oksanen

Organizations: Technical University of Munich, Germany; Chair of Agrimechatronics; Munich Institute of Robotics and Machine Intelligence (MIRMI)

Abstract

Operator manuals of agricultural implements are delivered in many languages, and the standards require that competent persons review every translation, which makes this review the bottleneck. This paper presents a proof of concept that compares two training workflows on the same LLM, EuroLLM-1.7B-Instruct, trained on operator manuals of one manufacturer. In approach A, a back-translator translates each language version back into English, and an English reader compares it with the English version. In approach B, a checker reads the English version and the translation together, where the checker reports differences. The test sets contain errors planted in numbers, units, opposite words, obligations, negations, and conditions in 24 languages, and they include unmodified text and meaning-preserving edits for alerts and false alarms. The trained back-translator concealed many planted errors, most of all weakened obligations, while translation-quality scores rose. In contrast, the separately trained checker detected most planted errors, although it was weak for units and for number changes unlike those in training. A check by the author confirmed the concealment. In addition, a simple rule check caught the number and unit changes, but it alerted on harmless rewrites. A check of alerts on real Finnish translations showed that most alerts were false, caused by language-specific wording, so the precision on real text remains a limitation. Within this setting, a trained checker might support, not replace, the review by competent persons, and firmer conclusions need more data.

Figures & tables

Error type	Planted by	Items	English version	Finnish version, planted or removed words in bold
number	rule	$1094$	after 30 hours of operation.	30 käyttötunnin jälkeen. $\rightarrow$ 300 käyttötunnin jälkeen.
unit	rule	$383$	Retighten the bolts to a tightening torque of 85 Nm.	Kiristä ruuvit kiristystiukkuuteen 85 Nm. $\rightarrow$ Kiristä ruuvit kiristystiukkuuteen 85 N .
polarity	LLM	$1098$	Switch off the PTO shaft drive.	Kytke nivelakselin käyttö pois päältä. $\rightarrow$ Kytke nivelakselin käyttö päälle .
obligation	LLM	$756$	The empty weight of the tractor must be greater than the weight of the machine.	Traktorin omapainon on oltava koneen painoa suurempi. $\rightarrow$ Traktorin omapainon tulisi olla koneen painoa suurempi.
negation	LLM	$988$	The electrical supply to the tractor must not be connected when the lighting equipment is being fitted.	Asentaessa valaistusjärjestelmää paikalleen traktorin virransyöttö ei saa olla liitettynä. $\rightarrow$ Asentaessa valaistusjärjestelmää paikalleen traktorin virransyöttö saa olla liitettynä.
condition	LLM	$1013$	Lubricate all bearings after cleaning.	Voitele kaikki laakerit puhdistuksen jälkeen . $\rightarrow$ Voitele kaikki laakerit.

Table 1: The six error types of the test sets, the planting method, and the number of test items in 24 languages, including one example per type from the Finnish version of the rake manual with the planted or removed words in bold.

Panel of Figure 1 — Figure 1: Approach A uses the LLM to translate the translation back into English for comparison by an English reviewer, whereas approach B uses the LLM to read the English version and the translation together to answer “Same meaning.” or “Check:” with the difference, using the same untuned or fine-tuned LLM in both approaches.

	A: back-translator	B: checker
training examples	$198030$ pairs	$17295$ examples
shown per pass (one per English text)	$10553$	$6947$
optimiser steps per pass	165	109
development measure	chrF	balanced accuracy
best pass, seeds 1 / 2 / 3	6 / 7 / 5	12 / 5 / 13
passes run (early stopping)	9 / 10 / 8	15 / 8 / 16

Table 2: Training of the back-translator for approach A and of the checker for approach B, including the values of seeds 1, 2, and 3.

Figure 2: Development score after every pass for three seeds per approach, where dots indicate the best pass kept as the model.

Models
LLM of both approaches	utter-project/EuroLLM-1.7B-Instruct, revision a25c7fa
COMET	Unbabel/wmt22-comet-da, revision 2760a223
helper models	Gemma 4 (26B), gpt-oss-120b, Clef; Ollama 0.35.1
audit of plantings	Claude (Opus 5.5), 192 plantings in the 16 added languages
Decoding and training
decoding	greedy; at most 512 new tokens (back-translation), 128 (checker)

Table 3: Settings of the models, decoding, and training, as well as the denominators of the test sets and of the checks by the author.

	Untuned	Fine-tuned	Seed range
A: back-translation of the unmodified test text
COMET, all units	80.0	84.9	84.8 to 84.9
COMET, new text	76.9	77.6	76.7 to 78.2
chrF, all units	50.9	70.7	70.1 to 71.1
chrF, text close to training text	52.6	83.0	81.1 to 84.7
chrF, new text	48.5	50.6	48.9 to 51.8

Table 4: Standard evaluation on the unmodified text of the two test manuals using the untuned and fine-tuned model, where fine-tuned values show the mean of three seeds and the range of the seeds.

	A (%)		B (%)
Error type	untuned	fine-tuned	untuned	fine-tuned
number	97	93	26	85
unit	100	88	15	62
polarity	70	60	17	93
obligation	94	30	20	96
negation	85	66	28	98

Table 5: Detection of planted errors per error type, alerts and false alarms in percent for the untuned and fine-tuned model, based on the mean of three seeds across 24 languages, with approach A decided by the element rules.

Figure 3: Detection per error type from the untuned model, represented by open circles, to the fine-tuned model, represented by filled circles with a mean of three seeds and a range band, showing approach A on the left and approach B on the right with alerts on unmodified text in the panel titles.

Figure 4: Share of planted errors detected in the back-translation for approach A, comparing the untuned, fine-tuned, and overtrained models after epoch 1 and epoch 2.

Figure 5: Detection of the number and unit changes of the test set for approach B, which differ from the changes in training, showing the untuned model as open circles and the fine-tuned mean as filled circles with a range band of three seeds and the number of test items n.

Figure 6: Change of detection by fine-tuning per language and error type in points, calculated as the mean of three seeds minus the untuned model, where blue indicates more errors detected and red indicates fewer, showing approach A on the left and approach B on the right. The changes were computed before rounding. Some cells are small, with as few as 3 items for a word-level type and 8 for units in one language.

Figure 7: Detection of planted errors per language and error type in percent for the untuned and fine-tuned model, based on the mean of three seeds for approach A and approach B. Some cells are small, with as few as 3 items for a word-level type and 8 for units in one language.

	number	unit	without conditions	alerts	false alarms
rule check	100	100	35	0.8	37.7
checker	85	62	90	3.4	12.7
rule check + checker	100	100	97	4.0	38.7

Table 6: Detection of planted errors per error type, alerts on unmodified text, and false alarms on meaning-preserving edits in percent, for the rule check alone, the fine-tuned checker alone (mean of three seeds), and both combined, across 24 languages.

		A (%)		B (%)
Similarity group	$n$	untuned	fine-tuned	untuned	fine-tuned
exact copy	1243	85	61	24	93
high (at least 0.8)	662	87	58	26	92
partial (0.5 to 0.8)	689	83	65	19	88
new text	1725	90	75	21	89

Table 7: Detection of planted errors without conditions in percent, per similarity group of the test text to the training text, for untuned and fine-tuned (mean of three seeds) versions of approach A and approach B.

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 90 papers

Machine-Readable Operator Manuals over the ISO 11783 Network for Offline, Multilingual LLM Assistance
Oct 22, 2026Timo Oksanen, Julius GunData Integration Multilingual QA
Chair of Agrimechatronics, Technical University of Munich, Germany · Munich Institute of Robotics and Machine Intelligence (MIRMI), Germany
Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators
Oct 24, 2026Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva +3MT Evaluation LLM-as-a-Judge
Yandex · HSE University
From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudication
May 31, 2026Máté Metzger, Nadnapang Phophichit, Hansa DhammahasoLLM Auditing Machine Translation
Independent Researcher, Hungary · Nibbana Meditation Centre, Thailand

Oct 22, 2026J/K move · Enter open · S save

Machine-Readable Operator Manuals over the ISO 11783 Network for Offline, Multilingual LLM Assistance

Timo Oksanen, Julius Gun

Chair of Agrimechatronics, Technical University of Munich, Germany · Munich Institute of Robotics and Machine Intelligence (MIRMI), Germany

Data Integration Multilingual QA Multilingual IR

Open paper AskPDF arXiv

If agricultural machinery is to support intelligent, offline assistance, then the system must access technical documentation dynamically. In this paper, the Semantic Synchronization of operator manuals over ISO 11783-based CAN buses is analyzed, defined as the process that brings the local knowledge base of a tractor-side assistant into agreement with the operator manual of the connected implement. Data transfer performance was benchmarked using Markdown, JSON, and XML; various open-weight Large Language Models (LLMs) were also evaluated in cross-lingual query tasks. The automatic evaluation of the answers was validated against four human raters. A complete manual was transferred over a physical ISO 11783 network in 31 to 89 s, depending on format and transport settings, which is comparable to the object pool upload that operators already wait for at the first coupling of an implement. Markdown produced the smallest files, while the question-answering performance was comparable across the three formats for most models. With retrieval-augmented generation, locally run models answered most questions on the English manual correctly, with no advantage of the largest model over 7B and 8B models, but performed worse on the five other language editions. Keyword retrieval failed across languages, whereas semantic retrieval provided the cross-lingual capability. The exchange requires only a new parameter group that uses the existing request and transport mechanisms of ISO 11783, which makes it a candidate for standardization. The study is a proof of concept with inference on PC-grade hardware; deployment on embedded agricultural hardware remains future work.

Abstract

Language Model Error Detection

Explore similar work

Machine-Readable Operator Manuals over the ISO 11783 Network for Offline, Multilingual LLM Assistance

Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudication