Dataset Collection and Fine-Tuning of NLLB for German--English Translation

SK Deeraj; Yash Ingle; Pruthwik Mishra

cs.CLNov 5, 2026

Dataset Collection and Fine-Tuning of NLLB for German--English Translation

Authors: SK Deeraj, Yash Ingle, Pruthwik Mishra

Organizations: Department of Artificial Intelligence, Sardar Vallabhbhai National Institute of Technology, India

Abstract

This paper describes the curation of a high-quality parallel corpus from OPUS Books and the fine-tuning of Meta's NLLB-200 models for German--English translation. We detail a dataset extraction pipeline that filters 45k raw OPUS sentence pairs to 25k high-quality examples through encoding validation, length filtering, language identification, deduplication, and quality scoring. We present a reproducible fine-tuning methodology optimized for consumer GPUs using mixed precision, gradient accumulation, and gradient checkpointing. Our fine-tuned NLLB-600M model achieves statistically significant improvements (+3.4 BLEU, +2.6 chrF++, -3.2 TER) over the base model on literary and technical German-English translation. We demonstrate practical deployment by integrating the fine-tuned model into LinguaHaru, an existing open-source document translation system, achieving format-preserving translation across PDF and Office documents. All dataset preparation scripts, training code, and evaluation tools are provided for reproducibility. The fine-tuned model is available at HuggingFace.

Figures & tables

Split	#Pairs	Avg. DE chars	Avg. EN chars
Train	20,000	175	150
Validation	2,500	172	149
Test	2,500	168	154
Total	25,000	171	151

Table 1: OPUS-derived dataset statistics after filtering

Parameter	Value
Base model	facebook/nllb-200-distilled-600M
Per-device batch size	1–4 (GPU dependent)
Gradient accumulation	32
Effective batch size	32
Learning rate	$2\times 10^{-5}$ with linear warmup
Warmup	10% of steps

Table 2: Training hyperparameters and practical settings

Model	BLEU	chrF++	TER
NLLB-200 Base	32.4	58.7	48.2
Fine-tuned (25k)	35.8	61.3	45.0
Improvement	+3.4	+2.6	-3.2
95% CI (Bootstrap)	±0.6	±0.4	±0.5

Table 3: Translation quality on held-out test set (2,500 pairs)

Length (tokens)	Count	Base	Fine-tuned
Short (5-15)	612	38.2	41.5 (+3.3)
Medium (16-40)	1,456	33.1	36.9 (+3.8)
Long (41-100)	432	26.7	30.2 (+3.5)

Table 4: BLEU scores by sentence length category

Error Type	Base	Fine-tuned
Rare terminology	18	12
Awkward phrasing	24	16
Gender/pronoun errors	9	6
Omissions	7	4
Mistranslations	6	3
Total errors	64	41

Table 5: Error type frequency (N=100 test samples)

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 275 papers

KletterMix: Climbing Toward High-Quality German Pretraining Data
Jun 2, 2026Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia +4Language Model Pretraining Synthetic Data Pretraining
AI & ML Group, TU Darmstadt · Lab1141 · Fraunhofer IAIS +3
North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+)
Sep 12, 2026Tom Kocmi, Alexandre Bérard, Phil Blunsom +20Machine Translation Language Model Post-Training
Cohere
ReMova: Fine-tuning LLMs for English to Belarusian translation
Sep 14, 2026Mikita Pilinka, Aliaksandr Kliujeŭ, David Samuel +1LLM Fine-Tuning Low-Resource MT
University of Oslo Department of Informatics · Independent Researcher

Jun 2, 2026J/K move · Enter open · S save

KletterMix: Climbing Toward High-Quality German Pretraining Data

Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia, Abbas Goher Khan+3

AI & ML Group, TU Darmstadt · Lab1141 · Fraunhofer IAIS+3

Language Model Pretraining Synthetic Data Pretraining Low-Resource Language Processing

Open paper AskPDF arXiv

High-quality pretraining data is a central ingredient in modern language models, but German-language resources remain far less developed than their English counterparts: they are often smaller, less carefully curated, weakly documented, and rarely validated through controlled training experiments. We introduce KletterMix, a high-quality German corpus for language model pretraining and annealing, designed as a reusable dataset artifact for the natural language processing and modeling community. KletterMix is built by translating a state-of-the-art English pretraining corpus into German while preserving document boundaries, metadata, source structure, and topical diversity. This construction yields a German corpus with the scale and diversity of a modern pretraining dataset, while enabling direct comparison to its English source. We document the dataset through a broad set of corpus-level analyses, including translation quality, document length distributions, topic coverage, source composition, and geographic metadata. Using COMETKiwi, we show that the translated documents achieve strong quality across diverse domains, suggesting that careful translation can preserve much of the semantic and stylistic richness of the original corpus. Beyond dataset construction, we evaluate KletterMix as training data. Through controlled pretraining and annealing ablations against established German corpora, we show that models trained on KletterMix achieve measurable improvements on German-language downstream evaluations. These results demonstrate that carefully curated translated data can substantially strengthen the German pretraining data ecosystem.

Abstract

Machine Translation

Explore similar work

KletterMix: Climbing Toward High-Quality German Pretraining Data

North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+)

ReMova: Fine-tuning LLMs for English to Belarusian translation