The University of Melbourne WMT 2026 CreoleMT Submission: A Domain-Balanced Approach to Low-Resource Pacific Creole Machine Translation

Raphaël Merx; Nick Thieberger; Ekaterina Vylomova

cs.CLSep 12, 2026

The University of Melbourne WMT 2026 CreoleMT Submission: A Domain-Balanced Approach to Low-Resource Pacific Creole Machine Translation

Authors: Raphaël Merx, Nick Thieberger, Ekaterina Vylomova

Organizations: The University of Melbourne

Abstract

For our submission to the WMT26 Creole Language Translation Shared Task, we focus on machine translation (MT) models for Pacific creoles: Tok Pisin, Bislama, and Solomon Pijin, with particular attention to broad domain performance. After pre-training on a large collection of domain-imbalanced data, we continue fine-tuning on a diverse mix of domain-balanced data. We rely on a number of data collection and preparation techniques, including LLM-assisted respelling and alignment, back-translation, and distillation from Gemini for domains originally not present in training data. Evaluated on Bouquet and a novel test set made of spoken language transcripts, our models beat open model baselines by 3+ chrF++ points in all directions with human-original references. Looking ahead, we plan to develop human-translated test sets for Solomon Pijin and Bislama, and to distil our best models into much smaller ones that retain broad domain coverage.

Figures & tables

Pre-train
Source	tpi	pis	bis
JW300 (OPUS)	371,822	–	–
Web scrape	72,319	13,809	17,142
NWT Bible	28,526	28,562	28,568
News, back-transl.	–	147	35,107
Total	472,667	42,518	80,817

Table 1: Training data mix, in number of sentences. Under eng

\to

creole, the jw.org, Watchtower , back-translated news and example-sentence rows form the Tok Pisin high-quality mix.

Panel of Table 2 — Table 2: Tok Pisin results, chrF++ on Bouquet (854 human-translated rows), FLORES-200 (2,009 rows) and held-out spoken transcripts (300 rows). Bold marks the best score in each column and every score within 1 chrF++ of it (our noise floor); the * row is the system we submit. Pre-training on all data barely moves the forward direction, but continued fine-tuning and Gemini distillation bring us level with Gemini 3.5 Flash, the strongest baseline, everywhere but FLORES. Table 3 reports the same systems and stages for Solomon Pijin and Bislama.

Solomon Pijin (pis)
	eng $\rightarrow$ creole		creole $\rightarrow$ eng
	Bouq.	Transc.	Bouq.	Transc.
NLLB-3.3B, base	23.9	19.3	40.8	28.3
MADLAD-400-3B	21.4	19.2	40.0	31.6
CreoleM2M	39.2	29.5	41.5	31.1
Gemma-4-31B, zero-shot	38.3	29.2	62.3	51.4

Table 3: Solomon Pijin and Bislama results, chrF++; same systems as Table 2 . Note Bouquet references are Gemini-generated from the Tok Pisin reference (+ lexicon and example sentences, Section 3.2 ). Spoken transcripts (300 rows/lang) are human-original in both directions. Unlike Tok Pisin, these two languages gain most from the broad pre-train, which already beats every baseline.

Ablation	Lang	w/o $\to$ w/	$\Delta$
Loanword respelling	pis	45.8 $\to$ 53.4	$+$ 7.7
Continued FT	tpi	46.9 $\to$ 50.0	$+$ 3.1
Gemini distil. (6k pairs)	tpi	48.5 $\to$ 50.9	$+$ 2.4
Joint training	bis	50.7 $\to$ 52.5	$+$ 1.8
	pis	44.4 $\to$ 45.8	$+$ 1.4
	tpi	45.6 $\to$ 46.3	$+$ 0.7

Table 4: Ablations, sorted by effect size; each varies one factor and holds the rest fixed. Scores are Bouquet en-xx chrF++. Respelling and alignment filtering use Gemma-4-31B for cleaning. Continued FT is the two-stage pre-train

\rightarrow

continued-FT recipe from Table 2 . Gemini distillation adds 6k synthetic eng

\rightarrow

tpi pairs (the 0 and 6k points of Figure 1 ). The joint-training rows compare each language’s single-language NLLB (w/o) against the joint model (w/) on raw-spelling data, so a positive

\Delta

means joint training helps that language.

Panel of Figure 1 — Figure 1: Translation quality gains from adding Gemini distillation in ChrF++ points (top) / ESA through LLM-judge (bottom). Gains are measurable with only 6k sentences (2k each of TweetEval / WikiLingua / ROCStories), then plateau.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation
Aug 25, 2026Surangika Ranathunga, Nisansa de Silva, Aloka Fernando +8Machine Translation Low-Resource MT
Massey University · University of Moratuwa · National University of Singapore +3
Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography
Feb 16, 2026Gianluca Vico, Jindřich LibovickýTokenization in Language Models Low-Resource Language Processing
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Praha, Czech Republic
Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models
Oct 1, 2026Grzegorz Kulik, Mikołaj Pokrywka, Adam Jatowt +1Low-Resource MT Machine Translation
NASK National Research Institute, Poland · University of Innsbruck, Austria

Aug 25, 2026J/K move · Enter open · S save

EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation

Surangika Ranathunga, Nisansa de Silva, Aloka Fernando, Kavindu Warnakulasuriya+7

Massey University · University of Moratuwa · National University of Singapore+3

Machine Translation Low-Resource MT MT Evaluation

Open paper AskPDF arXiv

Machine Translation (MT) for low-resource languages remains far behind that of high-resource languages, and the gap is widest in specialised domains, where parallel data is scarce or entirely absent. We present EnSiTa, a trilingual multi-domain parallel dataset and benchmark for English, Sinhala and Tamil. EnSiTa provides human post-edited training data for seven domains, plus manually translated test sets for those and one additional domain, all produced by professional translators under a multi-year, rigorously quality-controlled process. Using this dataset, we conduct an extensive study of domain-specific MT for all six language directions, fine-tuning a from-scratch Transformer, a pre-trained translation model (NLLB-600M), and decoder-only LLMs (Gemma 3 family, 1B-12B, and TranslateGemma) across training-data sizes, model scales, and in-domain, cross-domain, multilingual and multi-domain settings. To the best of our knowledge, this is the most extensive systematically documented multi-domain parallel data creation and benchmarking effort for low-resource MT. Our data and models will be publicly released.

Abstract

Machine Translation

Explore similar work

EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models