The Ongiini-Eval-OW Benchmark: A Concept Paper for the Planned Benchmarking of Machine Translation and Large Language Models on Oshindonga and Oshikwanyama

Sebastian Küpers

cs.CLSep 22, 2026

The Ongiini-Eval-OW Benchmark: A Concept Paper for the Planned Benchmarking of Machine Translation and Large Language Models on Oshindonga and Oshikwanyama

Authors: Sebastian Küpers

Organizations: Common Intelligence Foundation · Common Intelligence Foundation, Estonia Ongiini AI programme, Namibia

Abstract

Oshiwambo -- a cluster of mutually intelligible Bantu languages spoken by over a million people across northern Namibia and southern Angola, and the home language of roughly half of Namibian households -- has, to our knowledge, no published machine-translation evaluation benchmark. Major commercial services (Google Translate, DeepL, Microsoft Translator), open multilingual MT models (NLLB-200, MADLAD-400), and the open Masakhane checkpoint collection all lack coverage of either standardised dialect, Oshindonga or Oshikwanyama. We announce Ongiini-Eval-OW, a planned 600-item English-Oshindonga and English-Oshikwanyama benchmark with native-speaker references from two independent translators, a 30-item inter-translator agreement set, an 11-tag phenomenon-tagged stratification (at least 30 items per tag), a deterministic 30% blind split, and a reproducible scoring protocol over chrF++, BLEU, and COMET-22, supplemented by a 50-item human-evaluation round. We document the empirical coverage gap, the dataset composition, the launch-leaderboard model matrix across American, European, and Chinese frontier and open-weight systems, and the contribution pipeline. The dataset is targeted for first public release in Q4 2026; this v1.0 concept paper announces the design and the call for participation. Data and code will be released under CC-BY-4.0 and MIT respectively.

Figures & tables

Dialect	ISO 639-1	ISO 639-3	Guthrie	Primary regions
Oshikwanyama	kj	kua	R.21	Ohangwena and Omusati regions (Namibia); Cunene Province (Angola)
Oshindonga	ng	ndo	R.22	Oshana and Oshikoto regions (Namibia)

Table 1: The two standardised Oshiwambo dialects covered by this benchmark.

System	Claims Oshiwambo	What we checked, and what we found
NLLB-200 ( NLLB Team et al., 2022 )	No	Inspected special_tokens_map.json on Hugging Face; no Oshiwambo language code present
MADLAD-400 ( Kudugunta et al., 2023 )	No	Inspected model README and supported-language list; no Oshiwambo language code present
Masakhane open checkpoints	No	Surveyed the public Masakhane MT repositories; no Oshiwambo checkpoint located
Google Translate (web)	No	Consulted public language list; Oshiwambo not offered as a translation option
DeepL	No	Consulted public language list; Oshiwambo not offered as a translation option
Microsoft Translator	No	Consulted public language list; Oshiwambo not offered as a translation option

Table 2: Empirical coverage audit (May 2026, re-verified September 2026). Inspection-based for open systems; documentation-based for closed APIs.

Source	Items	Description
v1_retained	150	Phrasebook-style items, curated down from a larger v1 set; longer constructions preferred over greetings.
mined_paraphrased	180	Inspired by production WhatsApp conversation logs: PII-scrubbed, then paraphrased into clean English while preserving register and intent. The published English source is not verbatim user text — the label is honest about this. Capped at three derived items per user to prevent dominance.
crafted	210	Phenomenon-tagged items authored to ensure each phenomenon slice carries at least 30 items (negation, code-switching, noun-class agreement, politeness register, etc.).
formal_drafted	60	Government, health, education, legal, and Namibian news/broadcast register items, drafted to anchor the benchmark in institutional language.
Total	600

Table 3: Provenance composition of Ongiini-Eval-OW v1.0.

Tag	Items	Tests
numbers_dates	40	Currency (N$), dates, times, phone numbers, IDs
named_entities	30	Namibian places, ministries, common Namibian names
tense_aspect	30	Perfect / recent past / habitual (Bantu makes finer distinctions than English)
negation	30	Single, double, scope ambiguity — historically the #1 MT failure mode ( Hossain et al., 2020 )
code_switch	30	English loanwords embedded in Oshiwambo
pronoun_coreference	30	Ambiguous antecedents; Bantu noun-class pronouns force disambiguation

Table 4: Phenomenon-tag inventory. Items can carry multiple tags.

Use	Which items?
Running a system to see what it outputs	All 600
Per-phenomenon / per-length / per-domain diagnostics	All 600
Tuning a prompt or fine-tuning a model	Dev set only
Headline leaderboard number, publishable claim	chrF++ on blind set

Table 5: Use-case convention for the development/blind split.

When	Milestone
September 2026	This concept paper (v1.0) submitted to arXiv as cs.CL. Email outreach to external prospective submitters (Meyabase, Masakhane-affiliated teams) with the arXiv link, inviting submissions for the launch leaderboard. The 423-item internal v0.1 build has been fully translated into both dialects by both reference translators, validating the translation pipeline end-to-end at production scale.
Q4 2026	600-item English source set locked: new crafted and formal items authored, second Spark mining run executed and curated, pipeline scripts updated for the v1.0 schema. Translator handoff packets generated (one document per translator with their assigned items in randomised order), including the 30-item agreement-set overlap. Volunteer-pool onboarding for human evaluation begins in parallel. Claude Opus 5 and Gemma 4 26B baselines computed against the locked English source (independent of translator timeline).
End of 2026	Translator deliveries imported, references finalised, inter-translator agreement computed and published. Dataset v1.0 goes live on HuggingFace under CC-BY-4.0. First public leaderboard launches with the two Anthropic + Google baselines plus any external submissions received during the build phase.
Q1 2027	Volunteer-pool human-evaluation round 1 (50-item sample, 5–10 raters per dialect) completes. Krippendorff’s $\alpha$ computed within-volunteer and cross-group (with any external academic raters who participate). Concept paper v2 submitted to arXiv with finalised numbers and the live HuggingFace URL.
Q2 2027	Academic paper submission. Target venues (in order of preference): AfricaNLP Workshop (co-located with ACL or ICLR); LoResMT at EMNLP; ACL Findings; LREC. Submission cycle is bound by conference deadlines and runs after the artefact release.

Table 6: Roadmap from concept paper to first peer-reviewed paper.

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

Omnilingual MT: Machine Translation for 1,600 Languages
Mar 17, 2026Omnilingual MT Team, Belen Alastruey, Niyati Bafna +29Machine Translation Low-Resource MT
FAIR at Meta
Last Translation Benchmark
Sep 3, 2026Vilém Zouhar, Niyati Bafna, Mukund Choudhary +257Multimodal MT MT Evaluation
The State and Fate of Multilingual, Contextual Evaluation in the NLP World
Oct 27, 2026Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali +1Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
Microsoft Research India · IIT Gandhinagar

Mar 17, 2026J/K move · Enter open · S save

Omnilingual MT: Machine Translation for 1,600 Languages

Omnilingual MT Team, Belen Alastruey, Niyati Bafna, Andrea Caciolai+28

FAIR at Meta

Machine Translation Low-Resource MT Cross-Lingual Transfer MT Evaluation

Open paper AskPDF arXiv

High-quality machine translation (MT) can scale to hundreds of languages, setting a high bar for multilingual systems. However, compared to the world's 7,000 languages, current systems still offer only limited coverage: about 200 languages on the target side, and maybe a few hundreds more on the source side, supported due to cross-lingual transfer. And even these numbers have been hard to evaluate due to the lack of reliable benchmarks and metrics. We present Omnilingual Machine Translation (OMT), the first MT system supporting more than 1,600 languages. This scale is enabled by a comprehensive data strategy that integrates large public multilingual corpora with newly created datasets, including manually curated MeDLEY bitext. We explore two ways of specializing a Large Language model (LLM) for machine translation: as a decoder-only model (OMT-LLaMA) or as a module in an encoder-decoder architecture (OMT-NLLB). Notably, all our 1B to 8B parameter models match or exceed the MT performance of a 70B LLM baseline, revealing a clear specialization advantage and enabling strong translation quality in low-compute settings. Moreover, our evaluation of English-to-1,600 translations further shows that while baseline models can interpret undersupported languages, they frequently fail to generate them with meaningful fidelity; OMT-LLaMA models substantially expand the set of languages for which coherent generation is feasible. Additionally, OMT models improve in cross-lingual transfer, being close to solving the "understanding" part of the puzzle in MT for the 1,600 evaluated. Our leaderboard and main human-created evaluation datasets (BOUQuET and Met-BOUQuET) are dynamically evolving towards Omnilinguality and freely available.

Abstract

MT Evaluation

Explore similar work

Omnilingual MT: Machine Translation for 1,600 Languages

Last Translation Benchmark

The State and Fate of Multilingual, Contextual Evaluation in the NLP World