Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
Organizations: Queen’s University Belfast · University of Technology Nuremberg · Saarland University · University of Melbourne · University of Helsinki · IIIT Hyderabad · TCS Research · Johns Hopkins University · Imam Ja’afar Al-Sadiq University · Cohere · Last Token · Charles University · University of Edinburgh · ETH Zurich · University of Zurich
Abstract
Multilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evaluation---a design that is prone to contamination over time and overlooks locale and cultural considerations. We therefore advocate for source-contrastive evaluation and instantiate it with Cultivar, a localised subset of FLORES, which enables locale-specific translation evaluation. When paired with unlocalised counterparts, performance discrepancy allows the probing of data contamination and localisation robustness. We benchmark 32 open-weight models and find that MT-specialised models are less robust, a few models potentially overfit FLORES, and models tend to translate US content better than that of other locales, regardless of language.