cs.CLFeb 25, 2026

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

Authors: Hanna Yukhymenko, Anton Alexandrov, Martin Vechev

Organizations: INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich

Abstract

The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a fully automated framework designed to address these challenges by enabling scalable, high-quality translation of datasets and benchmarks. We demonstrate that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and our proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines. Our framework ensures that benchmarks preserve their original task structure and linguistic nuances during localization. We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Evaluations using both reference-based metrics and LLM-as-a-judge show that our translations surpass existing resources, resulting in more accurate downstream model assessment. We release both the framework and the improved benchmarks to facilitate robust and reproducible multilingual AI development.

Figures & tables

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite

    Apr 2, 2026Klaudia Thellmann, Bernhard Stadler, Michael FärberBenchmark AuditingMT Evaluation

  2. Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

    May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model EvaluationMT Evaluation

  3. Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

    Oct 14, 2026Zahraa Al Sahili, Ioannis Patras, Matthew PurverMultilingual Language Model EvaluationLanguage Model Safety Evaluation