cs.CLNov 11, 2026

Is Jev a Good Multilingual Judge?

Authors: Wei Liu, Yi Fan, Xiyan Fu

Organizations: Heidelberg Institute for Theoretical Studies · Independent Researcher

Abstract

Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention. We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. At approximately 1/311/31 to 1/611/61 of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

    Oct 22, 2026Asal Mehrabi, Mohammad Aliannejadi, Clemencia SiroLLM EvaluationLLM-as-a-Judge

  2. Lost in Translation: Do LVLM Judges Generalize Across Languages?

    Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5Multimodal Model EvaluationMultilingual VLM Evaluation