Is Jev a Good Multilingual Judge?
Organizations: Heidelberg Institute for Theoretical Studies · Independent Researcher
Abstract
Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention. We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. At approximately to of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge.
Figures & tables
| Benchmark | Task | Languages | Evaluation format | ||
|---|---|---|---|---|---|
| Binary | Pairwise | Graded | |||
| MEMERAG | RAG faithfulness | 5 | |||
| Global-MMLU-Lite | Answer verification | 23 | |||
| MENLO | Response quality | 47 | |||
| METAL | Summary evaluation | 10 | |||
| Judge | MEMERAG | Global-MMLU-Lite | METAL (Hallucination) | |||
|---|---|---|---|---|---|---|
| BAcc | AUROC | BAcc | AUROC | BAcc | AUROC | |
| Jev | 69.0 | 81.0 | 89.6 | 95.8 | 85.9 | 92.6 |
| Qwen | 74.1 | 82.0 | 82.9 | 91.0 | 84.9 | 92.3 |
| GPT-6.1 | 73.5 | — | 96.1 | — | 79.2 | — |
| Benchmark | Jev | Qwen | GPT-6.1 |
|---|---|---|---|
| MEMERAG | 0.082 | 1.240 (15.2 ) | 5.010 (61 ) |
| Global-MMLU-Lite | 0.026 | 0.160 (6.2 ) | 0.820 (32 ) |
| MENLO | 0.081 | 0.620 (7.8 ) | 2.950 (37 ) |
| METAL | 0.063 | 0.440 (7.1 ) | 1.960 (31 ) |
| Judge | Preference accuracy (%) |
|---|---|
| Jev | 74.6 |
| Qwen | 74.7 |
| GPT-6.1 | 72.0 |
| Judge | MENLO (Response rating) | METAL (Summary rating) | ||
|---|---|---|---|---|
| Macro-F1 | Spearman | Macro-F1 | Spearman | |
| Jev | 33.3 | 0.29 | 55.0 | 0.60 |
| Qwen | 31.9 | 0.31 | 56.2 | 0.65 |
| GPT-6.1 | 28.3 | 0.30 | 55.5 | 0.63 |
| Cross-lingual | Order | ||
|---|---|---|---|
| Judge | Agreement (%) | Fleiss’ | Agreement (%) |
| Jev | 93.4 | 0.81 | 93.2 |
| Qwen | 85.7 | 0.63 | 76.6 |
| GPT-6.1 | 97.8 | 0.95 | 94.6 |
| Task | English | Target language |
|---|---|---|
| Global-MMLU-Lite | 89.39 | 89.21 |
| MENLO (Preference) | 67.42 | 67.02 |
| MENLO (Rating) | 30.80 | 31.32 |
| No filtering | ||||||||
|---|---|---|---|---|---|---|---|---|
| Task | Acc. | Cov. | Acc. | Cov. | Acc. | Cov. | Acc. | Cov. |
| Global-MMLU-Lite (Verification) | 89.57 | 100.00 | 94.90 | 81.73 | 98.48 | 46.95 | 99.18 | 28.00 |
| MENLO (Preference) | 74.63 | 100.00 | 78.66 | 82.11 | 83.97 | 53.10 | 86.69 | 38.24 |
| MENLO (Rating) | 35.91 | 100.00 | 46.89 | 30.49 | 62.86 | 3.81 | 69.27 | 1.72 |
| Task | Jev | Jev GPT | Jev GPT | Jev Qwen | Jev Qwen |
|---|---|---|---|---|---|
| (random) | (confidence) | (random) | (confidence) | ||
| Global-MMLU-Lite | 89.57 | 90.22 | 92.73 | 88.90 | 89.90 |
| MENLO (Pairwise comparison) | 74.64 | 74.51 | 74.64 | 74.44 | 75.04 |
| METAL (Summary rating) | 55.01 | 55.12 | 55.67 | 55.28 | 55.69 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Language | Jev | Qwen | GPT-6.1 |
|---|---|---|---|
| English | 55.3 | 61.1 | 64.9 |
| French | 46.5 | 50.8 | 50.6 |
| Chinese | 48.4 | 45.9 | 48.1 |
| Hindi | 35.7 | 38.2 | 37.6 |
| Arabic | 49.3 | 48.8 | 45.3 |
| Bengali | 34.9 | 33.2 | 33.0 |
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 605 papers