Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention. We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. At approximately 1/31 to 1/61 of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge.
Figures & tables
Benchmark
Task
Languages
Evaluation format
Binary
Pairwise
Graded
MEMERAG
RAG faithfulness
5
✓
×
×
Global-MMLU-Lite
Answer verification
23
✓
×
×
MENLO
Response quality
47
×
✓
✓
METAL
Summary evaluation
10
✓
×
✓
Table 1: Overview of the four multilingual benchmarks.
Judge
MEMERAG
Global-MMLU-Lite
METAL (Hallucination)
BAcc ↑
AUROC ↑
BAcc ↑
AUROC ↑
BAcc ↑
AUROC ↑
Jev
69.0
81.0
89.6
95.8
85.9
92.6
Qwen
74.1
82.0
82.9
91.0
84.9
92.3
GPT-6.1
73.5
—
96.1
—
79.2
—
Table 3: Performance on RAG faithfulness, answer verification, and hallucination detection. Dashes indicate unavailable answer probabilities for GPT-6.1.
Benchmark
Jev
Qwen
GPT-6.1
MEMERAG
0.082
1.240 (15.2 × )
5.010 (61 × )
Global-MMLU-Lite
0.026
0.160 (6.2 × )
0.820 (32 × )
MENLO
0.081
0.620 (7.8 × )
2.950 (37 × )
METAL
0.063
0.440 (7.1 × )
1.960 (31 × )
Table 2: Evaluation cost per 1,000 samples (USD). Parentheses indicate how many times each model costs as much as Jev.
Judge
Preference accuracy (%) ↑
Jev
74.6
Qwen
74.7
GPT-6.1
72.0
Table 4: Pairwise preference accuracy on MENLO, using preferences derived from human ratings.
Judge
MENLO (Response rating)
METAL (Summary rating)
Macro-F1 ↑
Spearman ρ↑
Macro-F1 ↑
Spearman ρ↑
Jev
33.3
0.29
55.0
0.60
Qwen
31.9
0.31
56.2
0.65
GPT-6.1
28.3
0.30
55.5
0.63
Table 5: Response rating on MENLO and summary rating on METAL. Macro-F1 measures agreement with reference rating levels, while Spearman’s ρ measures rating correlation.
Figure 1: Agreement with English judgments across 22 non-English languages in Global-MMLU-Lite. Outer bands indicate high-, mid-, and low-resource language groups defined in the dataset ( Singh et al., 2025 ) .
Cross-lingual
Order
Judge
Agreement (%)
Fleiss’ κ
Agreement (%)
Jev
93.4
0.81
93.2
Qwen
85.7
0.63
76.6
GPT-6.1
97.8
0.95
94.6
Table 6: Cross-lingual and order consistency. On Global-MMLU-Lite, agreement compares each judge’s decisions with its English decisions, while Fleiss’ κ measures agreement across all language versions. On MENLO, order agreement measures how often a judge selects the same response when the two candidate responses are presented in reverse order.
Figure 2: Changes in judgments from English to other languages on Global-MMLU-Lite. False acceptances and false rejections are newly introduced errors, while their corrected counterparts are English errors corrected in the target language. Each segment shows a percentage of all candidate answers, averaged across languages in the resource group. Total bar heights give disagreement with English.
Figure 3: A Global-MMLU-Lite example of judgment changes from English to Welsh. Jev introduces a false acceptance and Qwen a false rejection, while GPT-6.1 maintains both correct judgments. Incorrect judgments are highlighted in red.
Figure 4: How well Jev’s confidence distinguishes correct from incorrect judgments, measured by AUROC. Colors distinguish judgment types, and the dashed line marks chance level (0.5).
Task
English
Target language
Global-MMLU-Lite
89.39
89.21
MENLO (Preference)
67.42
67.02
MENLO (Rating)
30.80
31.32
Table 7: Performance using English instructions or instructions translated into the language of the evaluated content.
No filtering
≥0.7
≥0.9
≥0.95
Task
Acc.
Cov.
Acc.
Cov.
Acc.
Cov.
Acc.
Cov.
Global-MMLU-Lite (Verification)
89.57
100.00
94.90
81.73
98.48
46.95
99.18
28.00
MENLO (Preference)
74.63
100.00
78.66
82.11
83.97
53.10
86.69
38.24
MENLO (Rating)
35.91
100.00
46.89
30.49
62.86
3.81
69.27
1.72
Table 8: Accuracy of retained judgments (Acc.) and percentage of samples retained (Cov.) at different confidence thresholds. “No filtering” retains all judgments, while ≥t retains only judgments with confidence at least t .
Task
Jev
Jev → GPT
Jev → GPT
Jev → Qwen
Jev → Qwen
(random)
(confidence)
(random)
(confidence)
Global-MMLU-Lite
89.57
90.22
92.73
88.90
89.90
MENLO (Pairwise comparison)
74.64
74.51
74.64
74.44
75.04
METAL (Summary rating)
55.01
55.12
55.67
55.28
55.69
Table 9: Performance of Jev alone and routing to GPT or Qwen. “Confidence” selects test samples below a threshold determined by the lowest-confidence 10% of development samples, while “random” selects the same number of test samples at random.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Prompts for Global-MMLU-Lite answer verification (top) and MEMERAG sentence-level faithfulness evaluation (bottom).
Figure 6: MENLO prompts for pairwise comparison and response rating, shown for cultural relevance.
Figure 7: METAL summary-rating prompts for linguistic acceptability and output content quality.
Figure 8: METAL prompts for task-quality rating and hallucination detection.
Language
Jev
Qwen
GPT-6.1
English
55.3
61.1
64.9
French
46.5
50.8
50.6
Chinese
48.4
45.9
48.1
Hindi
35.7
38.2
37.6
Arabic
49.3
48.8
45.3
Bengali
34.9
33.2
33.0
Appendix
Table 14: METAL summary rating by language. Scores are macro-F1 (%) averaged across three evaluation dimensions.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.