Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

Alhasan Mahmood; Samir Abdaljalil; Hasan Kurban

cs.CLAug 23, 2026

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

Authors: Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

Organizations: College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar · Department of Computer and Electrical Engineering, Texas A&M University, College Station, TX, USA

Abstract

Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchmark, the top-ranked backbone alternates across English, Arabic, Chinese, Hindi, Japanese, Spanish, Turkish, and Swahili, and 7 of 15 backbone pairs show statistically significant pairwise rank reversal. We treat this as a measurement problem. The multilingual judge score decomposes additively into task difficulty, backbone skill, and a language-backbone interaction term, the last of which is recoverable without human labels by double-centering the cell-mean score matrix. We make this estimator (Consensus-Based Calibration, CBC) explicit, give an $O(1/\sqrt{n})$ finite-sample concentration bound with variance constant $(1-\tfrac{1}{m})(1-\tfrac{1}{k})$ , and show that it is unbiased even when task-language interactions are present. Across 7{,}920 judge runs (6 backbones, 8 languages, 55 tasks, 3 frameworks), CBC raises held-out cross-task rank consistency $τ$ from 0.650 to 0.902 and agrees with the held-out additive-model oracle in 100% of per-language decisions versus 68.5% raw; these are consistency diagnostics, not human-grounded correctness measures. On a separately collected M-RewardBench panel (7 languages, 1{,}500 items per language, 10{,}500 language-item instances, 5 evaluators), panel agreement with the public human gold preferences rises from 68.7% to 76.6% (gain 7.9 percentage points, 95% CI $[6.0, 9.9]$ ), our strongest external evidence of downstream usefulness. The estimator is the standard two-way ANOVA interaction-recovery operation under sum-to-zero contrasts; our contribution is its application as a label-free post-hoc calibrator for multilingual LLM judges, an explicit finite-sample concentration bound, and an unbiasedness result that holds even under task-language misspecification.

Figures & tables

Figure 1: Estimated centered language-backbone interaction $\hat{\beta}(\ell,b)$ from the expanded multilingual Agent-as-a-Judge benchmark (8 languages, 6 backbones), computed by double-centering the framework-averaged task scores. Positive values: backbone scores higher than expected in that language. These cells are distinct from the raw pairwise gaps $d_{ij}$ reported in score points. Strongest cells: GPT-4o in Spanish ( $-20.61$ ) vs. English ( $+11.84$ ), Gemini in Arabic ( $+9.31$ ), GPT-5.4 in Swahili ( $+8.39$ ) vs. English ( $-7.44$ ). The simultaneous radius is about $10.0$ points at $\varepsilon=0.05$ under Proposition 3 , so moderate cells should be interpreted cautiously (Section Limitations ).

Pair	$\delta$ ( $\mathrm{points}^{2}$ )	Adj. $p$
GPT-4o vs. GPT-5.4	400.8	$9.7{\times}10^{-8}$
GPT-4o vs. Sonnet	221.1	0.0019
GPT-4o vs. Gemini	312.0	0.0019
GPT-4o vs. DeepSeek	140.7	0.0031
GPT-4o vs. Qwen	0.0	1.000
GPT-5.4 vs. Sonnet	95.9	$7.5{\times}10^{-6}$

Table 1: Rank-reversal strength (

\delta

\mathrm{points}^{2}

) for backbone pairs on the expanded eight-language benchmark. Each test uses one-sided task-level tests in the two witness languages (the language pair attaining the most negative product

d_{ij}(\ell_{a})\cdot d_{ij}(\ell_{b})

), combined by the intersection-union rule; adj.

p

uses Benjamini–Hochberg over all 15 pairs. Witness language pairs are listed in Appendix A .

Method	$\tau$ $\uparrow$	95% CI	Full-fit ∗
Raw (no calib.)	0.650	[0.610, 0.714]	0.629
Per-language norm.	0.650	[0.610, 0.714]	0.629
ComBat-EB	0.650	[0.610, 0.714]	0.629
Z-score	0.220	[-0.038, 0.586]	-0.086
Quantile norm.	-0.097	[-0.119, -0.068]	-0.118
Ensemble	0.013	[-0.086, 0.186]	-0.081

Table 2: Calibration on the observed eight-language benchmark (higher is better). In each bootstrap replicate, methods are fit on sampled training tasks and evaluated on out-of-bag held-out tasks. “Full-fit ∗ ” is an in-sample diagnostic on all 55 tasks. The oracle subtracts

\beta

on the held-out tasks themselves and is an unattainable additive-model upper bound. ∗ denotes an in-sample sanity check, not an independent generalization estimate.

Figure 2: Backbone ranks by language before (left) and after (right) CBC on the expanded eight-language benchmark. Raw rankings vary substantially across languages, whereas CBC aligns all eight languages to the same backbone order: Gemini $>$ GPT-4o $>$ GPT-5.4 $>$ Sonnet $>$ DeepSeek $>$ Qwen ( $\tau=1.000$ ). The shared post-CBC order is a full-fit in-sample sanity check, not an independent generalization estimate.

Method	Oracle selection acc. $\uparrow$	Regret $\downarrow$
Raw	68.5% [67.4, 69.5]	3.23 [0.90, 5.82]
CBC	100.0% [99.95, 100.0]	0.00 [0.00, 0.00]

Table 3: Decision-level backbone selection on the observed eight-language benchmark. Each bootstrap replicate produces 8 language-specific deployment decisions. Selection accuracy is reported with exact binomial 95% confidence intervals over the resulting 8,000 language-decisions; regret is the held-out additive-model oracle-score gap between the selected backbone and the held-out oracle winner, with percentile intervals over bootstrap replicates.

Method	$\tau$ $\uparrow$	95% CI	Full-fit ∗
Raw (no calib.)	0.430	[0.371, 0.486]	0.410
Judge-aware BTL †	0.405	[0.352, 0.467]	0.410
CBC	0.900	[0.790, 1.000]	1.000 ∗

Table 4: Scaled validation on self-collected evaluator scores over public 7-language, 5-evaluator M-RewardBench instances, with the same bootstrap train/OOB protocol as the main benchmark. “BTL † ” is an adapted Xu et al. [ 2026 ] pairwise-only baseline. “Full-fit ∗ ” is an in-sample diagnostic on 1,500 items per language (10,500 language-item instances total). ∗ denotes an in-sample sanity check, not an independent generalization estimate.

Method	Gold agree. $\uparrow$	95% CI
Raw panel mean margin	68.7%	[65.4, 71.7]
CBC-aligned	76.6%	[73.6, 79.3]

Table 5: Human-anchor validation on the M-RewardBench panel containing 1,500 items per language (10,500 language-item instances total). For each of the 7 languages we sample 100 items stratified by subset and compare the sign of the evaluator-panel mean margin against the public gold preference (higher is better). Ties count as non-agreement.

Figure 3: Evaluator ranks before (left) and after (right) CBC on the self-collected 7-language, 5-evaluator score panel built from public M-RewardBench instances. Raw rankings vary across languages, while CBC aligns all seven languages to the same order: DeepSeek $>$ Sonnet $>$ GPT-4o $>$ GPT-5.4 $>$ Gemini ( $\tau=1.000$ ). The shared post-CBC order is a full-fit in-sample sanity check, not an independent generalization estimate.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$m$	2	3	4	5	6
Rank $\tau$	0.901	0.907	0.900	0.898	0.911
Std.	0.206	0.105	0.075	0.044	—

Appendix

Table A1 : CBC calibration quality as a function of the number of backbones

m

. Rank

\tau

is averaged over all

\binom{6}{m}

subsets using the observed-language bootstrap/OOB protocol.

$n$	10	20	30	40	55
Rank $\tau$	0.759	0.838	0.853	0.891	0.906
Std.	0.103	0.061	0.057	0.054	0.051

Appendix

Table A2 : CBC calibration quality as a function of the number of tasks

n

. Averaged over 100 random subsamples.

Figure A1 : Convergence of CBC interaction-estimation error as a function of the number of tasks $n$ . The empirical mean absolute error $|\hat{\beta}-\beta_{\mathrm{oracle}}|$ decreases steadily with more tasks. The dashed reference curve plots the explicit Proposition 3 constant using $\hat{\sigma}=22.24$ , $\varepsilon=0.05$ , $m=6$ , and $k=8$ , so it should be read as the paper’s conservative finite-sample bound rather than as a generic asymptotic envelope.

Component	Calls	Input tokens	Output tokens	Est. cost
3-language extension (JA / ES / SW)	19,710	25.68M	6.64M	$54.93
M-RewardBench 1,500-items-per-language panel	105,000	52.06M	9.85M	$195.30

Appendix

Table A3 : Estimated API cost breakdown for the two added experimental components. “Calls” counts direct model invocations; the M-RewardBench panel contains 1,500 items per language (10,500 language-item instances) and scores both chosen and rejected responses separately, so 52,500 judged pairs correspond to 105,000 pointwise calls. Costs are estimates from recorded token counts and public list prices, not billing exports.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
Oct 22, 2026Asal Mehrabi, Mohammad Aliannejadi, Clemencia SiroLLM Evaluation LLM-as-a-Judge
University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands
A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
Oct 13, 2026Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6Bias in LLM-as-a-Judge LLM-as-a-Judge
University of Pisa · University of Trento · KU Leuven +1
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge

Oct 22, 2026J/K move · Enter open · S save

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro

University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands

LLM Evaluation LLM-as-a-Judge Multilingual Language Model Evaluation

Open paper AskPDF arXiv

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

Abstract

Multilingual Language Model Evaluation

Explore similar work

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories