Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

Alhasan Mahmood; Samir Abdaljalil; Hasan Kurban

cs.CLAug 23, 2026

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

Authors: Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

Organizations: College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar · Department of Computer and Electrical Engineering, Texas A&M University, College Station, TX, USA

Abstract

Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchmark, the top-ranked backbone alternates across English, Arabic, Chinese, Hindi, Japanese, Spanish, Turkish, and Swahili, and 7 of 15 backbone pairs show statistically significant pairwise rank reversal. We treat this as a measurement problem. The multilingual judge score decomposes additively into task difficulty, backbone skill, and a language-backbone interaction term, the last of which is recoverable without human labels by double-centering the cell-mean score matrix. We make this estimator (Consensus-Based Calibration, CBC) explicit, give an $O(1/\sqrt{n})$ finite-sample concentration bound with variance constant $(1-\tfrac{1}{m})(1-\tfrac{1}{k})$ , and show that it is unbiased even when task-language interactions are present. Across 7{,}920 judge runs (6 backbones, 8 languages, 55 tasks, 3 frameworks), CBC raises held-out cross-task rank consistency $τ$ from 0.650 to 0.902 and agrees with the held-out additive-model oracle in 100% of per-language decisions versus 68.5% raw; these are consistency diagnostics, not human-grounded correctness measures. On a separately collected M-RewardBench panel (7 languages, 1{,}500 items per language, 10{,}500 language-item instances, 5 evaluators), panel agreement with the public human gold preferences rises from 68.7% to 76.6% (gain 7.9 percentage points, 95% CI $[6.0, 9.9]$ ), our strongest external evidence of downstream usefulness. The estimator is the standard two-way ANOVA interaction-recovery operation under sum-to-zero contrasts; our contribution is its application as a label-free post-hoc calibrator for multilingual LLM judges, an explicit finite-sample concentration bound, and an unbiasedness result that holds even under task-language misspecification.

Figures & tables

Figure 1: Estimated centered language-backbone interaction $\hat{\beta}(\ell,b)$ from the expanded multilingual Agent-as-a-Judge benchmark (8 languages, 6 backbones), computed by double-centering the framework-averaged task scores. Positive values: backbone scores higher than expected in that language. These cells are distinct from the raw pairwise gaps $d_{ij}$ reported in score points. Strongest cells: GPT-4o in Spanish ( $-20.61$ ) vs. English ( $+11.84$ ), Gemini in Arabic ( $+9.31$ ), GPT-5.4 in Swahili ( $+8.39$ ) vs. English ( $-7.44$ ). The simultaneous radius is about $10.0$ points at $\varepsilon=0.05$ under Proposition 3 , so moderate cells should be interpreted cautiously (Section Limitations ).

Pair	$\delta$ ( $\mathrm{points}^{2}$ )	Adj. $p$
GPT-4o vs. GPT-5.4	400.8	$9.7{\times}10^{-8}$
GPT-4o vs. Sonnet	221.1	0.0019
GPT-4o vs. Gemini	312.0	0.0019
GPT-4o vs. DeepSeek	140.7	0.0031
GPT-4o vs. Qwen	0.0	1.000
GPT-5.4 vs. Sonnet	95.9	$7.5{\times}10^{-6}$

Table 1: Rank-reversal strength (

\delta

\mathrm{points}^{2}

) for backbone pairs on the expanded eight-language benchmark. Each test uses one-sided task-level tests in the two witness languages (the language pair attaining the most negative product

d_{ij}(\ell_{a})\cdot d_{ij}(\ell_{b})

), combined by the intersection-union rule; adj.

p

uses Benjamini–Hochberg over all 15 pairs. Witness language pairs are listed in Appendix A .

Method	$\tau$ $\uparrow$	95% CI	Full-fit ∗
Raw (no calib.)	0.650	[0.610, 0.714]	0.629
Per-language norm.	0.650	[0.610, 0.714]	0.629
ComBat-EB	0.650	[0.610, 0.714]	0.629
Z-score	0.220	[-0.038, 0.586]	-0.086
Quantile norm.	-0.097	[-0.119, -0.068]	-0.118
Ensemble	0.013	[-0.086, 0.186]	-0.081

Table 2: Calibration on the observed eight-language benchmark (higher is better). In each bootstrap replicate, methods are fit on sampled training tasks and evaluated on out-of-bag held-out tasks. “Full-fit ∗ ” is an in-sample diagnostic on all 55 tasks. The oracle subtracts

\beta

on the held-out tasks themselves and is an unattainable additive-model upper bound. ∗ denotes an in-sample sanity check, not an independent generalization estimate.

Figure 2: Backbone ranks by language before (left) and after (right) CBC on the expanded eight-language benchmark. Raw rankings vary substantially across languages, whereas CBC aligns all eight languages to the same backbone order: Gemini $>$ GPT-4o $>$ GPT-5.4 $>$ Sonnet $>$ DeepSeek $>$ Qwen ( $\tau=1.000$ ). The shared post-CBC order is a full-fit in-sample sanity check, not an independent generalization estimate.

Method	Oracle selection acc. $\uparrow$	Regret $\downarrow$
Raw	68.5% [67.4, 69.5]	3.23 [0.90, 5.82]
CBC	100.0% [99.95, 100.0]	0.00 [0.00, 0.00]

Table 3: Decision-level backbone selection on the observed eight-language benchmark. Each bootstrap replicate produces 8 language-specific deployment decisions. Selection accuracy is reported with exact binomial 95% confidence intervals over the resulting 8,000 language-decisions; regret is the held-out additive-model oracle-score gap between the selected backbone and the held-out oracle winner, with percentile intervals over bootstrap replicates.

Method	$\tau$ $\uparrow$	95% CI	Full-fit ∗
Raw (no calib.)	0.430	[0.371, 0.486]	0.410
Judge-aware BTL †	0.405	[0.352, 0.467]	0.410
CBC	0.900	[0.790, 1.000]	1.000 ∗

Table 4: Scaled validation on self-collected evaluator scores over public 7-language, 5-evaluator M-RewardBench instances, with the same bootstrap train/OOB protocol as the main benchmark. “BTL † ” is an adapted Xu et al. [ 2026 ] pairwise-only baseline. “Full-fit ∗ ” is an in-sample diagnostic on 1,500 items per language (10,500 language-item instances total). ∗ denotes an in-sample sanity check, not an independent generalization estimate.

Method	Gold agree. $\uparrow$	95% CI
Raw panel mean margin	68.7%	[65.4, 71.7]
CBC-aligned	76.6%	[73.6, 79.3]

Table 5: Human-anchor validation on the M-RewardBench panel containing 1,500 items per language (10,500 language-item instances total). For each of the 7 languages we sample 100 items stratified by subset and compare the sign of the evaluator-panel mean margin against the public gold preference (higher is better). Ties count as non-agreement.

Figure 3: Evaluator ranks before (left) and after (right) CBC on the self-collected 7-language, 5-evaluator score panel built from public M-RewardBench instances. Raw rankings vary across languages, while CBC aligns all seven languages to the same order: DeepSeek $>$ Sonnet $>$ GPT-4o $>$ GPT-5.4 $>$ Gemini ( $\tau=1.000$ ). The shared post-CBC order is a full-fit in-sample sanity check, not an independent generalization estimate.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$m$	2	3	4	5	6
Rank $\tau$	0.901	0.907	0.900	0.898	0.911
Std.	0.206	0.105	0.075	0.044	—

Appendix

Table A1 : CBC calibration quality as a function of the number of backbones

m

. Rank

\tau

is averaged over all

\binom{6}{m}

subsets using the observed-language bootstrap/OOB protocol.

$n$	10	20	30	40	55
Rank $\tau$	0.759	0.838	0.853	0.891	0.906
Std.	0.103	0.061	0.057	0.054	0.051

Appendix

Table A2 : CBC calibration quality as a function of the number of tasks

n

. Averaged over 100 random subsamples.

Figure A1 : Convergence of CBC interaction-estimation error as a function of the number of tasks $n$ . The empirical mean absolute error $|\hat{\beta}-\beta_{\mathrm{oracle}}|$ decreases steadily with more tasks. The dashed reference curve plots the explicit Proposition 3 constant using $\hat{\sigma}=22.24$ , $\varepsilon=0.05$ , $m=6$ , and $k=8$ , so it should be read as the paper’s conservative finite-sample bound rather than as a generic asymptotic envelope.

Component	Calls	Input tokens	Output tokens	Est. cost
3-language extension (JA / ES / SW)	19,710	25.68M	6.64M	$54.93
M-RewardBench 1,500-items-per-language panel	105,000	52.06M	9.85M	$195.30

Appendix

Table A3 : Estimated API cost breakdown for the two added experimental components. “Calls” counts direct model invocations; the M-RewardBench panel contains 1,500 items per language (10,500 language-item instances) and scores both chosen and rejected responses separately, so 52,500 judged pairs correspond to 105,000 pointwise calls. Costs are estimates from recorded token counts and public list prices, not billing exports.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Oct 22, 2026

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro

University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands

LLM Evaluation LLM-as-a-Judge Multilingual Language Model Evaluation

Ask PDF

Oct 13, 2026

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model family of the candidate answers influencing the judge decision rather than their quality, while additional biases may remain undocumented.

\textit{Multicalibration}

provides a natural framework to address this problem: each bias corresponds to miscalibration on an identifiable group,

\textit{e.g.}

, family bias manifests as overconfidence in answers generated by models belonging to the same family of the judge. The challenge is to fix miscalibration simultaneously across many, potentially unknown groups. To this end, we introduce

\mathrm{CaLLM}

, a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously. We prove that

\mathrm{CaLLM}

bounds the bias of every group whose error structure is captured by an available representation, and that augmenting it with explicit bias groups further reduces the residual bias. We then derive theoretical conditions under which

\mathrm{CaLLM}

improves the judge's accuracy, rather than trading it for calibration. Finally, we evaluate

\mathrm{CaLLM}

across pairwise preference benchmarks, showing that it reduces the multicalibration error on both specified and unspecified bias groups, while improving predictive performance of the judge.

Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6

University of Pisa · University of Trento · KU Leuven +1

Bias in LLM-as-a-Judge LLM-as-a-Judge Language Model Calibration Multicalibration

Ask PDF

Jun 21, 2026

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge

Shreyas KC

Multilingual Language Model Evaluation LLM-as-a-Judge Language Model Generation Evaluation Benchmark Auditing LLM Agent Evaluation LLM Reliability+2

Ask PDF

Abstract

Multilingual Language Model Evaluation

Explore similar work

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories