cs.CLJun 21, 2026

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

Authors: Shreyas KC

Abstract

LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge

Explore similar work

CardsList
Jun 17, 2026cs.CL

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.
Justin D. Norman, Michael U. Rivera, D. Alex Hughes
Llm-As-A-JudgeJudgesEvaluation BenchmarksAgreementValidityDiscriminative
AskPDF
Apr 16, 2026cs.AI

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: (1)\textbf{(1)} a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (ρˉ=0.8\barρ = 0.8-4.1%4.1\%), with 3333-67%67\% of documents exhibiting at least one directed 3-cycle; and (2)\textbf{(2)} split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed ≥(1−α)\geq(1{-}α) coverage, with set width serving as a per-instance reliability indicator (rs=+0.576r_s = {+}0.576, N=1,918N{=}1{,}918, p<10−100p < 10^{-100}, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (rˉ=0.32\bar{r} = 0.32-0.380.38), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size ≈3.0\approx 3.0) and coherence moderately so (avg. set size ≈3.9\approx 3.9), while fluency and consistency remain unreliable (avg. set size ≈4.9\approx 4.9). We release all code, prompts, and cached results.
Manan Gupta, Dhruv Kumar
Large Language Model JudgesJudgesConformal Prediction
AskPDF
Feb 6, 2026cs.CL

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

Existing LLM-as-a-Judge systems suffer from three fundamental limitations: limited adaptivity to task- and domain-specific evaluation criteria, systematic biases driven by non-semantic cues such as position, length, format, and model provenance, and evaluation inconsistency that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose FairJudge, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models judging behavior itself as a learnable and regularized policy. From a data-centric perspective, we construct a high-information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT-DPO-GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge consistently improves agreement and F1, reduces non-semantic biases, and outperforms substantially larger instruction-tuned LLMs. All resources will be publicly released after acceptance to facilitate future research.
Bo Yang, Lanfei Feng, Yunkui Chen +3
Llm-As-A-JudgeRaw Judge OutputsImplicit BiasNovel Modular Learning Framework
AskPDF