Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

Sunishchal Dev; Andrew Sloan; Joshua Kavner; Nicholas Kong; Morgan Sandler

cs.AIMar 5, 2026

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

Authors: Sunishchal Dev, Andrew Sloan, Joshua Kavner, Nicholas Kong, Morgan Sandler

Organizations: RAND Corporation, Santa Monica, CA, USA

Abstract

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness

Figures & tables

Figure 1: Workflow for the Judge Reliability Harness. Human-in-the-loop review is performed after generation of the JRH synthetic test suite and before evaluation by the judges.

Figure 2: Human in the Loop Review User Interface (Agentic mode). The original sample from the benchmark is shown at the top. The left shows the transcript provided as input along with edits made by the synthetic data pipeline in red and green. The right shows the edited transcript, which is a free text box the user may use to make further edits. The bottom gives the user an option to accept the sample if they are satisfied with the edits, or to reject the sample and move on to the next one.

Panel of Figure 3 — Figure 3: Reliability performance heatmaps across all four benchmarks: (a) FORTRESS, (b) HarmBench, (c) Persuade, and (d) AgentHarm.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Cost-Accuracy Trade-off Across Benchmarks. Each point represents a model’s total cost and average accuracy across all benchmarks. Full model names are Llama Maverick 4, Gemini 2.5 Pro, GPT-4o, and Claude Sonnet 4.5

Benchmark	Llama 4 Maverick 17B	Claude Sonnet 4.5	GPT 4o	Gemini 2.5 Pro
Fortress	0.0016	0.0230	0.0150	0.0108
Persuade	0.0008	0.0260	0.0430	0.0073
HarmBench	0.0015	0.0352	0.0190	0.0132
AgentHarm	0.0001	0.0050	0.0013	0.0007
Overall	0.0010	0.0223	0.0196	0.0080

Appendix

Table 1: Model Cost Efficiency Across Benchmarks (

per accuracy point) (

\downarrow$ )

Model	Input Cost / 1M tokens	Output Cost / 1M tokens
GPT 4o 1 1 1 Pricing from https://platform.openai.com/docs/pricing as of December 2025	$2.50	$10
Claude Opus 4.5 2 2 2 Pricing from https://platform.claude.com/docs/en/about-claude/pricing as of December 2025	$5	$25
Claude Sonnet 4.5	$3	$15
Gemini 2.5 Pro 3 3 3 Pricing from https://ai.google.dev/gemini-api/docs/pricing as of December 2025	$1.25	$10
Llama 4 Maverick 17B 4 4 4 Pricing from https://aws.amazon.com/bedrock/pricing/ as of December 2025	$0.24	$0.97

Appendix

Table 2: Cost of input/output tokens per model (per 1 Million tokens)

Model Name	Test Name	CCC	Pearson’s $\rho$	Spearman’s $\rho_{s}$	MAE
Claude Sonnet 4.5	format_invariance_1	0.343	0.595	0.616	1.700
	format_invariance_2	0.868	0.928	0.901	0.500
	format_invariance_3	0.819	0.915	0.914	0.600
	semantic_paraphrase	0.125	0.327	0.444	1.200
	stochastic_stability	0.739	0.943	0.958	0.900
	synthetic_ordinal	0.838	0.901	0.889	0.484

Appendix

Table 3: Performance on the Persuade benchmark across eight reliability tests, reporting the Concordance Correlation Coefficient (CCC), Pearson’s Correlation Coefficient (

\rho

), Spearman’s Correlation Coefficient (

\rho_{s}

), and Mean Absolute Error (MAE).

Benchmark	Judge	Mean Score	Std Dev	Min Score	Max Score
Persuade	Claude Sonnet 4.5	37.26%	17.18%	20.00%	60.00%
	GPT-4o	48.43%	16.61%	30.00%	77.42%
	Gemini 2.5 Pro	53.20%	11.10%	40.00%	70.00%
	LLama 4 Maverick	52.62%	15.05%	30.00%	70.97%
HarmBench	Claude Sonnet 4.5	60.50%	11.13%	40.00%	73.33%
	GPT-4o	62.33%	14.88%	40.00%	80.00%

Appendix

Table 4: Performance of Judges Across Non-Agentic Benchmarks

Benchmark	Test	Mean Score	Std Dev	Min Score	Max Score
Persuade	format_invariance_1	35.00%	12.91%	20.00%	50.00%
	format_invariance_2	52.50%	5.00%	50.00%	60.00%
	format_invariance_3	40.00%	0.00%	40.00%	40.00%
	semantic_paraphrase	60.00%	8.16%	50.00%	70.00%
	stochastic_stability	37.50%	22.17%	20.00%	70.00%
	synthetic_ordinal	68.02%	8.21%	58.06%	77.42%

Appendix

Table 5: Statistics of Tests Across Non-Agentic Benchmarks

Model	Accuracy	Error Rate	FPR	FNR
GPT-4o	0.906	0.094	0.063	0.125
Claude Opus 4.5	0.813	0.188	0.063	0.313
Gemini 2.5 Pro	0.813	0.188	0.250	0.125
Llama 4 Maverick 17B	0.906	0.094	0.063	0.125
Best Trio Ensemble	0.906	0.094	0.063	0.125

Appendix

Table 6: Model Performance Metrics on Agent Perturbation Reliability Tests (AgentHarm)

Explore similar work

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
Aug 27, 2026Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian +1LLM-as-a-Judge LLM Agent Evaluation
ServiceNow AI Hyderabad, India
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Apr 16, 2026Manan Gupta, Dhruv KumarLLM Evaluation LLM-as-a-Judge
1BITS Pilani, Pilani Campus, India

Jun 21, 2026J/K move · Enter open · S save

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

Shreyas KC

Multilingual Language Model Evaluation LLM-as-a-Judge Language Model Generation Evaluation Benchmark Auditing LLM Agent Evaluation LLM Reliability+2

Open paper AskPDF arXiv

LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge

Abstract

Explore similar work

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations