LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion. Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types. We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions. Given a scorer specification, implementation, and representative inputs, the pipeline first identifies scorer-specific capabilities, then automatically derives perturbation recipes and expected score relations for each. SANE also defines a scorer-independent validation framework for generated judge tests. It determines whether each recipe agrees with scorer intent and whether each instantiated case realizes the recipe and supports its expected relation. Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added. Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none. On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100).
Figures & tables
Figure 1: End-to-end comparison of native test-case generation and validation pipelines. Both SANE and JRH receive a scorer definition, implementation, and representative examples. Sane begins with a fixed set of behavioral requirements. It identifies capabilities per scorer, derives recipes per capability, then generates cases per recipe. Both harnesses instantiate cases and apply LLM validation. Sane separates recipe alignment from case realization and expected-relation applicability before assembling the validated test suite. Jrh uses a validator LLM with retries. Automating capability identification and recipe ideation makes Sane ’s aligned recipe inventory adapt to new scorers. Dashed boxes mark automated operations. Figure 3 defines the oracle decisions and validity accounting.
Figure 2: Concrete Sane toxicity test. The agent identifies implicit toxicity without profanity as a scorer capability and generates a monotonicity case that appends profanity-free belittling language. Blue text marks the generated addition. The expected relation states that text augmented to be more toxic should not receive a lower toxicity score, namely saug≥sbase−ϵ , with ϵ accounting for stochastic noise. This example passes our oracle judge panel unanimously for recipe alignment, realization, and relation applicability.
Figure 3: Validation from candidate recipes to test families. The recipe-level oracle evaluates scorer alignment. The case-level oracle separately evaluates perturbation realization and applicability of the expected relation (Figure 4 ). A family enters the test suite when a majority of sampled cases are joint-valid.
Figure 4: Oracle inputs and decisions. (a) The recipe-alignment oracle determines whether a candidate recipe and its expected relation agree with the scorer specification. For each aligned recipe, (b) The case-realization oracle separately determines whether the generated text implements the requested perturbation and whether the expected relation applies to that instance.
Figure 5: Cumulative aligned recipes across scorer subsets. Small points enumerate all subsets, large points give subset means, and bands give the full minimum-to-maximum range. The scorer set has no intrinsic order; enumerating every subset avoids imposing an arbitrary sequence of scorer additions. Sane ’s scorer-specific generation adds aligned recipes as scorers are added. Jrh ’s fixed library reapplies the same recipes regardless of scorer domain.
Figure 6: Recipe validation and fault detection. (a) The left plot tracks scorer-recipe pairs as they funnel through the evaluation pipeline. Sane ’s expanded scoring surface results in three times more fault-detecting test families than Jrh . (b) The right plot reports validated and fault-detecting test family counts for each scorer and method. Explicit zero labels mark method-scorer pairs with no fault-detecting family. Counts use each framework’s native expected relations. Sane detects faults across all scorer domains, including professionalism and toxicity, where Jrh produced no fault-detecting families.
Figure 7: Human alignment at the judge-model level. Each point represents one judge model measured by two independent summaries of scorer quality. FaithBench AUROC measures how well continuous beta-calibrated faithfulness scores distinguish human-labeled faithful and unfaithful examples. Harness pass rate measures how often the model satisfies expected relations over joint-valid faithfulness test cases. Spearman ρ and Kendall τb compare the rankings induced by these metrics, not their numerical values. Opus 5, which generated both test corpora, is excluded.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Scorer
Benchmark
AUROC
AUPRC
Spearman ρ
Kendall τb
Weighted Spearman ρw
Faithfulness
FaithBench
0.707
0.473
N/A
N/A
N/A
Fluency
SummEval
N/A
N/A
0.331
0.313
N/A
Coherence
SummEval
N/A
N/A
0.553
0.472
N/A
Professionalism
Stanford Politeness
N/A
N/A
0.336
0.259
0.277
Toxicity
Civil Comments
0.803
0.264
N/A
N/A
N/A
Appendix
Table 1: External human-label evidence for the scorer implementations. Metrics follow each benchmark’s target type, use sampling weights where applicable, and are not averaged across criteria. N/A marks metrics that do not apply to that target type. Stanford Politeness evaluates one component of professionalism. These labels evaluate the scorers, not the generated metamorphic cases.
Method
Pr(A)
Pr(Z∣A)
Pr(I∣A,Z)
Pr(J)
Sane
0.812
0.708
0.987
0.568
Jrh
0.714
0.860
0.986
0.606
Appendix
Table 2: Validity accounting from candidate recipe to joint-valid case. The analysis contains 133 candidate recipes and 1,330 potential cases for Sane , and 35 candidate recipes and 350 potential cases for Jrh . Each candidate recipe contributes 10 potential cases. We generate and validate cases for the 108 aligned Sane recipes and 25 aligned Jrh recipes. Unresolved oracle predictions count as invalid. Boldface marks the larger observed probability in each column.
Figure 8: Sensitivity of validated and fault-detecting test family counts to the family threshold τ , with n=10 sampled cases per aligned recipe. The dashed line marks the primary strict-majority choice, τ=6 . Counts are conservative: unresolved oracle predictions count as invalid, and unresolved scorer predictions count as not fault detecting. Sane retains more validated and fault-detecting test families than Jrh throughout the sweep. The direction of the comparison is stable across τ , while its magnitude narrows at stricter thresholds.
Figure 9: Validated and fault-detecting test families by scorer under three family thresholds. Labels report fault-detecting families over validated families. Dark bars give fault-detecting counts, and light extensions give the remaining validated families. Counts use conservative treatment of unresolved predictions.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness
Large language model (LLM) judges provide a flexible and scalable method for evaluating model and agent outputs, but their verdicts can be sensitive to incidental changes in the evaluated response, judge instructions, and scoring rubric. Existing systems examine important subsets of these failure modes, but auditing a configured judge requires testing both the judge instrument and the items it evaluates. We introduce MAWILE, a developer-facing workbench for auditing judge sensitivity across four surfaces: the judge prompt, judge rubric, target-system input, and target-system output. Given a user-supplied judge and representative evaluation items, MAWILE constructs and validates controlled perturbations, re-executes the judge, and localizes the resulting sensitivity. Each perturbation declares whether the verdict should remain invariant or change in a specified direction, allowing the same system to measure both robustness to irrelevant variations and sensitivity to meaningful changes. MAWILE audits binary, ordinal, and pairwise judges without requiring gold labels. The code for this tool is available at: github.com/megagonlabs/mawile-judge.
Jackson Hassell, Farima Fatahi Bayat, Pouya Pezeshkpour +1
LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge