Large language model (LLM) judges provide a flexible and scalable method for evaluating model and agent outputs, but their verdicts can be sensitive to incidental changes in the evaluated response, judge instructions, and scoring rubric. Existing systems examine important subsets of these failure modes, but auditing a configured judge requires testing both the judge instrument and the items it evaluates. We introduce MAWILE, a developer-facing workbench for auditing judge sensitivity across four surfaces: the judge prompt, judge rubric, target-system input, and target-system output. Given a user-supplied judge and representative evaluation items, MAWILE constructs and validates controlled perturbations, re-executes the judge, and localizes the resulting sensitivity. Each perturbation declares whether the verdict should remain invariant or change in a specified direction, allowing the same system to measure both robustness to irrelevant variations and sensitivity to meaningful changes. MAWILE audits binary, ordinal, and pairwise judges without requiring gold labels. The code for this tool is available at: github.com/megagonlabs/mawile-judge.
Figures & tables
Figure 1: The Mawile end-to-end judge-auditing workflow. Given a configured judge and representative audit items, the planner agent produces an editable suite of invariant and directional probes. Mawile then generates and validates these variants, re-executes the judge alongside repeated unmodified baselines, and reports robustness, degradation sensitivity, and—when gold labels are available—correctness.
Judge Instrument
Evaluated Instance
Developer-Facing Audit System
Method
Rubric R
Prompt P
Input x
Output y
Coupled x,y
FBI ( Doddapaneni et al., 2024 )
–
–
–
✓
–
–
reWordBench ( Wu et al., 2025 )
–
–
✓
✓
✓
–
CALM ( Ye et al., 2025 )
–
✓
–
✓
–
–
JudgeSense ( Bellibatlu et al., 2026 )
–
✓
–
–
–
–
JRH ( Dev et al., 2026 )
–
–
–
✓
–
✓
Table 1: Comparison with prior judge-auditing systems and perturbation frameworks. A checkmark indicates that the method automatically constructs or applies controlled variants of the corresponding surface as an intervention target, not that a given surface is merely configurable.
Figure 2: Screenshot of the Mawile Web UI’s editable perturbation catalog. During configuration, developers can inspect each operator’s family, construction method, target field, and expected verdict relation, then enable or disable it before execution. A planner agent can also suggest perturbations tailored to the dataset.
Figure 3: Screenshot of the Mawile web UI’s per-operator results for the MT-Bench walkthrough using DeepSeek-V4-Flash as the judge and Kimi K3 as the perturbation generator.
Baseline
Perturbation Response
Dataset
Perturbation Generator
Judge
Accuracy ↑
Repeat Flip Rate ↓
Invariant Flip Rate ↓
Degradation Detection Rate ↑
MT-Bench Ordinal
Qwen 3.8 Max
GPT-5.6 Luna
56.0
2.3
6.0
62.9
DeepSeek-V4-Flash
59.1
5.3
11.7
65.4
Gemma 3 4B
51.2
0.5
2.8
47.4
Kimi K3
GPT-5.6 Luna
53.6
3.2
5.7
65.4
DeepSeek-V4-Flash
59.0
4.0
12.2
68.9
Table 2: Aggregate audit results for ordinal, pairwise, and binary judges across two perturbation generators and three judge models. Within each dataset, bold and underlining mark the best and second-best judges in each metric, respectively. All numbers are percentages. For each dataset, a Kimi K3 planner selected one operator suite, which is held fixed across all six generator-judge combinations.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Screenshot of the Mawile web UI for configuring a judge and audit dataset.
Figure 5: Screenshot of the Mawile planning agent’s proposed perturbation suite for the MT-Bench walkthrough, with justification.
Figure 6: Screenshot of the Mawile web UI for defining a custom perturbation, with an example of an LLM-suggested custom perturbation.
Figure 7: Screenshot of the Mawile report overview for the MT-Bench walkthrough.
Figure 8: Screenshot of the Mawile review queue for inspecting flagged items from the MT-Bench walkthrough.
Invariant Perturbation Catalog
Surface
Perturbation
Gen.
Description
Judge prompt
Instruction ordering
Rule
Moves output-format instructions before the prompt and rubric.
Reasoning style
LLM
Requests direct, step-by-step, or criterion-by-criterion reasoning.
Role framing
LLM
Changes the evaluator persona without changing the task.
Prompt paraphrase
LLM
Rewords the judge instructions while preserving their meaning.
Judge rubric
Rubric criterion reorder
Rule
Reorders independent rubric blocks without changing their content.
Appendix
Table 3: Built-in invariant perturbations.
Directional Perturbation Catalog
Surface
Perturbation
Gen.
Description
Agent input
Unmet requirement injection
LLM
Adds a task requirement that the existing response does not satisfy.
Agent output
Partial completion
Rule
Removes the final separable part of a response.
Format violation
LLM
Breaks one previously satisfied output-format requirement.
Unsupported claim insertion
LLM
Adds one unsupported substantive claim to the response.
Factual inconsistency insertion
LLM
Introduces one factual or reasoning inconsistency.
Appendix
Table 4: Built-in directional degradation perturbations. Directional probes are available only for pointwise audits.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness
LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge
LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion. Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types. We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions. Given a scorer specification, implementation, and representative inputs, the pipeline first identifies scorer-specific capabilities, then automatically derives perturbation recipes and expected score relations for each. SANE also defines a scorer-independent validation framework for generated judge tests. It determines whether each recipe agrees with scorer intent and whether each instantiated case realizes the recipe and supports its expected relation. Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added. Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none. On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100).