MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluators

Jackson Hassell; Farima Fatahi Bayat; Pouya Pezeshkpour; Estevam Hruschka

cs.AISep 18, 2026

MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluators

Authors: Jackson Hassell, Farima Fatahi Bayat, Pouya Pezeshkpour, Estevam Hruschka

Organizations: Megagon Labs

Abstract

Large language model (LLM) judges provide a flexible and scalable method for evaluating model and agent outputs, but their verdicts can be sensitive to incidental changes in the evaluated response, judge instructions, and scoring rubric. Existing systems examine important subsets of these failure modes, but auditing a configured judge requires testing both the judge instrument and the items it evaluates. We introduce MAWILE, a developer-facing workbench for auditing judge sensitivity across four surfaces: the judge prompt, judge rubric, target-system input, and target-system output. Given a user-supplied judge and representative evaluation items, MAWILE constructs and validates controlled perturbations, re-executes the judge, and localizes the resulting sensitivity. Each perturbation declares whether the verdict should remain invariant or change in a specified direction, allowing the same system to measure both robustness to irrelevant variations and sensitivity to meaningful changes. MAWILE audits binary, ordinal, and pairwise judges without requiring gold labels. The code for this tool is available at: github.com/megagonlabs/mawile-judge.

Figures & tables

Figure 1: The Mawile end-to-end judge-auditing workflow. Given a configured judge and representative audit items, the planner agent produces an editable suite of invariant and directional probes. Mawile then generates and validates these variants, re-executes the judge alongside repeated unmodified baselines, and reports robustness, degradation sensitivity, and—when gold labels are available—correctness.

	Judge Instrument		Evaluated Instance			Developer-Facing Audit System
Method	Rubric $R$	Prompt $P$	Input $x$	Output $y$	Coupled $x,y$	Developer-Facing Audit System
FBI ( Doddapaneni et al., 2024 )	–	–	–	✓	–	–
reWordBench ( Wu et al., 2025 )	–	–	✓	✓	✓	–
CALM ( Ye et al., 2025 )	–	✓	–	✓	–	–
JudgeSense ( Bellibatlu et al., 2026 )	–	✓	–	–	–	–
JRH ( Dev et al., 2026 )	–	–	–	✓	–	✓

Table 1: Comparison with prior judge-auditing systems and perturbation frameworks. A checkmark indicates that the method automatically constructs or applies controlled variants of the corresponding surface as an intervention target, not that a given surface is merely configurable.

Figure 2: Screenshot of the Mawile Web UI’s editable perturbation catalog. During configuration, developers can inspect each operator’s family, construction method, target field, and expected verdict relation, then enable or disable it before execution. A planner agent can also suggest perturbations tailored to the dataset.

Figure 3: Screenshot of the Mawile web UI’s per-operator results for the MT-Bench walkthrough using DeepSeek-V4-Flash as the judge and Kimi K3 as the perturbation generator.

			Baseline		Perturbation Response
Dataset	Perturbation Generator	Judge	Accuracy $\uparrow$	Repeat Flip Rate $\downarrow$	Invariant Flip Rate $\downarrow$	Degradation Detection Rate $\uparrow$
MT-Bench Ordinal	Qwen 3.8 Max	GPT-5.6 Luna	56.0	2.3	6.0	62.9
		DeepSeek-V4-Flash	59.1	5.3	11.7	65.4
		Gemma 3 4B	51.2	0.5	2.8	47.4
	Kimi K3	GPT-5.6 Luna	53.6	3.2	5.7	65.4
		DeepSeek-V4-Flash	59.0	4.0	12.2	68.9

Table 2: Aggregate audit results for ordinal, pairwise, and binary judges across two perturbation generators and three judge models. Within each dataset, bold and underlining mark the best and second-best judges in each metric, respectively. All numbers are percentages. For each dataset, a Kimi K3 planner selected one operator suite, which is held fixed across all six generator-judge combinations.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Screenshot of the Mawile web UI for configuring a judge and audit dataset.

Figure 5: Screenshot of the Mawile planning agent’s proposed perturbation suite for the MT-Bench walkthrough, with justification.

Figure 6: Screenshot of the Mawile web UI for defining a custom perturbation, with an example of an LLM-suggested custom perturbation.

Figure 7: Screenshot of the Mawile report overview for the MT-Bench walkthrough.

Figure 8: Screenshot of the Mawile review queue for inspecting flagged items from the MT-Bench walkthrough.

Invariant Perturbation Catalog
Surface	Perturbation	Gen.	Description
Judge prompt	Instruction ordering	Rule	Moves output-format instructions before the prompt and rubric.
	Reasoning style	LLM	Requests direct, step-by-step, or criterion-by-criterion reasoning.
	Role framing	LLM	Changes the evaluator persona without changing the task.
	Prompt paraphrase	LLM	Rewords the judge instructions while preserving their meaning.
Judge rubric	Rubric criterion reorder	Rule	Reorders independent rubric blocks without changing their content.

Appendix

Table 3: Built-in invariant perturbations.

Directional Perturbation Catalog
Surface	Perturbation	Gen.	Description
Agent input	Unmet requirement injection	LLM	Adds a task requirement that the existing response does not satisfy.
Agent output	Partial completion	Rule	Removes the final separable part of a response.
	Format violation	LLM	Breaks one previously satisfied output-format requirement.
	Unsupported claim insertion	LLM	Adds one unsupported substantive claim to the response.
	Factual inconsistency insertion	LLM	Introduces one factual or reasoning inconsistency.

Appendix

Table 4: Built-in directional degradation perturbations. Directional probes are available only for pointwise audits.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Mar 5, 2026Sunishchal Dev, Andrew Sloan, Joshua Kavner +2LLM-as-a-Judge LLM Evaluation
RAND Corporation, Santa Monica, CA, USA
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
Oct 20, 2026Ethan Ludwick, Yu-Chung HsiaoLLM-as-a-Judge Automated Test Generation
Cisco Systems · Anduril Industries

Mar 5, 2026J/K move · Enter open · S save

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

Sunishchal Dev, Andrew Sloan, Joshua Kavner, Nicholas Kong+1

RAND Corporation, Santa Monica, CA, USA

LLM-as-a-Judge LLM Evaluation Language Model Robustness

Open paper AskPDF arXiv

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness

Abstract

LLM-as-a-Judge

Explore similar work

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges