Artificial intelligence has progressed from task-specific systems toward general models spanning perception, generation, reasoning, and autonomous interaction, making reliable evaluation essential for comparison, deployment, and post-training feedback. At scale, modern evaluation pipelines increasingly rely on LLM-as-a-Judge to produce scores, preferences, and critiques; in its fixed-context setting, the judge relies on the supplied evaluand and evaluation materials. Yet a reliable verdict may require evidence beyond the supplied evaluand and context---either because critical environmental states remain unrecorded in the static execution log, or because controversial factual claims require external verification. Passive judges cannot obtain that missing evidence before deciding, especially when evaluating agent trajectories, executable artifacts, or domain-specific reports. Recent work therefore develops active evaluators that seek missing evidence and update their grounds for judgment before deciding. We bring these advances together through Agent-as-a-Judge, focusing on how evaluators construct and control checking processes within the broader LLM-based judging landscape. We define the scope through evaluation-directed checks and the use of their results, covering both structured workflows and observation-driven control. We organize the design space by multi-agent collaboration, tool integration, memory, and planning, and show how these mechanisms supply the checks and evidence on which a verdict rests. We consolidate published scores and reproduce missing entries for key agentic judges; the comparison reveals uneven agentic gains and motivates dedicated agentic evaluation. We conclude by characterizing application demands and open challenges in cost, reliability, safety, and governance that condition trusted deployment of agentic judges at scale.
Figures & tables
Figure 1: Comparison between (a) LLM-as-a-Judge and (b) Agent-as-a-Judge. The former acts as a passive evaluator relying on the given context and model-internal knowledge, while the latter acts as an active evaluator that plans, gathers evidence, uses tools, and iteratively verifies its judgment.
Figure 3: Role lexicon of subagents in multi-agent judges. The four panels group cleaned subagent names into adversarial, perspective, analytical, and verdict-control roles.
Figure 4: From functional role names to implementation classes. The Sankey maps subagent and component instances from functional roles to implementation classes.
Figure 5: Distribution of 60 tool-interface assignments in a manually coded subset of 34 tool-augmented judges. Each bar shows one interface’s share of the total; papers using multiple interfaces contribute one count per interface type.
Figure 6: Functional view of tool use in Agent-as-a-Judge . Four overlapping regions show that judges can chain multiple tool roles within one evaluation.
Figure 7: Memory in Agent-as-a-Judge can be analyzed as a write–manage–read loop over evaluation state. The stored state may take the form of process memory, artifact memory, or experience memory. Later judgments either receive this state directly, access a filtered or summarized form, or target specific artifacts, failures, or templates through retrieval.
Figure 8: Planning in Agent-as-a-Judge systems can externalize either what to check or how to gather proof before the verdict.
System
Planned element
Decision point
Mechanism
What-to-check planning
DREAM [ DREAM ]
Query-specific evaluation protocol
Before protocol execution
Turns the query into report-specific fact checks and validation questions.
APRES [ apres2026 ]
Review rubric
During rubric search
Searches over candidate criteria and keeps the rubric used for revision.
Evaluation Agent [ zhang2025EvaluationAgent ]
Next sub-aspect
Across proposal rounds
Moves from one aspect to the next and stops when the accumulated evidence is sufficient.
How-to-check planning
EvalPlanner [ evalplanner2025 ]
Unconstrained evaluation plan
Before scoring
Writes the checking route first and then executes it.
Table 1: Planning mechanisms in Agent-as-a-Judge . What-to-check planning decides the evaluation target, while how-to-check planning decides the evidence-gathering route.
Figure 9: Representative adoption timeline of the four methodological mechanisms in Agent-as-a-Judge . A method may appear in more than one lane when it instantiates multiple mechanisms. Stars mark two sparse patterns discussed in the text: rubric construction on Planning and experience memory on Memory.
Figure 10: Evaluation–benchmark coverage map. Links indicate that a method reports results on, or is explicitly evaluated on, a reported benchmark. Right-node fill and pattern encode benchmark artifact type: ∙∙∙ fixed response or judgment sets packaged for direct comparison; / / / task problems whose reported results require candidate generation before judging; and green nodes, NLG evaluation datasets.
Benchmark / dataset
Evaluand form
Evaluand
Domain
Judgment signal
Scale
URL
RewardBench [ lambert2024rewardbench ]
pair-wise
response pair
chat, safety, reasoning, code/math
chosen–rejected preference
2,985
RM-Bench [ liu2024rmbench ]
pair-wise
response pair
chat, code, math, safety
subtle correctness under style variation
1,237
JudgeBench [ tan2025judgebench ]
pair-wise
response pair
knowledge, reasoning, math, code
objective correctness label
620
IFBench [ peng2025AgenticRM ]
pair-wise
response pair
instruction following
constraint satisfaction
444
VerifyBench [ yan2025verifybench ]
point-wise
completion
reference-based answer verification
binary correctness against reference
3,000
ProcessBench [ zheng2025processbench ]
step-wise
reasoning trace
mathematical reasoning
earliest-error step label
3,400
Table 2: Reward-model and verifier benchmarks.
Benchmark
Method
Model
Subsets
Overall
Chat
Chat-Hard
Safety
Reason
RewardBench
TIR-Judge [ xu2025TIRJudge ] †
TIR-Judge-8B
94.7
77.4
88.8
95.7
89.1
Agentic Reward Model [ peng2025AgenticRM ]
ArmoRM-Llama3-8B ∗
90.8
70.0
86.5
93.2
87.6
Multi-Agent Verification [ lifshitz2025multi ]
Qwen3-8B
78.5
80.2
86.1
86.7
82.9
VerifiAgent [ han2025VerifiAgent ]
Qwen3-8B
69.8
65.8
78.6
61.3
68.9
Themis [ li2024toolaugmented ]
Themis-7B
82.7
42.3
45.7
37.1
52.0
Table 3: Local reward-model results on shared judge-quality benchmarks.
Figure 11: Application map of Agent-as-a-Judge . General domains mainly stress verifiability, evidence gathering, interaction, and multimodal inspection, while professional domains add stronger requirements on expertise, traceability, safety, and consistency.
Domain
Judgment target
Evaluation gap
Agentic capability
Table 5: Application-specific requirements for Agent-as-a-Judge . Each domain is summarized by its judgment target, the limitation of fixed-context judging, and the agentic capability used to address it.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.
Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian +1
As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.
Wentao Shi, Yu Wang, Yuyang Zhao +8
University of Science and Technology of China · National University of Singapore · Meituan
Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access. In one case study, two frontier LLM judges scored a plausible agent response 0.85 and higher. But the trace told a different story: the agent had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000. We introduce GroundEval, a judge-free framework for evaluating agents against grounded, time-bounded, and access-controlled evidence. GroundEval uses a domain configuration to generate questions, lets the agent choose how to answer, and then scores both the final answer and the recorded trajectory that produced it. The benchmark targets three failures that LLM-as-judge evaluation struggles to detect: whether an agent checked before claiming absence, reasoned only from evidence available to the actor at the relevant time, and used the correct causal mechanism rather than a plausible one. These correspond to three tracks: Silence, Perspective, and Counterfactual. GroundEval exposes when plausible answers rest on invalid evidence paths, and produces structured per-question diagnostics that pair tool activity with the agent's turn-level narration, making each score inspectable rather than merely reported. Our case studies suggest this failure mode is common rather than exceptional, one that final-answer and judge-based evaluation cannot detect by construction.