AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning

Liang Ding

cs.AIMar 22, 2026

AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning

Authors: Liang Ding

Organizations: The University of Sydney

Abstract

Evaluating LLM agent trajectories is fundamentally task-specific: a code-debugging agent should be judged on Correctness and Error Handling, not on Fluency or Safety. Yet the dominant paradigm -- LLM-as-Judge with a fixed rubric -- applies the same static dimensions regardless of task, producing systematic mis-evaluation. We present AdaRubric, a framework that (i) adaptively generates task-specific evaluation rubrics from task descriptions via LLM, (ii) evaluates agent trajectories step-by-step with confidence-weighted, per-dimension scoring, and (iii) produces dense reward signals for preference learning. Three composable filtering strategies, including the novel DimensionAwareFilter that provably prevents dimension-level quality masking, yield high-quality DPO preference pairs. On WebArena, ToolBench, and AgentBench, AdaRubric achieves Pearson r = 0.79 human correlation (+0.15 over the strongest baseline), with strong reliability (Krippendorff's alpha = 0.83). DPO models trained on AdaRubric-generated pairs improve task success by +6.8-8.5% over the best baseline. AdaRubric also generalises zero-shot to unseen domains (SWE-bench) and extends to multimodal agents (VisualWebArena, OSWorld) without modification. Our code is available at: github.com/alphadl/AdaRubrics

Figures & tables

Panel of Figure 1 — Figure 1: Static evaluation vs. AdaRubric . Static LLM-as-Judge applies identical dimensions to all tasks ( $r{\approx}0.46$ ). AdaRubric synthesises task-specific rubrics ( $r{\approx}0.77$ ).

Panel of Figure 2 — Figure 2: AdaRubric pipeline. Stage 1 synthesises a task-adaptive rubric. Stage 2 evaluates trajectories per-step $\times$ per-dimension. Stage 3 applies composable filters and generates DPO pairs.

Panel of Figure 3 — Figure 3: Complete AdaRubric pipeline . All three stages are modular; any LLM can serve as $\mathcal{M}$ .

Method	WA	TB	AB	Avg	$\Delta$
ROUGE-L	0.31	0.26	0.29	0.29	$-0.35$
BERTScore	0.43	0.39	0.41	0.41	$-0.23$
G-Eval	0.54	0.49	0.52	0.52	$-0.12$
Prometheus	0.61	0.57	0.59	0.59	$-0.05$
GPT-4 Direct	0.64	0.60	0.62	0.62	—
AdaRubric -WM	0.74	0.70	0.72	0.72	$+0.10$

Table 1: Human correlation (Pearson

r

). AdaRubric -DA: DimensionAwareFilter variant.

\Delta

vs. GPT-4 Direct.

Method	WA	TB	AB
Base (zero-shot)	12.3	18.4	15.2
SFT – Success only	16.7	23.1	20.1
DPO – Random	17.4	24.8	21.3
DPO – G-Eval	20.1	27.6	24.5
DPO – Prometheus	21.0	29.3	26.4
DPO – AdaRubric -WM	24.3	34.2	30.8

Table 2: Multi-benchmark DPO training results. WebArena (SR%), ToolBench (TCR%), AgentBench (SR%). Qwen2.5-7B backbone.

\Delta

vs. Prometheus.

Method	WA	TB	Avg
G-Eval (GPT-4o)	0.64	0.61	0.63
Prometheus	0.71	0.68	0.70
GPT-4 Direct	0.69	0.66	0.68
AdaRubric -WM	0.81	0.79	0.80
AdaRubric -GM	0.83	0.80	0.82
AdaRubric -DA	0.85	0.82	0.84

Table 3: Evaluation reliability (Krippendorff’s

\alpha

). WA=WebArena, TB=ToolBench.

Method	$r$	$\alpha$	Res.%
G-Eval (GPT-4o)	0.51	0.63	8.2
Prometheus	0.56	0.70	9.1
GPT-4 Direct	0.59	0.68	9.8
AdaRubric -WM	0.72	0.82	12.4
AdaRubric -DA	0.77	0.84	14.7

Table 4: SWE-bench Lite evaluation. Pearson

r

with pass/fail oracle and resolve rate (%) after DPO fine-tuning of Llama-3.1-8B-Instruct. AdaRubric generalises to code-repair tasks with zero rubric engineering.

Variant	$r$	SR%
Fixed (generic)	0.51	19.1
Fixed (domain template)	0.65	22.4
Adaptive, no conf. wt.	0.72	24.0
Adaptive, no DAFilter	0.75	25.2
AdaRubric -DA (full)	0.79	27.8

Table 5: Ablation on WebArena. GPT-4o evaluator.

Backbone	$r$	$\alpha$	SR%
GPT-4 Direct	0.64	0.69	—
Prometheus (13B)	0.61	0.71	21.0
AR / GPT-4o	0.79	0.85	27.8
AR / Llama-70B	0.75	0.82	25.9
AR / Llama-8B	0.68	0.77	23.2

Table 6: Backbone generalisation (WebArena). Adaptive rubrics outperform GPT-4 Direct even with smaller models.

Prometheus baseline (static rubric)
Train Source	WA	TB	AB
WA $\to$ WA (in-dom.)	21.0	—	—
WA $\to$ TB (cross)	—	21.4	—
TB $\to$ WA (cross)	18.3	—	—
TB $\to$ TB (in-dom.)	—	29.3	—
AB $\to$ AB (in-dom.)	—	—	26.4

Table 7: Cross-domain transfer. “Train

\to

Test” denotes which benchmark’s DPO pairs are used for fine-tuning and which is the evaluation target. AdaRubric cross-domain exceeds Prometheus in-domain.

Reward	1K	3K	5K
GPT-4 Scalar	14.2	18.7	21.3
Prometheus	15.8	21.2	23.6
AdaRubric -WM	18.3	24.9	27.1
AdaRubric -DA	20.1	27.4	30.2

Table 8: PPO training with AdaRubric reward. WebArena SR% after 1K, 3K, and 5K rollout steps. Dense per-dimension rewards substantially accelerate convergence.

Rubric Source	Rel.	Orth.	Comp.
Generic (Help./Safety/Flu.)	2.1	3.8	1.6
Domain template (manual)	3.9	3.6	3.7
Expert-designed	4.6	4.4	4.5
AdaRubric -generated	4.3	4.2	4.1

Table 9: Rubric quality human study. Scores are mean Likert (1–5). Inter-annotator agreement

\kappa{=}0.79

. AdaRubric -generated rubrics approach expert-designed quality.

Filter	$r$	SR%	Retained
None	0.71	21.7	100%
AbsoluteThreshold	0.74	24.0	72.3%
PercentileFilter	0.73	23.4	80.0%
DimensionAwareFilter	0.79	27.8	61.5%
CompositeFilter	0.78	26.9	47.2%

Table 10: Filter strategy comparison (WebArena). DimensionAwareFilter achieves the best trade-off between quality and pair retention.

$\lambda$	$r$	SR%
0 (uniform)	0.71	23.5
0.25	0.75	25.4
0.5 (default)	0.79	27.8
1.0	0.77	26.9
2.0	0.72	24.8

Table 11: Effect of recency decay

\lambda

(WebArena).

\lambda{=}0.5

provides optimal balance between recent and early steps.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	VWA		OSWorld
Method	$r$	SR%	$r$	SR%
G-Eval	0.48	—	0.44	—
GPT-4V Direct	0.62	—	0.59	—
AdaRubric -DA	0.76	26.3	0.73	22.8

Appendix

Table 12: Multimodal agent evaluation. Pearson

r

and DPO SR% on VisualWebArena (VWA) and OSWorld.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape
Jun 7, 2026Hao Chen, Ziyu Han, Yukun Yan +3LLM Evaluation Rubric-Based Evaluation
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University
CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
Jul 31, 2026Mengting Chen, Yanshu Sun, Wanting Liang +5Open-Ended Generation LLM Evaluation
FinStep · StepFun
ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents
Jun 19, 2026Zihang Tian, Jingsen Zhang, Rui Li +3Process Reward Models Credit Assignment in RL
Renmin University of China

Jun 7, 2026J/K move · Enter open · S save

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape

Hao Chen, Ziyu Han, Yukun Yan, Qingfu Zhu+2

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University

LLM Evaluation Rubric-Based Evaluation

Open paper AskPDF arXiv

As Large Language Models (LLMs) advance toward open-ended autonomous agents, the mechanisms used to evaluate and guide their behavior must evolve accordingly. This work introduces the rubric as a unifying framework capturing this evolution, characterizing rubrics as a dynamic response to successive LLM paradigm shifts that recurs across otherwise independent efforts in evaluation, reinforcement learning, and safety alignment. We define rubrics as explicit criteria sets that transform complex quality judgments into structured and actionable standards, and demonstrate that their recurrence across these research threads is not coincidental. We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training. Rubrics manifest at three progressively deeper levels: at the evaluative level, they decompose holistic judgments into verifiable dimensions; at the training level, they serve as dense feedback signals providing process-level guidance where scalar rewards fall short; at the intrinsic level, they emerge dynamically from model behaviors, driving self-improvement. We further assess rubric reliability across generation quality, execution fidelity, theoretical constraints, and security threats, before surveying rubric-based benchmarks across diverse domains. By rendering assessment transparent and decomposable, rubrics translate human value expectations into machine-learnable signals, serving as the enduring bridge between human intentions and machine behavior.

Abstract

LLM Agent Evaluation

Explore similar work

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape

CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents