PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

Cheng Chang; Yining Mao; Peng Qi

cs.CLSep 28, 2026

PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

Authors: Cheng Chang, Yining Mao, Peng Qi

Organizations: Uniphore

Abstract

Language models are frequently employed to evaluate other language models. An LM evaluator scoring agentic behaviors across multiple criteria is valuable, provided that its decisions align with human judgment. We call the problem of evaluating this alignment Meta-Evaluation. Tackling it directly is difficult: collecting human data is expensive, absolute scoring is hard to align, and using an LM meta-evaluator recurses the question of trustworthiness. We adopt a reformulation of meta-evaluation as a preference judgment problem: rather than comparing human and LM evaluator scores of a trajectory, we ask whether their implied preferences align. Building on this, we introduce PADMÉ, a data synthesis method that generates reliable criterion-based meta-evaluation data for agentic settings. PADMÉ uses only small language models, requires no human involvement during evaluations, and operates under a low computational budget. We build a prototype of PADMÉ and synthesize a dataset of 1,000 samples across four agentic domains and three evaluation criteria. Human validation on a 150-sample subset demonstrates that PADMÉ improves agreement with human judgment from 73% to 85% over a naive baseline. Meta-evaluating 25 common models with our dataset demonstrates the correlations between evaluation performance and scoring granularity, leniency, and model size, among other factors.

Figures & tables

Figure 1: Overview of criterion-specific meta-evaluation. Given multi-step user-agent interaction trajectories evaluated under a specific criterion $c$ (e.g., communication clarity), an LM evaluator $f$ assigns a continuous score $s_{f}\in[0,1]$ . Meta-evaluation asks the question “Which of the evaluators ( $s_{1},s_{2},s_{3}$ ) should I trust?”, which PADMÉ attempts to solve through preference alignment.

Figure 2: Overview of the PADMÉ pipeline and meta-evaluation architecture. For each task-criterion cell $(t,c)$ from $\tau^{3}$ -bench, steering prompts generate trajectory pairs $p=(x^{-},x^{+})$ across three quality levels ( good , ok , bad ). Two LM filter judges discard pairs lacking visible quality contrast to yield dataset $\mathcal{D}_{K}$ , validated against a human annotation subset. Target evaluators rate individual trajectories ( $s=f(x;c)\in[0,1]$ ), and meta-evaluation assesses their preference alignment across verified pairs.

retry (r)	draws	kept	$J_{1}$ rejected	$J_{2}$ rejected	retention rate (%)	cumulative retention rate (%)
$0$	1,125	770	266	89	68.4	68.4
$1$	355	161	146	48	45.4	82.8
$2$	193	69	100	24	35.8	88.9
total	1,673	1,000	512	161	59.8	88.9

Table 1: Data curation funnel over 1,125 cells across retry attempts.

dataset	$\|\mathcal{D}\|$	yield	synthetic label accuracy	$\alpha$
$\mathcal{D}_{0}$ (unfiltered)	150	100%	73.3% [66–80]	+0.468
$\mathcal{D}_{1}$ ( $J_{1}$ )	114	76%	84.2% [76–90]	+0.686
$\mathcal{D}_{2}$ ( $J_{1}\&J_{2}$ )	104	69%	84.6% [76–90]	+0.694

Table 2: Agreement between synthetic labels and human majority vote across filter depths

\mathbf{\mathcal{D}_{k}}

. Yield indicates the proportion of the 150 annotated pairs surviving at each depth. Synthetic label accuracy reports

\mathrm{Agr}(\mathrm{maj},\star;\mathcal{D}_{k})

with bootstrap 95% confidence intervals in brackets.

\alpha

is Krippendorff’s

\alpha

for the same label-versus-majority decision.

			parameters			accuracy (%)	leniency	average	tie	by domain (%)				by criterion (%)			# distinct
#	evaluator	released	total	active	reas.	mean $\pm$ SD	(mean score)	score SD	(%)	airl.	bank.	retail	telec.	clar.	friend.	task	scores
1	claude-opus-5	2026-07	closed	closed	Yes	85.1 $\pm$ 0.93	0.324	0.023	4.1	84	82	86	87	73	96	85	69
3	kimi-k3	2026-07	2.8T	104B	Yes	83.6 $\pm$ 0.12	0.508	0.036	6.0	83	80	84	87	71	97	81	37
6	glm-5p2	2026-06	753B	$\sim$ 30B	Yes	81.5 $\pm$ 0.50	0.521	0.051	10.1	84	81	81	82	78	92	73	31
8	gpt-5.4-nano ‡	2026-03	closed	closed	No	79.4 $\pm$ 1.16	0.631	0.052	7.8	81	78	78	81	74	84	80	61
10	gpt-5.6-sol ‡	2026-07	closed	closed	Yes	78.7 $\pm$ 0.40	0.543	0.035	4.9	79	77	81	77	64	97	74	86

Table 3: Performance of evaluators across 1,000 trajectory pairs (

n=3

runs). 12 of 25 evaluators are selected to span the accuracy range and model families (full view in Table 18 ). All reported statistics are computed across all 25 evaluators. Released indicates public release month. Temperature = 0 except where unavailable ‡ . Preferences derive from individual trajectory score gaps; ties count as incorrect. Accuracy and score denote benchmark level accuracy and raw evaluator score, respectively. SD is standard deviation. Leniency is mean emitted score. Average score SD measures per data point scoring stability across 3 runs. Bold indicates best performance per column across all 25 evaluators. Accuracy and average score SD serve as the two primary evaluator metrics.

factor	$n$	$\rho$	$p_{\mathrm{holm}}$
tie rate	25	$-0.946$	$\mathbf{<0.0001}$
distinct score values	25	$+0.797$	$\mathbf{<0.0001}$
leniency (mean score)	25	$-0.725$	$\mathbf{0.0002}$
total parameters	15	$+0.725$	$\mathbf{0.0089}$

Table 4: Seven evaluator properties against accuracy, sorted by Spearman’s correlation coefficients.

p_{\mathrm{holm}}

is the Holm-Bonferroni corrected p-value across the seven tests, and bolded where it indicates significance.

n=15

on some rows because ten closed models do not publish parameter counts. Reasoning is a binary factor.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Generation
$t$	a task from the dataset, carrying its domain policy, tools, and user goal.
$c$	a criterion, supplied at runtime as free text: a name and a detailed description.
$\ell\in\mathcal{L}$	a steer level, $\mathcal{L}=\{\textsf{bad}\prec\textsf{ok}\prec\textsf{good}\}$ , ordered by intended quality.
$\theta$	everything held fixed while the steer level varies: agent model, user simulator, domain, decoding seed.
$s_{c,t,\ell}$	the steering instruction, written by a small model from $c$ , $t$ , and $\ell$ , Eq. ( 1 ).
$x$	a trajectory: the complete trace generated by an interactive dialogue between a user simulator and an agent executing a task, with tool calls and its outcome.

Appendix

Table 5: Notation, grouped by pipeline stage in the order the paper uses it.

level	pairs	description
Evaluation criterion
friendliness	355	Warmth and consideration toward the customer: whether the agent acknowledges their situation and how they feel about it, delivers unwelcome news with care, and leaves them feeling attended to. Judge the manner, not whether the request was resolved.
task resolution	329	Whether the customer’s actual problem was settled: did the agent establish what was needed, take the actions that would resolve it, and leave the customer with the outcome they came for. A correct refusal counts as resolution – if the request was not permitted, saying so plainly and explaining why resolves it, while quietly doing it anyway does not. Judge the outcome, not the manner or how well it was explained.
communication clarity	316	How easily the customer can follow the agent: whether the main point is findable, whether technical or policy language is explained, whether multi-part information is organised, and whether the customer is left knowing what is true and what happens next. Judge the presentation, not the warmth or the outcome.
Domain
retail	309	114 tasks; 1,158-word policy, 16 agent tools

Appendix

Table 6: A glossary of dataset subsets, along with the distribution of the 1,000 pairs across axes. Each pair carries exactly one level per axis, resulting in subtotal sums of 1,000 per block. Comprehensive retention rates per subset appear in Table 8 . Evaluation criterion descriptions reflect the full runtime definitions, also provided in Appendix H.1 . Stratified experimental breakdowns are reported in Appendix E .

	filter judge $J_{k}$	human annotator $a$	evaluator under test $f$
purpose	curates the dataset	validates the dataset	the object of measurement
is given	the criterion name and definition, the agent’s tools and the domain policy, and both trajectories	the same five fields, rendered in a purpose-built interface (Appendix D )	the criterion name and definition, the agent’s tools and the domain policy, and one trajectory
is not given	the steering prompts, the target levels, or which side was steered better	the same, and no filter verdict	the same, and never the other trajectory
is asked	which of the two is better on the criterion	which of the two is better on the criterion	to score the agent on the criterion from $0.0$ to $1.0$
slot order	randomized per pair and per judge	randomized per pair	no order arises: one trajectory per call
returns	a forced choice of one side, with at most three sentences of reasoning	a forced choice of one side, with a confidence rating of $\{0,1,2\}$	a score in $[0,1]$ , with at most three sentences of reasoning

Appendix

Table 7: A glossary of information exposure, task objectives, and output specifications across preference providers. Symmetrical blinding ensures human annotators directly benchmark filter judge decisions, whereas evaluators under test operate in single-trajectory pointwise mode to match real-world deployment.

axis	subset	cells filled	retention rate	cum. retention rate	draws per retention
By criterion
	friendliness	355/375	70.3%	95%	1.42 $\times$
	task resolution	329/375	58.9%	88%	1.70 $\times$
	communication clarity	316/375	51.9%	84%	1.93 $\times$
By domain
	airline	135/150	61.4%	90%	1.63 $\times$

Appendix

Table 8: Cell retention metrics by subset after 2 retries

(R=2)

. The retention rate measures the proportion of retained final pairs against the total number of generation draws across all attempts. The cumulative retention rate denotes the proportion of retained final pairs relative to the original set of target cells prior to retries. Draws per retention measures the average number of generation attempts required to yield one retained pair.

axis	slot-2 share range	binomial $p$ range
criterion	48.7–53.8%	0.17–0.91
domain	46.9–54.1%	0.27–0.73
agent model	45.5–54.3%	0.11–0.22
level contrast	49.2–54.0%	0.14–0.82
whole dataset	511/1,000 = 51.1%

Appendix

Table 9: Position balance audit of the target (better-steered) trajectory across data subsets. Across all groups, slot distribution remains statistically indistinguishable from uniform parity (binomial

p>0.10

), confirming that position bias is not a baseline confounder.

criterion	pairs	mean reward gap	better / worse	sign test $p$
task resolution	329	$+0.122$	49 / 9	$<0.001$
communication clarity	316	$+0.082$	40 / 14	$<0.001$
friendliness	355	$+0.017$	29 / 23	0.488

Appendix

Table 10: Environment reward verification comparing deterministic

\tau^{3}

-bench task success reward gaps across evaluation criteria. The reward gap represents the score of the higher-steered trajectory minus that of the lower-steered trajectory. Among the 1,000 pairs evaluated on both trajectories, 836 pairs (84%) exhibit identical environment rewards.

Figure 3: The interface used by the human panel. The two trajectories of a pair are presented together with the criterion being judged; the steering instruction and target levels are withheld, and the annotator returns a forced binary choice together with a confidence rating.

rater pool	$\mathcal{D}_{0}$	$\mathcal{D}_{1}$	$\mathcal{D}_{2}$
$(a_{1},a_{2},a_{3})$	$+0.350$	$+0.369$	$+0.383$
$(a_{i},a_{j},\star)$ , mean $\pm$ SD	$+0.372\pm 0.055$	$+0.479\pm 0.069$	$\mathbf{+0.496}\pm 0.065$
$\Delta$	$+0.022$	$+0.110$	$+0.113$

Appendix

Table 11: Krippendorff’s

\alpha

over a panel of three raters. The first row is the human panel

(a_{1},a_{2},a_{3})

. The second row substitutes one human rater with synthetic label

\star

and reports mean

\pm

SD across permutations

(a_{1},a_{2},\star)

(a_{1},a_{3},\star)

, and

(a_{2},a_{3},\star)

\Delta

represents the difference between hybrid and human-only panels.

measure	pooled (150)	panel 1 (75)	panel 2 (75)
Krippendorff’s $\alpha$	$\mathbf{+0.350}$ [+0.24, +0.45]	$+0.451$ [+0.29, +0.60]	$+0.240$ [+0.09, +0.38]
unanimous (3–0)	77/150 = 51%	44/75 = 59%	33/75 = 44%
Agreement between every two annotators of a panel
	agreement	Cohen’s $\kappa$	both said “certain”
panel 1, $(a_{1},a_{2})$	81.3% (61/75)	$\mathbf{+0.63}$	85% (28/33)
panel 1, $(a_{1},a_{3})$	65.3% (49/75)	$+0.32$	75% (24/32)

Appendix

Table 12: Reliability metrics for majority vote reference

\mathrm{maj}

across the 150 rated pairs in

\mathcal{D}_{0}

(75 per panel). Unanimous indicates a 3–0 vote. Lower blocks report pairwise agreement metrics for two-annotator panel subsets.

Kept against removed, per filter judge
subset	$\|\mathcal{D}\|$	mean confidence	SD
$\mathcal{D}_{0}$ (unfiltered)	150	1.49	0.43
$\mathcal{D}_{1}$ ( $J_{1}$ )	114	1.49	0.44
$\mathcal{D}_{2}$ ( $J_{1}\&J_{2}$ )	104	1.51	0.44
judge	kept	removed	difference ( $p$ )
$J_{1}$	1.49 ( $\mathcal{D}_{1}$ , 114)	1.49 ( $\mathcal{D}_{0}\setminus\mathcal{D}_{1}$ , 36)	$\mathbf{+0.00}$ (1.000)

Appendix

Table 13: Human annotator confidence and agreement metrics across filter depths

\mathcal{D}_{k}

. Per-pair confidence reflects the mean of three annotator ratings (

0=\text{guess}

1=\text{leaning}

2=\text{certain}

). Statistical significance (

p

) is computed via permutation tests over 100,000 shuffles.

\alpha

is Krippendorff’s

\alpha

subset	as reported	domain-weighted	criterion-weighted	agent-weighted	contrast-weighted
$\mathcal{D}_{0}$ (unfiltered)	73.3%	73.1%	73.3%	73.4%	73.7%
$\mathcal{D}_{1}$ ( $J_{1}$ )	84.2%	83.4%	83.8%	84.6%	84.6%
$\mathcal{D}_{2}$ ( $J_{1}\&J_{2}$ )	84.6%	83.9%	84.2%	85.0%	85.3%

Appendix

Table 14: Synthetic label accuracy (agreement with human majority vote) across raw sample estimates and population re-weighted strata (

\mathcal{D}_{k}

). Post-stratification shifts headline accuracy by less than 1.0 pt , confirming sample representativeness within a

\pm 3.7\text{pt}

standard error.

axis	subset	unfiltered ( $\mathcal{D}_{0}$ )	1 filter ( $\mathcal{D}_{1}$ )	2 filters ( $\mathcal{D}_{2}$ )	gain	retention
By criterion
	friendliness	90% (45/50)	93% (42/45)	93% (39/42)	$+3$ pt	84%
	task resolution	68% (34/50)	77% (30/39)	78% (28/36)	$+10$ pt	72%
	communication clarity	62% (31/50)	80% (24/30)	81% (21/26)	$+19$ pt	52%
By domain
	airline	75% (27/36)	92% (23/25)	92% (22/24)	$+17$ pt	67%

Appendix

Table 15: Agreement between synthetic labels

\star

and human majority vote across filter depths

\mathcal{D}_{k}

and dataset subsets. Gain reflects the accuracy change from two filters (

\mathcal{D}_{2}

) versus zero filters (

\mathcal{D}_{0}

); retention represents the proportion of target pairs retained after two filtering stages.

domain	policy words	agent tools	user tools	sims	median msgs	median tool calls	tool-error sims	mean reward
airline	1,313	14	0	440	18	5	21%	0.434
banking	926	16	2	855	30	7	3%	0.037
retail	1,158	16	0	980	22	6	34%	0.353
telecom	3,715	13	30	1,072	48	10	48%	0.243

Appendix

Table 16: Domain complexity metrics across evaluation environments. Tool name overlap between domain pairs ranges from 0.03 to 0.11 by Jaccard index; banking represents the only domain featuring active retrieval.

human panel	pairs removed	label was right	purity	yield
1	6	4	86.0 $\rightarrow$ 88.2% ( $+2.3$ pt)	$-8$ pt
2	4	4	82.5 $\rightarrow$ 81.1% ( $-1.3$ pt)	$-5$ pt
pooled	10	8	84.2 $\rightarrow$ 84.6% ( $\mathbf{+0.4}$ pt)	$\mathbf{-7}$ pt

Appendix

Table 17: Trade-off analysis for the second-stage filter

J_{2}

. Among the 10 pairs eliminated by stage two, human majority vote supports the original ground truth label in 8 cases.

evaluator	its own pairs	the other pairs	gap (pt)
	accuracy (%)
gpt-oss-120b	86.5 ( $n=334$ )	77.4 ( $n=666$ )	$\mathbf{+9.1}$
gpt-oss-20b	68.3 ( $n=325$ )	68.9 ( $n=675$ )	$-0.6$
nemotron-lightning-3.5	52.1 ( $n=341$ )	67.8 ( $n=659$ )	$\mathbf{-15.7}$

Appendix

Table 19: Contamination diagnostic comparing evaluator accuracy on self-generated versus external trajectory pairs. Values represent means across three independent evaluation runs.

Pipeline Stage	API Calls	Tokens	Token Share	Uncached Cost	Effective Cost
1. Steering Generation	1,125	2.56M	0.3%	$0.85	$0.67
2. Trajectory Generation	77,980	709.4M	95.0%	$51.16	$18.36
3. Filter Judges ( $K=2$ )	2,879	34.8M	4.7%	$4.42	$2.27
Total Pipeline	81,984	746.7M	100.0%	$56.43	$23.63

Appendix

Table 20: Computational overhead and curation cost. All costs are computed or estimated with the model service provider’s token-based pricing model.

model	prompt tokens	cached	rate
nemotron-lightning-3.5	350,084,755	312,862,256	89.4%
gpt-oss-20b	137,504,779	125,776,651	91.5%
qwen3-30b-a3b-instruct	107,519,458	102,523,608	95.4%
gpt-oss-120b	98,571,871	86,652,136	87.9%
total	693,680,863	627,814,651	90.5%

Appendix

Table 21: Prompt cache hit rates across all 77,980 trajectory generation calls.

asset	role in this work	license	terms of use
Benchmark substrate and serving infrastructure
$\tau^{3}$ -bench, sierra-research/ tau2-bench [ 20 , 21 , 22 ]	Task definitions, domain environments, tool sets, agent framework, and user simulator	MIT License	Free commercial and non-commercial use, modification, and redistribution provided the MIT notice (Copyright (c) 2025 Sierra Research) is preserved; evaluation runs are additionally subject to third-party LLM API terms
Fireworks AI serverless inference	Hosted inference for every open-weight model in the pipeline and the sweep	not applicable (service)	Fireworks AI terms of service
Open-weight models
gpt-oss-20b , gpt-oss-120b	Trajectory generation; 120b also generates steering instructions and serves as filter judge $J_{2}$ ; both are evaluators	Apache 2.0	model license and Fireworks AI terms of service
nemotron-lightning-3.5	Filter judge $J_{1}$ , trajectory generation, and evaluator	NVIDIA OpenMDW-1.1	as above

Appendix

Table 22: Existing assets used in this work. Model roles refer to the pipeline stages of Table 7 and the evaluator sweep of Section 5.3 .

Explore similar work

Jun 19, 2026cs.AI

Counsel: A Meta-Evaluation Dataset for Agentic Tasks

As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such as LLM-as-a-judge (LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on two agent benchmarks: tau-bench (customer support agents) and DA-Code (coding agents), and human meta-evaluations of these critiques. Human annotators label critiques on each flagged error as "spot on", "correct location but poor reasoning", or "should not have flagged", achieving reliable inter-annotator agreement (Krippendorff's alpha of 0.78). The resulting dataset stratifies LLMJ critiques by human alignment across both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated using open-weight models and is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators for agentic systems.

Sashank Pisupati, Henry Broomfield, Eujeong Choi +5

1Atla AI · 2Cohere AI · 3Mistral AI +1

Agentic Benchmarks Llm-As-A-Judge Human Annotators Counseling

Ask PDF

May 29, 2026cs.CL

Preference-Aware Rubric Learning for Personalized Evaluation

As Large Language Models (LLMs) evolve from general-purpose assistants to user-centric agents, personalization has become central to aligning model behavior with individual preferences, making the evaluation of personalized alignment a critical bottleneck. Existing evaluation methods-ranging from automatic metrics to LLM-as-a-judge approaches-fail to capture subjective, user-specific preferences embedded in long-term interaction histories. We identify three essential principles for reliable and effective personalized evaluation: Representativeness, User-Consistency, and Discriminativeness. To address these principles, we introduce Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment. Under this paradigm, we propose PARL (Preference-Aware Rubric Learning for Personalized Evaluation), a framework that learns to induce preference-aware evaluation rubrics directly from raw user histories and performs a self-validation mechanism to ensure consistency with the user's preferences. PARL integrates rubric induction with a discriminative reinforcement learning objective that contrasts user-authored responses against competitive personalized model outputs, enabling the learned rubrics to capture precise, user-specific decision boundaries. Experiments on real-world personalized text generation tasks show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users and tasks, while capturing stable stylistic preferences and fine-grained evaluative patterns. To ensure reproducibility, our code is available at https://github.com/SnowCharmQ/PARL.

Yilun Qiu, Xiaoyan Zhao, Yang Zhang +7

National University of Singapore · Xiaohongshu Inc. · The University of Tokyo

Large Language Model Personalization Large Language Model Evaluation

Ask PDF

May 15, 2026cs.LG

Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences

In many applications, human and LLM evaluators use assessments of relevant criteria to create an overall evaluation for an item or individual. For example, in admissions, committees assess candidates on attributes such as test scores, GPA, and research experience to evaluate their overall fit for the program. Another example arises in medical care where clinicians use patient reports of symptoms to consider preliminary diagnoses and assess risks. Each setting involves mapping multiple criteria to an overall evaluation -- a process that reflects the evaluator's underlying preferences. We focus on the fundamental question of learning these preferences. Many applications of this problem make specific modeling assumptions on evaluator preferences that may be substantially violated in the real world. We make the minimal assumption that the preference function is coordinate-wise non-decreasing, which is reasonable in a large number of evaluation settings. We theoretically characterize the severity of model mismatch for many common assumptions and show that it can lead to significant issues for learning evaluator preferences and other important downstream tasks. We then present an algorithm for learning evaluators' preferences that is robust to model mismatch. We prove theoretically that our algorithm can learn any preference function without sacrificing performance when the linearity assumption holds. Evaluations of our algorithm with synthetic simulations and real-world data confirm its ability to learn preferences robustly and illustrate key aspects of LLM and human preferences.

Madeline Celi Kitch, Nihar B. Shah

Carnegie Mellon University

Preference Alignment Learning Rater Mismatch Reliable

Ask PDF