cs.LGApr 15, 2026

AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals

Authors: Tadhg Looram, Lucas Nuzzi, Kyle Waters, Steven Dillmann

Organizations: PortexAI · Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Much of the focus in RL today is on evaluation design: building meaningful evals that serve simultaneously as benchmarks and as well-defined reward signals for post-training. Yet, many real-world tasks are governed by subjective, procedural, and domain-specific requirements that are difficult to encode as exact-match targets or open-ended preference judgments frequently used in RL pipelines today. In this work, we present AsymmetryZero, a framework for operationalizing human expert preferences as semantic evals. AsymmetryZero represents each task as a stable evaluation contract that makes grading criteria explicit: what is being graded, how each criterion is judged, and how criterion-level decisions are aggregated into a task outcome. The same contract can be executed using Inspect for model-only evaluations, as well as the Harbor Framework for agentic evaluations, enabling comparable scores and shared audit artifacts across both settings. We argue that the central challenge in post-training today is the faithful encoding of expert requirements into the evaluation itself. To that end, we present a study using Harbor that holds task contracts fixed and compares a five-model frontier jury against a five-model compact jury across four frontier-class solvers (Claude Opus 4.6, GPT-5.4, Grok-4.20, Gemini-3.1-Pro). We find that criterion-level frontier-vs-compact agreement ranges from 75.9%75.9\% to 89.6%89.6\% (strict common-subset agreement: 77.8%77.8\% to 92.1%92.1\%), while compact juries exhibit substantially higher internal dissent (3--2 split rate 28.7%28.7\%--32.4%32.4\%) than frontier juries (6.1%6.1\%--11.5%11.5\%). Verifier traces further show that compact juries reduce per-criterion judging cost to roughly 4.2%4.2\%--5.6%5.6\% of frontier and latency to roughly 21.7%21.7\%--27.1%27.1\%, even as aggregated task-level outcomes often remain comparatively stable.

Figures & tables

Figure 1: A shallow ontology of LLM-as-a-Judge: judgment interface (rating vs. ranking), evaluation contract (free-form vs. rubric-based), judge architecture (single vs. ensemble), and robustness strategies (bias auditing, protocol design).
More in this topic

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

Explore similar work

CardsList
Mar 11, 2026

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

The paradigm of LLM-as-a-judge relies on a critical assumption, namely that high inter-evaluator agreement indicates reliable and objective evaluation. We present two complementary findings that challenge this assumption. First, we demonstrate that this consensus is frequently illusory. We identify and formalize Evaluation Illusion, a phenomenon where LLM judges generate sophisticated critiques yet anchor scores on shared surface heuristics rather than substantive quality. Through a large-scale study of 105,600 evaluation instances (32 LLMs ×\times 3 frontier judges ×\times 100 tasks ×\times 11 temperatures), we show that model-level agreement (Spearman ρ=0.99ρ= 0.99) masks fragile sample-level agreement (Pearson rˉ=0.72\bar{r} = 0.72; absolute agreement ICC =0.67= 0.67), that merely sharing rubric structure restores 62% of total agreement, and that high-quality outputs paradoxically receive the least consistent evaluations. Second, we demonstrate that dynamically generating evaluation rubrics grounded in domain knowledge produces more meaningful assessment. We introduce MERG (Metacognitive Enhanced Rubric Generation), a knowledge-driven rubric generation framework whose domain-selective effects confirm this. Agreement increases in codified domains (Education +22%, Academic +27%) where knowledge anchors evaluators on shared standards, while it decreases in subjective domains where genuine evaluative pluralism emerges. These findings suggest that evaluation rubrics should be dynamically enriched with expert knowledge rather than relying on generic criteria, with implications for reward modeling in RLAIF.

Mingyang Song, Mao Zheng, Chenning Xu

Large Language Model Department, Tencent, China

LLM-as-a-JudgeRubric-Based EvaluationBias in LLM-as-a-Judge
AskPDF
Feb 6, 2026

JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-as-a-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domain-grounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to HealthBench and DR.BENCH, covering medical and 10-domain professional evaluation settings. Code and data are available at https://github.com/smiling-world/JADE.

Lanbo Lin, Jiayao Liu, Tianyuan Yang +5

Alibaba International Digital Commerce Group · Zhejiang University · Peking University

LLM-as-a-JudgeAI Agent EvaluationRubric-Based EvaluationAutomated Agent Evaluation
AskPDF
Oct 23, 2026

Distilling Expert Judgment at Scale

Frontier AI models are increasingly deployed on topics where the stakes are high and expert judgment is required, including healthcare, defense, finance, governance, and politics. We present a general methodology for distilling expert human judgment into automated evaluation systems that operate at scale. Our approach assembles networks of domain experts and develops techniques for extracting not just their conclusions but their reasoning processes, encoding those processes into structured editorial standards that automated judges can apply continuously. We ground our approach in a long tradition of research on the nature of expertise and the conditions under which expert judgment is reliable and more effective than other approaches, and argue that independent, expert-calibrated evaluation is necessary not only on epistemic grounds but as a matter of accountability and public trust. We validate the methodology on AI news reporting as a first application domain. On source quality, two independent domain experts achieve Krippendorff's α=0.83/0.69α= 0.83/0.69 on a held-out test set when applying the editorial standards, and our automated judge reaches F1=0.81/0.97F1 = 0.81/0.97 against expert consensus. On neutrality, experts achieve α=.72α= .72 on the test set, and the automated judge reaches F1=0.86F1 = 0.86 against expert consensus. Additionally, a controlled comparison shows that expert-calibrated judges substantially outperform uncalibrated frontier models on every source-quality metric, demonstrating that encoding expert reasoning is the critical step that prompt engineering alone cannot replace. We present this benchmark as a proof of concept for a broader evaluation infrastructure applicable to any domain where judgment matters and expertise is scarce.

Robbie Goldfarb, Andrew B. Hall, Jillian Fisher +2

Forum AI · Stanford University · University of Washington

Automated EvaluationLLM-as-a-Judge
AskPDF