Distilling Expert Judgment at Scale

Robbie Goldfarb; Andrew B. Hall; Jillian Fisher; Kathryn Salam; Matt Wilde

cs.AIOct 23, 2026

Distilling Expert Judgment at Scale

Authors: Robbie Goldfarb, Andrew B. Hall, Jillian Fisher, Kathryn Salam, Matt Wilde

Organizations: Forum AI · Stanford University · University of Washington

Abstract

Frontier AI models are increasingly deployed on topics where the stakes are high and expert judgment is required, including healthcare, defense, finance, governance, and politics. We present a general methodology for distilling expert human judgment into automated evaluation systems that operate at scale. Our approach assembles networks of domain experts and develops techniques for extracting not just their conclusions but their reasoning processes, encoding those processes into structured editorial standards that automated judges can apply continuously. We ground our approach in a long tradition of research on the nature of expertise and the conditions under which expert judgment is reliable and more effective than other approaches, and argue that independent, expert-calibrated evaluation is necessary not only on epistemic grounds but as a matter of accountability and public trust. We validate the methodology on AI news reporting as a first application domain. On source quality, two independent domain experts achieve Krippendorff's $α= 0.83/0.69$ on a held-out test set when applying the editorial standards, and our automated judge reaches $F1 = 0.81/0.97$ against expert consensus. On neutrality, experts achieve $α= .72$ on the test set, and the automated judge reaches $F1 = 0.86$ against expert consensus. Additionally, a controlled comparison shows that expert-calibrated judges substantially outperform uncalibrated frontier models on every source-quality metric, demonstrating that encoding expert reasoning is the critical step that prompt engineering alone cannot replace. We present this benchmark as a proof of concept for a broader evaluation infrastructure applicable to any domain where judgment matters and expertise is scarce.

Figures & tables

Dimension	Description	Categories in Our Benchmark
Domains	Substantive policy area, ensuring breadth across major political and geopolitical issue spaces.	Government & Politics; Economic Policy; Social Policy; Environmental & Energy; Technology & Society; Foreign Policy & Security; Transnational Issues; Conflict and War; Global Economy; + customer- or product-specific tasks
Task types	Functional form of the user request, capturing how the model is being asked to respond.	Informational queries; Perspective role-play; Policy deliverables; Charged framings; + customer- or product-specific tasks
Complexity	Cognitive and normative difficulty, from objective fact recall to value-laden reasoning.	Factual; Interpretive; Analytical; Normative; Strategic

Table 1: Coverage dimensions for prompt characterization.

Standard	Validation	Test $\alpha$
	$\alpha$ [95% CI]	$\alpha$ [95% CI]
Source type (categ.)	.82 [.76, .87]	.83 [.76, .89]
State-controlled (bin.)	.93 [.80, 1.00]	.69 [.52, .83]

Table 2: Source Quality: Inter-expert agreement (Human–Human). Krippendorff’s

\alpha

with item-bootstrap 95% CI on both the validation and test set.

Source type (categ.)
Standard	$n$	Prec.	Rec.	F1
macro avg	146	.74	.80	.75
weighted avg	146	.86	.79	.81
State-controlled (bin.)
macro avg	147	.98	.92	.95
weighted avg	147	.97	.97	.97

Table 3: Source Quality: Precision, recall, and F1 of the automated AI judge against the adjudicated human consensus on the test set. For Source, macro avg is the unweighted mean across the six tiers; weighted avg is support-weighted.

Agreement metrics (higher better)
Layer	Claude+Expert	Claude	Codex
Source ord. ( $\rho$ )	.76 [.67,.85]	.27 [.14,.39]	.59 [.49,.67]
Bad-source ( $\kappa$ )	.79 [.70,.86]	.03 [-.09,.14]	.30 [.17,.42]
Resp. agr. ( $\rho$ )	.37 [.26,.48]	.14 [.02,.26]	.16 [.04,.27]
Error metric (lower better)
Resp. err. (MAE)	12.8 [11.4,14.2]	28.9 [26.7,31.1]	27.6 [25.5,29.8]

Table 4: Expert-guided AI judges vs. uncalibrated AI judges. Mean metric with bootstrap 95% confidence intervals in brackets with

2K

samples. Best value is bolded; lower is better for MAE.

Standard	Validation	Test
	$\alpha$ [95% CI]	$\alpha$ [95% CI]
Neutrality (bin.)	.79 [.66,.90]	.72 [.60, .83]

Table 5: Neutrality: Inter-expert agreement (Human–Human). Krippendorff’s

\alpha

with item-bootstrap 95% CI on both the validation and test set.

Standard	Validation	Test
	$\alpha$ [95% CI]	$\alpha$ [95% CI]
Political Lean (categ.)	.68 [.55,.80]	.65 [.51, .78]

Table 6: Political Lean: Inter-expert agreement (Human–Human). Krippendorff’s

\alpha

with item-bootstrap 95% CI on both the validation and test set.

Standard	$n$	Prec.	Rec.	F1
Neutrality (bin.)
macro avg	159	.85	.85	.85
weighted avg	159	.86	.86	.86

Table 7: Neutrality: Precision, recall, and F1 of the automated AI judge against the adjudicated human consensus on the test set.

Class	$n$	Prec.	Rec.	F1
Political Lean (categ.)
macro avg	99	.72	.76	.73
weighted avg	99	.81	.78	.79

Table 8: Political Lean: Precision, recall, and F1 of the automated AI judge against the adjudicated human consensus on the test set.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

Oct 23, 2026

NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting

As frontier AI systems increasingly serve as sources of political and current-events information, evaluating the epistemic quality of their responses is now especially consequential. Existing evaluations often focus narrowly on ideological alignment, rely on measures not validated for this setting, or depend on reference labels derived from public opinion or non-expert annotation, potentially producing misleading or miscalibrated results. To address these limitations, we introduce NewsBench, an expert-grounded benchmark that reframes evaluation around editorial standards developed by senior practitioners in journalism, policy, and intelligence analysis. NewsBench operationalizes quality across three dimensions, source quality, factuality, and neutrality, using a two-phase pipeline in which domain experts first define and calibrate evaluation guidelines, which are then transferred to automated AI judges validated against expert-adjudicated gold labels. The benchmark includes approximately

3,000

evaluation prompts, with roughly

500

refreshed monthly to capture emerging current events, roughly

2,500

human expert-annotated items for calibration and validation, and a suite of AI judges designed to enable scalable evaluation. To validate our methodology, we report both inter-rater reliability among human experts and AI judge--human agreement across all three dimensions. Finally, we apply NewsBench to four frontier models, revealing distinct and non-overlapping failure profiles in sourcing, factuality, and neutrality.

Matt Wilde, Jillian Fisher, Andrew B. Hall +2

Forum AI · University of Washington · Stanford University

Generative AI Evaluation Factual Consistency Evaluation LLM-as-a-Judge

Ask PDF

Jul 22, 2026

Frontier Financial Judgement: Can agents tell what might move a stock?

We introduce Frontier Financial Judgement, a challenging new benchmark developed in collaboration with professional equity analysts to assess agents' ability to replicate expert human judgements. Rapidly identifying new information, evaluating its implications and determining its valuation impact is one of the most time-consuming and challenging aspects of real-world equity coverage. This is becoming ever more difficult and important as AI rapidly increases the quantity of new information to process. The strongest agent we evaluate on Frontier Financial Judgement matches all expert labels in only 52.4% of cases. We also find significant divergence in estimated false-positive rates among frontier agents, ranging from ~1% for GPT-5.6 Sol to ~32% for Claude Sonnet 4.6. To construct the benchmark and make it representative of real-world settings, we combine human-designed and labelled synthetic articles with live news articles and historical documents, creating 656 items for assessment. The resulting task requires agents to distinguish genuinely new, valuation-relevant financial information from stale, immaterial or misleading news under realistic conditions. We find substantial trade-offs among agent accuracy, cost, false positives and reliability that continue to hinder the reliable deployment of news-flow filtering in practice.

Joshua Harris

Sonto

AI Agent Evaluation AI Agent Benchmarks Finance LLM Agent Reliability

Ask PDF

Apr 15, 2026

AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals

Much of the focus in RL today is on evaluation design: building meaningful evals that serve simultaneously as benchmarks and as well-defined reward signals for post-training. Yet, many real-world tasks are governed by subjective, procedural, and domain-specific requirements that are difficult to encode as exact-match targets or open-ended preference judgments frequently used in RL pipelines today. In this work, we present AsymmetryZero, a framework for operationalizing human expert preferences as semantic evals. AsymmetryZero represents each task as a stable evaluation contract that makes grading criteria explicit: what is being graded, how each criterion is judged, and how criterion-level decisions are aggregated into a task outcome. The same contract can be executed using Inspect for model-only evaluations, as well as the Harbor Framework for agentic evaluations, enabling comparable scores and shared audit artifacts across both settings. We argue that the central challenge in post-training today is the faithful encoding of expert requirements into the evaluation itself. To that end, we present a study using Harbor that holds task contracts fixed and compares a five-model frontier jury against a five-model compact jury across four frontier-class solvers (Claude Opus 4.6, GPT-5.4, Grok-4.20, Gemini-3.1-Pro). We find that criterion-level frontier-vs-compact agreement ranges from

75.9\%

89.6\%

(strict common-subset agreement:

77.8\%

92.1\%

), while compact juries exhibit substantially higher internal dissent (3--2 split rate

28.7\%

32.4\%

) than frontier juries (

6.1\%

11.5\%

). Verifier traces further show that compact juries reduce per-criterion judging cost to roughly

4.2\%

5.6\%

of frontier and latency to roughly

21.7\%

27.1\%

, even as aggregated task-level outcomes often remain comparatively stable.

Tadhg Looram, Lucas Nuzzi, Kyle Waters +1

PortexAI · Stanford University

Automated Evaluation Human Preference Evaluation AI Agent Evaluation Rubric-Based Evaluation LLM-as-a-Judge

Ask PDF

Abstract

Automated Evaluation

Explore similar work

NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting

Frontier Financial Judgement: Can agents tell what might move a stock?

AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals