NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting

Matt Wilde; Jillian Fisher; Andrew B. Hall; Kathryn Salam; Robbie Goldfarb

cs.AIOct 23, 2026

NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting

Authors: Matt Wilde, Jillian Fisher, Andrew B. Hall, Kathryn Salam, Robbie Goldfarb

Organizations: Forum AI · University of Washington · Stanford University

Abstract

As frontier AI systems increasingly serve as sources of political and current-events information, evaluating the epistemic quality of their responses is now especially consequential. Existing evaluations often focus narrowly on ideological alignment, rely on measures not validated for this setting, or depend on reference labels derived from public opinion or non-expert annotation, potentially producing misleading or miscalibrated results. To address these limitations, we introduce NewsBench, an expert-grounded benchmark that reframes evaluation around editorial standards developed by senior practitioners in journalism, policy, and intelligence analysis. NewsBench operationalizes quality across three dimensions, source quality, factuality, and neutrality, using a two-phase pipeline in which domain experts first define and calibrate evaluation guidelines, which are then transferred to automated AI judges validated against expert-adjudicated gold labels. The benchmark includes approximately $3,000$ evaluation prompts, with roughly $500$ refreshed monthly to capture emerging current events, roughly $2,500$ human expert-annotated items for calibration and validation, and a suite of AI judges designed to enable scalable evaluation. To validate our methodology, we report both inter-rater reliability among human experts and AI judge--human agreement across all three dimensions. Finally, we apply NewsBench to four frontier models, revealing distinct and non-overlapping failure profiles in sourcing, factuality, and neutrality.

Figures & tables

Source	Total Number	Share
Real-World	1,500	47.8%
Expert-Generated	750	23.9%
Synthetic	885	28.2%
Total	3,135	100.0%

Table 1: Breakdown of NewsBench input sources in the final prompt set.

Component	Real-World	Expert-Generated	Synthetic	Total	Share
Evergreen	966	664	433	2,063	65.8%
Current Events	534	86	452	1,072	34.2%
Total	1,500	750	885	3,135	100.0%

Table 2: Breakdown of evergreen and current events components in the final NewsBench prompt set, by source type.

	Validation Set	Test Set
Standard	$\alpha$ [95% CI]	$\alpha$ [95% CI]
Source ( categorical )	.82 [.76, .87]	.83 [.76, .89]
State-Controlled ( binary )	.93 [.80, 1.00]	.69 [.52, .83]

Table 3: Human–human inter-rater reliability for the Source Quality standards, reported per set. The validation pairs were labeled during Phase 1 calibration; the test set was held out from both Phase 1 and 2. We report human-human reliability via Krippendorff’s

\alpha

, and 95% confidence intervals computed via item-bootstrap with

B=2,000

Source ( categorical )
Standard	Precision	Recall	F1
macro avg	.74	.80	.75
weighted avg	.86	.79	.81
State-Controlled ( binary )
macro avg	.98	.92	.95
weighted avg	.97	.97	.97

Table 4: Performance of the AI judge on Source Quality standards using the adjudicated human labels as gold labels. We report precision, recall, and F1 using a macro avg and weighted avg . For Source, macro avg is the unweighted mean across the six tiers; weighted avg is support-weighted.

	Validation Set	Test Set
Standard	$\alpha$ [95% CI]	$\alpha$ [95% CI]
Claim Verification ( binary )	.47 [.36, .58]	.39 [.26, .51]

Table 5: Human–human inter-rater reliability for Factuality, reported per set. The validation samples were labeled during Phase 1 calibration; the test set was held out from both Phase 1 and 2. We report human-human reliability via Krippendorff’s

\alpha

, and 95% confidence intervals computed via item-bootstrap with

B=2,000

Claim Verification ( binary )
Standard	Precision	Recall	F1
macro avg	.69	.75	.69
weighted avg	.80	.74	.75

Table 6: Performance of the AI judge on Factuality using the adjudicated human labels as gold labels. We report precision, recall, and F1 using a macro avg and weighted avg .

	Validation Set	Test Set
Standard	$\alpha$ [95% CI]	$\alpha$ [95% CI]
Neutrality ( binary )	.79 [.66, .90]	.72 [.60, .83]

Table 7: Human–human inter-rater reliability for Neutrality, reported per set. The validation pairs were labeled during Phase 1 calibration; the test set was held out from both Phase 1 and 2. We report human-human reliability via Krippendorff’s

\alpha

, and 95% confidence intervals computed via item-bootstrap with

B=2,000

Standard / Class	Precision	Recall	F1
Neutrality ( binary )
macro avg	.85	.85	.85
weighted avg	.86	.86	.86

Table 8: Performance of the AI judge on the Neutrality standard using the adjudicated human labels as gold, evaluated on the test set. We report precision, recall, and F1 using a macro avg and weighted avg .

	Validation Set	Test Set
Standard	$\alpha$ [95% CI]	$\alpha$ [95% CI]
Political Lean ( categorical )	.68 [.55, .80]	.65 [.51, .78]

Table 9: Human–human inter-rater reliability for Political Lean, reported per set. The validation samples were labeled during Phase 1 calibration; the test set was held out from both Phase 1 and 2. We report human-human reliability via Krippendorff’s

\alpha

, and 95% confidence intervals computed via item-bootstrap with

B=2,000

Political Lean ( categorical )
Standard / Class	Precision	Recall	F1
macro avg	.72	.76	.73
weighted avg	.81	.78	.79

Table 10: Performance of the AI judge on the Political Lean standard using the adjudicated human labels as gold, evaluated on the test set. We report precision, recall, and F1 using a macro avg and weighted avg .

Model	Neutrality Pass	Factual Pass	Avg Source-Quality Score	Directional Lean Ratio
Claude Opus 4.7	82.2%	58.9%	81.5	12.7x left
Gemini 3.1 Pro	83.0%	75.4%	78.8	4.9x left
ChatGPT (GPT-5.5)	74.8%	91.1%	74.3	15.5x left
Grok 4.3	70.1%	57.1%	73.7	4.2x right

Table 11: Model results on NewsBench. Neutrality Pass is the percentage of responses passing the neutrality judge; Factual Pass is the percentage of responses with no false verifiable claims; Avg Source-Quality Score is the mean 0–100 source-quality score (higher better); Directional Lean Ratio is the larger left/right direction among directional neutrality failures, excluding none/other cases.

Model	Responses with False Claim	# Claims / Response	False-Claim Rate
Claude Opus 4.7	41.1%	17.03	4.30%
Gemini 3.1 Pro	24.6%	11.47	3.07%
ChatGPT (GPT-5.5)	8.9%	13.86	0.85%
Grok 4.3	42.9%	19.00	3.62%

Table 12: Factuality by model. Responses with False Claim is the percentage of responses containing at least one false verifiable claim; # Claims / Response is the mean number of extracted verifiable claims per response; False-Claim Rate is the average response-level percentage of extracted claims labeled false.

Generative AI Evaluation

Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.

See all 309 papers

Distilling Expert Judgment at Scale
Oct 23, 2026Robbie Goldfarb, Andrew B. Hall, Jillian Fisher +2Automated Evaluation LLM-as-a-Judge
Forum AI · Stanford University · University of Washington
Evaluating Commercial AI Chatbots as News Intermediaries
May 21, 2026Mirac Suzgun, Emily Shen, Federico Bianchi +5LLM Evaluation Question Answering
1Stanford University · 2Independent Researcher · 3Together AI.
SourceBench: Can AI Answers Reference Quality Web Sources?
Feb 18, 2026Hexi Jin, Stephen Liu, Yuheng Li +2LLM Evaluation IR Evaluation
University of California, San Diego · GenseeAI Inc.

Oct 23, 2026J/K move · Enter open · S save

Distilling Expert Judgment at Scale

Robbie Goldfarb, Andrew B. Hall, Jillian Fisher, Kathryn Salam+1

Forum AI · Stanford University · University of Washington

Automated Evaluation LLM-as-a-Judge

Open paper AskPDF arXiv

Frontier AI models are increasingly deployed on topics where the stakes are high and expert judgment is required, including healthcare, defense, finance, governance, and politics. We present a general methodology for distilling expert human judgment into automated evaluation systems that operate at scale. Our approach assembles networks of domain experts and develops techniques for extracting not just their conclusions but their reasoning processes, encoding those processes into structured editorial standards that automated judges can apply continuously. We ground our approach in a long tradition of research on the nature of expertise and the conditions under which expert judgment is reliable and more effective than other approaches, and argue that independent, expert-calibrated evaluation is necessary not only on epistemic grounds but as a matter of accountability and public trust. We validate the methodology on AI news reporting as a first application domain. On source quality, two independent domain experts achieve Krippendorff's

α= 0.83/0.69

on a held-out test set when applying the editorial standards, and our automated judge reaches

F1 = 0.81/0.97

against expert consensus. On neutrality, experts achieve

α= .72

on the test set, and the automated judge reaches

F1 = 0.86

against expert consensus. Additionally, a controlled comparison shows that expert-calibrated judges substantially outperform uncalibrated frontier models on every source-quality metric, demonstrating that encoding expert reasoning is the critical step that prompt engineering alone cannot replace. We present this benchmark as a proof of concept for a broader evaluation infrastructure applicable to any domain where judgment matters and expertise is scarce.

Abstract

Generative AI Evaluation

Explore similar work

Distilling Expert Judgment at Scale

Evaluating Commercial AI Chatbots as News Intermediaries

SourceBench: Can AI Answers Reference Quality Web Sources?