DREAM: Deep Research Evaluation with Agentic Metrics

Elad Ben Avraham; Changhao Li; Ron Dorfman; Roy Ganz; Oren Nuriel; Amir Dudai; Aviad Aberdam; Noah Flynn

cs.AIFeb 21, 2026

DREAM: Deep Research Evaluation with Agentic Metrics

Authors: Elad Ben Avraham, Changhao Li, Ron Dorfman, Roy Ganz, Oren Nuriel, Amir Dudai, Aviad Aberdam, Noah Flynn,

Organizations: AWS Agentic AI · Georgia Institute of Technology

Abstract

Deep Research Agents generate analyst-grade reports, yet evaluating them remains challenging due to the absence of a single ground truth and the multidimensional nature of research quality. Recent benchmarks propose distinct methodologies, yet they suffer from the Mirage of Synthesis, where strong surface-level fluency and citation alignment can obscure underlying factual and reasoning defects. We characterize this gap by introducing a taxonomy across four verticals that exposes a critical capability mismatch: static evaluators inherently lack the tool-use capabilities required to assess temporal validity and factual correctness. To address this, we propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that instantiates the principle of capability parity by making evaluation itself agentic. DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent, enabling temporally aware coverage, grounded verification, and systematic reasoning probes. Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks, offering a scalable, reference-free evaluation paradigm.

Figures & tables

Figure 1: Capturing Overlooked Dimensions of Research Quality. DREAM actively verifies the reasoning of generated reports by probing external sources (left), detects factual errors injected in a controlled experiment (middle), and captures time-sensitive validity gaps by penalizing outdated reports (right).

Benchmark	Vertical	Metric Name	Creation	Execution
DeepResearchGym	Presentation Quality	Clarity	N/A	LLM
Coelho et al. (2025)	Task Compliance	Report Relevance	LLM + Human	LLM
	Analytical Depth	Insightfulness	N/A	LLM
	Source Quality	Retrieval Faithfulness	N/A	Workflow
DeepResearch Bench	Presentation Quality	RACE (Presentation Quality)	LLM	LLM
Du et al. (2025)	Task Compliance	RACE (Inst., Comp.)	LLM	LLM

Table 1: Taxonomic breakdown of deep research evaluation benchmarks. Metrics from existing benchmarks mapped to our four-dimensional taxonomy, alongside their creation and execution methodologies.

Figure 2: DREAM Overview. Our framework operates in two phases. Left: Protocol Creation, where query-independent Static Metrics are combined with Adaptive Metrics constructed by an agent equipped with web search tools and optional tools to access external data. Right: Protocol Execution, where each metric is routed to the appropriate evaluator, either an LLM, agent with tool access, or workflow.

Figure 3: DREAM Protocol Execution Evaluators. (a) LLM Evaluator assesses writing quality (WQ) and key-information coverage (KIC); (b) Agent Evaluator evaluates reasoning quality (RQ) using external tools; (c) Workflow Evaluator performs factuality assessment via evidence retrieval, citation integrity (CI) verification through claim-source validation, and domain authoritativeness (DA) scoring via credibility assessment of extracted citations.

Metric	Method	Rel.	Verif.	Clar.	Valid.	Avg.
KIC	LLM	0.84	0.73	0.80	–	0.79
	Agent	0.88	0.75	0.87	–	0.83
	Agent + Ret.	0.94	0.91	0.92	–	0.92
RQ	LLM	0.82	0.51	0.78	0.70	0.70
	Agent	0.89	0.67	0.88	0.92	0.84
	Agent + Ret.	0.94	0.80	0.97	0.99	0.93

Table 2: Human evaluation of KIC and RQ items. Performance by metric and method. The agent with retrieval achieves the best performance across all axes.

Figure 4: Temporal Awareness in KIC Evaluation. Comparison of evaluation criteria for a TikTok legal status query, showing DeepResearch Bench’s static criteria (left) versus DREAM’s KIC criteria (right) that incorporate time-sensitive facts (e.g., mid-December 2025 joint venture deal and January 23, 2026 deadline).

Figure 5: Reasoning flaws detection. Relative score degradations between well-reasoned and malformed reports. DREAM–RQ centers around $40.1\%$ degradation, while RACE centers around $9.1\%$ , with several malformed reports outscoring well-reasoned ones.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Agentic Taxonomy Sub-Dimension Clustering Visualization. Each point represents an evaluation criterion from one of four deep research benchmarks (shape), colored by its LLM-assigned taxonomy dimension. Criteria assigned to the same dimension tend to form localized regions in embedding space, suggesting coherent semantic groupings across the four verticals.

Dimension	Sub-dimension (weight)	Evaluation Prompt (Exact Description)
Ideas and Content	Main Idea Clarity (0.25)	This dimension assesses the clarity and specificity of the main idea expressed in the section summary. A high-quality section will present a focused and well-articulated central idea that is tightly aligned with the report question. Summaries lacking precision, or that simply list general topics without insight or framing, should be penalized. Do not give high scores if the main idea is vague, overgeneralized, or merely implied.
	Detail Relevance (0.25)	This dimension focuses on how well the supporting details in the summary reinforce the main idea. Bullet points should be specific, relevant, and purposefully selected. Low-quality summaries may include off-topic, overly generic, or redundant details that do not support the section’s main message. Do not reward high scores based on the amount of content alone—focus on alignment and purpose.
	Information Density (0.25)	This dimension measures the information richness of the section summary. High-density summaries use each bullet to convey important, non-obvious, and topic-specific content. Shallow summaries repeat known facts, use vague language, or include fluff. Length alone should not be rewarded—focus on content value per line.
	Conceptual Synthesis (0.25)	This dimension evaluates the structural and conceptual integration in the summary. Look for signs of synthesis such as: grouping related points, identifying contrasts, cause-effect relationships, or thematic framing. Poor summaries are unordered lists with no visible logic. Do not reward correctness alone—this dimension rewards insight, not just content.
Organization	Heading Structure (0.3)	This dimension assesses the use and clarity of headings in the section. High-quality summaries include headings that meaningfully segment the content, reflect topic hierarchy, and help orient the reader. Avoid rewarding default, generic, or misaligned headings. Headings should reflect actual conceptual boundaries.
	Bullet Grouping Logic (0.4)	This dimension evaluates the internal logic of bullet groupings. High-quality summaries group related points together according to thematic, temporal, causal, or hierarchical logic. Low-quality groupings mix unrelated ideas, interrupt flow, or reflect no discernible principle.

Appendix

Table 3: Writing Quality Rubric. The exact descriptions used as system prompts for the LLM Evaluator to score each sub-dimension.

ID	Topic	DRB-RACE Comp.			DRB-RACE Insight			DREAM-KIC
ID	Topic	Upd.	Jan-25	Jan-24	Upd.	Jan-25	Jan-24	Upd.	Jan-25	Jan-24
1	Current papal leadership & policies	48.29	53.80	55.75	48.55	52.12	55.23	64.00	0.00	0.00
2	DOGE agency authority & funding	54.17	40.46	23.73	56.44	33.56	17.62	73.33	13.33	0.00
3	US-China LLM competition (DeepSeek)	50.62	47.10	44.75	50.14	45.90	39.76	84.38	56.13	31.25
4	US-Venezuela military tensions	46.96	47.07	49.15	44.38	44.32	47.28	46.88	6.12	0.00
5	Sudan civil war status	48.32	48.72	0.00	46.21	48.46	0.00	81.67	52.00	0.00

Appendix

Table 4: Temporal sensitivity analysis. Comparison of DRB-RACE and DREAM-KIC scores across current (Upd.) and simulated outdated knowledge cutoffs (Jan 2025, Jan 2024) for the DRA. Unlike baselines, DREAM-KIC scores degrade on reports generated with older knowledge cutoffs, effectively capturing information lag.

#	RACE Criteria	DREAM KIC Checklist Items
	Comprehensiveness
1	Identification and Explanation of Core Legislation : Assesses if the article clearly identifies the specific U.S. law mandating the potential ban (e.g., the ‘Protecting Americans from Foreign Adversary Controlled Applications Act’) and explains its key provisions. This is the foundational context for the entire update.	Does the report mention the Protecting Americans from Foreign Adversary Controlled Applications Act (PAFACA) as the specific law requiring ByteDance to divest TikTok?
2	Detailed Breakdown of the Divestiture Deadline and Extensions : Evaluates if the article provides the exact initial deadline for divestiture and, crucially, clarifies the conditions and duration of any potential presidential extensions. This directly addresses the task’s specific requirement to clarify the deadline’s ‘proximity.’	Does the report state that the current divestiture deadline is January 23, 2026?
3	Coverage of Ongoing Legal Challenges : Checks if the article discusses the lawsuits filed by TikTok/ByteDance or other parties to challenge the law. This is essential for a comprehensive understanding of the current ‘legal status,’ as these proceedings could impact the deadline and enforcement.	Does the report accurately calculate and convey that approximately 31 days or one month remains until the deadline from late December 2025?
4	Analysis of Key Stakeholder Positions : Assesses whether the report covers the stances and actions of key stakeholders, including the U.S. Executive and Legislative branches, ByteDance, and the Chinese government (regarding its approval of tech exports). This provides a complete picture of the geopolitical and corporate landscape.	Does the report mention that ByteDance signed a deal in mid-December 2025 to create a U.S. joint venture, with closing scheduled for January 22, 2026?
5	Explanation of the Ban’s Enforcement Mechanism : Evaluates if the article explains how the ban would be implemented—specifically, the nature of the nationwide app store ban and its implications for app distribution and hosting services. This clarifies the practical consequences of non-compliance.	Does the report specify the ownership structure of the joint venture deal, including that ByteDance will retain 19.9% and U.S. investors will hold majority control?

Appendix

Table 5: Task Alignment Criteria Comparison. DeepResearch Bench RACE rubric criteria alongside DREAM KIC Checklist items for the query: “Prepare a regulatory compliance update on the legal status of TikTok in the U.S. Specifically, clarify the proximity of the deadline for ByteDance to divest its U.S. operations to avoid a nationwide app store ban.”

ID	Query
1	How will a 10% price cut affect unit sales next month for an e-commerce product line?
2	What assumptions about inflation, interest rates, and currency stability most influence predictions of stock market volatility in emerging economies over the next five years?
3	How will the adoption of renewable energy technologies impact global oil demand over the next decade?
4	What assumptions about hospital staffing, patient flow, and technology adoption most influence predictions of emergency department wait times over the next five years?
5	How will AI-powered recommendation engines affect consumer purchase diversity in online retail markets over the next two years?
6	What assumptions about monetary policy, housing supply, and demographic trends most influence predictions of urban housing prices over the next decade?

Appendix

Table 6: Research queries for the “Reasoning Flaws Detection” experiment in Section 4.3 .

ID	Topic	Variant	Claim	URL Domain
1	15-Min Cities	T	15-minute cities are an urban planning concept to place essential services within walking distance; they do not restrict movement.	fullfact
1	15-Min Cities	F	The ‘15-minute city’ is a lockdown plan to restrict residents’ movement and fine them for leaving their zones.	spiked-online
2	GMO Safety	T	Commercially available GMO foods are safe for consumption and do not pose health risks.	natl-academies
2	GMO Safety	F	There is no scientific consensus on GMO safety; they pose potential long-term risks to health and biodiversity.	ensser
3	Climate Change	T	Rapid warming since the mid-20th century is primarily driven by human activities, specifically fossil fuel emissions.	nasa
3	Climate Change	F	Climate forecasting has fundamental uncertainties regarding natural variability; warming may not be as catastrophic as models predict.	judithcurry

Appendix

Table 7: Adversarial Factuality Dataset. The 15 adversarial claim pairs used to measure factual sensitivity. Ground Truth ( T ) variants are backed by reliable evidence, while False ( F ) variants are paired with specious citations that satisfy text-alignment criteria but contradict objective reality.

Dimension	Scoring Guidelines
Relevance	Directly addresses the specific requirements and key aspects of the research query.
Verifiability	Can be objectively confirmed using accessible external evidence or well-defined logic.
Clarity	Is formulated precisely and unambiguously to ensure consistent interpretation.
Validation	Is methodologically correct, free of circular reasoning, and capable of rigorously confirming the answer.

Appendix

Table 8: Human evaluation rubric . Criteria used to guide expert assessment of created protocols.

Indicator	Kendall’s W	p-value
Relevance	0.58	.030
Verifiability	0.78	.009
Clarity	0.58	.030
Validation Soundness	0.86	.006
Average	0.69	.016

Appendix

Table 9: Agreement analysis among human annotators, measured using Kendall’s W. All results are statistically significant (

p<.05

		Static Metrics				Adaptive Metrics
Dataset	DRA	WQ	Fact.	CI	DA	KIC	RQ
DeepResearchBench	LangChain Open DR	63.69	46.02	14.43	86.32	64.99	53.34
	Smolagents Open DR	63.30	57.64	2.45	9.44	74.15	63.68
	Tongyi Deep Research	63.95	58.58	0.00	0.00	58.53	40.56
LiveResearchBench	LangChain Open DR	60.75	38.11	11.90	84.00	63.66	59.04
	Smolagents Open DR	61.71	56.90	9.67	24.41	73.55	68.82

Appendix

Table 10: DREAM evaluation. Static and adaptive metric scores for leading DRAs across three datasets.

Figure 7: Factuality Label Distribution. Distribution of factuality judgments (Full Support, Partial Support, Contradict, Unverifiable) for each model across three datasets.

Figure 8: Citation Faithfulness Label Distribution. Distribution of citation faithfulness labels assessing the alignment between claims and cited source text across three datasets.

Figure 9: Citation Integrity Components. Citation Faithfulness and Citation Attribution visualized for each model across three benchmarks.

Figure 10: Report length distributions across DRAs and datasets. Word count distributions across DRAs and datasets. Smolagents Open Deep Research consistently exhibits the highest variance and mean length, while LangChain Open Deep Research and Tongyi Deep Research are more concise and consistent.

Dataset	DRA	Mean $\pm$ Std	Min	Max
DeepResearchBench	LangChain Open DR	$1628.3\pm 398.3$	836	2650
	Smolagents Open DR	$3092.0\pm 1728.0$	1022	8511
	Tongyi Deep Research	$1431.0\pm 248.7$	670	2055
LiveResearchBench	LangChain Open DR	$1792.0\pm 459.9$	701	2784
	Smolagents Open DR	$3009.3\pm 1627.8$	352	9261
	Tongyi Deep Research	$1505.2\pm 350.9$	254	2602

Appendix

Table 11: Report length statistics across DRAs and datasets. Word counts are reported as Mean

\pm

Standard Deviation. Smolagents consistently produces the longest reports, while Tongyi Deep Research is the most concise.

		Static Metrics				Adaptive Metrics
Judge Model	DRA	WQ	Fact.	CI	DA	KIC	RQ
Claude Sonnet 4.5	LangChain Open DR	63.69	46.02	14.43	86.32	64.99	53.34
	Smolagents Open DR	63.30	57.64	2.45	9.44	74.15	63.68
	Tongyi Deep Research	63.95	58.58	0.00	0.00	58.53	40.56
DeepSeek V3.2	LangChain Open DR	66.19	47.35	17.24	86.14	63.86	67.68
	Smolagents Open DR	67.10	54.45	6.02	60.72	73.57	71.89

Appendix

Table 12: Backbone Model Comparison. Metric scores on DeepResearchBench across three judge backbones. Agent rankings remain largely consistent regardless of the judge model used.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 519 papers

Jan 13, 2026

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmarks often either under-evaluate a system's ability to produce meaningful insights and high-quality writing, or adopt coarse or LLM-defined criteria that are hard to verify and can diverge from human expert judgment. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRAs. It contains 132 grounded research tasks across 22 domains; for each task, an agent must produce a research report that is evaluated by a set of 9,430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are verifiable and aligned with human expert judgment. We evaluate several state-of-the-art deep-research agents on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRAs and human experts. We release the benchmark, evaluation scripts, and all rubrics at https://github.com/imlrz/DeepResearch-Bench-II to facilitate future research on deep-rearch agents.

Ruizhe Li, Mingxuan Du, Benfeng Xu +3

University of Science and Technology of China · Metastone Technology, Beijing, China

Deep Research Agents LLM Agent Evaluation AI Agent Benchmarks Rubric-Based Evaluation Automated Evaluation

Ask PDF

Apr 16, 2026

DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation

Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR

^{3}

-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR

^{3}

-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR

^{3}

-Agent based on multiple state-of-the-art language models demonstrate that DR

^{3}

-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.

Qianqian Xie, Qingheng Xiong, He Zhu +16

Nanjing University · M-A-P · Nanjing University of Science and Technology +2

LLM Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks

Ask PDF

May 17, 2026

Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

Frontier deep research agents (DRAs) are being deployed in enterprise workflows faster than they are being evaluated. Existing benchmarks measure factual recall, single-hop QA, or generic agentic skill, and miss the multi-document, decision-grade deliverables DRAs are asked to produce. We introduce a benchmark of 70 SME-authored management consulting prompts, each embedding cognitive traps that penalize surface-pattern reasoning. Three frontier agents, namely Claude Opus~~4.6, OpenAI o3-deep-research and Gemini~~3.1Pro deep-research, are scored on two complementary layers: deterministic binary verifiers (mean 14.9 per task) and a five-criterion 0--3 SME rubric (Data Integrity, Analytical Rigor, Relevance & Focus, Execution Precision, Format & Deliverability), combined into a Verifier-Rubric Score (VRS, 0--100). Acceptance under a joint threshold (rubric mean $\geq 2.5$ and verifier pass rate $\geq 80\%$ ) is uniformly low: o3 15.7%, Claude 12.9%, Gemini 12.9%. Pairwise differences are statistically indistinguishable. On the continuous VRS, o3 leads (61.4[CI: 55.2,,67.5]), followed by Gemini (52.6) and Claude (38.5); the o3--Claude gap (

Δ{=}22.9

p{<}0.001

) survives Bonferroni correction. No agent averages above the rubric's ``adequate'' threshold of 2.0; no agent's mean verifier pass rate reaches the 80% acceptance floor. Each agent fails distinctively: Claude leads on data fabrication and file-access failures; o3 propagates cascading computation errors; Gemini oscillates between the highest perfect-verifier rate and the most catastrophic collapses. The benchmark, evaluation code, and full prompt corpus are publicly released.

Tanmay Asthana, Aman Saksena, Divyansh Sahu

Deccan AI Research

AI Agent Reliability Deep Research Agents LLM Agent Evaluation LLM Agent Verification AI Agent Benchmarks Agentic Reasoning+1

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

DR3^{3}3-Eval: Towards Realistic and Reproducible Deep Research Evaluation

Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation