cs.AIFeb 21, 2026
SaveDREAM: Deep Research Evaluation with Agentic Metrics
Organizations: AWS Agentic AI · Georgia Institute of Technology
Abstract
Deep Research Agents generate analyst-grade reports, yet evaluating them remains challenging due to the absence of a single ground truth and the multidimensional nature of research quality. Recent benchmarks propose distinct methodologies, yet they suffer from the Mirage of Synthesis, where strong surface-level fluency and citation alignment can obscure underlying factual and reasoning defects. We characterize this gap by introducing a taxonomy across four verticals that exposes a critical capability mismatch: static evaluators inherently lack the tool-use capabilities required to assess temporal validity and factual correctness. To address this, we propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that instantiates the principle of capability parity by making evaluation itself agentic. DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent, enabling temporally aware coverage, grounded verification, and systematic reasoning probes. Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks, offering a scalable, reference-free evaluation paradigm.
Figures & tables
Figure 1: Capturing Overlooked Dimensions of Research Quality. DREAM actively verifies the reasoning of generated reports by probing external sources (left), detects factual errors injected in a controlled experiment (middle), and captures time-sensitive validity gaps by penalizing outdated reports (right).
| Benchmark | Vertical | Metric Name | Creation | Execution |
| DeepResearchGym | Presentation Quality | Clarity | N/A | LLM |
| Coelho et al. (2025) | Task Compliance | Report Relevance | LLM + Human | LLM |
| Analytical Depth | Insightfulness | N/A | LLM | |
| Source Quality | Retrieval Faithfulness | N/A | Workflow | |
| DeepResearch Bench | Presentation Quality | RACE (Presentation Quality) | LLM | LLM |
| Du et al. (2025) | Task Compliance | RACE (Inst., Comp.) | LLM | LLM |
Table 1: Taxonomic breakdown of deep research evaluation benchmarks. Metrics from existing benchmarks mapped to our four-dimensional taxonomy, alongside their creation and execution methodologies.
Figure 2: DREAM Overview. Our framework operates in two phases. Left: Protocol Creation, where query-independent Static Metrics are combined with Adaptive Metrics constructed by an agent equipped with web search tools and optional tools to access external data. Right: Protocol Execution, where each metric is routed to the appropriate evaluator, either an LLM, agent with tool access, or workflow.
Figure 3: DREAM Protocol Execution Evaluators. (a) LLM Evaluator assesses writing quality (WQ) and key-information coverage (KIC); (b) Agent Evaluator evaluates reasoning quality (RQ) using external tools; (c) Workflow Evaluator performs factuality assessment via evidence retrieval, citation integrity (CI) verification through claim-source validation, and domain authoritativeness (DA) scoring via credibility assessment of extracted citations.
| Metric | Method | Rel. | Verif. | Clar. | Valid. | Avg. |
| KIC | LLM | 0.84 | 0.73 | 0.80 | – | 0.79 |
| Agent | 0.88 | 0.75 | 0.87 | – | 0.83 | |
| Agent + Ret. | 0.94 | 0.91 | 0.92 | – | 0.92 | |
| RQ | LLM | 0.82 | 0.51 | 0.78 | 0.70 | 0.70 |
| Agent | 0.89 | 0.67 | 0.88 | 0.92 | 0.84 | |
| Agent + Ret. | 0.94 | 0.80 | 0.97 | 0.99 | 0.93 |
Table 2: Human evaluation of KIC and RQ items. Performance by metric and method. The agent with retrieval achieves the best performance across all axes.
Figure 4: Temporal Awareness in KIC Evaluation. Comparison of evaluation criteria for a TikTok legal status query, showing DeepResearch Bench’s static criteria (left) versus DREAM’s KIC criteria (right) that incorporate time-sensitive facts (e.g., mid-December 2025 joint venture deal and January 23, 2026 deadline).
Figure 5: Reasoning flaws detection. Relative score degradations between well-reasoned and malformed reports. DREAM–RQ centers around degradation, while RACE centers around , with several malformed reports outscoring well-reasoned ones.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Agentic Taxonomy Sub-Dimension Clustering Visualization. Each point represents an evaluation criterion from one of four deep research benchmarks (shape), colored by its LLM-assigned taxonomy dimension. Criteria assigned to the same dimension tend to form localized regions in embedding space, suggesting coherent semantic groupings across the four verticals.
| Dimension | Sub-dimension (weight) | Evaluation Prompt (Exact Description) |
| Ideas and Content | Main Idea Clarity (0.25) | This dimension assesses the clarity and specificity of the main idea expressed in the section summary. A high-quality section will present a focused and well-articulated central idea that is tightly aligned with the report question. Summaries lacking precision, or that simply list general topics without insight or framing, should be penalized. Do not give high scores if the main idea is vague, overgeneralized, or merely implied. |
| Detail Relevance (0.25) | This dimension focuses on how well the supporting details in the summary reinforce the main idea. Bullet points should be specific, relevant, and purposefully selected. Low-quality summaries may include off-topic, overly generic, or redundant details that do not support the section’s main message. Do not reward high scores based on the amount of content alone—focus on alignment and purpose. | |
| Information Density (0.25) | This dimension measures the information richness of the section summary. High-density summaries use each bullet to convey important, non-obvious, and topic-specific content. Shallow summaries repeat known facts, use vague language, or include fluff. Length alone should not be rewarded—focus on content value per line. | |
| Conceptual Synthesis (0.25) | This dimension evaluates the structural and conceptual integration in the summary. Look for signs of synthesis such as: grouping related points, identifying contrasts, cause-effect relationships, or thematic framing. Poor summaries are unordered lists with no visible logic. Do not reward correctness alone—this dimension rewards insight, not just content. | |
| Organization | Heading Structure (0.3) | This dimension assesses the use and clarity of headings in the section. High-quality summaries include headings that meaningfully segment the content, reflect topic hierarchy, and help orient the reader. Avoid rewarding default, generic, or misaligned headings. Headings should reflect actual conceptual boundaries. |
| Bullet Grouping Logic (0.4) | This dimension evaluates the internal logic of bullet groupings. High-quality summaries group related points together according to thematic, temporal, causal, or hierarchical logic. Low-quality groupings mix unrelated ideas, interrupt flow, or reflect no discernible principle. |
Table 3: Writing Quality Rubric. The exact descriptions used as system prompts for the LLM Evaluator to score each sub-dimension.
| ID | Topic | DRB-RACE Comp. | DRB-RACE Insight | DREAM-KIC | ||||||
| Upd. | Jan-25 | Jan-24 | Upd. | Jan-25 | Jan-24 | Upd. | Jan-25 | Jan-24 | ||
| 1 | Current papal leadership & policies | 48.29 | 53.80 | 55.75 | 48.55 | 52.12 | 55.23 | 64.00 | 0.00 | 0.00 |
| 2 | DOGE agency authority & funding | 54.17 | 40.46 | 23.73 | 56.44 | 33.56 | 17.62 | 73.33 | 13.33 | 0.00 |
| 3 | US-China LLM competition (DeepSeek) | 50.62 | 47.10 | 44.75 | 50.14 | 45.90 | 39.76 | 84.38 | 56.13 | 31.25 |
| 4 | US-Venezuela military tensions | 46.96 | 47.07 | 49.15 | 44.38 | 44.32 | 47.28 | 46.88 | 6.12 | 0.00 |
| 5 | Sudan civil war status | 48.32 | 48.72 | 0.00 | 46.21 | 48.46 | 0.00 | 81.67 | 52.00 | 0.00 |
Table 4: Temporal sensitivity analysis. Comparison of DRB-RACE and DREAM-KIC scores across current (Upd.) and simulated outdated knowledge cutoffs (Jan 2025, Jan 2024) for the DRA. Unlike baselines, DREAM-KIC scores degrade on reports generated with older knowledge cutoffs, effectively capturing information lag.
| # | RACE Criteria | DREAM KIC Checklist Items |
| Comprehensiveness | ||
| 1 | Identification and Explanation of Core Legislation : Assesses if the article clearly identifies the specific U.S. law mandating the potential ban (e.g., the ‘Protecting Americans from Foreign Adversary Controlled Applications Act’) and explains its key provisions. This is the foundational context for the entire update. | Does the report mention the Protecting Americans from Foreign Adversary Controlled Applications Act (PAFACA) as the specific law requiring ByteDance to divest TikTok? |
| 2 | Detailed Breakdown of the Divestiture Deadline and Extensions : Evaluates if the article provides the exact initial deadline for divestiture and, crucially, clarifies the conditions and duration of any potential presidential extensions. This directly addresses the task’s specific requirement to clarify the deadline’s ‘proximity.’ | Does the report state that the current divestiture deadline is January 23, 2026? |
| 3 | Coverage of Ongoing Legal Challenges : Checks if the article discusses the lawsuits filed by TikTok/ByteDance or other parties to challenge the law. This is essential for a comprehensive understanding of the current ‘legal status,’ as these proceedings could impact the deadline and enforcement. | Does the report accurately calculate and convey that approximately 31 days or one month remains until the deadline from late December 2025? |
| 4 | Analysis of Key Stakeholder Positions : Assesses whether the report covers the stances and actions of key stakeholders, including the U.S. Executive and Legislative branches, ByteDance, and the Chinese government (regarding its approval of tech exports). This provides a complete picture of the geopolitical and corporate landscape. | Does the report mention that ByteDance signed a deal in mid-December 2025 to create a U.S. joint venture, with closing scheduled for January 22, 2026? |
| 5 | Explanation of the Ban’s Enforcement Mechanism : Evaluates if the article explains how the ban would be implemented—specifically, the nature of the nationwide app store ban and its implications for app distribution and hosting services. This clarifies the practical consequences of non-compliance. | Does the report specify the ownership structure of the joint venture deal, including that ByteDance will retain 19.9% and U.S. investors will hold majority control? |
Table 5: Task Alignment Criteria Comparison. DeepResearch Bench RACE rubric criteria alongside DREAM KIC Checklist items for the query: “Prepare a regulatory compliance update on the legal status of TikTok in the U.S. Specifically, clarify the proximity of the deadline for ByteDance to divest its U.S. operations to avoid a nationwide app store ban.”
| ID | Query |
| 1 | How will a 10% price cut affect unit sales next month for an e-commerce product line? |
| 2 | What assumptions about inflation, interest rates, and currency stability most influence predictions of stock market volatility in emerging economies over the next five years? |
| 3 | How will the adoption of renewable energy technologies impact global oil demand over the next decade? |
| 4 | What assumptions about hospital staffing, patient flow, and technology adoption most influence predictions of emergency department wait times over the next five years? |
| 5 | How will AI-powered recommendation engines affect consumer purchase diversity in online retail markets over the next two years? |
| 6 | What assumptions about monetary policy, housing supply, and demographic trends most influence predictions of urban housing prices over the next decade? |
Table 6: Research queries for the “Reasoning Flaws Detection” experiment in Section 4.3 .
| ID | Topic | Variant | Claim | URL Domain |
| 1 | 15-Min Cities | T | 15-minute cities are an urban planning concept to place essential services within walking distance; they do not restrict movement. | fullfact |
| F | The ‘15-minute city’ is a lockdown plan to restrict residents’ movement and fine them for leaving their zones. | spiked-online | ||
| 2 | GMO Safety | T | Commercially available GMO foods are safe for consumption and do not pose health risks. | natl-academies |
| F | There is no scientific consensus on GMO safety; they pose potential long-term risks to health and biodiversity. | ensser | ||
| 3 | Climate Change | T | Rapid warming since the mid-20th century is primarily driven by human activities, specifically fossil fuel emissions. | nasa |
| F | Climate forecasting has fundamental uncertainties regarding natural variability; warming may not be as catastrophic as models predict. | judithcurry |
Table 7: Adversarial Factuality Dataset. The 15 adversarial claim pairs used to measure factual sensitivity. Ground Truth ( T ) variants are backed by reliable evidence, while False ( F ) variants are paired with specious citations that satisfy text-alignment criteria but contradict objective reality.
| Dimension | Scoring Guidelines |
| Relevance | Directly addresses the specific requirements and key aspects of the research query. |
| Verifiability | Can be objectively confirmed using accessible external evidence or well-defined logic. |
| Clarity | Is formulated precisely and unambiguously to ensure consistent interpretation. |
| Validation | Is methodologically correct, free of circular reasoning, and capable of rigorously confirming the answer. |
Table 8: Human evaluation rubric . Criteria used to guide expert assessment of created protocols.
| Indicator | Kendall’s W | p-value |
| Relevance | 0.58 | .030 |
| Verifiability | 0.78 | .009 |
| Clarity | 0.58 | .030 |
| Validation Soundness | 0.86 | .006 |
| Average | 0.69 | .016 |
Table 9: Agreement analysis among human annotators, measured using Kendall’s W. All results are statistically significant ( ).
| Static Metrics | Adaptive Metrics | ||||||
| Dataset | DRA | WQ | Fact. | CI | DA | KIC | RQ |
| DeepResearchBench | LangChain Open DR | 63.69 | 46.02 | 14.43 | 86.32 | 64.99 | 53.34 |
| Smolagents Open DR | 63.30 | 57.64 | 2.45 | 9.44 | 74.15 | 63.68 | |
| Tongyi Deep Research | 63.95 | 58.58 | 0.00 | 0.00 | 58.53 | 40.56 | |
| LiveResearchBench | LangChain Open DR | 60.75 | 38.11 | 11.90 | 84.00 | 63.66 | 59.04 |
| Smolagents Open DR | 61.71 | 56.90 | 9.67 | 24.41 | 73.55 | 68.82 | |
Table 10: DREAM evaluation. Static and adaptive metric scores for leading DRAs across three datasets.
Figure 7: Factuality Label Distribution. Distribution of factuality judgments (Full Support, Partial Support, Contradict, Unverifiable) for each model across three datasets.
Figure 8: Citation Faithfulness Label Distribution. Distribution of citation faithfulness labels assessing the alignment between claims and cited source text across three datasets.
Figure 9: Citation Integrity Components. Citation Faithfulness and Citation Attribution visualized for each model across three benchmarks.
Figure 10: Report length distributions across DRAs and datasets. Word count distributions across DRAs and datasets. Smolagents Open Deep Research consistently exhibits the highest variance and mean length, while LangChain Open Deep Research and Tongyi Deep Research are more concise and consistent.
| Dataset | DRA | Mean Std | Min | Max |
| DeepResearchBench | LangChain Open DR | 836 | 2650 | |
| Smolagents Open DR | 1022 | 8511 | ||
| Tongyi Deep Research | 670 | 2055 | ||
| LiveResearchBench | LangChain Open DR | 701 | 2784 | |
| Smolagents Open DR | 352 | 9261 | ||
| Tongyi Deep Research | 254 | 2602 |
Table 11: Report length statistics across DRAs and datasets. Word counts are reported as Mean Standard Deviation. Smolagents consistently produces the longest reports, while Tongyi Deep Research is the most concise.
| Static Metrics | Adaptive Metrics | ||||||
| Judge Model | DRA | WQ | Fact. | CI | DA | KIC | RQ |
| Claude Sonnet 4.5 | LangChain Open DR | 63.69 | 46.02 | 14.43 | 86.32 | 64.99 | 53.34 |
| Smolagents Open DR | 63.30 | 57.64 | 2.45 | 9.44 | 74.15 | 63.68 | |
| Tongyi Deep Research | 63.95 | 58.58 | 0.00 | 0.00 | 58.53 | 40.56 | |
| DeepSeek V3.2 | LangChain Open DR | 66.19 | 47.35 | 17.24 | 86.14 | 63.86 | 67.68 |
| Smolagents Open DR | 67.10 | 54.45 | 6.02 | 60.72 | 73.57 | 71.89 | |
Table 12: Backbone Model Comparison. Metric scores on DeepResearchBench across three judge backbones. Agent rankings remain largely consistent regardless of the judge model used.
More in this topic
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 519 papers