cs.LGAug 14, 2026

WANDR: A Benchmark for Wide and Deep Research

Authors: Vitaliy Polshkov, Marcin Pitera, Jeremy Yang, Kirill Priemko, Maksim Gaiduk, Aleksandr Nikolenko, Denis Bykov, Clare Southern, +2 more

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

WANDR (Wide ANd Deep Research) is a benchmark of 500 realistic, challenging data-collection tasks for research agents. Each task requires a system to discover a large set of entities that satisfy specified criteria (breadth), investigate each entity through multiple coordinated web searches (depth), and return independently verifiable records with supporting sources and excerpts. Tasks are represented as qualification key hierarchies that specify the entities, relationships, evidence, and required count at each level; a hierarchy with n companies, m employees per company, and k sources per employee requires n x m x k records. This structure supports diverse workflows such as market mapping, due diligence, literature review, product comparison, and talent sourcing, with targets ranging from dozens to thousands of records. WANDR replaces static gold answer sets with task-specific judges that refetch cited pages and verify each record against its evidence, allowing evaluation of current and changing facts. Record verdicts are aggregated into soft and hard precision, recall, and F1 scores that distinguish factual quality, coverage, and hierarchical completeness. The tasks are derived from de-identified product-usage logs and produced through a semi-automated pipeline with automated checks, empirical audits, and human review where needed. We evaluate six production research systems and find that the benchmark is far from saturated: at high effort, the strongest system reaches only 0.363 soft F1 and 0.133 hard F1. Performance degrades as target volume and hierarchy depth increase, with incomplete discovery, missing enrichment, and incomplete evidence construction remaining major bottlenecks. The benchmark and evaluation harness are available at https://github.com/perplexityai/wandr.

Explore similar work

CardsList
Date pendingcs.CL

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmarks often either under-evaluate a system's ability to produce meaningful insights and high-quality writing, or adopt coarse or LLM-defined criteria that are hard to verify and can diverge from human expert judgment. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRAs. It contains 132 grounded research tasks across 22 domains; for each task, an agent must produce a research report that is evaluated by a set of 9,430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are verifiable and aligned with human expert judgment. We evaluate several state-of-the-art deep-research agents on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRAs and human experts. We release the benchmark, evaluation scripts, and all rubrics at https://github.com/imlrz/DeepResearch-Bench-II to facilitate future research on deep-rearch agents.

Ruizhe Li, Mingxuan Du, Benfeng Xu +3

University of Science and Technology of China · Metastone Technology, Beijing, China

Deep Research AgentsLLM Agent EvaluationAI Agent BenchmarksRubric-Based EvaluationAutomated Evaluation
AskPDF
Aug 3, 2026cs.AI

From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

Deep research benchmarks require expert-level tasks and reliable evaluation grounded in task-specific knowledge. Existing benchmarks rely heavily on expert authoring or pre-existing human-authored materials, while fully automatic construction struggles to ensure consistent and traceable verification. To address this gap, we introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories, with three query forms designed to probe complementary capabilities required for deep research. The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline that progressively transforms simple questions into deep research tasks. Each task is represented as a directed acyclic graph (DAG) of atomic steps and associated checkpoints, enabling the query, DAG, and rubrics to evolve together in a controlled manner. Experiments demonstrate that the benchmark clearly discriminates among models and query types, while its fact-grounded pointwise rubrics enable fine-grained, human-aligned, and stable evaluation. Our data, implementation, and results are publicly available.

Can Wang, Haoran Chen, Haowen Gao +3

Shandong Key Laboratory of Digital Service Computing Technology and Systems · Alibaba Token Hub · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences +1

LLM EvaluationBenchmark DesignSynthetic Benchmark GenerationAutomated Evaluation
AskPDF
Apr 16, 2026cs.AI

DR3^{3}-Eval: Towards Realistic and Reproducible Deep Research Evaluation

Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR3^{3}-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR3^{3}-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR3^{3}-Agent based on multiple state-of-the-art language models demonstrate that DR3^{3}-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.

Qianqian Xie, Qingheng Xiong, He Zhu +16

Nanjing University · M-A-P · Nanjing University of Science and Technology +2

LLM Agent EvaluationLong-Horizon Agent EvaluationAI Agent Benchmarks
AskPDF