Towards Knowledgeable Deep Research: Framework and Benchmark

Wenxuan Liu; Zixuan Li; Long Bai; Chunmao Zhang; Fenghui Zhang; Zhuo Chen; Wei Li; Yuxin Zuo

cs.AIApr 9, 2026

Towards Knowledgeable Deep Research: Framework and Benchmark

Authors: Wenxuan Liu, Zixuan Li, Long Bai, Chunmao Zhang, Fenghui Zhang, Zhuo Chen, Wei Li, Yuxin Zuo, +8 more

Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Also with University of Chinese Academy of Sciences. · National University of Singapore, Singapore, Singapore

Abstract

Deep Research (DR) requires LLM agents to autonomously perform multi-step information seeking, processing, and reasoning to generate comprehensive reports. In contrast to existing studies that mainly focus on unstructured web content, a more challenging DR task should additionally utilize structured knowledge to provide a solid data foundation, facilitate quantitative computation, and lead to in-depth analyses. In this paper, we refer to this novel task as Knowledgeable Deep Research (KDR), which requires DR agents to generate reports with both structured and unstructured knowledge. Furthermore, we propose the Hybrid Knowledge Analysis framework (HKA), a multi-agent architecture that reasons over both kinds of knowledge and integrates the texts, figures, and tables into coherent multimodal reports. The key design is the Structured Knowledge Analyzer, which utilizes both coding and vision-language models to produce figures, tables, and corresponding insights. To support systematic evaluation, we construct KDR-Bench, which covers 9 domains, includes 41 expert-level questions, and incorporates a large number of structured knowledge resources (e.g., 1,252 tables). We further annotate the main conclusions and key points for each question and propose three categories of evaluation metrics including general-purpose, knowledge-centric, and vision-enhanced ones. Experimental results demonstrate that HKA consistently outperforms most existing DR agents on general-purpose and knowledge-centric metrics, and even surpasses the Gemini DR agent on vision-enhanced metrics, highlighting its effectiveness in deep, structure-aware knowledge analysis. Finally, we hope this work can serve as a new foundation for structured knowledge analysis in DR agents and facilitate future multimodal DR studies.

Figures & tables

Figure 1. Demonstration of the proposed HKA framework.

Figure 2. The construction procedure of KDR-Bench: (a) Dataset construction process; (b) Evaluation framework.

Figure 3. The statistics on tables in KDR-Bench.

	General-Purpose					Knowledge-Centric
Methods	Avg.	Comp.	Depth	Coher.	Read.	Main.	Key.	Support.
LLM with search tools
Hunyuan2.0	40.6	40.9	38.8	41.4	41.2	61.0	46.7	-
GLM4.6	40.8	40.8	36.4	42.6	43.5	59.1	40.1	-
Qwen3-Max	44.6	43.9	40.5	46.6	47.4	64.2	43.5	-
Minimax-M2	44.7	44.8	41.1	46.8	46.0	58.2	50.5	-

Table 1. Results on General-purpose and Knowledge-centric metrics. The best results are in boldface, and the second-best results are underlined.

Methods	Agr.	P&E	E&E	F&I	M&E	Soc.	Art	Tech.	Trans.	Avg.
GLM4.6	91.7	100.0	100.0	100.0	100.0	100.0	83.3	100.0	100.0	97.6
Minimax-M2	100.0	80.0	100.0	83.3	100.0	83.3	100.0	100.0	100.0	92.2
Perplexity	66.7	91.7	60.0	66.7	70.0	83.3	66.7	50.0	100.0	72.0
Grok	66.7	100.0	100.0	58.3	80.0	83.3	66.7	100.0	100.0	81.7
Gemini	50.0	66.6	60.0	41.7	60.0	50.0	66.7	50.0	100.0	56.1
Enterprise	66.7	100.0	90.0	66.7	100.0	83.3	83.3	66.7	100.0	82.9

Table 2. Results on Vision-Enhanced metrics. Greener cells indicate a larger advantage for HKA, while redder cells indicate a larger disadvantage for HKA.

	G.P. Avg.	Main.	Key.
HKA	48.4	82.1	61.7
wo. S.K.A.	46.7 ↓1.7	74.1 ↓8.0	52.7 ↓9.0
wo. U.K.A.	45.4 ↓3.0	75.2 ↓6.9	58.7 ↓3.0
wo. Rerank	46.1 ↓2.3	79.3 ↓2.8	59.9 ↓1.9
wo. Schema	48.1 ↓0.3	81.7 ↓0.4	60.3 ↓1.4

Table 3. Results of ablation study.

Method	Metric	Default	Claude
Minimax-M2	G.P. Avg.	44.7	45.6
	Main.	58.2	58.1
	Key.	50.5	50.5
Perplexity	G.P. Avg.	46.5	46.8
	Main.	70.1	67.1
	Key.	49.8	50.9

Table 4. Results on different judge LLMs.

Method	Metric	Run 1	Run 2	Run 3	S.D.
Minimax-M2	G.P. Avg.	44.7	45.1	44.6	0.2
	Main.	58.2	58.9	58.7	0.3
	Key.	50.5	49.7	50.1	0.3
Enterprise	G.P. Avg.	46.0	46.2	45.6	0.2
	Main.	72.3	73.1	71.6	0.6
	Key.	51.8	50.6	51.8	0.6

Table 5. Results on different generation runs.

Explore similar work

Jan 13, 2026

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmarks often either under-evaluate a system's ability to produce meaningful insights and high-quality writing, or adopt coarse or LLM-defined criteria that are hard to verify and can diverge from human expert judgment. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRAs. It contains 132 grounded research tasks across 22 domains; for each task, an agent must produce a research report that is evaluated by a set of 9,430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are verifiable and aligned with human expert judgment. We evaluate several state-of-the-art deep-research agents on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRAs and human experts. We release the benchmark, evaluation scripts, and all rubrics at https://github.com/imlrz/DeepResearch-Bench-II to facilitate future research on deep-rearch agents.

Ruizhe Li, Mingxuan Du, Benfeng Xu +3

University of Science and Technology of China · Metastone Technology, Beijing, China

Deep Research Agents LLM Agent Evaluation AI Agent Benchmarks Rubric-Based Evaluation Automated Evaluation

Ask PDF

Apr 16, 2026

DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation

Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR

^{3}

-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR

^{3}

-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR

^{3}

-Agent based on multiple state-of-the-art language models demonstrate that DR

^{3}

-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.

Qianqian Xie, Qingheng Xiong, He Zhu +16

Nanjing University · M-A-P · Nanjing University of Science and Technology +2

LLM Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks

Ask PDF

Mar 26, 2026

From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

Deep Research Agents (DRAs) aim to answer complex questions by searching the web, checking evidence, and synthesizing conclusions across heterogeneous sources. We introduce a category-theoretic framework for evaluating and improving such agents. The framework treats deep research as a structured mapping from user intent to evidence-grounded conclusions, making retrieval traces, cross-source alignment, and final synthesis explicit. Guided by this view, we derive a mechanism-aware benchmark of 296 bilingual questions. The benchmark targets four structural skills central to real research: following multi-hop evidence chains, verifying claims across sources, re-ordering fragmented information, and rejecting unsupported assumptions. We evaluate 16 frontier systems with human verification and find that these structural tasks remain highly challenging: the best system reaches only 19.9% average accuracy. The results show that strong agents can sometimes reorganize evidence and detect false premises, but still struggle with long-horizon synthesis and intersection-heavy verification. Beyond evaluation, the same theory also leads to practical system improvements. We instantiate theory-guided interventions such as tracked search, which preserves retrieval traces, and category tools, which add explicit verification and synthesis steps. These interventions yield measurable gains in API-based deep research systems. Our work therefore provides both a challenging benchmark and concrete design guidance for building more reliable research agents.

Shuoling Liu, Zhiquan Tan, Kun Yi +6

The Hong Kong University of Science and Technology · E Fund Management Co., Ltd. · The Hong Kong Polytechnic University

Deep Research Agents AI Agent Benchmarks Long-Horizon Agent Evaluation

Ask PDF

Abstract

Explore similar work

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

DR3^{3}3-Eval: Towards Realistic and Reproducible Deep Research Evaluation

From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation