cs.CLJan 18, 2026

TaxoBench: Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies

Authors: Ming Zhang, Jiabao Zhuang, Wenqing Jing, Kexin Tan, Ziyu Kong, Jingyi Deng, Yujiong Shen, Yuhui Wang, 

Organizations: Fudan University · Hunyuan Team, Tencent

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deep Research Agents increasingly automate survey writing, yet existing benchmarks do not jointly test whether they retrieve the papers experts consider essential and organize those papers into paper-grounded taxonomies. We introduce TaxoBench, a benchmark built from 72 highly cited LLM surveys, 3,815 cited papers, and their expert-authored taxonomies. TaxoBench evaluates systems in two settings: Deep Research mode measures end-to-end retrieval and organization from a topic, while Bottom-Up mode provides the expert paper set and isolates organization. We evaluate leaf-level assignments with ARI and V-Measure and hierarchy-level structure with US-TED, US-NTED, and Sem-Path. Across 7 Deep Research Agents and 16 LLM configurations, the best agent retrieves only 20.92% of expert-cited papers, and none of 70 standard Bottom-Up runs reaches the experts' average depth of 4.86. A controlled probe shows that models which match this depth do so by fragmenting the taxonomy, reducing alignment with the expert reference. We further find that raw Sem-Path remains near a no-organization floor even when a newer model generation gains 3.68 pp ARI; after depth matching, humans lead on all 10 matched surveys by 13.27 pp. These results identify retrieval and hierarchical organization as separate bottlenecks and show why hierarchy metrics must be calibrated before they are used to compare models.

Figures & tables

Figure 1: Deep Research Agents for survey generation: given a topic, retrieve papers, then organize them into a taxonomy.
Figure 2: Overview of TaxoBench . In Deep Research mode , agents receive only a topic, retrieve papers with web tools, and build a taxonomy. In Bottom-Up mode , we provide the expert paper set to isolate organization. We compare the output with the expert reference at the leaf level using ARI and V-Measure and at the hierarchy level using US-TED / US-NTED and Sem-Path .
Figure 3
Table 2: Retrieval capability of Deep Research Agents when given the same survey topic as human experts. All values are percentages; best results are bold ; 3-run Recall std ≤3.01\leq 3.01 pp (App. L.4 ).
Figure 4: Retrieval is the dominant bottleneck in Deep Research mode. (a) End-to-end ARI vs. ARI ∩ on retrieved expert papers, with intersection size shown (App. E ). (b) F1 vs. Sem-Path across the 7 agents ( ρ=0.89\rho=0.89 , p=0.007p=0.007 ).
Table 3: Organization capability in Deep Research mode . ∣P∗ ⁣∩ ⁣P^∣|\mathcal{P}^{*}\!\cap\!\hat{\mathcal{P}}| is the mean number of retrieved expert papers; the ∩ metrics average over surveys with at least two aligned papers (App. E ). Best results are bold ; 3-run Sem-Path std ≤2.14\leq 2.14 pp.
Table 4: Organization capability in Bottom-Up mode : the newest reachable model of each family on Title+Abstract inputs for all 72 surveys, at the lowest reasoning effort. Highest-effort rows, a second GPT tier, the previous generation, and 3-run standard deviations are in Apps. H , I , and L.4 . Best results are bold . Every model is shallower than the expert depth of 4.86, and Sem-Path lies only 0.6–2.5 pp above the 27.49% Flat floor.
Figure 5: Two organization failures in Bottom-Up mode. (a) Homogeneity above Completeness indicates over-segmentation; the gap Δ\Delta (pp) is largest for GPT-5.6-sol and negative for Gemini-3.1-Pro. (b) Every raw Sem-Path score lies within 2.5 pp of the 27.49% Flat floor; depth matching raises scores to 32–42% (App. B ). Panels show 4 and 8 of the 16 configurations for legibility.
Figure 6: Human baseline on the matched 10 surveys (App. K.3 ). (a) Agreement between annotators, expert, and models; the dotted line marks annotator–annotator agreement, which models exceed on ARI and V-Measure but not on Sem-Path . (b) Per-survey Sem-Path : the bar spans the two annotators, the slate marker is their mean, and the clay marker is the strongest model configuration. Humans lead on all 10 surveys.
Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5: Soft set-matching baselines (NSR, NSP, Soft F1) for taxonomy evaluation.
TaxonomySem-PathConstruction
Singleton26.00%root →\to one category per paper
Flat27.49%root →\to one category →\to every paper
Random28.11%expert labels rewired into a random 3-level tree
12 configs, prev. gen.28.13–29.16%Bottom-Up, Title+Abstract
16 configs, current gen.28.12–30.03%Bottom-Up, one generation later
Truncated47.30%expert tree collapsed to 3 levels, placement correct
Appendix
Table 6: Sem-Path calibration. Random and Shuffle are averaged over 3 seeds ( ±0.02\pm 0.02 and ±0.27\pm 0.27 pp). Two facts follow. (i) The model bands sit 0.6–2.5 pp above Flat , a taxonomy performing no organization at all. (ii) Humans given the same Bottom-Up task clear the floor by 26.42 pp (App. K.3 ), while Shuffle destroys placement yet still scores 49.45%, just below the 50.56% at which two independent annotators agree with each other.
Reference frameCandidateSem-Path
Full expert treecollapsed to 3 levels, placement correct ( Truncated )47.30%
full tree, placement random ( Shuffle )49.43%
Collapsed expert treeitself (definitional ceiling)100.00%
itself with placement random77.29%
model trees (Table 8 )38–42%
Appendix
Table 7: Calibrating the depth-matched variant in its own reference frame , with the expert tree collapsed to three levels, averaged over 3 seeds and all 72 surveys. The top block reproduces the published raw-frame values and confirms the implementation. The bottom block tests the placement claim directly. Randomizing placement while keeping expert labels costs only 100% →\to 77.29%, yet models score 38–42%, so the model deficit is dominated by label and topology divergence rather than by placement.
Table 8: Depth-matched Sem-Path on the previous generation (Bottom-Up, Title+Abstract; the 8 configurations with retained tree outputs covering 66–71 of the 72 surveys). Each model is additionally scored against the expert tree collapsed to 3 category levels, which removes the unmatched-ancestor penalty arising purely from depth. The two reference rows are computed in the same collapsed frame as the model column; models fall well below the placement-random floor, so the shortfall is not attributable to placement (Table 7 ).
Finding / NumberLevelWhat it consumes
Best Recall 20.92% (F1)L3P∗\mathcal{P}^{*} only; no taxonomy
Depth deficit 0.89–1.87 levels (Finding 6)L2Expert depth (a scalar); no judge
Branching 3.16 vs. 2.52L2Expert branching (a scalar)
Over-segmentation, #cat. and singletonsL2Expert category count (a scalar)
Best ARI 34.92% ( saturated ; alignment only)L1Full expert assignment U∗U^{*}
Sem-Path band 28–30% (Finding 5)L1Full expert hierarchy Th∗T_{h}^{*}
Appendix
Table 9: Expert information used by each finding (L1–L3, Section 3.4 ). L3 uses the expert paper set but no taxonomy; L2 uses only scalar shape statistics of the expert hierarchy; L1 uses the hierarchy or assignment in full. ARI is marked saturated because models exceed human–human agreement on it. The LLM-judged defect rates at the bottom have no working expert control (App. D ).
Table 10: Expert-paired structural statistics and the induced Sem-Path ceiling (Bottom-Up, Title+Abstract; the 8 configurations whose tree outputs were fully retained). Nothing here uses an LLM judge or a tunable threshold. Depth counts category levels including the root and excluding paper nodes, matching Table 1 ; Chain is the mean per-paper ancestor-chain length, which is the quantity Sem-Path actually operates on; Deficit is the mean of qd−pdq_{d}-p_{d} over aligned papers; Ceiling is Equation 30 , averaged per paper. A singleton is a paper category holding exactly one paper. The depth deficit holds for every configuration without exception; the category-count and singleton gaps hold only for Claude-Sonnet-4.5 and the two GPT-5 variants.
Defect CategoryModelExpertBasis
Semantic overlap between siblings75.9%80.6%GPT-4o rubric; cosine re-impl. at τ=0.6\tau{=}0.6
Structural imbalance83.4%63.9%GPT-4o rubric (criterion cites reference tree)
MECE violations51.2%—GPT-4o rubric
Inconsistent classification criteria66.0%—GPT-4o rubric
Missing core branches77.5%—GPT-4o rubric + domain knowledge
Appendix
Table 11: LLM-judged defect rates with an expert control. All rates come from the GPT-4o rubric, which reaches κ=0.89\kappa=0.89 against human raters on the 5-point scale (App. J ). The Expert column applies an embedding re-implementation of the same criterion to the 72 expert taxonomies. Expert trees show higher sibling overlap than model trees, so this category does not discriminate. A dash marks criteria that we could not re-implement without a judge.
Table 12: Depth-controlled probe with 864 calls and 857 parseable trees. Condition C makes GPT-5 match the expert’s depth exactly but costs it 2.77 pp Sem-Path and half its ARI, because the extra levels are produced by fragmentation. Rising homogeneity under C reflects splitting (smaller clusters are purer).
Table 13: Depth and size decomposition (Bottom-Up). Each cell is the average over the corresponding bin.
MetricFree- KKConstrained- KK
ARI13.05%20.54% (+7.49 pp)
Homogeneity73.84%62.87% ( −- 10.97 pp)
Appendix
Table 14: Constrained- KK probe (15 surveys, GPT-5). Forcing the model to use exactly KK paper categories (the expert count) raises ARI by 7.49 pp but lowers homogeneity, showing both granularity and semantic assignment as bottlenecks.
Table 15: Average paper categories per input setting. Richer input increases the number of fine-grained categories, consistent with the over-segmentation explanation.
 Cov.Org.LogicTopo.Avg
Sem-Path0.9490.6320.7740.9641.000
Appendix
Table 16: Spearman correlation of Sem-Path with LLM-as-Judge scores (Deep Research mode, n=7n{=}7 ).
Table 17: Reasoning-effort settings, as sent and as observed. The low setting is the weakest value each endpoint accepts, found by probing: GPT accepts none while the others reject it and require minimal . max_tokens is 65,536 throughout, temperature is left at the endpoint default, and the prompt is byte-identical across both rows. Three cases show why the parameter alone is not evidence of a contrast: minimal does not disable reasoning (Gemini still spends 2,717 tokens); DeepSeek-V4-Pro appears to reason at capacity regardless of the setting; and Claude-Sonnet-5 reports no reasoning tokens at all, so its contrast is unverified.
Table 18: Highest reasoning effort , same prompts and surveys as Table 4 ( n=72n{=}72 , except Grok-4 at 71), plus both settings of the preceding GPT tier. † Claude-Sonnet-5 reports zero reasoning tokens in both settings, so this is the one pair we could not verify as two distinct conditions and it should not be counted as evidence about reasoning.
Table 19: Previous generation , Bottom-Up, Title+Abstract. * denotes Preview versions. † The two Gemini-3-Pro rows are not two distinct conditions : their outputs are byte-identical on all 72 surveys because the endpoint silently ignored the thinking flag, so any difference between these rows is not attributable to reasoning mode. The successor-model comparison uses per-call token verification (App. H ); the three other families differ on 72/72 surveys.
Table 20: Cohen’s Kappa coefficient between human and GPT-4o evaluations.
Table 21: LLM-as-Judge evaluation results across Deep Research and Bottom-Up modes.
Table 22: Ablation on input granularity (Bottom-Up mode). Models marked with * denote the Preview versions. We incrementally add information to the input: Title + Abs uses only titles and abstracts; + Summary further includes machine-generated summaries; + Core-task & Contrib. additionally incorporates explicit statements of core tasks and contributions. Best results within each model group are bold .
Figure 7: Top 3 error types by evaluation category. Error rates computed from qualitative analysis of 1000 model-generated taxonomies.
Figure 8: Ablation study on input settings in Bottom-Up mode. (a) ARI scores decrease as more information is provided, indicating reduced alignment with expert classifications. (b) US-NTED shows mixed or improving trends, suggesting that richer input does not necessarily help models generate taxonomies more aligned with expert standards.
DimensionError TypeFreq. (%)
Semantic CoverageMissing Core Branches77.5%
Irrelevant Additions70.8%
Conceptual Deviation67.9%
Missing Subfields58.0%
Incomplete Recall16.6%
Sibling OrganizationSemantic Overlap75.9%
Appendix
Table 23: Error-type statistics on 1,000 model-generated taxonomies. Errors are sorted by frequency within each dimension. Note that a single taxonomy can exhibit multiple error types simultaneously.
Table 24: Human baseline on a matched 10-survey subset. Humans and models receive the same Title+Abstract Bottom-Up task and are scored with the same code. SP dm is depth-matched Sem-Path ; bold marks the better of the human mean and the models on each metric. Model rows are macro-averages over the same 10 surveys; Kimi-K3 is the strongest ARI configuration on this subset and Claude-Sonnet-5 the strongest raw Sem-Path . The two annotators separate from every model on Sem-Path , US-NTED , and depth, but not on ARI, where they disagree with each other more than they disagree with the models.
ComparisonARI ↑\uparrowV-Meas. ↑\uparrowSem-Path ↑\uparrowSP dm↑{}_{\mathrm{dm}}\uparrowUS-NTED ↓\downarrow
Annotator vs. annotator22.92%63.54%50.56%50.81%31.99%
Annotator vs. expert46.27%77.09%53.91%54.09%29.80%
Model vs. expert35.76%75.91%28.21%38.71%77.04%
Model vs. model46.04%81.83%47.82%49.95%44.35%
Appendix
Table 25: Four-way agreement on the matched 10 surveys , one scoring pipeline throughout. Model rows pool all 16 current-generation Bottom-Up configurations. Models agree with the expert more than two independent annotators agree with each other on ARI and V-Measure , so those metrics are saturated relative to the reproducibility of a single reference. On Sem-Path and US-NTED the ordering reverses: models reach 0.56 ×\times the annotator-agreement level on Sem-Path and sit 2.4 ×\times further from the expert on US-NTED than the annotators sit from each other. Model–model agreement above annotator–annotator agreement shows that model taxonomies are more homogeneous than human ones. Substituting the excluded third delivery for annotator A2 changes no entry by more than 2.2 pp and preserves every ordering in this table.
Figure 9: Comparison between human expert taxonomy (left) and Kimi-K2-Thinking generated taxonomy (right) . The model-generated tree exhibits over-segmentation with many singleton clusters (marked as [Singleton] ), while the expert taxonomy uses broader thematic groupings.
Figure 10: Comparison of Human-Expert Taxonomies. (a) Human-Expert Taxonomy.
Figure 10: Comparison of Human-Expert Taxonomies and Human-Curated Taxonomies. (b) Human-Curated Taxonomy.
Table 26: Temporal filtering verification. Recall, Precision, and F1 are exactly unchanged when the reference list is restricted to papers with publication date ≤ts\leq t_{s} . This formalizes the structural argument that temporal under-specification cannot inflate the 20.92% Recall ceiling.
Table 27: 3-run error bars (Bottom-Up mode). Mean ±\pm std over three independent runs.
Table 28: 3-run error bars (Deep Research mode). Variance is higher due to non-deterministic retrieval, but ranking patterns are preserved.
MetricEncodero3GrokGem.Pplx.DSQwenDoub.
US-TEDMiniLM2157364
BGE-large2157364
Stella-1.5B2157364
US-NTEDMiniLM2167354
BGE-large2167453
Stella-1.5B3167452
Appendix
Table 29: Embedding sensitivity of structural metrics (Deep Research mode). Numbers are rank positions among 7 agents. US-TED and Sem-Path rankings are exactly identical across the three encoders.
Input conditionARI ↑\uparrowSem-Path ↑\uparrow
humanmachinehumanmachine
+Core-task, No-Thinking25.5%25.5%28.0%28.7%
+Core-task, Thinking26.8%27.1%28.3%28.7%
+Summary, No-Thinking27.8%25.6%28.0%28.7%
+Summary, Thinking27.3%26.8%28.3%28.5%
Appendix
Table 30: Human-written input control on all 72 surveys, averaged over six models: Claude-Sonnet-4.5, GPT-5, Gemini-3-Pro, DeepSeek-V3.2, Qwen3-Max, and Kimi-K2. The human columns replace all machine-generated auxiliary text with hand-written text. The machine columns use Claude-Sonnet-4.6 as in Table 22 . Differences are at most 2.2 pp for ARI and 0.75 pp for Sem-Path .
Table 31: Multi-category paper sensitivity. Sem-Path after removing all expert-cited papers that the survey authors placed in multiple categories. Differences are within ≤1.5\leq 1.5 pp, confirming no systematic bias.
MetricCross-domain avg.AI/ML avg.
ARI23.93%27.25%
Sem-Path30.90%29.16%
Homogeneity85.35%82.27%
Appendix
Table 32: Cross-domain pilot. Claude-Sonnet-4.5 on two interdisciplinary surveys (finance, medical), Bottom-Up mode. The same qualitative pattern emerges as for AI/ML surveys.
Table 33: Deep Research re-test with current agents. Percentages over the 72 surveys, with failed runs scored as zero; Papers is the mean number of papers listed per survey. Over repeated runs on a 15-survey subset, the standard deviation of F1 is below 1 pp.
Stage# surveys retained
0. Initial pool of LLM-related surveys200+
1. Taxonomy filter (explicit taxonomy figure)∼\sim 140
2. Impact filter (high citation count)∼\sim 110
3. Scope filter (well-defined survey scope)∼\sim 90
4. Expert verification (Ph.D. researcher review)72
Appendix
Table 34: Survey selection pipeline (four-stage). Counts are after each filter; the final 72 surveys span 8 subdomains and have an average of 354.5 citations.
Figure 11: Prompt for generating a taxonomy tree in Bottom-Up mode.
Figure 12: The prompt for taxonomy tree generation based on paper summaries.
Figure 13: The prompt for taxonomy tree construction based on Core-Task and Contribution.
Figure 14: Prompt for Deep Research mode, as given to o3, Grok, and Perplexity, with {TOPIC} replaced by the survey title. App. R.2 lists the variants the other agents received.
Figure 15: Markdown variant of the Deep Research prompt given to Gemini, reproduced verbatim, including the sentence that names the field as LLM alignment for every survey.
Figure 16: The prompt for paper summary extraction.
Figure 17: The prompt for core-task extraction.
Figure 18: The prompt for contribution extraction.
Figure 19: The prompt for LLM-as-Judge evaluation on taxonomy trees.
Figure 19: The prompt for LLM-as-Judge evaluation on taxonomy trees. (Continued)
More in this topic

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,482 papers

Explore similar work

CardsList
Jan 13, 2026

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmarks often either under-evaluate a system's ability to produce meaningful insights and high-quality writing, or adopt coarse or LLM-defined criteria that are hard to verify and can diverge from human expert judgment. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRAs. It contains 132 grounded research tasks across 22 domains; for each task, an agent must produce a research report that is evaluated by a set of 9,430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are verifiable and aligned with human expert judgment. We evaluate several state-of-the-art deep-research agents on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRAs and human experts. We release the benchmark, evaluation scripts, and all rubrics at https://github.com/imlrz/DeepResearch-Bench-II to facilitate future research on deep-rearch agents.

Ruizhe Li, Mingxuan Du, Benfeng Xu +3

University of Science and Technology of China · Metastone Technology, Beijing, China

Deep Research AgentsLLM Agent EvaluationAI Agent BenchmarksRubric-Based EvaluationAutomated Evaluation
AskPDF
Jun 18, 2026

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks offer limited coverage of research intents and insufficient support for scalable answer construction and reproducible multi-turn evaluation. We propose ScholarQuest, a taxonomy-guided benchmark constructed from over 1,000 computer science topics and organized around four research intents. To construct answer sets at scale, we combine multi-source retrieval, citation expansion, and multi-stage relevance adjudication. We further build ScholarBase, a million-scale literature environment with unified retrieval and citation-expansion interfaces, together with a shared multi-turn search harness. We benchmark complete literature search systems and compare language models within this harness, evaluating retrieval coverage, search efficiency, and robustness across research intents. Results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing search system achieves only 0.312 Recall@100 and 0.352 Recall@All. Comparisons under the shared harness reveal intent-dependent model strengths, with higher Planner token consumption not consistently yielding higher recall. We release the benchmark, retrieval backend, and evaluation harness for reproducible research

Tingyue Pan, Mingyue Cheng, Daoyu Wang +4

State Key Lab of Cognitive Intelligence, University of Science and Technology of China

Agentic SearchInformation RetrievalIR EvaluationAgent Benchmarks
AskPDF
Mar 4, 2026

AgentIR: Reasoning-Aware Retrieval for Deep Research Agents

Deep Research agents are rapidly emerging as primary consumers of modern retrieval systems. Unlike human users who issue and refine queries without documenting their intermediate thought processes, Deep Research agents generate explicit natural language reasoning before each search call, revealing rich intent and contextual information that existing retrievers entirely ignore. To exploit this overlooked signal, we introduce: (1) Reasoning-Aware Retrieval, a retrieval paradigm that jointly embeds the agent's reasoning trace alongside its query; and (2) DR-Synth, a data synthesis method that generates Deep Research retriever training data from standard QA datasets. We demonstrate that both components are independently effective, and their combination yields a trained embedding model, AgentIR-4B, with substantial gains. On the challenging BrowseComp-Plus benchmark, AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch, compared to 50% with conventional embedding models twice its size, and 37% with BM25. Code and data are available at: https://texttron.github.io/AgentIR/.

Zijian Chen, Xueguang Ma, Shengyao Zhuang +3

University of Waterloo · University of Queensland · Carnegie Mellon University

Agentic RetrievalDense RetrievalDeep Research Agents
AskPDF