cs.CLJan 18, 2026
SaveTaxoBench: Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies
Organizations: Fudan University · Hunyuan Team, Tencent
Abstract
Deep Research Agents increasingly automate survey writing, yet existing benchmarks do not jointly test whether they retrieve the papers experts consider essential and organize those papers into paper-grounded taxonomies. We introduce TaxoBench, a benchmark built from 72 highly cited LLM surveys, 3,815 cited papers, and their expert-authored taxonomies. TaxoBench evaluates systems in two settings: Deep Research mode measures end-to-end retrieval and organization from a topic, while Bottom-Up mode provides the expert paper set and isolates organization. We evaluate leaf-level assignments with ARI and V-Measure and hierarchy-level structure with US-TED, US-NTED, and Sem-Path. Across 7 Deep Research Agents and 16 LLM configurations, the best agent retrieves only 20.92% of expert-cited papers, and none of 70 standard Bottom-Up runs reaches the experts' average depth of 4.86. A controlled probe shows that models which match this depth do so by fragmenting the taxonomy, reducing alignment with the expert reference. We further find that raw Sem-Path remains near a no-organization floor even when a newer model generation gains 3.68 pp ARI; after depth matching, humans lead on all 10 matched surveys by 13.27 pp. These results identify retrieval and hierarchical organization as separate bottlenecks and show why hierarchy metrics must be calibrated before they are used to compare models.
Figures & tables
Figure 1: Deep Research Agents for survey generation: given a topic, retrieve papers, then organize them into a taxonomy.
Figure 2: Overview of TaxoBench . In Deep Research mode , agents receive only a topic, retrieve papers with web tools, and build a taxonomy. In Bottom-Up mode , we provide the expert paper set to isolate organization. We compare the output with the expert reference at the leaf level using ARI and V-Measure and at the hierarchy level using US-TED / US-NTED and Sem-Path .
Table 2: Retrieval capability of Deep Research Agents when given the same survey topic as human experts. All values are percentages; best results are bold ; 3-run Recall std pp (App. L.4 ).
Figure 4: Retrieval is the dominant bottleneck in Deep Research mode. (a) End-to-end ARI vs. ARI ∩ on retrieved expert papers, with intersection size shown (App. E ). (b) F1 vs. Sem-Path across the 7 agents ( , ).
Table 3: Organization capability in Deep Research mode . is the mean number of retrieved expert papers; the ∩ metrics average over surveys with at least two aligned papers (App. E ). Best results are bold ; 3-run Sem-Path std pp.
Table 4: Organization capability in Bottom-Up mode : the newest reachable model of each family on Title+Abstract inputs for all 72 surveys, at the lowest reasoning effort. Highest-effort rows, a second GPT tier, the previous generation, and 3-run standard deviations are in Apps. H , I , and L.4 . Best results are bold . Every model is shallower than the expert depth of 4.86, and Sem-Path lies only 0.6–2.5 pp above the 27.49% Flat floor.
Figure 5: Two organization failures in Bottom-Up mode. (a) Homogeneity above Completeness indicates over-segmentation; the gap (pp) is largest for GPT-5.6-sol and negative for Gemini-3.1-Pro. (b) Every raw Sem-Path score lies within 2.5 pp of the 27.49% Flat floor; depth matching raises scores to 32–42% (App. B ). Panels show 4 and 8 of the 16 configurations for legibility.
Figure 6: Human baseline on the matched 10 surveys (App. K.3 ). (a) Agreement between annotators, expert, and models; the dotted line marks annotator–annotator agreement, which models exceed on ARI and V-Measure but not on Sem-Path . (b) Per-survey Sem-Path : the bar spans the two annotators, the slate marker is their mean, and the clay marker is the strongest model configuration. Humans lead on all 10 surveys.
Appendix figures & tables45 assets
Supplementary material from the paper’s appendix.
Appendix
Table 5: Soft set-matching baselines (NSR, NSP, Soft F1) for taxonomy evaluation.
| Taxonomy | Sem-Path | Construction |
| Singleton | 26.00% | root one category per paper |
| Flat | 27.49% | root one category every paper |
| Random | 28.11% | expert labels rewired into a random 3-level tree |
| 12 configs, prev. gen. | 28.13–29.16% | Bottom-Up, Title+Abstract |
| 16 configs, current gen. | 28.12–30.03% | Bottom-Up, one generation later |
| Truncated | 47.30% | expert tree collapsed to 3 levels, placement correct |
Table 6: Sem-Path calibration. Random and Shuffle are averaged over 3 seeds ( and pp). Two facts follow. (i) The model bands sit 0.6–2.5 pp above Flat , a taxonomy performing no organization at all. (ii) Humans given the same Bottom-Up task clear the floor by 26.42 pp (App. K.3 ), while Shuffle destroys placement yet still scores 49.45%, just below the 50.56% at which two independent annotators agree with each other.
| Reference frame | Candidate | Sem-Path |
| Full expert tree | collapsed to 3 levels, placement correct ( Truncated ) | 47.30% |
| full tree, placement random ( Shuffle ) | 49.43% | |
| Collapsed expert tree | itself (definitional ceiling) | 100.00% |
| itself with placement random | 77.29% | |
| model trees (Table 8 ) | 38–42% |
Table 7: Calibrating the depth-matched variant in its own reference frame , with the expert tree collapsed to three levels, averaged over 3 seeds and all 72 surveys. The top block reproduces the published raw-frame values and confirms the implementation. The bottom block tests the placement claim directly. Randomizing placement while keeping expert labels costs only 100% 77.29%, yet models score 38–42%, so the model deficit is dominated by label and topology divergence rather than by placement.
Table 8: Depth-matched Sem-Path on the previous generation (Bottom-Up, Title+Abstract; the 8 configurations with retained tree outputs covering 66–71 of the 72 surveys). Each model is additionally scored against the expert tree collapsed to 3 category levels, which removes the unmatched-ancestor penalty arising purely from depth. The two reference rows are computed in the same collapsed frame as the model column; models fall well below the placement-random floor, so the shortfall is not attributable to placement (Table 7 ).
| Finding / Number | Level | What it consumes |
| Best Recall 20.92% (F1) | L3 | only; no taxonomy |
| Depth deficit 0.89–1.87 levels (Finding 6) | L2 | Expert depth (a scalar); no judge |
| Branching 3.16 vs. 2.52 | L2 | Expert branching (a scalar) |
| Over-segmentation, #cat. and singletons | L2 | Expert category count (a scalar) |
| Best ARI 34.92% ( saturated ; alignment only) | L1 | Full expert assignment |
| Sem-Path band 28–30% (Finding 5) | L1 | Full expert hierarchy |
Table 9: Expert information used by each finding (L1–L3, Section 3.4 ). L3 uses the expert paper set but no taxonomy; L2 uses only scalar shape statistics of the expert hierarchy; L1 uses the hierarchy or assignment in full. ARI is marked saturated because models exceed human–human agreement on it. The LLM-judged defect rates at the bottom have no working expert control (App. D ).
Table 10: Expert-paired structural statistics and the induced Sem-Path ceiling (Bottom-Up, Title+Abstract; the 8 configurations whose tree outputs were fully retained). Nothing here uses an LLM judge or a tunable threshold. Depth counts category levels including the root and excluding paper nodes, matching Table 1 ; Chain is the mean per-paper ancestor-chain length, which is the quantity Sem-Path actually operates on; Deficit is the mean of over aligned papers; Ceiling is Equation 30 , averaged per paper. A singleton is a paper category holding exactly one paper. The depth deficit holds for every configuration without exception; the category-count and singleton gaps hold only for Claude-Sonnet-4.5 and the two GPT-5 variants.
| Defect Category | Model | Expert | Basis |
| Semantic overlap between siblings | 75.9% | 80.6% | GPT-4o rubric; cosine re-impl. at |
| Structural imbalance | 83.4% | 63.9% | GPT-4o rubric (criterion cites reference tree) |
| MECE violations | 51.2% | — | GPT-4o rubric |
| Inconsistent classification criteria | 66.0% | — | GPT-4o rubric |
| Missing core branches | 77.5% | — | GPT-4o rubric + domain knowledge |
Table 11: LLM-judged defect rates with an expert control. All rates come from the GPT-4o rubric, which reaches against human raters on the 5-point scale (App. J ). The Expert column applies an embedding re-implementation of the same criterion to the 72 expert taxonomies. Expert trees show higher sibling overlap than model trees, so this category does not discriminate. A dash marks criteria that we could not re-implement without a judge.
Table 12: Depth-controlled probe with 864 calls and 857 parseable trees. Condition C makes GPT-5 match the expert’s depth exactly but costs it 2.77 pp Sem-Path and half its ARI, because the extra levels are produced by fragmentation. Rising homogeneity under C reflects splitting (smaller clusters are purer).
Table 13: Depth and size decomposition (Bottom-Up). Each cell is the average over the corresponding bin.
| Metric | Free- | Constrained- |
| ARI | 13.05% | 20.54% (+7.49 pp) |
| Homogeneity | 73.84% | 62.87% ( 10.97 pp) |
Table 14: Constrained- probe (15 surveys, GPT-5). Forcing the model to use exactly paper categories (the expert count) raises ARI by 7.49 pp but lowers homogeneity, showing both granularity and semantic assignment as bottlenecks.
Table 15: Average paper categories per input setting. Richer input increases the number of fine-grained categories, consistent with the over-segmentation explanation.
| Cov. | Org. | Logic | Topo. | Avg | |
| Sem-Path | 0.949 | 0.632 | 0.774 | 0.964 | 1.000 |
Table 16: Spearman correlation of Sem-Path with LLM-as-Judge scores (Deep Research mode, ).
Table 17: Reasoning-effort settings, as sent and as observed. The low setting is the weakest value each endpoint accepts, found by probing: GPT accepts none while the others reject it and require minimal . max_tokens is 65,536 throughout, temperature is left at the endpoint default, and the prompt is byte-identical across both rows. Three cases show why the parameter alone is not evidence of a contrast: minimal does not disable reasoning (Gemini still spends 2,717 tokens); DeepSeek-V4-Pro appears to reason at capacity regardless of the setting; and Claude-Sonnet-5 reports no reasoning tokens at all, so its contrast is unverified.
Table 18: Highest reasoning effort , same prompts and surveys as Table 4 ( , except Grok-4 at 71), plus both settings of the preceding GPT tier. † Claude-Sonnet-5 reports zero reasoning tokens in both settings, so this is the one pair we could not verify as two distinct conditions and it should not be counted as evidence about reasoning.
Table 19: Previous generation , Bottom-Up, Title+Abstract. * denotes Preview versions. † The two Gemini-3-Pro rows are not two distinct conditions : their outputs are byte-identical on all 72 surveys because the endpoint silently ignored the thinking flag, so any difference between these rows is not attributable to reasoning mode. The successor-model comparison uses per-call token verification (App. H ); the three other families differ on 72/72 surveys.
Table 20: Cohen’s Kappa coefficient between human and GPT-4o evaluations.
Table 21: LLM-as-Judge evaluation results across Deep Research and Bottom-Up modes.
Table 22: Ablation on input granularity (Bottom-Up mode). Models marked with * denote the Preview versions. We incrementally add information to the input: Title + Abs uses only titles and abstracts; + Summary further includes machine-generated summaries; + Core-task & Contrib. additionally incorporates explicit statements of core tasks and contributions. Best results within each model group are bold .
Figure 7: Top 3 error types by evaluation category. Error rates computed from qualitative analysis of 1000 model-generated taxonomies.
Figure 8: Ablation study on input settings in Bottom-Up mode. (a) ARI scores decrease as more information is provided, indicating reduced alignment with expert classifications. (b) US-NTED shows mixed or improving trends, suggesting that richer input does not necessarily help models generate taxonomies more aligned with expert standards.
| Dimension | Error Type | Freq. (%) |
| Semantic Coverage | Missing Core Branches | 77.5% |
| Irrelevant Additions | 70.8% | |
| Conceptual Deviation | 67.9% | |
| Missing Subfields | 58.0% | |
| Incomplete Recall | 16.6% | |
| Sibling Organization | Semantic Overlap | 75.9% |
Table 23: Error-type statistics on 1,000 model-generated taxonomies. Errors are sorted by frequency within each dimension. Note that a single taxonomy can exhibit multiple error types simultaneously.
Table 24: Human baseline on a matched 10-survey subset. Humans and models receive the same Title+Abstract Bottom-Up task and are scored with the same code. SP dm is depth-matched Sem-Path ; bold marks the better of the human mean and the models on each metric. Model rows are macro-averages over the same 10 surveys; Kimi-K3 is the strongest ARI configuration on this subset and Claude-Sonnet-5 the strongest raw Sem-Path . The two annotators separate from every model on Sem-Path , US-NTED , and depth, but not on ARI, where they disagree with each other more than they disagree with the models.
| Comparison | ARI | V-Meas. | Sem-Path | SP | US-NTED |
| Annotator vs. annotator | 22.92% | 63.54% | 50.56% | 50.81% | 31.99% |
| Annotator vs. expert | 46.27% | 77.09% | 53.91% | 54.09% | 29.80% |
| Model vs. expert | 35.76% | 75.91% | 28.21% | 38.71% | 77.04% |
| Model vs. model | 46.04% | 81.83% | 47.82% | 49.95% | 44.35% |
Table 25: Four-way agreement on the matched 10 surveys , one scoring pipeline throughout. Model rows pool all 16 current-generation Bottom-Up configurations. Models agree with the expert more than two independent annotators agree with each other on ARI and V-Measure , so those metrics are saturated relative to the reproducibility of a single reference. On Sem-Path and US-NTED the ordering reverses: models reach 0.56 the annotator-agreement level on Sem-Path and sit 2.4 further from the expert on US-NTED than the annotators sit from each other. Model–model agreement above annotator–annotator agreement shows that model taxonomies are more homogeneous than human ones. Substituting the excluded third delivery for annotator A2 changes no entry by more than 2.2 pp and preserves every ordering in this table.
Figure 9: Comparison between human expert taxonomy (left) and Kimi-K2-Thinking generated taxonomy (right) . The model-generated tree exhibits over-segmentation with many singleton clusters (marked as [Singleton] ), while the expert taxonomy uses broader thematic groupings.
Figure 10: Comparison of Human-Expert Taxonomies. (a) Human-Expert Taxonomy.
Figure 10: Comparison of Human-Expert Taxonomies and Human-Curated Taxonomies. (b) Human-Curated Taxonomy.
Table 26: Temporal filtering verification. Recall, Precision, and F1 are exactly unchanged when the reference list is restricted to papers with publication date . This formalizes the structural argument that temporal under-specification cannot inflate the 20.92% Recall ceiling.
Table 27: 3-run error bars (Bottom-Up mode). Mean std over three independent runs.
Table 28: 3-run error bars (Deep Research mode). Variance is higher due to non-deterministic retrieval, but ranking patterns are preserved.
| Metric | Encoder | o3 | Grok | Gem. | Pplx. | DS | Qwen | Doub. |
| US-TED | MiniLM | 2 | 1 | 5 | 7 | 3 | 6 | 4 |
| BGE-large | 2 | 1 | 5 | 7 | 3 | 6 | 4 | |
| Stella-1.5B | 2 | 1 | 5 | 7 | 3 | 6 | 4 | |
| US-NTED | MiniLM | 2 | 1 | 6 | 7 | 3 | 5 | 4 |
| BGE-large | 2 | 1 | 6 | 7 | 4 | 5 | 3 | |
| Stella-1.5B | 3 | 1 | 6 | 7 | 4 | 5 | 2 |
Table 29: Embedding sensitivity of structural metrics (Deep Research mode). Numbers are rank positions among 7 agents. US-TED and Sem-Path rankings are exactly identical across the three encoders.
| Input condition | ARI | Sem-Path | ||
| human | machine | human | machine | |
| +Core-task, No-Thinking | 25.5% | 25.5% | 28.0% | 28.7% |
| +Core-task, Thinking | 26.8% | 27.1% | 28.3% | 28.7% |
| +Summary, No-Thinking | 27.8% | 25.6% | 28.0% | 28.7% |
| +Summary, Thinking | 27.3% | 26.8% | 28.3% | 28.5% |
Table 30: Human-written input control on all 72 surveys, averaged over six models: Claude-Sonnet-4.5, GPT-5, Gemini-3-Pro, DeepSeek-V3.2, Qwen3-Max, and Kimi-K2. The human columns replace all machine-generated auxiliary text with hand-written text. The machine columns use Claude-Sonnet-4.6 as in Table 22 . Differences are at most 2.2 pp for ARI and 0.75 pp for Sem-Path .
Table 31: Multi-category paper sensitivity. Sem-Path after removing all expert-cited papers that the survey authors placed in multiple categories. Differences are within pp, confirming no systematic bias.
| Metric | Cross-domain avg. | AI/ML avg. |
| ARI | 23.93% | 27.25% |
| Sem-Path | 30.90% | 29.16% |
| Homogeneity | 85.35% | 82.27% |
Table 32: Cross-domain pilot. Claude-Sonnet-4.5 on two interdisciplinary surveys (finance, medical), Bottom-Up mode. The same qualitative pattern emerges as for AI/ML surveys.
Table 33: Deep Research re-test with current agents. Percentages over the 72 surveys, with failed runs scored as zero; Papers is the mean number of papers listed per survey. Over repeated runs on a 15-survey subset, the standard deviation of F1 is below 1 pp.
| Stage | # surveys retained |
| 0. Initial pool of LLM-related surveys | 200+ |
| 1. Taxonomy filter (explicit taxonomy figure) | 140 |
| 2. Impact filter (high citation count) | 110 |
| 3. Scope filter (well-defined survey scope) | 90 |
| 4. Expert verification (Ph.D. researcher review) | 72 |
Table 34: Survey selection pipeline (four-stage). Counts are after each filter; the final 72 surveys span 8 subdomains and have an average of 354.5 citations.
Figure 11: Prompt for generating a taxonomy tree in Bottom-Up mode.
Figure 12: The prompt for taxonomy tree generation based on paper summaries.
Figure 13: The prompt for taxonomy tree construction based on Core-Task and Contribution.
Figure 14: Prompt for Deep Research mode, as given to o3, Grok, and Perplexity, with {TOPIC} replaced by the survey title. App. R.2 lists the variants the other agents received.
Figure 15: Markdown variant of the Deep Research prompt given to Gemini, reproduced verbatim, including the sentence that names the field as LLM alignment for every survey.
Figure 16: The prompt for paper summary extraction.
Figure 17: The prompt for core-task extraction.
Figure 18: The prompt for contribution extraction.
Figure 19: The prompt for LLM-as-Judge evaluation on taxonomy trees.
Figure 19: The prompt for LLM-as-Judge evaluation on taxonomy trees. (Continued)
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,482 papers