RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents

Delin Wang; Sinuo Wang; Rui Sun; Beidi Luan; Qiming Shi; Jing Li; Daxin Jiang; Zuo Bai

cs.AIOct 27, 2026

RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents

Authors: Delin Wang, Sinuo Wang, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai

Organizations: StepFun · Renmin University of China · University of Adelaide · Zhejiang University ·

Abstract

Deep research agents conduct multi-step Web searches and synthesise retrieved information into analytical reports. Evaluating these outputs requires checking which analyses they deliver, beyond the presence of relevant facts. Query-specific binary rubrics make individual requirements explicit, but task-level checklists may leave the overall analytical structure underexamined. We introduce RubricAtlas, which derives type-level analytical requirements from expert reports on different objects within the same research type and organises detailed checks around recurring analytical questions. The requirements give related tasks a shared standard for what to analyse, without prescribing their answers. Our framework links binary checkpoints to report passages and evaluates completeness separately from argument validity. Built from 3,079 Chinese financial research reports, RubricAtlas covers 99 research types. Reference-report selection experiments show that individual reports cover only about half of their type's analytical questions, and that selecting requirements from different reports changes evaluation scores and agent rankings. Across ten agents, two judge pipelines achieve checkpoint agreement of $κ$ = 0.71 and agent-ranking agreement of Kendall $τ$ = 0.87. Controlled deletions reduce scores on the affected analyses, while padding produces no detectable increase. These findings support type-level rubrics for comparing the analytical completeness of Web deep research outputs, with individual decisions traceable to report evidence.

Figures & tables

Figure 1. Construction and evaluation. (a) Questions from expert reports on different firms are aligned and abstracted into type-level requirements. (b) Aligned report passages feed separate completeness and validity evaluations, yielding scores and analytical profiles. (c) An oil-and-gas question has three Targets; the peer-comparison Target is expanded into checkpoints. A named-peer comparison passes; an industry average triggers the negative criterion. Source questions are paraphrased; outcomes are schematic. Panel a shows questions about resource advantages and sales-network stability from different firms within one research type, aligned into a shared competitive-advantage question and abstracted into the RubricAtlas hierarchy. Panel b shows a sampled task producing a Web research report. A judge selects block IDs and a program retrieves original passages, which feed independent Target and General evaluations. Outputs are report completeness with question and Target profiles, and a separate violation profile. Panel c shows a consensus question supported by 14 of 21 reports from 13 institutions, with three Targets: capabilities and metrics, named-peer comparison, and persistence conditions. A dashed detail leader expands the second Target into three checkpoints. Schematic named-peer comparisons pass, while industry averages trigger a negative criterion that sets the Target score to zero.

Benchmark	Construction / reference basis	Scope
DeepResearch Bench ( Du et al., 2025 )	LLM-generated, with one reference report	task
DRB II ( Li et al., 2026 )	extracted from one expert report	task
ResearchRubrics ( Sharma et al., 2025 )	expert-written for each prompt	prompt
FinResearchBench ( Sun et al., 2025 )	report-level logic trees; shared structural metrics and LLM judgments	shared criteria
FinResearchBench II ( Luan et al., 2026 )	multi-system reports for one query; consistency and distinguishability filters	query
RubricAtlas (ours)	30 expert reports per type; 14 supporting reports and 9 institutions per question	research type

Table 1. Source aggregation in related evaluations. All RubricAtlas counts are medians.

Question tier	Questions	Reports	Institutions
Consensus	503	21	11
Differentiating	418	8	6
Multi-source support among differentiating questions
At least three supporting reports			96%
At least three supporting institutions			90%

Table 2. Multi-source question support. Report and institution counts are medians per question; percentages concern differentiating questions.

$k$	Cov.	$\tau$	$\tau$ (5th)	$\tau$ rand.	Gain	Gain rand.	Top-1 (rand.)
1	51%	0.69	0.51	0.78	4.6	0.7	24% (7%)
3	79%	0.79	0.64	0.89	2.0	0.3	0% (0%)
10	96%	0.95	0.87	0.97	0.3	0.0	0% (0%)

Table 3.

k

-report question subsets per type (200 draws; smaller pools use all reports). Cov.: mean union coverage over types/draws, not median per-report coverage.

\tau

: mean correlation with the full ranking; 5th: fifth percentile. Rand.: count-matched random questions, without tier matching. Gain: mean score increase (points). Top-1: fraction changing the overall leader.

Figure 2. Full-atlas coverage by

k

reports per type (a) and ranking Kendall

\tau

(b), with size-matched random questions. Shading spans the report draws’ 5th percentile to mean. Equally spaced ticks denote discrete

k

conditions; smaller pools use all reports. Two line plots show atlas coverage and ranking correlation as the number of supporting reports per type increases. The ranking plot includes a size-matched random question-subset comparison and a shaded band down to the fifth percentile of report-support draws.

Figure 3. Completeness diagnostics (levels on 0–100). (a) Ten agents’ report T , ordered by A; Kendall

\tau

summarises ranking agreement. (b) Mean completeness on 27 target questions before (solid) and after (pale) deletion. (c) Net report T changes after 40% irrelevant or duplicated text, with paired 95% intervals against unchanged re-evaluation. The dashed A length-slope reference is not an equivalence margin. A paired dot plot displays the same ten agents' completeness scores under judges A and B. A grouped bar plot compares mean target-question completeness before and after deletion. Four padding point estimates have paired confidence intervals, a zero line, and a dashed reference calculated from judge A's observed length slope.

	Judge A (DeepSeek-V4.1-Flash)						Judge B (GPT-6-luna)
Agent	T	$\pm$	Rank	P(1st)	Cons.	Diff.	T	$\pm$	Rank	P(1st)
Claude-Opus-5.5	59.5	4.3	1–2	92%	67.1	54.2	50.6	4.1	1–3	84%
GLM-5.3	57.5	4.1	1–6	5%	67.4	51.1	49.6	3.9	1–4	15%
Mimo-V2.6-Pro	57.0	4.3	2–7	1%	65.8	51.1	48.1	4.0	2–7	1%
Mimo-V2.6-Flash	57.0	4.6	2–7	2%	65.5	51.8	48.3	4.2	2–7	1%
DeepSeek-V4.1-Flash	56.5	4.3	2–7	1%	66.1	50.4	47.5	4.2	3–7	0%

Table 4. Completeness T and tier scores on 98 common tasks (0–100).

\pm

: 95% half-width from task bootstrap (2,000 resamples) and shared repeat-run variance. Rank: task-resampling 95% range; P(1st): first-place fraction. Cons./Diff.: consensus/differentiating scores. Rows follow unrounded judge-A means. Absolute levels depend on the judge.

Rule	A	B	Both	$\kappa$	$\tau$	$r$
	Question level				Agent level
G04	3,030	1,194	898	0.29	0.60	0.75
G06	1,516	1,362	567	0.28	0.22	0.49
G02	2,481	1,896	875	0.22	0.47	0.64
G07	1,138	711	271	0.22	0.64	0.91
G03	640	339	101	0.17	0.69	0.73

Table 5. Generic rules on 9,196 shared questions: A/B flags, overlap and Cohen’s

\kappa

. Agent level: flag-rate Kendall

\tau

and Pearson

r

(ten agents).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Scheme	Cons.:Diff.	$\tau$	First
Both tiers retained
Equal	1:1	0.96	Claude
More differentiating	1:3	0.96	Claude
More consensus	2:1	0.73	Claude
Inverse frequency	varies	0.91	Claude
Frequency	varies	0.78	Claude

Appendix

Table 6. Aggregation sensitivity under A. Ratios are consensus:differentiating;

\tau

compares with the main agent ranking.

Judge	Target/General effort	Tokens	$r$
A	low/none	5.8k	0.94
A	medium/low	8.1k	0.96
A	high/high	8.1k	0.96
B	medium/low	1.6k	0.93
B	high/high	3.5k	0.95

Appendix

Table 7. Settings on 100 reports. Tokens: mean reasoning tokens per Target call;

r

: report- T correlation with A’s main run on the paired sample.

Threat	Evidence	Residual risk
Circular coverage	Held-out 53%; out-of-half 56%; composition-matched control	Split halves reuse the atlas questions
Judge dependence	Two families: $\kappa$ = 0.71, $\tau$ = 0.87; re-run SD 4.1 points	Author audit is not independent validation
Length	Padding estimate $+$ 1.1 points; length-adjusted leader unchanged	Modest reward not excluded; matched and adjusted rankings differ
Self-preference	DeepSeek shift net of length $+$ 0.4 [ $-$ 1.4, 2.1]	One agent is also a judge
Leakage	1.7% of reads; no detected source-title exposure gain	Search API has no date filter
Frequency $\neq$ importance	Model-screened, author-verified depth; weights fixed before agent results; same leader under five weightings	Weights remain a design choice

Appendix

Table 8. Main threats to validity, the evidence we report and the residual risk.

Figure 4. Tier completeness under both judges on 98 common tasks. Paired means are computed within reports, then averaged over tasks. Panels share a score axis and follow Table 4 ’s judge-A ordering. Two horizontal paired-dot panels show ten agents under judges A and B. Open circles indicate consensus completeness and filled squares indicate differentiating completeness. Grey lines pair the two tier means for each agent. Both panels use the same agent order and an axis from 30 to 70 on the 0 to 100 score scale. The figure displays descriptive means without confidence intervals or significance markers.

Procedure	Questions retained	Tiers
Leave-one-out	Remove the measured report’s support; drop questions below 3 reports	Not used
Split-half	Support from at least two reports in the half (three in the variant)	Within-half
Source removal	Remove task-source support; drop questions below 3 reports	Fixed
$k$ reports	Supported by at least one of $k$ randomly sampled reports	Fixed
Task source	Supported by the report used to construct the task	Fixed

Appendix

Table 9. Reference-report procedures. Fixed: original tiers retained. Split-half recomputes tiers from within-half support.

Author / model labels	Judge A	Judge B
Yes / yes	125	121
Yes / no	52	56
No / yes	25	29
No / no	98	94
Both judges yes ( $n=100$ )	91%	91%
Both judges no ( $n=100$ )	78%	78%

Appendix

Table 10. First-author checkpoint audit. Confusion counts use the author’s label as the row and model label as the column, not as an adjudicated ground truth. The lower panel reports agreement within sampling strata.

Agent Evaluation

Agent evaluation measures how reliably AI systems act, use tools, and complete tasks across interactive environments.

See all 144 papers

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Jan 13, 2026Ruizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research Agents LLM Agent Evaluation
University of Science and Technology of China · Metastone Technology, Beijing, China
FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality
Jul 14, 2026Beidi Luan, Rui Sun, Sinuo Wang +5LLM-as-a-Judge Rubric-Based Evaluation
1StepFun · University of Adelaide · 2FinStep +1
DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents
Jun 15, 2026Minghang Zhu, Chuyang Wei, Junhao Xu +3Deep Research Agents Rubric-Based RL
Shandong University, Qingdao, China · Zhongguancun Academy, Beijing, China · Fudan University, Shanghai, China

Jan 13, 2026J/K move · Enter open · S save

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

Ruizhe Li, Mingxuan Du, Benfeng Xu, Chiwei Zhu+2

University of Science and Technology of China · Metastone Technology, Beijing, China

Deep Research Agents LLM Agent Evaluation AI Agent Benchmarks Rubric-Based Evaluation Automated Evaluation

Open paper AskPDF arXiv

Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmarks often either under-evaluate a system's ability to produce meaningful insights and high-quality writing, or adopt coarse or LLM-defined criteria that are hard to verify and can diverge from human expert judgment. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRAs. It contains 132 grounded research tasks across 22 domains; for each task, an agent must produce a research report that is evaluated by a set of 9,430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are verifiable and aligned with human expert judgment. We evaluate several state-of-the-art deep-research agents on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRAs and human experts. We release the benchmark, evaluation scripts, and all rubrics at https://github.com/imlrz/DeepResearch-Bench-II to facilitate future research on deep-rearch agents.

Abstract

Agent Evaluation

Explore similar work

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents