Scientific agents make consequential choices about evidence, next actions and research strategy, but choices embedded in generated text are difficult to evaluate independently. Decision-Centric Research (DCR) is introduced to make individual choices the common unit of supervision, prediction and evaluation through typed predictions over an explicit Research State. DCR defines 24 decision types, ten of which are instantiated in the Research Decision Benchmark (RD-Bench) of 2,000 rule-generated cases. The Decision-Centric Research Model (DCRM) shares a candidate-token readout across types to produce answers and option probabilities in one forward pass. Under a held-out research-domain split, a 0.6B DCRM (state-aware) achieves 89.4% Top-1 accuracy, compared with 27.8% for DCRM (question-only) trained with the same recipe and 36.4% for DeepSeek-V4.1-Flash. All 30 type-seed comparisons against DCRM (question-only) are significant after Holm correction. After held-out calibration, the resulting prediction sets contain a mean of 1.515 candidates at 97.1% empirical label coverage, reducing mean set size by 57.4% relative to DCRM (question-only) at similar coverage. These results establish a compact state-aware decision layer with accurate typed prediction and an inspectable uncertainty interface.
Figures & tables
Fig. 1: Decision-Centric Research models scientific choices as state-conditioned, typed predictions. (a) Research State and the decision taxonomy define the context and candidate space. (b) A shared language model with Low-Rank Adaptation (LoRA) produces a candidate-token distribution in one forward pass, supporting both decision selection and calibrated prediction sets. (c) The Research Decision Process (RDP) connects decisions, actions, evidence and persistent state updates. Dashed arrows indicate the RDP feedback path.
Category
Guiding question
Decision types
A. State understanding
What is the situation?
A1 Research stage (7-way label), A2 State completeness (4 levels), A3 Primary objective, A4 Open-problem ranking
E1 Tool selection, E2 Model routing (3 tiers), E3 Self-verification need, E4 Agent continuation vs. termination
F. Human and advisor escalation
When should a person be involved?
F1 Advisor review need (3 levels), F2 User confirmation need (binary), F3 Expert consultation trigger
TABLE 1: Research decision taxonomy with six categories and 24 types, organized by their guiding question.
DCRM (state-aware)
DCRM (question-only)
DeepSeek- V4.1-Flash
Gain
Decision type
Top-1
Top-3
Top-1
Top-3
Top-1
(pp)
Research stage
0.767
0.867
0.107
0.327
0.260
+66.0
State completeness
0.687
0.947
0.187
0.820
0.440
+50.0
Evidence sufficiency
1.000
1.000 †
0.353
1.000 †
0.340
+64.7
Claim-evidence consistency
0.827
0.967
0.227
0.793
0.240
+60.0
Next action
0.720
0.933
0.273
0.720
0.340
+44.7
TABLE 2: Decision accuracy and candidate ranking across ten types
Fig. 2: Decision accuracy across the complete ten-type suite. DCRM (state-aware) and DCRM (question-only) share their base model and training recipe. Error bars show the standard error over three training seeds, whereas DeepSeek-V4.1-Flash was evaluated in one API evaluation run on the same cases. See Table 1 for type definitions and benchmark answer spaces.
DCRM (state-aware) vs DCRM (question-only)
DCRM (state-aware) vs DeepSeek-V4.1-Flash
DCRM training seed
DCRM training seed
Decision type
42
43
44
42
43
44
Research stage
6.93×10−7
1.49×10−9
4.19×10−9
6.39×10−3
5.01×10−6
1.42×10−4
State completeness
1.18×10−6
6.44×10−6
3.71×10−5
0.0541
0.115
0.132
Evidence sufficiency
1.49×10−8
1.34×10−9
1.49×10−8
5.12×10−9
5.12×10−9
5.12×10−9
Claim-evidence consistency
3.71×10−5
7.57×10−10
3.71×10−5
2.53×10−7
8.15×10−10
9.54×10−7
TABLE 3: Holm-adjusted McNemar p-values for state-aware DCRM comparisons
DCRM (question-only)
DeepSeek-V4.1-Flash
Seed
Diff.
95% CI
Diff.
95% CI
42
+0.580
[0.517, 0.643]
+0.490
[0.413, 0.563]
43
+0.663
[0.603, 0.720]
+0.573
[0.500, 0.640]
44
+0.573
[0.507, 0.637]
+0.487
[0.410, 0.560]
TABLE 4: Paired accuracy differences and confidence intervals
Fig. 3: DCRM (state-aware) produces smaller prediction sets than DCRM (question-only) at similar overall empirical label coverage. (a) Mean set cardinality by decision type. (b) Empirical ground-truth label inclusion, with the dashed line marking nominal coverage 0.95. Each type has 50 test cases, and metrics average three training seeds. The highlighted Overall results average all ten types using unrounded values.
Decision type
Signal
Accuracy
ECE
Brier
AUROC
Evidence sufficiency
softmax
0.860
0.055
0.089
0.882
Evidence sufficiency
P(IK)
0.860
0.153
0.159
0.791
Next action
softmax
0.600
0.226
0.286
0.542
Next action
P(IK)
0.600
0.238
0.295
0.638
Research stage
softmax
0.300
0.371
0.325
0.743
Research stage
P(IK)
0.300
0.339
0.332
0.585
TABLE 5: Confidence quality and correctness ranking
Variant
Research stage
Evidence sufficiency
Next action
DCRM (state-aware)
0.347
0.760
0.633
History-ablated DCRM
0.373
0.727
0.593
DCRM (question-only)
0.247
0.447
0.440
TABLE 6: State-input ablation on three decision types
Model
Format
Top-1
API end-to-end (s)
Local GPU forward (ms)
Completion tokens
DeepSeek-V4.1-Flash, 2 runs
plain
0.347 / 0.327
4.4–4.9
—
790–905
DeepSeek-V4.1-Flash
structured
0.340
3.9
—
700
DeepSeek-V4.1-Flash
plain
0.320
5.2
—
870
DCRM (state-aware), 0.6B
structured state
0.580
—
13.1
—
TABLE 7: Decision quality and runtime across inference configurations
Model size
Top-1
Seeds
Forward time (ms)
Peak GPU memory (GiB)
0.6B
0.580
3
13.1
1.3
1.7B
0.589
3
14.4
3.4
4B
0.573
1
26.0
7.7
TABLE 8: Model size, accuracy and resource footprint
Model
Assumed unit price
Cost per decision ( 10−6 USD)
DCRM (state-aware), 0.6B
0.35 USD/GPU-hour
1.278
DeepSeek-V4.1-Flash
1.10 USD per million completion tokens
1057
TABLE 9: Component costs under fixed accounting coefficients
Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmarks often either under-evaluate a system's ability to produce meaningful insights and high-quality writing, or adopt coarse or LLM-defined criteria that are hard to verify and can diverge from human expert judgment. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRAs. It contains 132 grounded research tasks across 22 domains; for each task, an agent must produce a research report that is evaluated by a set of 9,430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are verifiable and aligned with human expert judgment. We evaluate several state-of-the-art deep-research agents on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRAs and human experts. We release the benchmark, evaluation scripts, and all rubrics at https://github.com/imlrz/DeepResearch-Bench-II to facilitate future research on deep-rearch agents.
Ruizhe Li, Mingxuan Du, Benfeng Xu +3
University of Science and Technology of China · Metastone Technology, Beijing, China
Frontier deep research agents (DRAs) are being deployed in enterprise workflows faster than they are being evaluated. Existing benchmarks measure factual recall, single-hop QA, or generic agentic skill, and miss the multi-document, decision-grade deliverables DRAs are asked to produce. We introduce a benchmark of 70 SME-authored management consulting prompts, each embedding cognitive traps that penalize surface-pattern reasoning. Three frontier agents, namely Claude Opus4.6, OpenAI o3-deep-research and Gemini3.1Pro deep-research, are scored on two complementary layers: deterministic binary verifiers (mean 14.9 per task) and a five-criterion 0--3 SME rubric (Data Integrity, Analytical Rigor, Relevance & Focus, Execution Precision, Format & Deliverability), combined into a Verifier-Rubric Score (VRS, 0--100). Acceptance under a joint threshold (rubric mean ≥2.5 and verifier pass rate ≥80%) is uniformly low: o3 15.7%, Claude 12.9%, Gemini 12.9%. Pairwise differences are statistically indistinguishable. On the continuous VRS, o3 leads (61.4[CI: 55.2,,67.5]), followed by Gemini (52.6) and Claude (38.5); the o3--Claude gap (Δ=22.9, p<0.001) survives Bonferroni correction. No agent averages above the rubric's ``adequate'' threshold of 2.0; no agent's mean verifier pass rate reaches the 80% acceptance floor. Each agent fails distinctively: Claude leads on data fabrication and file-access failures; o3 propagates cascading computation errors; Gemini oscillates between the highest perfect-verifier rate and the most catastrophic collapses. The benchmark, evaluation code, and full prompt corpus are publicly released.
AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence. ForeSci contains 500 tasks across four fast-moving AI domains and four decision families. Each task is paired with a cutoff-aligned offline knowledge base; post-cutoff papers are hidden during generation and used only for validation. To avoid random future-event prediction, tasks are derived from pre-cutoff taxonomy branches and evidence signals, and answer-generation backbones are selected to precede the task cutoffs. We evaluate native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones. Results show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family. Diagnostics reveal a recurring evidence-decision decoupling: agents may cite relevant evidence while forecasting the wrong research object. ForeSci turns forward-looking AI research judgement into a controlled benchmark for evaluating research agents as decision-making systems.
Qiuyu Tian, Haojie Yin, Yingce Xia +2
1Southeast University, Nanjing, China · 3Duke Kunshan University, Kunshan, China · 2Beijing Zhongguancun Academy, Beijing, China