Decision-Centric Research: A Calibrated Decision Model and Benchmark for Scientific Agents
Abstract
Scientific agents make consequential choices about evidence, next actions and research strategy, but choices embedded in generated text are difficult to evaluate independently. Decision-Centric Research (DCR) is introduced to make individual choices the common unit of supervision, prediction and evaluation through typed predictions over an explicit Research State. DCR defines 24 decision types, ten of which are instantiated in the Research Decision Benchmark (RD-Bench) of 2,000 rule-generated cases. The Decision-Centric Research Model (DCRM) shares a candidate-token readout across types to produce answers and option probabilities in one forward pass. Under a held-out research-domain split, a 0.6B DCRM (state-aware) achieves 89.4% Top-1 accuracy, compared with 27.8% for DCRM (question-only) trained with the same recipe and 36.4% for DeepSeek-V4.1-Flash. All 30 type-seed comparisons against DCRM (question-only) are significant after Holm correction. After held-out calibration, the resulting prediction sets contain a mean of 1.515 candidates at 97.1% empirical label coverage, reducing mean set size by 57.4% relative to DCRM (question-only) at similar coverage. These results establish a compact state-aware decision layer with accurate typed prediction and an inspectable uncertainty interface.
Figures & tables
| Category | Guiding question | Decision types |
|---|---|---|
| A. State understanding | What is the situation? | A1 Research stage (7-way label), A2 State completeness (4 levels), A3 Primary objective, A4 Open-problem ranking |
| B. Evidence and validity | Is the evidence adequate and sound? | B1 Evidence sufficiency (3 levels), B2 Claim-evidence consistency (4 levels), B3 Experiment validity, B4 Result anomaly, B5 Reproducibility risk |
| C. Planning and next action | What should happen next? | C1 Next action (4 candidates), C2 Action priority, C3 Plan continuation vs. revision, C4 Stopping condition (3 levels) |
| D. Research strategy | Should the direction change? | D1 Direction decision (4-way label), D2 Hypothesis revision, D3 Scope change, D4 Methodology paradigm |
| E. Agent control | How should the agent operate? | E1 Tool selection, E2 Model routing (3 tiers), E3 Self-verification need, E4 Agent continuation vs. termination |
| F. Human and advisor escalation | When should a person be involved? | F1 Advisor review need (3 levels), F2 User confirmation need (binary), F3 Expert consultation trigger |
| DCRM (state-aware) | DCRM (question-only) | DeepSeek- V4.1-Flash | Gain | |||
|---|---|---|---|---|---|---|
| Decision type | Top-1 | Top-3 | Top-1 | Top-3 | Top-1 | (pp) |
| Research stage | 0.767 | 0.867 | 0.107 | 0.327 | 0.260 | +66.0 |
| State completeness | 0.687 | 0.947 | 0.187 | 0.820 | 0.440 | +50.0 |
| Evidence sufficiency | 1.000 | 1.000 † | 0.353 | 1.000 † | 0.340 | +64.7 |
| Claim-evidence consistency | 0.827 | 0.967 | 0.227 | 0.793 | 0.240 | +60.0 |
| Next action | 0.720 | 0.933 | 0.273 | 0.720 | 0.340 | +44.7 |
| DCRM (state-aware) vs DCRM (question-only) | DCRM (state-aware) vs DeepSeek-V4.1-Flash | |||||
|---|---|---|---|---|---|---|
| DCRM training seed | DCRM training seed | |||||
| Decision type | 42 | 43 | 44 | 42 | 43 | 44 |
| Research stage | ||||||
| State completeness | ||||||
| Evidence sufficiency | ||||||
| Claim-evidence consistency | ||||||
| DCRM (question-only) | DeepSeek-V4.1-Flash | |||
|---|---|---|---|---|
| Seed | Diff. | 95% CI | Diff. | 95% CI |
| 42 | +0.580 | [0.517, 0.643] | +0.490 | [0.413, 0.563] |
| 43 | +0.663 | [0.603, 0.720] | +0.573 | [0.500, 0.640] |
| 44 | +0.573 | [0.507, 0.637] | +0.487 | [0.410, 0.560] |
| Decision type | Signal | Accuracy | ECE | Brier | AUROC |
|---|---|---|---|---|---|
| Evidence sufficiency | softmax | 0.860 | 0.055 | 0.089 | 0.882 |
| Evidence sufficiency | P(IK) | 0.860 | 0.153 | 0.159 | 0.791 |
| Next action | softmax | 0.600 | 0.226 | 0.286 | 0.542 |
| Next action | P(IK) | 0.600 | 0.238 | 0.295 | 0.638 |
| Research stage | softmax | 0.300 | 0.371 | 0.325 | 0.743 |
| Research stage | P(IK) | 0.300 | 0.339 | 0.332 | 0.585 |
| Variant | Research stage | Evidence sufficiency | Next action |
|---|---|---|---|
| DCRM (state-aware) | 0.347 | 0.760 | 0.633 |
| History-ablated DCRM | 0.373 | 0.727 | 0.593 |
| DCRM (question-only) | 0.247 | 0.447 | 0.440 |
| Model | Format | Top-1 | API end-to-end (s) | Local GPU forward (ms) | Completion tokens |
|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash, 2 runs | plain | 0.347 / 0.327 | 4.4–4.9 | — | 790–905 |
| DeepSeek-V4.1-Flash | structured | 0.340 | 3.9 | — | 700 |
| DeepSeek-V4.1-Flash | plain | 0.320 | 5.2 | — | 870 |
| DCRM (state-aware), 0.6B | structured state | 0.580 | — | 13.1 | — |
| Model size | Top-1 | Seeds | Forward time (ms) | Peak GPU memory (GiB) |
|---|---|---|---|---|
| 0.6B | 0.580 | 3 | 13.1 | 1.3 |
| 1.7B | 0.589 | 3 | 14.4 | 3.4 |
| 4B | 0.573 | 1 | 26.0 | 7.7 |
| Model | Assumed unit price | Cost per decision ( USD) |
|---|---|---|
| DCRM (state-aware), 0.6B | 0.35 USD/GPU-hour | 1.278 |
| DeepSeek-V4.1-Flash | 1.10 USD per million completion tokens | 1057 |