Organizations: Shanghai Artificial Intelligence Laboratory, Fudan University, East China Normal University, Shanghai Jiao Tong University, Renmin University of China, Peking University
Research agents achieve increasingly strong benchmark scores, yet these scores do not necessarily translate into reliable delivery in ongoing research projects. Many research-agent benchmarks emphasize standalone tasks and relatively self-contained workflows in clean or curated workspaces. However, researchers ask agents to complete one step within an ongoing project, where requests depend on earlier session decisions and workspaces contain unfinished files, outdated versions, and failed attempts. We introduce ARE-bench to evaluate whether research agents advance ongoing projects while respecting prior decisions and existing workspace state, grounded in real session context and in-progress workspaces. ARE-bench includes 75 tasks built on agent sessions and full workspaces from recruited researchers' own projects, spanning seven research stages, 33 requirements, and 150 sub-requirements, with 1,173 binary checklist items in total. Each task is anchored at a real user request, preserves its requirements, and summarizes the earlier requests still in effect as background. The complete workspace is replayed to that boundary without cleanup, preserving artifacts from earlier turns. Each checklist is compiled from an intent-aligned rubric and credits any valid solution. Across seven frontier models, the strongest passes every checklist item on only 66.2% of tasks, and Design and Execution rank among the hardest stages for every model. Three models pass about 69% of tasks when only delivery contracts and general norms are checked, but just 28.4% to 40.0% once historical and material or workspace requirements are also checked. Evaluating delivery without these contextual requirements can therefore overestimate research agents and mask differences between them.
Figures & tables
Figure 1: ISR of seven models on 75 tasks, averaged over three rounds. Left: overall ISR, with error bars for the sample standard deviation across rounds. Right: ISR by research stage, with the radial axis from 35% to 95%.
Figure 2: An illustrative contrast between a standalone research task and one real research session in ARE-bench. Top: the standalone task has an end-to-end objective, while the real session moves back and forth between research stages. The task is cut at request T31, with still-effective requests from T1 to T30 summarized in Background. Bottom: a clean or curated starting repository is contrasted with the replayed in-progress workspace, including superseded drafts, misleadingly named files, and paused runs.
Benchmark
Real request
Session-derived
In-progress workspace
Research workflow
Scoring
SWE-bench ( Jimenez et al., 2023 )
✔
✘
✘
✘
Tests
Terminal-Bench (2.0) ( Merrill et al., 2026 )
✔
✘
✔
✘
Varies
OSWorld ( Xie et al., 2024 )
✔
✘
✔
✘
Varies
τ -bench ( Yao et al., 2024 )
✔
✘
✔
✘
Varies
MLE-bench ( Chan et al., 2024 )
✘
✘
✘
✘
Task metric
ScienceAgentBench ( Chen et al., 2024 )
✘
✘
✔
✘
Varies
Table 1: Comparison of benchmark designs for research-project continuation. Real request denotes a documented genuine work need, regardless of source. Session-derived denotes a task or request whose provenance is an actual recorded work session; earlier conversation transfer is optional. In-progress workspace denotes task-relevant files or application state containing prior activity or intermediate progress available when evaluation starts; a base repository, raw data, dependency environment, or empty scaffold alone does not qualify. Research workflow denotes suite-level coverage of Survey, Design, Data & Environment, Code, Execution, Analysis, and Writing. Scoring names the primary acceptance basis: Tests, State check, Answer check, Task metric, Rubric, a stable combination written with ‘+‘, or Varies when mechanisms differ across tasks or sub-benchmarks. A tick marks a documented feature and a cross means it is not established under these definitions.
Figure 3: Overview of ARE-bench construction, illustrated by a paper restructuring task cut at request T31 of a 42-turn session. (a) The Curator normalizes raw researcher sessions, verifies requirements against the full interaction, labels them with the requirement taxonomy, and selects sessions by rules S1 to S4. (b) The Builder compiles the target request and still-valid history into the actor-visible instruction, replays the workspace to that request, and compiles a hidden rubric into a binary checklist. (c) The Validator calibrates the judge, verifies the task package, runs actors, and sends each task to human review, which returns failing tasks to the Builder.
Research stage
Requirements
Sub-requirements
Checklist items
Tasks
Survey
2
20
111
26
Design
3
25
268
41
Data & Env.
5
16
45
20
Code
5
23
217
32
Execution
7
26
173
33
Analysis
4
13
92
34
Table 2: Observed requirement coverage across the 75 tasks. Tasks count distinct tasks within each stage, and the total counts their union.
Evaluated model
Archived model identifier
Actor harness
GPT-6.1-Sol
gpt-6.1-sol
Codex
Claude Opus 5.5
claude-opus-5-5
Claude Code
DeepSeek-V4.1-Flash
deepseek-v4.1-flash
Claude Code
Qwen3.8-Max
qwen3.8-max
Claude Code
GLM-5.3-Flash
glm-5.3-flash
Claude Code
Kimi-K3
kimi-k3
Claude Code
Table 3: Evaluated models and actor harnesses. Model identifiers follow the main evaluation archive. The specific historical GLM deployment identifiers are retained in individual run records.
Task batch
Tasks
CPUs
Memory (MB)
GPUs
Configured actor time limit
CPU tasks
67
2–16
4,096–49152
0
30 minutes–6 hours
GPU tasks
8
8–32
32,768–262,144
1–4
2, 3, 4, 6, or 48 hours, depending on the task
Table 4: Declared task resources and configured time limits. CPU, memory, and GPU counts come from the task configuration bound to each selected source. Time limits are calculated from overrides and multipliers in the run configuration and specify the actor execution budget. The physical hosts and GPU models remain to be verified.
Table 8
Model
Completeness ISR
Correctness ISR
Constraint compliance ISR
Failed runs with correctness errors
GPT-6.1-Sol
83.11 ± 5.39
71.11 ± 5.05
93.78 ± 0.77
65/76 (85.5%)
Claude Opus 5.5
87.56 ± 3.08
62.67 ± 3.53
89.33 ± 4.62
84/97 (86.6%)
DeepSeek-V4.1-Flash
82.67 ± 2.67
47.11 ± 1.54
88.89 ± 5.05
119/136 (87.5%)
Qwen3.8-Max
79.11 ± 2.04
43.56 ± 8.57
84.89 ± 0.77
127/140 (90.7%)
GLM-5.3-Flash
80.44 ± 0.77
39.11 ± 3.08
81.78 ± 1.54
137/156 (87.8%)
Kimi-K3
76.44 ± 1.54
34.22 ± 2.04
83.56 ± 1.54
148/162 (91.4%)
Table 6: ISR by completeness, correctness, and constraint compliance. Each dimension requires all its applicable checklist items to pass. The final column reports the count and share of failed runs containing correctness errors. Error categories can overlap. Darker blue indicates higher ISR on a shared quadratic percentage scale; darker orange indicates a greater share of failures with correctness errors.
Failure manifestation
Effect on research delivery
Archived example
Task degradation
A simpler implementation replaces the original method, retaining its name while changing the required capability
In the round 1 DeepSeek repair of CylindFormer, the backbone named KPConv applies only Linear, LayerNorm, and ReLU to individual points, without neighborhood or kernel-point aggregation, failing O4
Insufficient investigation and evidence checking
Cited work does not support the mechanism or subject discussed, leaving conclusions without reliable evidence
In the round 1 Qwen3.8-27B revision of a cuproptosis review, ferroptosis-resistance research is described as a cuproptosis mechanism and cancer subtypes are conflated, failing O6
Requirement omission
Existing constraints are not implemented, affecting subsequent use despite local functional progress
In the round 2 DeepSeek adaptation of Graph RAG, datasets share a global index directory instead of isolated indexes, failing O11
Inadequate validation design
Missing comparison rules prevent attribution of improvements to the factor under study
In the round 1 GLM-5.3-Flash plan for 4D spacetime tubes, costs are recorded but executable rules for matching inference budgets across comparison arms are absent, failing O6
Conclusions exceeding the evidence
A limited diagnostic result is extended into an untested causal explanation
In the round 2 GLM-5.3-Flash diagnosis of network-security generalization, adversarial validation is treated as a test of conditional distributions, and a failure to detect differences is attributed to a single cause, failing O13
Table 7: Failure manifestations and archived examples. These selected qualitative examples explain why checklist items fail. They do not constitute a uniform classification or frequency analysis of all failures.
Model
Delivery contracts + general norms
Add historical and material or workspace requirements
Full checklist
DeepSeek-V4.1-Flash
69.33
40.00
39.56
GLM-5.3-Flash
69.33
31.56
30.67
Kimi-K3
69.78
28.44
28.00
Table 8: ISR (%) as the scope of checklist evaluation expands. Every model covers the same 75 tasks with three rollouts per task, and values are three-round means. Each column requires all applicable items within its scope to pass, reusing the same deliveries and item verdicts. Delivery contracts and general norms correspond to contract + norm . The second scope adds inherited + anchored , while the full checklist also includes latent items discovered through subsequent rework but inferable at the task cut point. Darker blue indicates higher ISR on a shared quadratic percentage scale.
Model
Survey
Design
Data & Env.
Code
Execution
Analysis
Writing
GPT-6.1-Sol
88.46 ± 6.66
73.17 ± 4.88
90.00 ± 0.00
85.42 ± 9.55
76.77 ± 7.00
86.27 ± 6.12
80.65 ± 5.59
Claude Opus 5.5
84.62 ± 3.85
76.42 ± 2.82
85.00 ± 5.00
77.08 ± 7.86
66.67 ± 3.03
79.41 ± 2.94
64.52 ± 11.63
DeepSeek-V4.1-Flash
79.49 ± 4.44
62.60 ± 2.82
95.00 ± 0.00
71.88 ± 5.41
61.62 ± 4.63
76.47 ± 0.00
63.44 ± 6.72
Qwen3.8-Max
78.21 ± 4.44
58.54 ± 7.32
83.33 ± 12.58
67.71 ± 6.51
50.51 ± 4.63
60.78 ± 1.70
67.74 ± 5.59
GLM-5.3-Flash
76.92 ± 3.85
56.91 ± 6.14
85.00 ± 5.00
63.54 ± 1.80
59.60 ± 1.75
71.57 ± 6.12
58.06 ± 5.59
Kimi-K3
70.51 ± 4.44
47.97 ± 2.82
86.67 ± 7.64
73.96 ± 4.77
47.47 ± 3.50
64.71 ± 0.00
47.31 ± 3.72
Table 9: ISR across seven research stages. A task passes a stage only if all its applicable checklist items in that stage pass. Tasks without items in the stage are excluded from its denominator. Values are three-round means ± sample standard deviations, and one task may involve multiple stages. Darker blue indicates higher ISR on a shared quadratic percentage scale.
Research stage
Tasks covered
Checklist items
Seven-model mean stage ISR
Best-to-worst model gap
Survey
26
111
77.66
23.08
Design
41
268
59.47
35.77
Data & Env.
20
45
86.43
15.00
Code
32
217
71.28
26.04
Execution
33
173
57.43
37.37
Analysis
34
92
71.15
27.45
Table 10: Stage coverage and descriptive difficulty. The seven-model mean weights models equally, and the best-to-worst gap is in percentage points. Task counts indicate stage coverage in the benchmark, while effective denominators follow the scored records in each round. Darker blue indicates higher mean ISR, and darker orange indicates larger model gaps, on a shared quadratic percentage scale.
Position relative to the complete session midpoint
CSR (%)
Before the midpoint, original segment 2
95.92
Near the midpoint, original segment 3
91.63
After the midpoint, original segment 4
97.66
Table 11: CSR at and around the midpoint of the complete session. Each of 10 sessions is divided into five consecutive segments by user-turn count. We show the original segments 2–4, each containing 10 tasks, for 30 tasks in total. GLM-5.3 runs on the instruction and restored workspace at each cut point, and results weight tasks equally. Blue intensity is scaled over the displayed CSR range, with darker cells indicating higher values.
Figure 5: Mean peak context, estimated cost, and ISR per run. Peak context is the largest single API call input within a run, and K denotes thousands. Estimates apply the manuscript’s frozen prices dated 2026-11-02 to the measured uncached input, cache reads, cache writes, and output tokens of every call, covering model API charges. Cache writes use the ordinary input price, except that Claude Opus 5.5 uses its cache-write price of USD 5 per million tokens. DeepSeek’s context and cost use 222 runs with token records, while its ISR uses all 225 scored runs. Other models use their available records.
DeepSeek verdict / Qwen verdict
Pass
Fail
Total
Pass
8,043
67
8,110
Fail
648
1,116
1,764
Total
8,691
1,183
9,874
Table 12: DeepSeek and Qwen binary verdicts on matched inputs and checklist items. Starting from 657 matched scoring pairs, matching checklist-item IDs and content hashes yields 9,874 paired binary verdicts. The judges are deepseek-v4.1-flash-1017 and qwen3.8-flash . These results cover a subset of tasks. Darker blue indicates larger joint counts; marginal totals use neutral shading.
Figure 6: Cross-judge CSR ranks on the 20-task common-coverage subset. The columns show the historical default GLM reference, DeepSeek judge, and Qwen judge. Nodes use model logos and short names, and the same actor model is linked by the same color. More stable ranks produce more horizontal lines. The subset contains 147 actor trajectories, 21 per model. Scores first average included rounds within each task and then weight tasks equally. Current DeepSeek and Qwen inputs match, while historical default scores are a reference only. Tasks have different numbers of included rounds, and the subset retains the stage package’s source selection and policy of deferring low-scoring sources.
Evaluation aspect
Successful execution, 20/20
Execution with unmet items, 16/20
Training progress
Sustained nonzero loss and parameter updates
Also sustained training, nonzero loss, and parameter updates
O7, fusion coordinates
Projection indexes agree with the actual feature grid
Inconsistent coordinates clip some samples to boundary columns
O9, overlap labels
Separate 2D and 3D overlap checks retain positive, negative, and ignored regions
A single projection ratio replaces independent 2D and 3D overlap tests
O11, coarse matching loss
L2 distance, overlap weights, and specified settings are retained
Squared distance replaces L2 without corresponding boundary-parameter changes, and weights are altered
O12, fine matching loss
Bidirectional loss and the coarse/fine loss combination are retained
The changed distance and weight expressions are reused
Training and delivery items
Archived review accepts checkpoints, limited validation, and the report
The other 16 items pass, including training recovery, checkpoints, validation, and the report
Table 13: Two executions of the CylindFormer training-repair task. The task ID is cylindformer-training-repair . Both executions start at the same repair boundary and must resume training according to the reference paper. Scores follow the archived review. Appendix E contains the complete inputs, restored workspace, and 20-item checklist.
Figure 7: Divergent executions of the same training-repair request. The comparison concerns method consistency during recovery. Differences in final registration accuracy still require separate measurement.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Research stage
Requirement
Sub-requirement
Capability primitives
Work object
Search for relevant literature
Apply: implement; Analyze: differentiate
Evidence
Verify bibliographic information
Evaluate: critique
Evidence
Extract key points from literature
Analyze: differentiate; Understand: summarize
Evidence
Verify literature content
Evaluate: check
Evidence
Assess literature quality
Evaluate: critique
Evidence
Establish evidence traceability
Analyze: organize; Evaluate: check
Evidence
Appendix
Table 14: Research requirement taxonomy represented in the benchmark. Each row describes a sub-requirement, its parent requirement and research stage, and the associated capability primitives and work object.
Research stage
Representative starting workspace state Wc
Survey
Partial reading notes and a candidate research question, with no experiment code yet written
Design
An existing problem statement and prior baseline results that the new design must extend or contrast
Data & Env.
A partially installed environment and raw or partially downloaded data left from an earlier setup attempt
Code
A repository with a partial implementation, stub functions, and tests that do not yet pass
Execution
Configured run scripts together with logs or checkpoints from an interrupted or failed run
Analysis
Raw experiment outputs and an unfinished or outdated analysis script
Appendix
Table 15: Representative starting workspace state Wc for a task cut at each research stage. Every state is replayed from the source session rather than authored for the task, so earlier work is present regardless of stage.
Figure 8: Session with all six user turns and brief assistant excerpts. The star marks the cut after the repair request. Later turns appear below the boundary and are excluded from task construction.
Figure 9: Task inputs and assessment. Background, Requirements, and Environment form instruction.md ; the other two input regions describe the provided files and execution resources. Paths are normalized. The complete outcome rubric appears in Table 16 .
Figure 10: Directory view of the starting inputs: all 18 project files and four materials files, plus separately prepared data and model resources. Repeated raw-data samples use a filename pattern; weight entries name their models. The output directory is the requested handoff destination.
Figure 11: Two independent executions of the same task. Green and red backgrounds mark the decisive findings. The scores summarize archived outcome assessments; the comparison establishes method consistency or deviation, rather than a measured change in final registration accuracy.
ID
Criterion
Source
O1
Deliver a reusable repaired project in /workspace/output/, with an index locating source code, configuration, and reproduction entry points. Internal filenames and project organization are unrestricted.
Explicit; Requirements
O2
Construct image–point-cloud training pairs with real relative poses and filter them using the paper’s at-least-50% overlap rule. Pairing only an image with its own back-projected depth, or declaring min_overlap without computing and applying it, does not meet this check. Verifiable equivalent pairing is accepted; unavailable paper-internal pair IDs need not be restored.
Inferred; paper
O3
Separate training and evaluation according to the official 7-Scenes sequence splits. Validation must not train the model or select it using test performance. When the paper’s internal validation pairs are unavailable, an independent validation set constructed from training sequences is accepted if its scope is explained.
Inferred; paper
O4
The point-cloud backbone must aggregate neighborhood features using kernel points and produce multiscale coarse and fine features, consistent with the paper’s three-stage indoor KPConv-FPN. A per-point MLP merely named KPConv does not meet this check. Correct open-source components or equivalent reimplementations are accepted.
Inferred; paper
O5
Retain the paper’s ResNet-FPN and DINOv2 image-feature paths. DINOv2 must load pretrained weights and contribute to model output; disabling it or silently replacing it with image average pooling does not meet this check. Freezing or caching unchanged pretrained features is accepted.
Inferred; prior context and paper
O6
Cylindrical self-attention must enhance the key with local fusion information, implementing the paper’s Q(K+L)-transpose attention semantics. Disconnected local information or plain self-attention does not meet this check. Tensor implementations with the same mathematical meaning are accepted.
Inferred; paper
Appendix
Table 16: Complete outcome rubric. Each item is a binary check; source labels identify explicit or inferred grounding.
Setting
Value
Actor harness
Codex for GPT-6.1-Sol, Claude Code for other models
Harness version
Codex 0.118.0 or Claude Code 2.1.283 on RJob, varying by run on E2B
Judge
GLM-5.3 (GLM-5.3-Flash for a small subset of runs), one call per checklist item
Execution backend
E2B for 920 CPU runs, RJob/Docker for the other 655 runs
As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced scientific judgment. Consequently, frontier agents remain unable to fully replace human researchers. To bridge this gap, we conceptualize the AARR (Act As a Real Researcher) benchmark series. Unlike existing benchmarks that primarily assess macro-level execution capabilities, AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios. In this work, we propose AARRI-Bench (Act As a Real Research Intern), the first benchmark in this series. We conduct extensive experiments across frontier models and agentic systems, revealing that even the best-performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success rate, frequently overlooking subtle yet critical details that are obvious to real human researchers. Our results indicate that developing researcher-like AI requires further exploration of research behavior, rather than merely complex scaffolding. Our data is released at https://github.com/AARR-bench/AARRI-bench.
LLM agents are judged by what they deliver, and in professional work what gets delivered is a project, not a task. A project pays off only if the whole package holds together, and one weak part can limit the value of the rest. Yet existing benchmarks mostly test agents on one task at a time, leaving a gap between the capability we measure and the outcome that makes agents useful. To address this gap, we introduce DeliverBench, which tests whether agents can carry a project from brief to final delivery. Reconstructed from real materials with 13 domain experts, its 11 projects span settings such as financial due diligence, legal contract drafting, and drug safety reporting. They comprise 150 subtasks, with 2-15 subsequent releases of new materials and client feedback per project. Later subtasks build on the agent's own earlier deliverables, so unresolved errors can carry into downstream work. Expert rubrics assess each subtask submission and the final delivery package. Across seven model-harness systems, the best scores 65.0/100 on final delivery and passes 50.0% of subtasks at a 90-point threshold. Together, these results suggest that task-level capability does not simply compose into delivery; delivery also depends on keeping the work coherent as it evolves.
Yumou Liu, Mingju Chen, Zhoufutu Wen +21
Humanlaya · Theseus Labs · Shanghai Jiao Tong University +4
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.
Yaolun Zhang, Tianyi Xu, Yujie Zhao +3
Oregon State University · AG2 AI · University of California, San Diego +1