ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces
Organizations: Shanghai Artificial Intelligence Laboratory, Fudan University, East China Normal University, Shanghai Jiao Tong University, Renmin University of China, Peking University
Abstract
Research agents achieve increasingly strong benchmark scores, yet these scores do not necessarily translate into reliable delivery in ongoing research projects. Many research-agent benchmarks emphasize standalone tasks and relatively self-contained workflows in clean or curated workspaces. However, researchers ask agents to complete one step within an ongoing project, where requests depend on earlier session decisions and workspaces contain unfinished files, outdated versions, and failed attempts. We introduce ARE-bench to evaluate whether research agents advance ongoing projects while respecting prior decisions and existing workspace state, grounded in real session context and in-progress workspaces. ARE-bench includes 75 tasks built on agent sessions and full workspaces from recruited researchers' own projects, spanning seven research stages, 33 requirements, and 150 sub-requirements, with 1,173 binary checklist items in total. Each task is anchored at a real user request, preserves its requirements, and summarizes the earlier requests still in effect as background. The complete workspace is replayed to that boundary without cleanup, preserving artifacts from earlier turns. Each checklist is compiled from an intent-aligned rubric and credits any valid solution. Across seven frontier models, the strongest passes every checklist item on only 66.2% of tasks, and Design and Execution rank among the hardest stages for every model. Three models pass about 69% of tasks when only delivery contracts and general norms are checked, but just 28.4% to 40.0% once historical and material or workspace requirements are also checked. Evaluating delivery without these contextual requirements can therefore overestimate research agents and mask differences between them.
Figures & tables
| Benchmark | Real request | Session-derived | In-progress workspace | Research workflow | Scoring |
|---|---|---|---|---|---|
| SWE-bench ( Jimenez et al., 2023 ) | ✔ | ✘ | ✘ | ✘ | Tests |
| Terminal-Bench (2.0) ( Merrill et al., 2026 ) | ✔ | ✘ | ✔ | ✘ | Varies |
| OSWorld ( Xie et al., 2024 ) | ✔ | ✘ | ✔ | ✘ | Varies |
| -bench ( Yao et al., 2024 ) | ✔ | ✘ | ✔ | ✘ | Varies |
| MLE-bench ( Chan et al., 2024 ) | ✘ | ✘ | ✘ | ✘ | Task metric |
| ScienceAgentBench ( Chen et al., 2024 ) | ✘ | ✘ | ✔ | ✘ | Varies |
| Research stage | Requirements | Sub-requirements | Checklist items | Tasks |
|---|---|---|---|---|
| Survey | 2 | 20 | 111 | 26 |
| Design | 3 | 25 | 268 | 41 |
| Data & Env. | 5 | 16 | 45 | 20 |
| Code | 5 | 23 | 217 | 32 |
| Execution | 7 | 26 | 173 | 33 |
| Analysis | 4 | 13 | 92 | 34 |
| Evaluated model | Archived model identifier | Actor harness |
|---|---|---|
| GPT-6.1-Sol | gpt-6.1-sol | Codex |
| Claude Opus 5.5 | claude-opus-5-5 | Claude Code |
| DeepSeek-V4.1-Flash | deepseek-v4.1-flash | Claude Code |
| Qwen3.8-Max | qwen3.8-max | Claude Code |
| GLM-5.3-Flash | glm-5.3-flash | Claude Code |
| Kimi-K3 | kimi-k3 | Claude Code |
| Task batch | Tasks | CPUs | Memory (MB) | GPUs | Configured actor time limit |
|---|---|---|---|---|---|
| CPU tasks | 67 | 2–16 | 4,096–49152 | 0 | 30 minutes–6 hours |
| GPU tasks | 8 | 8–32 | 32,768–262,144 | 1–4 | 2, 3, 4, 6, or 48 hours, depending on the task |
| Model | Completeness ISR | Correctness ISR | Constraint compliance ISR | Failed runs with correctness errors |
|---|---|---|---|---|
| GPT-6.1-Sol | 83.11 5.39 | 71.11 5.05 | 93.78 0.77 | 65/76 (85.5%) |
| Claude Opus 5.5 | 87.56 3.08 | 62.67 3.53 | 89.33 4.62 | 84/97 (86.6%) |
| DeepSeek-V4.1-Flash | 82.67 2.67 | 47.11 1.54 | 88.89 5.05 | 119/136 (87.5%) |
| Qwen3.8-Max | 79.11 2.04 | 43.56 8.57 | 84.89 0.77 | 127/140 (90.7%) |
| GLM-5.3-Flash | 80.44 0.77 | 39.11 3.08 | 81.78 1.54 | 137/156 (87.8%) |
| Kimi-K3 | 76.44 1.54 | 34.22 2.04 | 83.56 1.54 | 148/162 (91.4%) |
| Failure manifestation | Effect on research delivery | Archived example |
|---|---|---|
| Task degradation | A simpler implementation replaces the original method, retaining its name while changing the required capability | In the round 1 DeepSeek repair of CylindFormer, the backbone named KPConv applies only Linear, LayerNorm, and ReLU to individual points, without neighborhood or kernel-point aggregation, failing O4 |
| Insufficient investigation and evidence checking | Cited work does not support the mechanism or subject discussed, leaving conclusions without reliable evidence | In the round 1 Qwen3.8-27B revision of a cuproptosis review, ferroptosis-resistance research is described as a cuproptosis mechanism and cancer subtypes are conflated, failing O6 |
| Requirement omission | Existing constraints are not implemented, affecting subsequent use despite local functional progress | In the round 2 DeepSeek adaptation of Graph RAG, datasets share a global index directory instead of isolated indexes, failing O11 |
| Inadequate validation design | Missing comparison rules prevent attribution of improvements to the factor under study | In the round 1 GLM-5.3-Flash plan for 4D spacetime tubes, costs are recorded but executable rules for matching inference budgets across comparison arms are absent, failing O6 |
| Conclusions exceeding the evidence | A limited diagnostic result is extended into an untested causal explanation | In the round 2 GLM-5.3-Flash diagnosis of network-security generalization, adversarial validation is treated as a test of conditional distributions, and a failure to detect differences is attributed to a single cause, failing O13 |
| Model | Delivery contracts + general norms | Add historical and material or workspace requirements | Full checklist |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 69.33 | 40.00 | 39.56 |
| GLM-5.3-Flash | 69.33 | 31.56 | 30.67 |
| Kimi-K3 | 69.78 | 28.44 | 28.00 |
| Model | Survey | Design | Data & Env. | Code | Execution | Analysis | Writing |
|---|---|---|---|---|---|---|---|
| GPT-6.1-Sol | 88.46 6.66 | 73.17 4.88 | 90.00 0.00 | 85.42 9.55 | 76.77 7.00 | 86.27 6.12 | 80.65 5.59 |
| Claude Opus 5.5 | 84.62 3.85 | 76.42 2.82 | 85.00 5.00 | 77.08 7.86 | 66.67 3.03 | 79.41 2.94 | 64.52 11.63 |
| DeepSeek-V4.1-Flash | 79.49 4.44 | 62.60 2.82 | 95.00 0.00 | 71.88 5.41 | 61.62 4.63 | 76.47 0.00 | 63.44 6.72 |
| Qwen3.8-Max | 78.21 4.44 | 58.54 7.32 | 83.33 12.58 | 67.71 6.51 | 50.51 4.63 | 60.78 1.70 | 67.74 5.59 |
| GLM-5.3-Flash | 76.92 3.85 | 56.91 6.14 | 85.00 5.00 | 63.54 1.80 | 59.60 1.75 | 71.57 6.12 | 58.06 5.59 |
| Kimi-K3 | 70.51 4.44 | 47.97 2.82 | 86.67 7.64 | 73.96 4.77 | 47.47 3.50 | 64.71 0.00 | 47.31 3.72 |
| Research stage | Tasks covered | Checklist items | Seven-model mean stage ISR | Best-to-worst model gap |
|---|---|---|---|---|
| Survey | 26 | 111 | 77.66 | 23.08 |
| Design | 41 | 268 | 59.47 | 35.77 |
| Data & Env. | 20 | 45 | 86.43 | 15.00 |
| Code | 32 | 217 | 71.28 | 26.04 |
| Execution | 33 | 173 | 57.43 | 37.37 |
| Analysis | 34 | 92 | 71.15 | 27.45 |
| Position relative to the complete session midpoint | CSR (%) |
|---|---|
| Before the midpoint, original segment 2 | 95.92 |
| Near the midpoint, original segment 3 | 91.63 |
| After the midpoint, original segment 4 | 97.66 |
| DeepSeek verdict / Qwen verdict | Pass | Fail | Total |
|---|---|---|---|
| Pass | 8,043 | 67 | 8,110 |
| Fail | 648 | 1,116 | 1,764 |
| Total | 8,691 | 1,183 | 9,874 |
| Evaluation aspect | Successful execution, 20/20 | Execution with unmet items, 16/20 |
|---|---|---|
| Training progress | Sustained nonzero loss and parameter updates | Also sustained training, nonzero loss, and parameter updates |
| O7, fusion coordinates | Projection indexes agree with the actual feature grid | Inconsistent coordinates clip some samples to boundary columns |
| O9, overlap labels | Separate 2D and 3D overlap checks retain positive, negative, and ignored regions | A single projection ratio replaces independent 2D and 3D overlap tests |
| O11, coarse matching loss | L2 distance, overlap weights, and specified settings are retained | Squared distance replaces L2 without corresponding boundary-parameter changes, and weights are altered |
| O12, fine matching loss | Bidirectional loss and the coarse/fine loss combination are retained | The changed distance and weight expressions are reused |
| Training and delivery items | Archived review accepts checkpoints, limited validation, and the report | The other 16 items pass, including training recovery, checkpoints, validation, and the report |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Research stage | Requirement | Sub-requirement | Capability primitives | Work object |
|---|---|---|---|---|
| Search for relevant literature | Apply: implement; Analyze: differentiate | Evidence | ||
| Verify bibliographic information | Evaluate: critique | Evidence | ||
| Extract key points from literature | Analyze: differentiate; Understand: summarize | Evidence | ||
| Verify literature content | Evaluate: check | Evidence | ||
| Assess literature quality | Evaluate: critique | Evidence | ||
| Establish evidence traceability | Analyze: organize; Evaluate: check | Evidence |
| Research stage | Representative starting workspace state |
|---|---|
| Survey | Partial reading notes and a candidate research question, with no experiment code yet written |
| Design | An existing problem statement and prior baseline results that the new design must extend or contrast |
| Data & Env. | A partially installed environment and raw or partially downloaded data left from an earlier setup attempt |
| Code | A repository with a partial implementation, stub functions, and tests that do not yet pass |
| Execution | Configured run scripts together with logs or checkpoints from an interrupted or failed run |
| Analysis | Raw experiment outputs and an unfinished or outdated analysis script |
| ID | Criterion | Source |
|---|---|---|
| O1 | Deliver a reusable repaired project in /workspace/output/, with an index locating source code, configuration, and reproduction entry points. Internal filenames and project organization are unrestricted. | Explicit; Requirements |
| O2 | Construct image–point-cloud training pairs with real relative poses and filter them using the paper’s at-least-50% overlap rule. Pairing only an image with its own back-projected depth, or declaring min_overlap without computing and applying it, does not meet this check. Verifiable equivalent pairing is accepted; unavailable paper-internal pair IDs need not be restored. | Inferred; paper |
| O3 | Separate training and evaluation according to the official 7-Scenes sequence splits. Validation must not train the model or select it using test performance. When the paper’s internal validation pairs are unavailable, an independent validation set constructed from training sequences is accepted if its scope is explained. | Inferred; paper |
| O4 | The point-cloud backbone must aggregate neighborhood features using kernel points and produce multiscale coarse and fine features, consistent with the paper’s three-stage indoor KPConv-FPN. A per-point MLP merely named KPConv does not meet this check. Correct open-source components or equivalent reimplementations are accepted. | Inferred; paper |
| O5 | Retain the paper’s ResNet-FPN and DINOv2 image-feature paths. DINOv2 must load pretrained weights and contribute to model output; disabling it or silently replacing it with image average pooling does not meet this check. Freezing or caching unchanged pretrained features is accepted. | Inferred; prior context and paper |
| O6 | Cylindrical self-attention must enhance the key with local fusion information, implementing the paper’s Q(K+L)-transpose attention semantics. Disconnected local information or plain self-attention does not meet this check. Tensor implementations with the same mathematical meaning are accepted. | Inferred; paper |
| Setting | Value |
|---|---|
| Actor harness | Codex for GPT-6.1-Sol, Claude Code for other models |
| Harness version | Codex 0.118.0 or Claude Code 2.1.283 on RJob, varying by run on E2B |
| Judge | GLM-5.3 (GLM-5.3-Flash for a small subset of runs), one call per checklist item |
| Execution backend | E2B for 920 CPU runs, RJob/Docker for the other 655 runs |
| Harbor version | 0.23.0 in the 650 readable run protocols |
| Rollouts per task | 3 |
| Metric | Value |
|---|---|
| Tasks | 75 |
| Checklist items | 1,173 |
| Avg. checklist items per task | 15.6 |
| Research stages covered | 7 |
| Avg. earlier requests in Background | 10.3 |
| Avg. workspace files at cut point | 41.1 |
Agent Evaluation
Agent evaluation measures how reliably AI systems act, use tools, and complete tasks across interactive environments.
See all 155 papers