ResearchFlow-Eval: Tracing the Execution Gap in End-to-End Multi-Claim Autoresearch
Organizations: Tsinghua University · Center for Life Sciences and Artificial Intelligence, School of Life Sciences, Tsinghua University, Beijing, China · IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China · Tsinghua-Peking Center for Life Sciences, Tsinghua University, Beijing, China ·
Abstract
Autoresearch systems increasingly undertake entire research projects. In practice, scientific research requires systems to pursue multiple interdependent claims through a continuous, multi-stage workflow and revise experiments as evidence accumulates. Existing evaluations of autoresearch systems, however, typically score isolated tasks or final artifacts, leaving these process-level capabilities largely unmeasured. We therefore introduce ResearchFlow-Eval, a fine-grained framework for assessing continuous scientific execution and iteration at both the stage and claim levels. Each evaluation is grounded in a published anchor paper, from which we derive stage-specific rubrics that decompose each research stage into dimensions and separately scored rubric items. Using these rubrics, we score each research stage and examine experimental execution in detail, focusing on multi-claim completion and revisions across attempts. We apply the evaluation framework to general-purpose agent harnesses, skill-augmented harnesses, and systems with research-specific orchestration. Across all evaluated systems, experimental execution consistently lags behind design and planning. Within execution, systems struggle to produce high-quality results and meet claim requirements, and repeated attempts rarely yield measurable improvements. Moreover, neither skill augmentation nor research-specific orchestration reliably yields higher scores. ResearchFlow-Eval thus extends autoresearch evaluation beyond isolated, single-objective runs to continuous workflows in which systems pursue multiple claims and refine their work across iterations.
Figures & tables
| Research parts | Evaluation features | ||||||
|---|---|---|---|---|---|---|---|
| Benchmark | Design plan | Exp. execute | Analyze write | Full E2E | Multi- claim | Paper rubric | Iteration diagnostics |
| ScienceAgentBench ( Chen et al., 2025b ) | |||||||
| PaperBench ( Starace et al., 2025 ) | |||||||
| MLR-Bench ( Chen et al., 2025a ) | |||||||
| ResearchGym ( Garikaparthi et al., 2026 ) | |||||||
| FML-bench ( Zou et al., 2025 ) | |||||||
| Design & planning | Experimental execution | Analysis & writing | |||||||||
| System | Lit. | Claims | Meth. | Plan | Module | Code | Result | Claim | Anal. | Writ. | Overview |
| General-purpose agent harness | |||||||||||
| Codex / GPT-5.5 | 68.8 | 47.5 | 56.2 | 41.7 | 53.6 | 50.0 | 30.0 | 42.5 | 42.5 | 32.5 | 45.1 |
| Codex / GPT-5.6 Sol | 83.3 | 65.0 | 78.1 | 66.7 | 60.7 | 52.5 | 32.5 | 60.0 | 55.0 | 62.5 | 59.5 |
| Codex / GPT-6 Astra | 72.9 | 65.0 | 84.4 | 70.8 | 64.3 | 57.5 | 47.5 | 65.0 | 62.5 | 70.0 | 64.9 |
| Claude Code / Opus 4.8 | 81.2 | 65.0 | 81.2 | 66.7 | 71.4 | 57.5 | 42.5 | 57.5 | 50.0 | 55.0 | 61.3 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Research activity | Paper-derived inputs | Required output | Budget (h) |
|---|---|---|---|---|
| S1 | Literature review | Research brief (S0) | Literature review | 4 |
| S2 | Research claims | S0–S1 | Claims with success criteria | 2 |
| S3 | Method design | S0–S2 | Method specification | 4 |
| S4 | Experiment planning | S0–S3 and data card | Experiment plan | 4 |
| S5 | Experimental execution | S0–S4 with data and environment | Code and results with their execution records | 48 |
| S6 | Results analysis | S0–S5 | Per-claim conclusions | 4 |
| Input | Content | Example excerpt |
|---|---|---|
| S0 research brief content.md | Research question and core hypothesis, with the data constraints and the success boundary | “Treating the neural population as an unordered, variable-size set of units, and making the decoding function invariant under unit permutation, may be better suited to cross-session decoding than models that rely on fixed channel/unit identity.” |
| S1 literature related_work.md | Four research areas with their capabilities and limitations, and a table that maps each gap to a claim | On cross-session alignment, “some methods also need held-out labels, test-time optimization or model-parameter updates.” |
| S2 claims claims.md | The ten required claims with their scope and success criteria, and the extrapolations they do not license | Claim MC2 reads “When unit rows of the neural window and the calibration identity are permuted synchronously, the decoder output remains permutation-invariant.” It is marked as “not equivalent to unconditional robustness to unit deletion.” |
| S3 method method.md | The decoder design component by component, with each component linked to the claims it serves | The identity encoder computes , and “mean pooling makes the result invariant to the order of calibration trials.” |
| S4 plan experiment_plan.md | The session split and the baselines, with one execution contract per claim | M1 uses four held-in sessions split into 2 train, 1 validation, and 1 test with seed 20260824 . Viability requires a maximum output difference of at most under synchronized unit permutation. |
| S5 execution execution_results.md | Reference results of the required experiments, each marked as exact or approximate | The candidate system (GF-FSU) reaches of on M1, on M2, and on H1. |
| Entry | Content |
|---|---|
| AGENTS.md | Research topic, run scope, and stage order. It also lists the inputs and the resource budget with the work boundaries. |
| TASK.md | The instruction of every stage in the run, with its goal, procedure, and completion condition |
| RESOURCES.md | Descriptions of the data, the environment, and the paper template. It also states the readable paths outside the workspace, the evaluation mode, and the rules for external sources. |
| OUTPUTS.md | The required artifact paths with their purpose, the optional and diagnostic outputs, and the terminal contract |
| inputs/upstream | Read-only paper-derived inputs of the preceding stages |
| inputs/runtime | Formats of the claim log and claim state, with the scripts that record and validate them |
| Source | Content | Example excerpt |
|---|---|---|
| initial_prompt.md (all runs) | Read the four root documents and work autonomously until the stage goals are met. Rerun after a poor result, and poll background experiments. | “Read AGENTS.md, TASK.md, RESOURCES.md, and OUTPUTS.md, then autonomously advance the current research task. […] Background experiments must be actively polled rather than exiting to wait.” |
| initial_prompt.md (runs with execution) | The sequential-claim rule and the stop line at 90% of the budget, with the commands that record claim events | “Advance one claim at a time in the frozen order until it reaches its completion standard, or until any actually provided budget cap reaches 90%.” The recording command is python3 inputs/runtime/record_claim_event.py record . |
| TASK.md (end-to-end run) | Instructions of all seven stages in order, with the condition for entering each stage | “Move on to the next stage only after the current stage’s instruction completion conditions and all required outputs in OUTPUTS.md belonging to that stage are satisfied.” |
| TASK.md (stagewise run) | Instruction of one stage and its upstream inputs, with a ban on later stages | “This run completes only ‘Method and model design’. […] do not read other run spaces, and do not continue on to later research stages.” |
| Pilot behavior or rule gap | Revision | Where it is stated |
|---|---|---|
| Runs ended after using about 1% of their budget once their outputs had the required form. The rules also left success criteria and claim order open during execution. | Claims and their success criteria are frozen before execution, and claims are pursued one at a time in a fixed order. An unmet claim is revised and rerun until it meets its criterion or the run reaches 90% of its budget. Each rerun is compared with the best attempt so far, and the number of attempts is not fixed. | Instructions of S2, S4, and S5, together with TASK.md and the initial instruction |
| Systems narrowed the scope of the research when they wrote their own claims | The research-claims stage requires at least two method claims that cover the proposed innovation and at least four empirical claims. The empirical claims include one against named baselines and one against named ablations, so that the claims can support a paper at the level of ICLR. | Instruction of S2 |
| Claude Code ended its turn to wait for background experiments and did not resume | The initial instruction requires polling background experiments and continuing with every dependent step at once. It states that running jobs, pending notifications, or a lack of output are not valid reasons to exit. | operator/initial_prompt.md |
| An end-to-end run could move on before a stage was complete and produce a short paper | TASK.md contains the seven stage instructions in order, with the condition for entering each stage. A run may exit only after every stage meets its own completion condition and has produced its outputs in OUTPUTS.md . | TASK.md (end-to-end runs) |
| Subject | Neural input | Behavior target | NWB assets | Local sessions |
|---|---|---|---|---|
| M1 | Macaque precentral spikes | 16-muscle EMG | 11 | 2/1/1 |
| M2 | Macaque precentral spikes | 2-D finger velocity | 20 | 2/1/1 |
| H1 | Human intracortical spikes | 7-DoF reach and grasp velocity | 40 | 4/1/1 |
| ID | Claim | Depends on | Success criterion |
|---|---|---|---|
| MC1 | Unit tokenization and set decoder | – | Unit-window tokens, context-dependent unit identity, and behavior-query cross-attention are implemented. Forward and backward passes succeed with a variable number of units. |
| MC2 | Permutation invariance | MC1 | Permuting the rows of the neural window and the calibration identity together changes the output by at most . |
| MC3 | Gradient-free calibration | MC1 | The calibration identity of a target session reads only neural activity. Traces show no optimizer step at inference and no behavior labels in adaptation. |
| MC4 | Dynamic channel dropout | – | Training-only dynamic channel dropout is implemented and verifiable from configuration and runtime traces. It is compared with a variant that differs only by having no dropout. |
| E1 | Decoding against baselines | MC1, MC3 | On M1, M2, and H1, the decoder exceeds every peer baseline, namely Wiener filter ZS, RNN ZS, and the calibration-aware linear and RNN FSU baselines. |
| E2 | Few-shot calibration | MC3, E1 | For every subject, the fewest-shot condition exceeds the zero predictor. M1 one-shot is at least full-calibration minus . |
| Stage | Dimension | Items | Weight in stage | Stage weight in part |
|---|---|---|---|---|
| Design and planning (30% of the overview score) | ||||
| S1 Literature review | Intracortical motor decoding | 3 | 25% | 20% |
| Cross-session neural alignment | 3 | 25% | ||
| Neuroscience foundation models | 3 | 25% | ||
| Set-structured (permutation-invariant) models | 3 | 25% | ||
| S2 Research claims | Method claims MC1–MC4 | 4 | 40% | 20% |
| Rating | Module development | Code execution | Result quality | Claim completion |
|---|---|---|---|---|
| 0 | Missing, fundamentally wrong, or unable to satisfy the method claim | No credible execution | No admissible result, or a key element such as the comparator or provenance is missing | No corresponding work, no execution, or invalid evidence |
| 1 | Only stubs, interfaces, or module names | Only scripts and configurations, or claims of execution without support | A complete, valid experiment fails to support or contradicts the claim | Only implementation, a smoke test, or an initial attempt |
| 2 | Substantive code with an incomplete key mechanism or data flow | The real path was run and left credible partial evidence, but the key path is unfinished | Real and admissible but mixed or weak support | Substantive work lacking a key protocol element, comparator, or piece of evidence |
| 3 | Core implementation correct, with one limited gap in interface, tests, or integration | Run completed, with one limited protocol or evidence gap | Credible result that mostly meets the criterion, with the unmet parts stated | Complete, adjudicable execution whose result misses the criterion |
| 4 | Complete and correct implementation, with enough code or tests to support the judgment | Qualified run with correct protocol and controls, and with its commands and logs | Admissible result that meets the criterion with correctly bounded scope | Qualified execution whose admissible evidence meets the criterion |
| Gate | Trigger |
|---|---|
| Forbidden source | Use of the anchor paper with its supplementary material, code, or checkpoints, or use of sources released after the cutoff |
| Reference leak | Reference outputs or rubric contents enter the workspace or the submission, or so does the mapping from reference entries to paper locations |
| Test protocol | Unlogged test access, behavior labels in a calibration or input described as unlabeled, or optimized test results described as untouched |
| Fabricated execution | An unexecuted experiment or a paper-reported number presented as a local result |
| Session mismatch | Subject, session, or sample mapping inconsistent with the frozen split manifest |
| Unsupported claim | A claim that exceeds its evidence or depends on a failed, missing, or invalid experiment |
| System | Mode | Overview | Wall-clock (h) | Tokens (M) |
|---|---|---|---|---|
| Codex / GPT-5.5 | End to end | 45.1 | 0.30 | 4.7 |
| Codex / GPT-5.6 Sol | End to end | 59.5 | 1.72 | 57.2 |
| Codex / GPT-6 Astra | End to end | 64.9 | 1.10 | 12.8 |
| Claude Code / Opus 4.8 | End to end | 61.3 | 5.73 | 126.5 |
| Kimi Code / K3 | End to end | 64.2 | 20.56 | 92.0 |
| ARIS | End to end | 44.5 | 0.74 | 15.1 |
| System | Harness or system version | Model | Effort | Context |
|---|---|---|---|---|
| Codex / GPT-5.5 | Codex CLI 0.153.4 | gpt-5.5 | high | 272K |
| Codex / GPT-5.6 Sol | Codex CLI 0.153.4 | gpt-5.6-sol | high | 872K |
| Codex / GPT-6 Astra | Codex CLI 0.153.4 | gpt-6-astra | high | 872K |
| Claude Code / Opus 4.8 | Claude Code 2.1.269 | claude-opus-4-8 | high | 1M |
| Kimi Code / K3 | Kimi Code 0.39.1 | k3 | high | 1M |
| ARIS | Codex CLI 0.153.4 with ARIS skills (commit 94d8093 ) | gpt-5.5 | high | 272K |
| Stage | S5 dimension | ||||||||||||
| System | Mode | S1 | S2 | S3 | S4 | S5 | S6 | S7 | Mod. | Code | Qual. | Comp. | Overview |
| Codex / GPT-5.5 | End to end | 68.8 | 47.5 | 56.2 | 41.7 | 44.0 | 42.5 | 32.5 | 53.6 | 50.0 | 30.0 | 42.5 | 45.1 |
| Stagewise | 83.3 | 67.5 | 90.6 | 83.3 | 77.6 | 90.0 | 65.0 | 67.9 | 65.0 | 85.0 | 92.5 | 78.5 | |
| Claude Code / Opus 4.8 | End to end | 81.2 | 65.0 | 81.2 | 66.7 | 57.2 | 50.0 | 55.0 | 71.4 | 57.5 | 42.5 | 57.5 | 61.3 |
| Stagewise | 89.6 | 72.5 | 90.6 | 70.8 | 86.1 | 77.5 | 62.5 | 89.3 | 67.5 | 92.5 | 95.0 | 81.0 | |
| Kimi Code / K3 | End to end | 72.9 | 65.0 | 78.1 | 68.8 | 61.6 | 62.5 | 57.5 | 71.4 | 62.5 | 47.5 | 65.0 | 64.2 |
| System | MC1 | MC2 | MC3 | MC4 | E1 | E2 | E3 | E4 | E5 | E6 | Met | Rep. | Att. |
| General-purpose agent harness, end to end | |||||||||||||
| Codex / GPT-5.5 | 4/4 | 2/3 | 2/2 | 0/0 | 2/0 | 2/1 | 0/0 | 1/0 | 2/2 | 2/0 | 1 | 0 | 9 |
| Codex / GPT-5.6 Sol | 4/4 | 4/4 | 4/4 | 1/0 | 2/0 | 3/1 | 0/0 | 2/0 | 2/0 | 2/0 | 3 | 8 | 24 |
| Codex / GPT-6 Astra | 4/4 | 4/4 | 4/4 | 0/0 | 3/1 | 3/2 | 0/0 | 2/0 | 4/4 | 2/0 | 4 | 8 | 10 |
| Claude Code / Opus 4.8 | 4/4 | 2/3 | 2/2 | 2/2 | 3/1 | 3/1 | 0/0 | 2/0 | 2/2 | 3/2 | 1 | 3 | 9 |
| Kimi Code / K3 | 4/4 | 4/4 | 4/4 | 2/2 | 2/0 | 3/2 | 0/0 | 2/0 | 2/2 | 3/1 | 3 | 2 | 18 |
Autonomous Research Agents
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
See all 319 papers