ResearchFlow-Eval: Tracing the Execution Gap in End-to-End Multi-Claim Autoresearch

Peng Jiang; Dongyang Li; Quanying Liu; Xiaoxuan Jia

cs.AIOct 13, 2026

ResearchFlow-Eval: Tracing the Execution Gap in End-to-End Multi-Claim Autoresearch

Authors: Peng Jiang, Dongyang Li, Quanying Liu, Xiaoxuan Jia

Organizations: Tsinghua University · Center for Life Sciences and Artificial Intelligence, School of Life Sciences, Tsinghua University, Beijing, China · IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China · Tsinghua-Peking Center for Life Sciences, Tsinghua University, Beijing, China ·

Abstract

Autoresearch systems increasingly undertake entire research projects. In practice, scientific research requires systems to pursue multiple interdependent claims through a continuous, multi-stage workflow and revise experiments as evidence accumulates. Existing evaluations of autoresearch systems, however, typically score isolated tasks or final artifacts, leaving these process-level capabilities largely unmeasured. We therefore introduce ResearchFlow-Eval, a fine-grained framework for assessing continuous scientific execution and iteration at both the stage and claim levels. Each evaluation is grounded in a published anchor paper, from which we derive stage-specific rubrics that decompose each research stage into dimensions and separately scored rubric items. Using these rubrics, we score each research stage and examine experimental execution in detail, focusing on multi-claim completion and revisions across attempts. We apply the evaluation framework to general-purpose agent harnesses, skill-augmented harnesses, and systems with research-specific orchestration. Across all evaluated systems, experimental execution consistently lags behind design and planning. Within execution, systems struggle to produce high-quality results and meet claim requirements, and repeated attempts rarely yield measurable improvements. Moreover, neither skill augmentation nor research-specific orchestration reliably yields higher scores. ResearchFlow-Eval thus extends autoresearch evaluation beyond isolated, single-objective runs to continuous workflows in which systems pursue multiple claims and refine their work across iterations.

Figures & tables

Figure 1: Overview of the multi-claim research workflow. Design and planning (S1–S4) precede experimental execution (S5) and analysis and writing (S6–S7). The inset shows claim-wise iteration, in which an unmet criterion returns the claim to diagnosis and a met one moves the system to the next claim.

	Research parts			Evaluation features
Benchmark	Design plan	Exp. execute	Analyze write	Full E2E	Multi- claim	Paper rubric	Iteration diagnostics
ScienceAgentBench ( Chen et al., 2025b )	$\times$	$\triangle$	$\triangle$	$\times$	$\times$	$\checkmark$	$\checkmark$
PaperBench ( Starace et al., 2025 )	$\times$	$\checkmark$	$\triangle$	$\triangle$	$\checkmark$	$\checkmark$	$\checkmark$
MLR-Bench ( Chen et al., 2025a )	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\triangle$
ResearchGym ( Garikaparthi et al., 2026 )	$\triangle$	$\checkmark$	$\times$	$\triangle$	$\triangle$	$\triangle$	$\checkmark$
FML-bench ( Zou et al., 2025 )	$\times$	$\checkmark$	$\times$	$\triangle$	$\times$	$\times$	$\checkmark$

Table 1: Comparison with related autoresearch benchmarks. The research-part columns mark which stages a benchmark evaluates. The evaluation-feature columns mark whether it scores a full end-to-end run and multiple claims per run, whether its rubric is derived from a paper, and whether it diagnoses iteration within a task.

\checkmark

\triangle

, and

\times

indicate that a benchmark evaluates an item explicitly, partially, or not at all.

Figure 2: Execution modes and scoring hierarchy. (a) Stagewise and end-to-end runs. A stagewise run starts each stage from the paper’s outputs for the preceding stages, whereas an end-to-end run builds on its own earlier outputs. (b) The overview score combines three research parts. Each stage box shows the stage’s share of it, and the bottom boxes list representative rubric dimensions.

General-purpose agent harness
	Design & planning				Experimental execution				Analysis & writing
System	Lit.	Claims	Meth.	Plan	Module	Code	Result	Claim	Anal.	Writ.	Overview
Codex / GPT-5.5	68.8	47.5	56.2	41.7	53.6	50.0	30.0	42.5	42.5	32.5	45.1
Codex / GPT-5.6 Sol	83.3	65.0	78.1	66.7	60.7	52.5	32.5	60.0	55.0	62.5	59.5
Codex / GPT-6 Astra	72.9	65.0	84.4	70.8	64.3	57.5	47.5	65.0	62.5	70.0	64.9
Claude Code / Opus 4.8	81.2	65.0	81.2	66.7	71.4	57.5	42.5	57.5	50.0	55.0	61.3

Table 2: End-to-end scores (0–100) of the nine systems. Stage columns abbreviate literature review (Lit.), research claims (Claims), method design (Meth.), experiment planning (Plan), results analysis (Anal.), and paper writing (Writ.). Bold marks the highest value in each column.

Figure 3: Stage profiles in end-to-end and stagewise runs. Scores (0–100) of Codex with GPT-5.5, Claude Code with Opus 4.8, and Kimi Code with K3, the three general-purpose harnesses run in both modes, end to end (solid blue) and stagewise (dashed vermillion).

Figure 4: Claim progress and metric trajectories during experimental execution. (a) Claims the system reported as successful and (b) claims whose criterion the evaluator confirmed as met, cumulative over log time. Filled circles mark run endpoints, and the open triangle marks AutoResearchClaw, which made no scientific attempt. (c) An example iteration trajectory of GPT-5.6 Sol.

Figure 5: Performance against resource use. Overview score of the nine end-to-end runs against wall-clock time (left) and total tokens (right), on logarithmic axes.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Research activity	Paper-derived inputs	Required output	Budget (h)
S1	Literature review	Research brief (S0)	Literature review	4
S2	Research claims	S0–S1	Claims with success criteria	2
S3	Method design	S0–S2	Method specification	4
S4	Experiment planning	S0–S3 and data card	Experiment plan	4
S5	Experimental execution	S0–S4 with data and environment	Code and results with their execution records	48
S6	Results analysis	S0–S5	Per-claim conclusions	4

Appendix

Table 3: Stagewise inputs, required outputs, and wall-clock budgets of the seven stages. The stage budgets sum to 72 hours, the budget of one end-to-end run. In an end-to-end run, each stage instead receives the evaluated system’s own earlier artifacts.

Input	Content	Example excerpt
S0 research brief content.md	Research question and core hypothesis, with the data constraints and the success boundary	“Treating the neural population as an unordered, variable-size set of units, and making the decoding function invariant under unit permutation, may be better suited to cross-session decoding than models that rely on fixed channel/unit identity.”
S1 literature related_work.md	Four research areas with their capabilities and limitations, and a table that maps each gap to a claim	On cross-session alignment, “some methods also need held-out labels, test-time optimization or model-parameter updates.”
S2 claims claims.md	The ten required claims with their scope and success criteria, and the extrapolations they do not license	Claim MC2 reads “When unit rows of the neural window and the calibration identity are permuted synchronously, the decoder output remains permutation-invariant.” It is marked as “not equivalent to unconditional robustness to unit deletion.”
S3 method method.md	The decoder design component by component, with each component linked to the claims it serves	The identity encoder computes $E_{i}=\psi(\mathrm{mean}_{j}\,\phi(X^{C}_{ij}))$ , and “mean pooling makes the result invariant to the order of calibration trials.”
S4 plan experiment_plan.md	The session split and the baselines, with one execution contract per claim	M1 uses four held-in sessions split into 2 train, 1 validation, and 1 test with seed 20260824 . Viability requires a maximum output difference of at most $10^{-5}$ under synchronized unit permutation.
S5 execution execution_results.md	Reference results of the required experiments, each marked as exact or approximate	The candidate system (GF-FSU) reaches $R^{2}$ of $0.66\pm 0.07$ on M1, $0.26\pm 0.13$ on M2, and $0.29\pm 0.15$ on H1.

Appendix

Table 4: Paper-derived inputs of the task derived from SPINT. Each input is placed under inputs/upstream/ of a stagewise workspace. In stagewise runs of results analysis and paper writing, the execution input replaces the system’s own execution evidence. Its values are marked as reference values so that the judge can tell them apart from local evidence.

Entry	Content
AGENTS.md	Research topic, run scope, and stage order. It also lists the inputs and the resource budget with the work boundaries.
TASK.md	The instruction of every stage in the run, with its goal, procedure, and completion condition
RESOURCES.md	Descriptions of the data, the environment, and the paper template. It also states the readable paths outside the workspace, the evaluation mode, and the rules for external sources.
OUTPUTS.md	The required artifact paths with their purpose, the optional and diagnostic outputs, and the terminal contract
inputs/upstream	Read-only paper-derived inputs of the preceding stages
inputs/runtime	Formats of the claim log and claim state, with the scripts that record and validate them

Appendix

Table 5: Contents of a research workspace. The four documents at the root are rendered from the task package. The directories below them hold the inputs, the work in progress, and the graded artifacts.

Source	Content	Example excerpt
initial_prompt.md (all runs)	Read the four root documents and work autonomously until the stage goals are met. Rerun after a poor result, and poll background experiments.	“Read AGENTS.md, TASK.md, RESOURCES.md, and OUTPUTS.md, then autonomously advance the current research task. […] Background experiments must be actively polled rather than exiting to wait.”
initial_prompt.md (runs with execution)	The sequential-claim rule and the stop line at 90% of the budget, with the commands that record claim events	“Advance one claim at a time in the frozen order until it reaches its completion standard, or until any actually provided budget cap reaches 90%.” The recording command is python3 inputs/runtime/record_claim_event.py record .
TASK.md (end-to-end run)	Instructions of all seven stages in order, with the condition for entering each stage	“Move on to the next stage only after the current stage’s instruction completion conditions and all required outputs in OUTPUTS.md belonging to that stage are satisfied.”
TASK.md (stagewise run)	Instruction of one stage and its upstream inputs, with a ban on later stages	“This run completes only ‘Method and model design’. […] do not read other run spaces, and do not continue on to later research stages.”

Appendix

Table 6: Instructions that start a run and set its scope.

Pilot behavior or rule gap	Revision	Where it is stated
Runs ended after using about 1% of their budget once their outputs had the required form. The rules also left success criteria and claim order open during execution.	Claims and their success criteria are frozen before execution, and claims are pursued one at a time in a fixed order. An unmet claim is revised and rerun until it meets its criterion or the run reaches 90% of its budget. Each rerun is compared with the best attempt so far, and the number of attempts is not fixed.	Instructions of S2, S4, and S5, together with TASK.md and the initial instruction
Systems narrowed the scope of the research when they wrote their own claims	The research-claims stage requires at least two method claims that cover the proposed innovation and at least four empirical claims. The empirical claims include one against named baselines and one against named ablations, so that the claims can support a paper at the level of ICLR.	Instruction of S2
Claude Code ended its turn to wait for background experiments and did not resume	The initial instruction requires polling background experiments and continuing with every dependent step at once. It states that running jobs, pending notifications, or a lack of output are not valid reasons to exit.	operator/initial_prompt.md
An end-to-end run could move on before a stage was complete and produce a short paper	TASK.md contains the seven stage instructions in order, with the condition for entering each stage. A run may exit only after every stage meets its own completion condition and has produced its outputs in OUTPUTS.md .	TASK.md (end-to-end runs)

Appendix

Table 7: Revisions of the workspace instructions made after pilot runs. Each row pairs a pilot behavior or rule gap with the revision that addresses it and the workspace file that states it.

Subject	Neural input	Behavior target	NWB assets	Local sessions
M1	Macaque precentral spikes	16-muscle EMG	11	2/1/1
M2	Macaque precentral spikes	2-D finger velocity	20	2/1/1
H1	Human intracortical spikes	7-DoF reach and grasp velocity	40	4/1/1

Appendix

Table 8: The three FALCON subjects of the task. Local sessions are train/validation/test counts selected from public held-in sessions and are not the private benchmark split.

ID	Claim	Depends on	Success criterion
MC1	Unit tokenization and set decoder	–	Unit-window tokens, context-dependent unit identity, and behavior-query cross-attention are implemented. Forward and backward passes succeed with a variable number of units.
MC2	Permutation invariance	MC1	Permuting the rows of the neural window and the calibration identity together changes the output by at most $10^{-5}$ .
MC3	Gradient-free calibration	MC1	The calibration identity of a target session reads only neural activity. Traces show no optimizer step at inference and no behavior labels in adaptation.
MC4	Dynamic channel dropout	–	Training-only dynamic channel dropout is implemented and verifiable from configuration and runtime traces. It is compared with a variant that differs only by having no dropout.
E1	Decoding against baselines	MC1, MC3	On M1, M2, and H1, the decoder exceeds every peer baseline, namely Wiener filter ZS, RNN ZS, and the calibration-aware linear and RNN FSU baselines.
E2	Few-shot calibration	MC3, E1	For every subject, the fewest-shot condition exceeds the zero predictor. M1 one-shot $R^{2}$ is at least full-calibration $R^{2}$ minus $0.05$ .

Appendix

Table 9: The ten required claims, their dependencies, and their frozen success criteria. Method claims (MC) state what must be implemented, whereas empirical claims (E) state what the experiments must show.

R^{2}

values are pooled means over the local test sessions.

Stage	Dimension	Items	Weight in stage	Stage weight in part
Design and planning (30% of the overview score)
S1 Literature review	Intracortical motor decoding	3	25%	20%
	Cross-session neural alignment	3	25%
	Neuroscience foundation models	3	25%
	Set-structured (permutation-invariant) models	3	25%
S2 Research claims	Method claims MC1–MC4	4	40%	20%

Appendix

Table 10: Rubric structure of the task, with its dimensions, item counts, and weights. The literature-review dimensions each contain items for the sources found, the comparison with prior work, and the resulting gap. The experimental protocol of experiment planning covers the data, metrics, baselines, and training. The two paper-writing dimensions are scored in separate judge calls.

Figure 6: Main-result ( $R^{2}$ ) trajectories of DeepScientist, ARIS, and Kimi K3 on claim E1, with one point per attempt, as in Figure 4 c.

Rating	Module development	Code execution	Result quality	Claim completion
0	Missing, fundamentally wrong, or unable to satisfy the method claim	No credible execution	No admissible result, or a key element such as the comparator or provenance is missing	No corresponding work, no execution, or invalid evidence
1	Only stubs, interfaces, or module names	Only scripts and configurations, or claims of execution without support	A complete, valid experiment fails to support or contradicts the claim	Only implementation, a smoke test, or an initial attempt
2	Substantive code with an incomplete key mechanism or data flow	The real path was run and left credible partial evidence, but the key path is unfinished	Real and admissible but mixed or weak support	Substantive work lacking a key protocol element, comparator, or piece of evidence
3	Core implementation correct, with one limited gap in interface, tests, or integration	Run completed, with one limited protocol or evidence gap	Credible result that mostly meets the criterion, with the unmet parts stated	Complete, adjudicable execution whose result misses the criterion
4	Complete and correct implementation, with enough code or tests to support the judgment	Qualified run with correct protocol and controls, and with its commands and logs	Admissible result that meets the criterion with correctly bounded scope	Qualified execution whose admissible evidence meets the criterion

Appendix

Table 11: Rating anchors of the four experimental-execution dimensions. Result quality and claim completion are rated once per required claim against its frozen success criterion.

Gate	Trigger
Forbidden source	Use of the anchor paper with its supplementary material, code, or checkpoints, or use of sources released after the cutoff
Reference leak	Reference outputs or rubric contents enter the workspace or the submission, or so does the mapping from reference entries to paper locations
Test protocol	Unlogged test access, behavior labels in a calibration or input described as unlabeled, or optimized test results described as untouched
Fabricated execution	An unexecuted experiment or a paper-reported number presented as a local result
Session mismatch	Subject, session, or sample mapping inconsistent with the frozen split manifest
Unsupported claim	A claim that exceeds its evidence or depends on a failed, missing, or invalid experiment

Appendix

Table 12: Integrity gates and their triggers. Each gate zeros only the rubric items that depend on the affected evidence, unless the contamination cannot be bounded.

System	Mode	Overview	Wall-clock (h)	Tokens (M)
Codex / GPT-5.5	End to end	45.1	0.30	4.7
Codex / GPT-5.6 Sol	End to end	59.5	1.72	57.2
Codex / GPT-6 Astra	End to end	64.9	1.10	12.8
Claude Code / Opus 4.8	End to end	61.3	5.73	126.5
Kimi Code / K3	End to end	64.2	20.56	92.0
ARIS	End to end	44.5	0.74	15.1

Appendix

Table 13: Scored workflows, overview scores, and resource use. Wall-clock time covers the whole workflow. For stagewise runs it sums the seven stage runs and excludes queueing. Token counts include cached input. The count of AutoResearchClaw is a lower bound because its backend records do not cover the whole run, and n/a marks the stagewise run of Kimi Code, whose receipts record no token use.

System	Harness or system version	Model	Effort	Context
Codex / GPT-5.5	Codex CLI 0.153.4	gpt-5.5	high	272K
Codex / GPT-5.6 Sol	Codex CLI 0.153.4	gpt-5.6-sol	high	872K
Codex / GPT-6 Astra	Codex CLI 0.153.4	gpt-6-astra	high	872K
Claude Code / Opus 4.8	Claude Code 2.1.269	claude-opus-4-8	high	1M
Kimi Code / K3	Kimi Code 0.39.1	k3	high	1M
ARIS	Codex CLI 0.153.4 with ARIS skills (commit 94d8093 )	gpt-5.5	high	272K

Appendix

Table 14: Configuration of the evaluated systems. All runs request high reasoning effort and allow no fallback model. Context is the model context window. GPT-5.6 Sol and GPT-6 Astra set it explicitly to 872K tokens. The GPT-5.5 runs set no window, so Codex uses its default of 272K tokens for this model and makes 95% (258.4K) of it usable, as the retained sessions of ARIS, Orchestra, and DeepScientist record. Claude Code also calls claude-haiku-4-5 as an auxiliary model.

		Stage							S5 dimension
System	Mode	S1	S2	S3	S4	S5	S6	S7	Mod.	Code	Qual.	Comp.	Overview
Codex / GPT-5.5	End to end	68.8	47.5	56.2	41.7	44.0	42.5	32.5	53.6	50.0	30.0	42.5	45.1
	Stagewise	83.3	67.5	90.6	83.3	77.6	90.0	65.0	67.9	65.0	85.0	92.5	78.5
Claude Code / Opus 4.8	End to end	81.2	65.0	81.2	66.7	57.2	50.0	55.0	71.4	57.5	42.5	57.5	61.3
	Stagewise	89.6	72.5	90.6	70.8	86.1	77.5	62.5	89.3	67.5	92.5	95.0	81.0
Kimi Code / K3	End to end	72.9	65.0	78.1	68.8	61.6	62.5	57.5	71.4	62.5	47.5	65.0	64.2

Appendix

Table 15: Stage scores, execution-dimension scores, and overview scores of the three general-purpose harnesses in end-to-end and stagewise evaluation. Mod., Code, Qual., and Comp. are module development, code execution, result quality, and claim completion.

System

MC1

MC2

MC3

MC4

Met

Rep.

Att.

General-purpose agent harness, end to end

Codex / GPT-5.5

4/4

2/3

2/2

0/0

2/0

2/1

0/0

1/0

2/2

2/0

Codex / GPT-5.6 Sol

4/4

1/0

2/0

3/1

0/0

2/0

Codex / GPT-6 Astra

4/4

0/0

3/1

3/2

0/0

2/0

4/4

2/0

Claude Code / Opus 4.8

4/4

2/3

2/2

3/1

0/0

2/0

2/2

3/2

Kimi Code / K3

4/4

2/2

2/0

3/2

0/0

2/0

2/2

3/1

Appendix

Table 16: Per-claim ratings as claim completion/result quality on the 0–4 scale. Shaded bold cells mark claims whose success criterion is met. Met is the number of claims met. Rep. is the number of claims the evaluated system reported as attained by the end of execution. Att. is the number of attempts on required claims. Reported claims and attempts are not tracked for the stagewise execution runs.

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 319 papers

AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation
May 4, 2026Rajesh Kumar, Waqar Ali, Junaid Ahmed +2Execution-Guided Code Generation Scientific Workflow Automation
International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · Department of Computer Science, College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou 325060, China · Computer Systems Engineering Department, Sukkur IBA University, Sindh, Pakistan +1
ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs
Aug 13, 2026Jiale Cui, Yueyao Yuan, Kaixi Zhong +3Human Preference Evaluation AI Agent Benchmarks
School of Software Technology, Zhejiang University
AutoResearch: Insight In, Hallucination Out
Aug 18, 2026Xiang Liu, Qumeng Sun, Bruno +3Scientific Hypothesis Generation Research Idea Generation
Infinite Evolution Lab, EvoMap

May 4, 2026J/K move · Enter open · S save

AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation

Rajesh Kumar, Waqar Ali, Junaid Ahmed, Abdullah Aman Khan+1

International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · Department of Computer Science, College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou 325060, China · Computer Systems Engineering Department, Sukkur IBA University, Sindh, Pakistan+1

Execution-Guided Code Generation Scientific Workflow Automation Autonomous Research Agents LLM Agent Reliability

Open paper AskPDF arXiv

Automated research agents increasingly generate code, retrieve literature, and draft scientific artifacts, but they often fail to verify whether generated experiments execute correctly or whether cited sources support generated claims. We present AutoResearch, an execution-grounded multi-agent framework for reliable research workflow automation. AutoResearch couples sandboxed Python/PyTorch execution, iterative code repair, citation verification, claim-support auditing, decision control, and structured \LaTeX{} artifact generation. The system treats runtime errors, citation-verification failures, and review-agent feedback as practical filtering signals for generated research artifacts. In controlled evaluations on HumanEval, MBPP, a SciCode subset, citation-validation tasks, claim-support auditing, and small end-to-end workflow stress tests, AutoResearch improves execution success, citation validity, local claim support, and workflow completion relative to directly comparable baselines. Code-oriented agents are reported separately as partial comparisons. AutoResearch is intended as a reliability-oriented research assistant, not as a fully autonomous scientist or a standalone manuscript-quality benchmark. Source Code: https://github.com/raja21068/AutoResearch

Abstract

Autonomous Research Agents

Explore similar work

AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation

ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs

AutoResearch: Insight In, Hallucination Out