Toward Autonomous Long-Horizon Engineering for ML Research

Guoxin Chen; Jie Chen; Lei Chen; Jiale Zhao; Fanzhe Meng; Wayne Xin Zhao; Ruihua Song; Cheng Chen

cs.CLApr 14, 2026

Toward Autonomous Long-Horizon Engineering for ML Research

Authors: Guoxin Chen, Jie Chen, Lei Chen, Jiale Zhao, Fanzhe Meng, Wayne Xin Zhao, Ruihua Song, Cheng Chen,

Abstract

Agentic systems increasingly automate pieces of AI research. Yet turning underspecified research objectives into runnable, experimentally validated ML systems remains a central bottleneck. We study this operational setting as long-horizon ML research engineering: converting a research specification into a runnable ML system through repeated implementation, experimentation, and refinement. The central challenge is to sustain cumulative project progress across heterogeneous stages under delayed, confounded feedback. We introduce AiScientist, a multi-agent system built around thin control over thick state: a lightweight hierarchical research team coordinates through a File-as-Bus workspace that preserves decision-relevant artifacts across roles and invocations. On PaperBench, AiScientist improves over the strongest matched baselines by 9.92 and 11.15 points with Gemini-3-Flash and GLM-5, respectively. On MLE-Bench Lite, it reaches 81.82 Any Medal% under both backbones, improving over the strongest matched baselines by 4.55 and 16.67 points, and exceeding a Codex/GPT-5.5 xhigh frontier harness reference by 13.64 Any Medal points. Ablations and process analyses show that durable project state is central to later-round refinement: removing File-as-Bus lowers PaperBench score by 6.41 points and MLE-Bench Lite Any Medal% by 31.82 points. These results suggest that long-horizon AI research is not only a problem of stronger local reasoning, but a systems problem of maintaining cumulative, inspectable project progress.

Figures & tables

Figure 1: AiScientist autonomously improving performance on a competition-style ML task over 23 hours. On MLE-Bench Lite’s Detecting Insults task, it conducted 74 experiment cycles without human intervention, raising validation AUC from 0.903 to 0.982 through 18 best-so-far updates.

Figure 2: Overview of AiScientist. A top-level Orchestrator maintains thin control through stage-level directives, concise summaries, and a compact workspace map. Specialized agents perform paper comprehension, task prioritization, implementation, experimentation, and auxiliary exploration, while coordinating through a File-as-Bus workspace that serves as the system of record. This design turns transient agent invocations into durable role-level and project-level progress by externalizing decision-relevant state into persistent project artifacts.

Task Name	GPT-5.5	Gemini-3-Flash				GLM-5
Task Name	Codex	BasicAgent	IterAgent	AiScientist	$\Delta$	BasicAgent	IterAgent	AiScientist	$\Delta$
adaptive-pruning	33.42	24.53	3.05	27.25	+2.72	30.82	11.93	33.26	+2.44
all-in-one	52.93	20.86	45.13	46.29	+1.16	33.78	44.43	49.47	+5.04
bam	56.65	48.46	45.04	56.59	+8.13	51.45	47.91	61.11	+9.66
bbox	17.30	15.43	8.30	33.79	+18.36	23.55	19.28	30.02	+6.47
bridging-data-gaps	14.16	12.59	12.44	23.09	+10.50	9.80	12.50	26.46	+13.96

Table 1: Main results on PaperBench full evaluation. Codex with GPT-5.5 xhigh is shown as a frontier harness reference; red values indicate AiScientist’s gains over the strongest matched baseline. Bold and underlined denote the best and second-best results within each backbone block.

Agent	Model	Valid Submission	Above Median	Bronze	Silver	Gold	Any Medal
Official MLE-Bench Leaderboard Results
InternAgent	Deepseek-R1	$100.00$ $\pm 0.00$	$78.79$ $\pm 5.46$	$10.61$ $\pm 1.52$	$16.67$ $\pm 3.03$	$34.85$ $\pm 1.52$	$62.12$ $\pm 3.03$
ML-Master	Deepseek-R1	$100.00$ $\pm 0.00$	$74.24$ $\pm 1.52$	$4.55$ $\pm 2.62$	$13.64$ $\pm 0.00$	$30.30$ $\pm 3.03$	$48.48$ $\pm 1.52$
AIRA-dojo	o3	$100.00$ $\pm 0.00$	$70.45$ $\pm 1.60$	$7.95$ $\pm 0.86$	$12.73$ $\pm 1.42$	$34.32$ $\pm 1.02$	$55.00$ $\pm 1.47$
ML-Master 2.0	Deepseek-V3.2-Spe	$100.00$ $\pm 0.00$	$84.85$ $\pm 1.52$	$13.64$ $\pm 2.62$	$31.82$ $\pm 5.25$	$30.30$ $\pm 3.03$	$75.76$ $\pm 1.52$
R&D-Agent	GPT-5	$77.27$ $\pm 0.00$	$74.24$ $\pm 1.52$	$12.12$ $\pm 4.01$	$22.73$ $\pm 0.00$	$33.33$ $\pm 3.03$	$68.18$ $\pm 2.62$

Table 2: Main results on MLE-Bench Lite. Values are percentages reported as mean

\pm

SEM over three runs/seeds following the MLE-Bench convention. Official leaderboard rows are contextual references, the Codex/GPT-5.5 xhigh row is a frontier harness reference evaluated under our setup, and controlled evaluation rows are matched comparisons. Bold and underlined denote the best and second-best Any Medal performance.

Panel of Figure 3 — Figure 3: Long-horizon improvement dynamics on MLE-Bench Lite under GLM-5. (a) Mean validation best-so-far normalized score with $\pm 1$ SEM across task–seed trajectories. (b) Pairwise lead rate of AiScientist against AIDE and Codex, with binomial standard error. (c) Cumulative distribution of when runs first reach their final best score. AiScientist starts slower but continues improving after both references largely plateau. Its advantage first becomes broad across paired runs, and many runs require late-budget refinement to reach their final best.

Figure 4: Mechanism analysis of AiScientist under GLM-5. Left: PaperBench ablations show that both hierarchical organization and File-as-Bus contribute to replication performance; removing File-as-Bus alone lowers score by 6.41 points. Right: on MLE-Bench Lite, File-as-Bus removal preserves first-pass validity but substantially reduces medal-level outcomes, indicating that durable state continuity primarily supports later-round refinement.

Figure 5: Stage-level workflow-step allocation on PaperBench under GLM-5. Grouping steps by workflow stage enables comparison across different methods. Codex is a strong low-step reference, while AiScientist concentrates more steps in the core engineering loop.

Figure 6: Workflow-step allocation for higher- and lower-scoring PaperBench trajectories under GLM-5. Tasks are split by final PaperBench score, separately for AiScientist and its variant without File-as-Bus. With File-as-Bus, higher-scoring trajectories spend more of their later effort on validation and closure; without File-as-Bus, higher-scoring trajectories use substantially more implementation and experimentation steps, suggesting less efficient reuse of prior evidence.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Artifact	Purpose	Update Mode	Writer	Readers
paper_analysis/summary.md	Concise implementation-oriented summary of the paper, including task goal, target result, and key assumptions.	Versioned artifact	Paper Comprehension	Orchestrator, Prioritization, Implementation, Experimentation
paper_analysis/structured.md	Structured extraction of datasets, metrics, methods, baselines, training details, and resource requirements.	Versioned artifact	Paper Comprehension	Orchestrator, Prioritization, Implementation, Experimentation
paper_analysis/algorithm.md	Algorithm-level reconstruction of the proposed method and implementation-relevant details.	Versioned artifact	Paper Comprehension	Implementation, Experimentation, Orchestrator
paper_analysis/baseline.md	Baseline methods, expected comparisons, and result targets that should be reproduced or approximated.	Versioned artifact	Paper Comprehension	Prioritization, Implementation, Experimentation, Orchestrator
agent/prioritized_task.md	Current ordered task plan, including dependencies, expected impact, feasibility, and next actions.	Versioned artifact	Prioritization	Orchestrator, Implementation, Experimentation
submission/	Runnable repository under construction, including source code, configuration, scripts, and generated project files.	Mutable runnable artifact	Implementation	Implementation, Experimentation, Orchestrator

Appendix

Table 3: File-as-Bus artifact schema used by AiScientist. Each artifact has an explicit purpose, update mode, writer, and reader set. This schema turns the shared workspace into a collaboration contract rather than an unstructured scratchpad.

Figure 7: Delegation ratio across PaperBench tasks under GLM-5. Subagent delegation is not concentrated in a small number of tasks; it appears across most tasks, indicating that the hierarchical team is used as a systematic control mechanism rather than as an occasional fallback.

Figure 8: Budget-dependent medal outcomes on MLE-Bench Lite under GLM-5. The figure provides a supplementary view of how competitive outcomes emerge over the 24-hour budget.

Long-Horizon LLM Agents

Long-horizon large language model agents plan, use tools, and preserve task state across extended, multi-step interactions.

See all 446 papers

ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond
Aug 14, 2026Mingming Zhao, Jiqian Dong, Kangping Xu +16Long-Horizon LLM Agents Autonomous Research Agents
Noah’s Ark Lab, Huawei
MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering
Oct 15, 2026Shangheng Du, Jinxin Shi, Shuaiyu Zhang +7LLM Agent Training Automated Machine Learning
East China Normal University · Shanghai Artificial Intelligence Laboratory
ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments
Mar 17, 2026Ziyang Cai, Amir Saeidi, Harkirat BehlAutonomous Research Agents LLM Agent Training
Arizona State University · Microsoft Research

Aug 14, 2026J/K move · Enter open · S save

ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond

Mingming Zhao, Jiqian Dong, Kangping Xu, Zadid Hasan+15

Noah’s Ark Lab, Huawei

Long-Horizon LLM Agents Autonomous Research Agents Autonomous Scientific Discovery LLM Agent Workflow Optimization

Open paper AskPDF arXiv

Enabling LLM agents to sustain productive, stable, and goal-aligned research over extended horizons is a central challenge for autonomous machine learning and scientific discovery, as progress hinges on continuously managing evolving state, exploration decisions, and computational resources. Pioneering autoresearch agents, despite great success, still lack mechanisms for continuity, recovery from dead ends, and value-driven compute allocation, which inherently undermines overall search efficiency, wastes computational resources, and lowers the chance of ultimate success. To bridge this gap, we introduce ScienceFlow, an end-to-end autoresearch agent framework that organizes long-horizon research work into research segments grounded in executable workspaces. It represents research progress as recoverable executable states, enabling efficient exploration, revision, and execution. Transitions between research segments are governed by Executable-State Transition through Re-Anchoring (ESTRA), which selects either the live state or an archived state as the next anchor and determines whether to continue or redirect the research trajectory. An evidence-aware execution controller allocates resources to physical jobs based on resource availability, remaining budget, and validated progress. We evaluate ScienceFlow on tasks spanning machine learning, scientific modeling, and mathematical optimization. Results on diverse long-horizon benchmarks demonstrate its ability to sustain effective research processes, highlighted by a SOTA 70.22 percent Any-Medal score on the full MLE-bench within a 24-hour budget, outperforming prior reported results by 4.92 percentage points. The efficacy of ScienceFlow further demonstrates that efficient state management, adaptive exploration, and objective-aligned execution are critical for scaling autonomous research beyond short-horizon interactions.

Abstract

Long-Horizon LLM Agents

Explore similar work

ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond

MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering

ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments