ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution

Yubang Wang; Chenxi Zhang; Bowen Chen; Zezheng Huai; Zihao Dai; Xinchi Chen; Yuxin Wang; Yining Zheng

cs.SEMar 6, 2026

ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution

Authors: Yubang Wang, Chenxi Zhang, Bowen Chen, Zezheng Huai, Zihao Dai, Xinchi Chen, Yuxin Wang, Yining Zheng,

Organizations: Shanghai Innovation Institution · College of Computer Science and Artificial Intelligence, Fudan University · OpenMOSS · Wuhan University ·

Abstract

Autonomous agents are increasingly expected to support scientific research, and recent benchmarks report progress in code repair and autonomous experimentation. However, these evaluations typically assume a pre-configured execution environment, which requires resolving complex software dependencies, aligning hardware and framework versions, and configuring distributed execution, yet this capability remains largely unbenchmarked. We introduce ResearchEnvBench, a benchmark for environment synthesis in research code execution. Given a research repository, documentation, and a target execution setting, agents must construct an environment that successfully executes at runtime. Evaluations on diverse research repositories reveal a substantial gap in current SOTA agents, with failures dominated by incomplete dependency resolution and brittle version coupling. ResearchEnvBench provides a realistic testbed for advancing autonomous agents toward reproducible scientific research.

Figures & tables

Figure 1: Agentic closed-loop resolution of research environments under implicit constraints. Starting from a raw research repository (left), an LLM-driven agent iteratively observes execution signals, diagnoses missing dependencies and compatibility constraints, acts by installing/pinning packages or switching toolchains, and verifies by re-running probes (center), aiming to reach an executable research-ready state (right). The multi-layer dependency stack highlights that compatibility constraints propagate downward (Python $\rightarrow$ system $\rightarrow$ CUDA $\rightarrow$ driver/hardware), while failures typically surface upward as logs/tracebacks. Evaluation reports importability ( $C_{0}$ ), runtime CPU/GPU checks ( $C_{1}$ – $C_{4}$ ), and an auditable, reproducible report and logs ( $C_{5}$ ).

Benchmark	Repos	Langs	Domain	HW Aware	Evaluation
Installamatic [ 17 ]	40	Py	Gen. SWE	✗	Build success
EnvBench [ 5 ]	994	Py, JVM	Gen. SWE	✗	Missing imports
SUPER [ 18 ]	45	Py	ML	✗	CPU execution
Multi-Docker-Eval [ 6 ]	40	9 langs	Gen. SWE	✗	Build + tests
SetupBench [ 16 ]	54	7 langs	DevOps	✗	Functional checks
ResearchEnvBench (Ours)	44	Py, C++	AI/HPC	✓	Runtime pyramid ( $C_{0}$ – $C_{4}$ )

Table 1: Comparison of ResearchEnvBench with state-of-the-art environment setup and execution benchmarks. Prior benchmarks often target general software engineering (Gen. SWE) or DevOps (development/operations) and evaluate via build success, missing-import checks, unit tests (e.g., pytest ), or functional checks; ResearchEnvBench focuses on AI/HPC research and uses a hardware-aware runtime pyramid (

C_{0}

–

C_{4}

Figure 2: Repository composition of ResearchEnvBench. We curate 44 post-2024 ML research repositories spanning eight categories: GenVis (generative vision), Depth (depth estimation), Audio (audio & speech), LLM-Inf (LLM inference & acceleration), TrainEng (training & engineering frameworks), VisMM (vision/multimodal foundations), DocAI (document AI / OCR / translation), and AppsEval (applications & evaluation). Slice sizes denote the fraction of repositories per category by count, and each slice shows a representative flagship repository from that category.

Figure 3: The Pyramid of Runtime Verification. We structure environment synthesis into a hierarchy of increasing complexity, ranging from static dependency resolution ( $C_{0}$ ) to multi-GPU distributed training ( $C_{4}$ ). The right panel illustrates our auxiliary metrics: Efficiency (Time, Token, Size) and Capability Hallucination ( $C_{5}$ ), which quantifies the discrepancy between the agent’s self-reported success (Report) and the ground-truth execution (Real Result).

Agent	$C_{0}$ missing/total $\downarrow$	$C_{1}$ CPU SR	$C_{2}$ CUDA SR	$C_{3}$ 1-GPU SR	$C_{4}$ DDP SR	$C_{5}$ path	$C_{5}$ version	$C_{5}$ cap.	$C_{5}$ total $\downarrow$
Codex (GPT-5.1-Codex)	675/2858 (23.6%)	17/29 (58.6%)	35/44 (79.5%)	19/43 (44.2%)	11/32 (34.4%)	0	0	4	4
Claude Code Agent (GLM-4.7)	761/2858 (26.6%)	16/29 (55.2%)	40/44 (90.9%)	21/43 (48.8%)	12/32 (37.5%)	0	1	17	18
Claude Code Agent (Sonnet 4.5)	728/2858 (25.5%)	15/29 (51.7%)	41/44 (93.2%)	18/43 (41.9%)	12/32 (37.5%)	0	0	20	20
NexAU (DeepSeek-V3.1-Nex-N1)	722/2858 (25.3%)	16/29 (55.2%)	41/44 (93.2%)	21/43 (48.8%)	11/32 (34.4%)	1	1	14	16

Table 2: Main results on ResearchEnvBench.

C_{0}

reports missing imports over total imports after synthesis;

C_{1}

–

C_{4}

report stage success rates (success/total on applicable repositories).

C_{5}

counts hallucinations by type (Path/Version/Capability) and totals them (

\downarrow

is better).

Agent	Avg. Tokens (k)	Avg. Setup Time (min)	Avg. Env Size (GB)	$C_{4}$ Success Rate (%)
Codex (GPT-5.1-Codex)	48.4	6.8	6.06	34.4 (11/32)
Claude Code (GLM-4.7)	–	9.8	6.65	37.5 (12/32)
Claude Code (Sonnet 4.5)	–	13.3	6.88	37.5 (12/32)
NexAU (DeepSeek-V3.1-Nex-N1)	957.1	9.2	6.55	34.4 (11/32)

Table 3: Resource Consumption Comparison. Avg. Tokens (k) is the total prompt+completion tokens consumed during synthesis. Avg. Setup Time is agent-phase wall time. Avg. Env Size comes from the env-size stage ( env_size_avg_gb ).

C_{4}

SR is computed as success/(total

-

skipped), excluding repos where

C_{4}

is inapplicable.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,482 papers

AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation
May 4, 2026Rajesh Kumar, Waqar Ali, Junaid Ahmed +2Execution-Guided Code Generation Scientific Workflow Automation
International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · Department of Computer Science, College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou 325060, China · Computer Systems Engineering Department, Sukkur IBA University, Sindh, Pakistan +1
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
May 28, 2026Wanghan Xu, Shuo Li, Tianlin Ye +48LLM Agent Evaluation AI Agent Benchmarks
POSEIDON: Scaling Training Environments for Scientific Agents from Code Repositories
Oct 23, 2026Qifan Zhang, Yuhan Li, Nuo Chen +8AI Agents for Scientific Discovery LLM Agent Training
The Hong Kong University of Science and Technology (Guangzhou) · Hunyuan Team, Tencent

May 4, 2026J/K move · Enter open · S save

AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation

Rajesh Kumar, Waqar Ali, Junaid Ahmed, Abdullah Aman Khan+1

International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · Department of Computer Science, College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou 325060, China · Computer Systems Engineering Department, Sukkur IBA University, Sindh, Pakistan+1

Execution-Guided Code Generation Scientific Workflow Automation Autonomous Research Agents LLM Agent Reliability

Open paper AskPDF arXiv

Automated research agents increasingly generate code, retrieve literature, and draft scientific artifacts, but they often fail to verify whether generated experiments execute correctly or whether cited sources support generated claims. We present AutoResearch, an execution-grounded multi-agent framework for reliable research workflow automation. AutoResearch couples sandboxed Python/PyTorch execution, iterative code repair, citation verification, claim-support auditing, decision control, and structured \LaTeX{} artifact generation. The system treats runtime errors, citation-verification failures, and review-agent feedback as practical filtering signals for generated research artifacts. In controlled evaluations on HumanEval, MBPP, a SciCode subset, citation-validation tasks, claim-support auditing, and small end-to-end workflow stress tests, AutoResearch improves execution success, citation validity, local claim support, and workflow completion relative to directly comparable baselines. Code-oriented agents are reported separately as partial comparisons. AutoResearch is intended as a reliability-oriented research assistant, not as a fully autonomous scientist or a standalone manuscript-quality benchmark. Source Code: https://github.com/raja21068/AutoResearch

Abstract

AI Agent Benchmarks

Explore similar work

AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

POSEIDON: Scaling Training Environments for Scientific Agents from Code Repositories