LabFactory: Building and Evaluating Executable AI Labs

Jinge Wu; Hongjian Zhou; Mingde Zeng; Jiayuan Zhu; Junde Wu; Jiazhen Pan; Lei Clifton; Andrew Liu

cs.LGSep 23, 2026

LabFactory: Building and Evaluating Executable AI Labs

Authors: Jinge Wu, Hongjian Zhou, Mingde Zeng, Jiayuan Zhu, Junde Wu, Jiazhen Pan, Lei Clifton, Andrew Liu, +1 more

Organizations: University of Oxford

Abstract

Scientific tasks specify a desired capability, but realizing it often requires building a computational system tailored to the task---acquiring data, designing representations, training models, implementing tools, and deciding how they are used at inference. We present LabFactory, a framework in which an AI builder turns a scientific brief into an executable AI lab: a task-specific solver that integrates models, knowledge resources, tools, and a controller behind a fixed interface. The builder develops and packages the lab in a metered workspace; a separate host then executes the delivered artifact on held-out inputs, with reference labels kept outside the solver's input interface, and scores its outputs under the task's protocol. This makes the delivered system, rather than the builder's account of its progress, the object of evaluation. We document 28 selected constructions across seven scientific task categories---from molecular and genomic prediction to physiological signals, clinical decision support, and biomedical text---whose delivered labs exceeded their configured reference values on all 33 subtests under host-side execution. Ten contain predictive models fitted during construction; the others assemble retrieval systems, executable analysis environments, and tool-driven workflows around a fixed platform LLM. Together they show that an AI agent can carry a scientific brief all the way to a working lab that can still be invoked, inspected, and checked after construction ends.

Figures & tables

Figure 1: The LabFactory framework. Construction produces a solver composed of models, knowledge, tools, and a controller. The runtime records actions and resource consumption; independent evaluation runs the delivered artifact. “Hidden reference” denotes the host-side reference. Illustrated modalities indicate possible task interfaces, not the reported result set.

Figure 2: Performance across 28 scientific tasks. Colored columns show host-side results of the delivered artifacts; dark horizontal ticks mark the task-specific reference bars. Colors distinguish scientific categories. Scores use heterogeneous metrics on their native scale (MAE cases use a higher-is-better skill transform), so column heights and gaps are not comparable effect sizes across tasks. Column labels abbreviate the task names of Table 2 , which lists the tasks in the same order.

Figure 3: Recorded time and resource consumption of the 28 selected runs. Each point is one episode, and colors identify task groups. These are descriptive construction-episode measurements, not inference costs.

Task group	Cases	Hours	Tokens M	GPU min	Credits M
Molecules & proteins	2	19.54	23.86	544.5	1.04
Genomics & omics	4	12.59	39.16	299.4	2.10
Imaging	1	20.09	28.80	217.4	0.49
Physiological signals	3	18.72	59.38	957.7	0.45
Knowledge & reasoning	8	15.39	15.45	46.9	4.37
Clinical decision support	2	17.01	36.11	9.5	9.98

Table 1: Task-group counts and median selected-run resource use. Each resource column is a within-group median, not a sum. Imaging contains one case. Heterogeneous performance metrics are not averaged.

Figure 4: Construction of the polyadenylation solver. The builder implements build utilities and a task interface, obtains data, constructs a 5-mer retrieval index, trains a CNN predictor, connects the LLM, performs development self-checks, and packages the artifact. The train, validation, and development-test split belongs to construction; the final box is the evaluation of § 3.5 .

Figure 5: Internal structure of the polyadenylation solver. A candidate DNA window enters the solver and produces a functional or non-functional prediction. The decision controller coordinates sequence tools, a CNN ensemble, biological guidance, and 5-mer retrieval; curved arrows summarize tool interaction and review at inference time. The LLM reviews every case; the box labelled “LLM review; uncertainty-gated adoption” marks where its decision is adopted, only inside the uncertainty band $0.44<p(x)<0.60$ . The positive-rate guard caps positives at 62% of the batch before output (Eq. ( 4 ) and the four stages in the text).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Subtest	Metric	$n$	Solver	Reference bar	Reference source
Protein–protein interaction	Bernett PPI	AUROC	3000	0.9874	0.7000	Liu et al.,2025
Protein–ligand binding affinity	PDBbind	Pearson $r$	285	0.9141	0.8030	Graber et al., 2025 7
Causal eQTL variants	eQTL	AUPRC	8862	0.9192	0.5960	Wang et al., 2025 §
Bioinformatics data analysis	BixBench	Judge acc.	296	0.4561	0.2280	Ghareeb et al., 2026 1
5mC methylation sites	5mC	AUROC	1407	0.9530	0.7830	Feng et al.,2025
Polyadenylation signal detection	Poly(A) signal	F1	3000	0.8977	0.8800	Shen et al.,2026

Appendix

Table 2: All subtests of the 28 reported cases, comprising 33 subtests. Solver is the delivered solver’s value under independent evaluation and Reference bar the configured reference value, both on the primary metric; the MAE rows are in skill space (native errors in Appendix A ). These metric types (Appendix A ) are reported separately and should not be interpreted interchangeably; jury scores and MAE skill values are not classification accuracies. Reference-source markers and notes 1–7 are explained below the table.

Figure 6: Constructed solvers versus the raw-LLM baseline. All 12 matched accuracy comparisons from nine cases, sorted by the solver-minus-baseline difference. Open circles denote the raw LLM and filled circles the constructed solver; colors follow the scientific categories in Figures 2 and 3 . Coincident markers indicate equal accuracy. The right column reports changes in percentage points (pp). Row labels give an abbreviated task name and the subtest; “Medical QA” is the medical reasoning task of Table 2 .

Subtest	$n$	Raw LLM	Solver	$\Delta$ (pp)
MedMCQA	4183	0.7930	0.8735	+8.05
MEDIQA	1107	0.7868	0.8193	+3.25
MetaMedQA	1373	0.8267	0.8529	+2.62
LabSafety	632	0.8481	0.8718	+2.37
MEDEC	597	0.8375	0.8526	+1.51
Medbullets	308	0.8994	0.9123	+1.29

Appendix

Table 3: The 12 matched accuracy comparisons plotted in Figure 6 , in the same order. Raw LLM and Solver are accuracies on a 0–1 scale;

\Delta

is Solver minus Raw LLM in percentage points.

Explore similar work

May 6, 2026eess.SY

Experiment-as-Code Labs: A Declarative Stack for AI-Driven Scientific Discovery

To unleash the full potential of AI for Science, we must untether the agents from a purely digital environment. The agent's ability to control and explore in real-world labs is essential because the physical lab remains foundational to scientific discovery. While some tasks can be performed on a computer (e.g., data analysis, running simulated experiments), Eureka moments could occur at any time while operating lab instruments (e.g., when a scientist notices unexpected clues, intuition may prompt a real-time course change). Although autonomous labs are on the rise, which expose programmable APIs to control scientific instruments via software, bridging the gap between increasingly powerful AI agents and automated lab equipment requires innovation that draws insights from computer systems. We propose a new paradigm called ``Experiment-as-Code (EaC) Labs,'' where a core concept is to encode experiments as declarative configurations that can be compiled down to device-level APIs. AI agents come up with hypotheses and experiments, written as an ensemble of declarative configurations. The systems layer performs program analysis, safety checks, resource assignment, and job orchestration. Finally, programmatic experimentation occurs via actuating the device APIs. This is a general stack that is science-, lab-, and instrument-independent, representing a novel synthesis across the physical, systems, and intelligence layers to unleash the next breakthrough in AI for Science.

Zhenning Yang, Yuhan Chen, Patrick Tser Jern Kon +5

University of Michigan

Laboratory Artificial Intelligence Scientists Scientific Discovery Case Study

Ask PDF

May 15, 2026cs.AI

From Prompts to Protocols: An AI Agent for Laboratory Automation

Automating science laboratories enables faster, safer, more accurate, and more reproducible execution of protocols, accelerating the discovery and testing of new materials, drugs, and more. However, setting up and running autonomous labs requires coordinating numerous instruments and robots, forcing scientists to write code, manage configuration files, and navigate complex software infrastructure. We present an AI agent architecture that integrates large language models with laboratory orchestration, enabling scientists to interactively create and monitor automated lab protocols using natural language. Integrated into the Experiment Orchestration System (EOS), the AI agent operates under an agentic loop with automated validation and error correction, and supports the complete experimental lifecycle: creating protocols, running and monitoring both protocols and closed-loop optimization campaigns, and analyzing results. A visual graph editor renders protocols as interactive node-based diagrams synchronized with the AI agent's protocol representation, enabling seamless alternation between AI-assisted and manual protocol construction. Evaluated on three simulated automated labs spanning chemistry, biology, and materials science, the AI agent achieves a 97% first-attempt protocol generation success rate and an order of magnitude reduction in required interface actions.

Angelos Angelopoulos, James F. Cahoon, Ron Alterovitz

Department of Computer Science, University of North Carolina at Chapel Hill, NC 27599, USA · Department of Chemistry, University of North Carolina at Chapel Hill, NC 27599, USA

Laboratory Artificial Intelligence Agents Agent Loop Orchestration Protocol Prompt Engineering

Ask PDF

Jul 25, 2026cs.AI

Stress-testing large language model agents in a robotic chemistry laboratory

AI is evaluated through knowledge, reasoning and plan generation, yet scientific agency requires reliable physical action and adaptation to evidence. Here, we use a robotic chemistry laboratory as a physical-world testbed to make scientific agency measurable. Its 45 modular workstations exposed as machine-readable skills enabled 4,608 trials. Only 3.3% of trials produced expert-assessed executable workflows under laboratory constraints; even the best system achieved 28.1%. Long-horizon planning remained a challenge: only three executable workflows exceeded 30 operations, although the longest contained 44. Across five rounds, experimental feedback prompted local adjustments but no workflow-level replanning or analytical-method redesign. By making physical executability and evidence-driven replanning measurable, our study provides an evidence-based assessment of deployment readiness and a diagnostic framework to guide closed-loop improvements towards physically grounded autonomous research.

Lulu Guo, Yingkai Sun, Xiaobo Li +13

State Key Laboratory of Precision and Intelligent Chemistry, Hefei National Research Center for Physical Sciences at the Microscale, School of Chemistry and Materials Science, University of Science and Technology of China, Hefei, China · Center for Scientific Intelligence Innovation, Hefei, China · School of Chemistry, School of Computer Science, University of Birmingham, Birmingham, UK

Laboratory Large Language Model Agents Scientific Discovery Iterative Re-Planning

Ask PDF