AlphaLab: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs

Brendan R. Hogan; Xiwen Chen; James T. Wilson; Kashif Rasul; Adel Boyarsky; Thomas Kamei; Anderson Schneider; Yuriy Nevmyvaka

cs.LGMar 31, 2026

AlphaLab: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs

Authors: Brendan R. Hogan, Xiwen Chen, James T. Wilson, Kashif Rasul, Adel Boyarsky, Thomas Kamei, Anderson Schneider, Yuriy Nevmyvaka

Organizations: Morgan Stanley

Abstract

We present AlphaLab, an autonomous research harness that leverages frontier LLM agentic capabilities to automate the full experimental cycle in quantitative, computation-intensive domains. Given only a dataset and a natural-language objective, AlphaLab proceeds through three phases without human intervention: (1) it adapts to the domain and explores the data, writing analysis code and producing a research report; (2) it constructs and adversarially validates its own evaluation framework; and (3) it runs large-scale GPU experiments via a Strategist/Worker loop, accumulating domain knowledge in a persistent playbook that functions as a form of online prompt optimization. All domain-specific behavior is factored into adapters generated by the model itself, so the same pipeline handles qualitatively different tasks without modification. We evaluate AlphaLab with two frontier LLMs (GPT-5.2 and Claude Opus 4.6) on three domains: CUDA kernel optimization, where it writes GPU kernels that run 4.4x faster than torch.compile on average (up to 91x); LLM pretraining, where the full system achieves 22% lower validation loss than a single-shot baseline using the same model; and traffic forecasting, where it beats standard baselines by 23-25% after researching and implementing published model families from the literature. The two models discover qualitatively different solutions in every domain (neither dominates uniformly), suggesting that multi-model campaigns provide complementary search coverage. We additionally report results on financial time series forecasting in the appendix, and release all code at https://brendanhogan.github.io/alphalab-paper/.

Figures & tables

Figure 1: AlphaLab pipeline overview. Given a dataset and objective, the system writes all its own code across four phases. Phase 0: configures domain-specific prompts/metrics. Phase 1: explores the dataset and researches prior work via web search. Phase 2: adversarial Builder/Critic/Tester loop constructs the evaluation framework. Phase 3: Strategist proposes experiments, Workers implement them, and jobs run on a GPU cluster via Slurm; a persistent Playbook evolves with each experiment. A Supervisor monitors health. The human can interact with the Strategist to guide search; we run fully autonomously in this paper for fair comparison.

Model	Best val_bpb	Best config	Cost
AlphaLab + GPT-5.2	0.9697	8L $\times$ 512d, GQA, cosine	$\sim$ $150
AlphaLab + Sonnet 4.6	0.8686	11L $\times$ 768d, QK-norm, Muon	$\sim$ $120
AlphaLab + Opus 4.6	0.7578	10L $\times$ 752d, QK-norm	$\sim$ $200
AlphaLab + GPT-5.1-mini	— †	(no valid results)	$\sim$ $40
Greedy loop (GPT-5.2)	1.020	12L $\times$ 768d, LLaMA, AdamW	$\sim$ $50
Single-shot (GPT-5.2)	1.248	27.4M LLaMA-style	$<$ $1

Table 1: LLM pretraining speedrun results. Task: train a

<

100M-parameter language model from scratch on the PleIAs SYNTH corpus under a 20-minute wall-clock budget on a single H100, then measure validation bits-per-byte (val_bpb, lower is better) on a held-out set. Configs describe the best architecture found: “L” = layers, “d” = model dimension, “GQA” = grouped-query attention, “QK-norm” = query/key normalization for training stability, “cosine”/“Muon” = learning rate schedule or optimizer. The greedy loop is a Karpathy-style AutoResearch baseline ( Karpathy, 2026 ) : the LLM proposes a modification, trains, and keeps or reverts based on the metric (50 iterations, 3 improvements). The single-shot baseline is one LLM call with one retry. † GPT-5.1-mini ran 33 experiments but failed to correctly implement the BPB metric; all results were broken (see text).

Variant	Removed	Best val_bpb	$\Delta$ vs. full
Full AlphaLab	(nothing)	0.9697	—
Skip Phase 1	Exploration	1.0908	$+$ 0.121 ( $+$ 12.5%)
No playbook	Knowledge acc.	0.9941	$+$ 0.024 ( $+$ 2.5%)
Variance (5 runs)	(nothing)	$0.994\pm 0.025$	—

Table 2: Ablation results on LLM speedrun (GPT-5.2, 50-experiment budget). Each ablation removes a single pipeline component while holding everything else constant (same hardware, shared Phase 2 harness). “Skip Phase 1” removes the data exploration phase where the system researches prior work on arXiv and analyzes the dataset before experimenting. “No playbook” removes the persistent knowledge document that accumulates lessons across experiments; without it, the system cannot remember what worked or failed in earlier experiments and must rediscover these lessons from scratch. The “Variance” row reports the mean

\pm

standard deviation of the best val_bpb across five runs of the full system.

Direct comparison (54 tasks, both models correct, H100, vs torch.compile )
System	Model	Correct	Mean spd.	$\textbf{fast}_{1}$
AlphaLab	GPT-5.2	54	5.17 $\times$	91%
AlphaLab	Opus 4.6	53	4.63 $\times$	70%
Full run (H100, vs torch.compile )
AlphaLab	GPT-5.2	110/119	4.40 $\times$	83%
AlphaLab	Opus 4.6	76/87	4.00 $\times$	70%

Table 3: CUDA kernel optimization results. Task: write a CUDA kernel matching PyTorch’s output but faster, on KernelBench ( Ouyang et al., 2025 ) (100 single-op + 100 fusion tasks). Top: head-to-head on 54 tasks where both models produced correct kernels – same tasks, hardware, baseline. Middle: each model’s full campaign (budget-capped at 50 experiments, so

<

200 tasks attempted). Bottom: AlphaLab re-reported against torch.native to enable comparison with external baselines. Sakana AI ( Lange et al., 2025b ) : 5-model ensemble, excludes contaminated tasks. KernelBench: L40S GPU,

\text{fast}_{1}

as L1/L2%. Cross-system comparison is approximate due to hardware and baseline differences.

Model	Lit. RMSE	AlphaLab RMSE	Cost
AlphaLab + Opus 4.6 (TFT, dropout 0.3)	—	0.02142	$\sim$ $200
AlphaLab + GPT-5.2 (iTransformer ctx336)	—	0.02204	$\sim$ $180
DeepAR ( Salinas et al., 2019 )	0.0249	0.0251	—
PatchTST ( Nie et al., 2023 )	0.0311	0.0226	—
TiDE ( Das et al., 2023 )	0.0281	0.0230	—
DLinear ( Zeng et al., 2023 )	0.0335	0.0234	—

Table 4: Traffic forecasting results (RMSE, lower is better). Task: predict hourly road occupancy 24h ahead for 862 freeway sensors. Seasonal Naïve(168) repeats the value from one week ago (0.0287). “Lit. RMSE” = published result using each paper’s protocol; “ AlphaLab RMSE” = best result when AlphaLab independently discovered and tuned that model family on its own harness. Discrepancies reflect tuning and protocol differences. Single-shot = one LLM call; greedy loop = 50 sequential propose-then-train iterations. The greedy loop’s worse-than-single-shot result reflects path dependence: its sequential propose-and-revert structure can converge to a local optimum that a single lucky call avoids.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Description
shell_exec	Full shell access: execute arbitrary commands, write and run code, install packages, manage files.
read_file	Read any file in the workspace.
grep_file	Search file contents by pattern.
web_search	Search the internet for papers, documentation, and best practices.
view_image	View plots and visualizations the agent has generated.
spawn_agent	Launch a sub-agent: a new instance of $\mathcal{M}$ with its own context and full tool access.

Appendix

Table 5: Full tool set available to all AlphaLab agents.

Panel of Figure 2 — Figure 2: Representative plots generated autonomously by AlphaLab ’s Explorer (Phase 1) and Reporter (Phase 3) agents across three domains. These plots are generated by LLM-written Python scripts and viewed by the agent via the view_image tool to inform subsequent analysis. Font sizes reflect the system’s raw output; we reproduce them unmodified to preserve fidelity.

Domain	Model	Best metric	vs. baseline	Cost
CUDA	GPT-5.2	5.14 $\times$ mean	—	$190
CUDA	Opus	4.48 $\times$ mean	—	$170
LLM	GPT-5.2	0.970 BPB	—	$150
	Sonnet	0.869 BPB	$-$ 10%	$120
	Opus	0.758 BPB	$-$ 22%	$200
Traffic	Opus	0.0214 RMSE	$-$ 25%	$200

Appendix

Table 6: Cross-domain summary. GPT-5.2 produces faster CUDA kernels (mean speedup on the 66-task overlap), while Opus 4.6 achieves lower validation loss on LLM pretraining (22% better than GPT-5.2) and lower RMSE on traffic forecasting. Neither model dominates uniformly. “vs. baseline” for LLM is relative to GPT-5.2; for traffic, relative to Seasonal Naïve(168). Cost is approximate API spend per campaign.

Domain	Metric	Dir.	Budget	Hardware	Time Limit
CUDA Kernel	speedup ( $\times$ )	max	50	4 $\times$ H100	7,200 s
LLM Pretraining	val_bpb	min	50	4 $\times$ H100	1,200 s
Traffic	RMSE	min	50	4 $\times$ H100	7,200 s

Appendix

Table 7: Experimental configuration summary.

Experiment	val_bpb	Wall clock
GPT-5.2
8L $\times$ 512d, GQA, query+answer, AdamW+cosine, no compile	0.9697	1200 s
24L $\times$ 512d, GQA, SwiGLU, AdamW+WSD	0.9786	1200 s
8L $\times$ 512d, answer-only, AdamW+cosine	0.9852	1200 s
8L $\times$ 640d, answer-only, no compile	0.9954	1200 s
12L $\times$ 512d, answer-only, no compile	1.0102	1200 s

Appendix

Table 8: Top-5 LLM speedrun experiments per model.

Configuration	val_bpb	Params	tok/s
8L $\times$ 512d, GQA, Q+A, cosine, no compile	0.970	33.7M	169K
24L $\times$ 512d, GQA, SwiGLU, WSD	0.979	91.4M	306K
8L $\times$ 512d, answer-only, cosine	0.985	33.7M	843K
8L $\times$ 640d, answer-only, no compile	0.995	52.6M	321K
12L $\times$ 512d, answer-only, no compile	1.010	50.5M	288K
16L $\times$ 704d, GPT-2, WSD	1.044	96.2M	276K

Appendix

Table 9: All GPT-5.2 LLM speedrun experiments with valid val_bpb (sorted). 45 total experiments; 28 with valid metrics, 12 cancelled, 5 with degenerate metrics (

>

3.0, indicating training failures).

Configuration	val_bpb	Params	tok/s
10L $\times$ 752d, shallow	0.758	98.2M	163K
12L $\times$ 672d, QK-norm, LR=3e-3, 10 shards	0.759	92.5M	121K
12L $\times$ 672d, wide shallow	0.762	92.5M	168K
12L $\times$ 704d, “allstar” 6 tricks	0.772	98.8M	113K
12L $\times$ 672d, small batch, more steps	0.776	92.5M	118K
12L $\times$ 672d, 10 data shards	0.780	92.5M	125K

Appendix

Table 10: All Opus 4.6 LLM speedrun experiments with valid val_bpb (sorted). 50 total; 36 analyzed, 14 cancelled.

Figure 3: LLM pretraining speedrun: outcome distribution. The blue curve shows a Gaussian fit to the best val_bpb from five independent AlphaLab + GPT-5.2 runs with identical inputs (same Phase 1/2 outputs, hardware, and 50-experiment budget); tick marks indicate individual runs. Dashed lines mark single-campaign AlphaLab results with Opus 4.6 (orange, 0.758) and Sonnet 4.6 (green, 0.869); dotted lines mark GPT-5.2 baselines without AlphaLab : greedy loop (1.020) and single-shot (1.248). The key observation is that while any single campaign has meaningful variance (

{\sim}

0.056 BPB spread from identical starting conditions), in practice a user can launch multiple campaigns—potentially with different models—and select the best result, effectively sampling the left tail of the distribution. Multi-model campaigns are especially powerful: Opus 4.6’s result lies entirely to the left of every GPT-5.2 run, so a campaign that includes both models accesses regions of the search space that neither would find alone.

Parameter	Opus (0.758 bpb)	GPT-5.2 (0.970 bpb)
Layers	10	8
d_model	752	512
d_ff	2256 (SwiGLU)	1365 (SwiGLU)
Heads	8	8 (GQA, kv=2)
Parameters	98.2M	33.7M
Optimizer	AdamW (fused)	AdamW

Appendix

Table 11: Winning architecture comparison: Opus vs. GPT-5.2 on LLM speedrun.

Figure 4: Per-task speedup over torch.compile for GPT-5.2 and Opus 4.6 on the 66-task direct comparison subset (log scale). Dashed line indicates 1

\times

parity. Both models achieve extreme speedups on normalization and reduction kernels but fail to beat torch.compile on convolutions.

		Level 1 (single-op)			Level 2 (fusion)
System	Model	Correct	$\textbf{fast}_{1}^{\text{nat}}$	$\textbf{fast}_{1}^{\text{comp}}$	Correct	$\textbf{fast}_{1}^{\text{nat}}$	$\textbf{fast}_{1}^{\text{comp}}$
AlphaLab	GPT-5.2	61/65	66%	82%	49/54	88%	84%
AlphaLab	Opus 4.6	45/52	89%	82%	31/35	84%	52%
KernelBench †	R1 (iter.)	—	43%	—	—	72%	—
KernelBench †	R1 (1-shot)	—	12%	—	—	36%	—
KernelBench †	o1 (1-shot)	—	10%	—	—	24%	—

Appendix

Table 12: Per-level CUDA kernel results. AlphaLab on H100 NVL; KernelBench baselines on L40S.

\text{fast}_{1}

is reported as fraction of correct tasks (for AlphaLab ) or fraction of 100 total tasks (for baselines).

Operation	Sakana AI	AlphaLab GPT-5.2	AlphaLab Opus
LayerNorm (fwd)	0.18 $\times$	11.19 $\times$	2.28 $\times$
RMSNorm (fwd)	2.39 $\times$	1.97 $\times$	2.07 $\times$
Cross entropy (fwd)	24.87 $\times$	5.14 $\times$	5.41 $\times$
ResNet block (fwd)	2.59 $\times$	—	—
Aggregate mean spd. (vs torch.native )		3.47 $\times$	3.27 $\times$
Sakana decontaminated mean (vs torch.native )		1.49 $\times$

Appendix

Table 13: Per-operation comparison with Sakana AI CUDA Engineer ( Lange et al., 2025b ) . All results on H100. Sakana results from robust-kbench (12 kernels); AlphaLab results from the nearest KernelBench task. Speedups vs torch.compile .

Experiment	RMSE
GPT-5.2
iTransformer ctx336 + time features + static cat	0.02204
iTransformer ctx336 + hour-of-week + RevIN + horizon weights	0.02247
iTransformer ctx336 + hour-of-week + RevIN	0.02256
iTransformer ctx672 + hour-of-week + static cat	0.02289
N-HiTS ctx336 + multiscale + time features	0.02295

Appendix

Table 14: Top-5 traffic forecasting experiments per model.

Model	Configuration	RMSE
GPT-5.2 (top-10 span 5 architectures)
GPT-5.2	iTransformer ctx336 + timefeat + static cat	0.02204
GPT-5.2	iTransformer ctx336 + hour-of-week + RevIN + horizon wt.	0.02247
GPT-5.2	iTransformer ctx336 + hour-of-week + RevIN	0.02256
GPT-5.2	iTransformer ctx672 + hour-of-week + static cat	0.02289
GPT-5.2	N-HiTS ctx336 + multiscale + timefeat	0.02295

Appendix

Table 15: Traffic forecasting: top-10 for each model plus baselines.

Figure 5: Convergence curves across all three domains. Each line shows the running-best metric (lower is better for LLM and traffic; higher for CUDA) as a function of analyzed experiment count. Both models improve rapidly in the first 10–15 experiments and plateau by 25–30.

Figure 6: Phase 1 dashboard: Explorer agent plan. The Explorer agent has autonomously generated a detailed to-do list ( plan.md , center pane) for the exchange-rate forecasting domain, covering data loading, temporal structure analysis, stationarity tests, cross-country dependencies, baseline comparisons, and domain context research. Items are checked off as the agent completes them. The left pane shows the workspace file tree (scripts, notes, plots being generated); the right pane shows the agent’s real-time thinking and tool calls.

Figure 7: Phase 1 dashboard: data exploration in progress. The Explorer agent has generated analytical plots (center pane shows a rolling volatility time series across currency pairs) and is viewing them via the view_image tool (right pane, with additional correlation heatmaps and scatter plots). The left pane shows the growing collection of analysis scripts and generated plots in the workspace.

Figure 8: Phase 3 dashboard: GPU experimentation. The Kanban board (center, top) shows experiments in various stages: to_implement , implemented , checked , queued , and running . Each card displays the experiment name, hypothesis, and current status. The leaderboard (center, bottom) ranks completed experiments by the primary metric. The left pane shows the workspace file tree with experiment directories, checkpoints, and logs. The right pane shows the conversation stream of the currently active Worker agent implementing an experiment.

Panel of Figure 9 — Figure 9: Phase 1 exploration plots generated autonomously by AlphaLab for the exchange-rate domain. The Explorer wrote the analysis scripts, generated these visualizations, and used the view_image tool to inspect them before recording findings.

Rank	Sharpe	RMSE	Experiment
1 †	4.214	0.0121	TimesNet multivariate Student-T
2 †	2.780	0.0141	PatchTST factor-first PCA(3) + reconstruct
3	0.748	0.0163	Transformer policy net (Sharpe loss)
4	0.697	0.0185	Stacking ensemble (turnover-penalized)
5	0.395	0.0125	TCN multivariate risk-parity
6	0.121	—	TSMixer global returns

Appendix

Table 16: Exchange-rate experiment leaderboard (annualized Sharpe, higher is better). All experiments use GPT-5.2. Sharpe values are computed over only 5 rolling-origin windows per currency (40 trades total), making individual estimates high-variance.

\dagger

denotes experiments flagged by the system’s own debrief as potentially unreliable.

Panel of Figure 10 — Figure 10: Equity curves for the top-ranked and most interesting exchange-rate experiments, generated by AlphaLab . The TimesNet result (left) exhibits suspiciously smooth performance from only 5 trades per currency; the Transformer policy net (right) shows more realistic variance.

Domain	Model	Input tokens	Output tokens	API calls	Est. cost
LLM Speedrun	GPT-5.2	84.3M	979K	2,892	$\sim$ $150
LLM Speedrun	Opus 4.6	116.9M	1.5M	2,512	$\sim$ $200
LLM Speedrun	Sonnet 4.6	195.3M	3.4M	4,192	$\sim$ $120
Traffic	GPT-5.2	73.5M	1.4M	3,415	$\sim$ $180
Traffic	Opus 4.6	92.2M	1.7M	2,787	$\sim$ $200
Electricity	GPT-5.2	133.8M	1.8M	4,963	$\sim$ $190

Appendix

Table 17: Token usage and API call counts for primary campaigns. Output tokens are consistently

{\sim}

1–2% of input tokens.

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 318 papers

AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensive
May 12, 2026Taicheng Guo, Nitesh V. Chawla, Olaf Wiest +1Multi-Fidelity Optimization Large Language Model-Guided Optimization
University of Notre Dame
AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
Jun 3, 2026Zhangchen Xu, Junda Chen, Yue Huang +16Long-Horizon Agent Evaluation AI Agent Benchmarks
University of Washington · Bake AI · UCSD +11
PostTrainBench: Can LLM Agents Automate LLM Post-Training?
Mar 9, 2026Ben Rank, Hardik Bhatnagar, Ameya Prabhu +4LLM Post-Training Autonomous Research Agents
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +2

May 12, 2026J/K move · Enter open · S save

AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensive

Taicheng Guo, Nitesh V. Chawla, Olaf Wiest, Xiangliang Zhang

University of Notre Dame

Multi-Fidelity Optimization Large Language Model-Guided Optimization LLM Agent Training Autonomous Experimentation

Open paper AskPDF arXiv

Effectively configuring scalable large language model (LLM) experiments, spanning architecture design, hyperparameter tuning, and beyond, is crucial for advancing LLM research, as poor configuration choices can waste substantial computational resources and prevent models from realizing their full potential. Prior automated methods are designed for low-cost settings where repeated trial and error is feasible, but scalable LLM experiments are too expensive for such extensive iteration. To our knowledge, no work has addressed the automation of high-cost LLM experiment configurations, leaving this problem labor-intensive and dependent on expert intuition. Motivated by this gap, we propose AutoLLMResearch, an agentic framework that mimics how human researchers learn generalizable principles from low-fidelity experiments and extrapolate to efficiently identify promising configurations in expensive LLM settings. The core challenge is how to enable an agent to learn, through interaction with a multi-fidelity experimental environment that captures the structure of the LLM configuration landscape. To achieve this, we propose a systematic framework with two key components: 1) LLMConfig-Gym, a multi-fidelity environment encompassing four critical LLM experiment tasks, supported by over one million GPU hours of verifiable experiment outcomes; 2) A structured training pipeline that formulates configuration research as a long-horizon Markov Decision Process and accordingly incentivizes cross-fidelity extrapolation reasoning. Extensive evaluation against diverse strong baselines on held-out experiments demonstrates the effectiveness, generalization, and interpretability of our framework, supporting its potential as a practical and general solution for scalable real-world LLM experiment automation.

Abstract

Autonomous Research Agents

Explore similar work

AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensive

AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

PostTrainBench: Can LLM Agents Automate LLM Post-Training?