Can LLMs predict the outcomes of machine learning experiments?

Ayush Bansal; Aldo Pareja; Sean Klein; Serge Didenko; Artem Lukoianov

cs.LGOct 15, 2026

Can LLMs predict the outcomes of machine learning experiments?

Authors: Ayush Bansal, Aldo Pareja, Sean Klein, Serge Didenko, Artem Lukoianov

Organizations: Autolab AI · Institute of Foundation Models · MIT

Abstract

Recent advances in AI make Recursive Self-Improvement (RSI) increasingly plausible. Large Language Models (LLMs) can already propose, select, and run Machine Learning (ML) experiments. However, the space of possible design choices in ML is vast. An LLM that proposes an experiment needs a reliable intuition of whether it is worth running given the timing constraints of long training runs. In this work, we measure such intuition in current frontier LLMs by evaluating how accurately they predict the impact of self-generated ML experiment ideas before these are put into training runs. To this end, we construct a dataset of 288 LLM-proposed ideas on eight MLE-bench tasks. We then prompt five frontier LLMs (GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1, Kimi K3, and GLM-5.3) to predict the impact of each idea and compare their predictions with the actual outcomes after training. We report four main findings. First, LLM predictions are substantially better than random chance but far from accurate: even the best model, Claude Opus 5.5, explains only half of the variance in outcomes (R^2 = 0.49). Second, we notice that all tested models systematically underestimate the magnitude of the changes. Third, prediction quality varies widely across tasks. Finally, the choice of the coding harness that actually implements the ideas has little effect on outcomes, and we find no evidence of self-preference. Higher reasoning settings and additional context lead to modest gains. Overall, our results show that current LLMs have useful but limited intuition about which ML ideas will work. We release our dataset to support further study.

Figures & tables

Figure 1: We prompt LLMs to predict how much an ML experiment idea will change the final training outcome. Left: predicted versus actual change for each idea, for the best model (Opus 5.5) and the worst model (Kimi K3), with each model’s line of best fit. A perfect predictor would lie on the dashed line. Points beyond

\pm 3\sigma

are not shown. Right:

R^{2}

and slope for all five models. Even the best model explains only half of the variance, and every fitted line is flatter than the diagonal, so all models underestimate large changes.

Figure 2: Left: starting from the baseline, a model proposes four ideas and then two follow-up ideas for each. Each node is a working solution and each edge is an idea. Right: a real example from our dataset, showing the idea, its code diff, and the predicted versus actual score after training. Here, Opus 5.5 predicts the right direction but underestimates the size of the improvement.

Figure 3:

R^{2}

per task. Each row is one MLE-bench task and each marker one model. The same model can be highly accurate on one task and poor on another, and Opus 5.5 beats Kimi K3 on every task. The two image tasks, dog breeds and document denoising, are the hardest for the best model.

Figure 4: Each point is one idea, for the best model (Opus 5.5) and the worst model (Kimi K3). Points in the shaded quadrants have the correct predicted direction, and crosses mark ideas where the model predicted the wrong direction. Most ideas cluster near zero, so most have only a modest effect on the score. The table shows how often each model predicts the right direction, for all ideas and for ideas that changed the score by at least

0.25\sigma

. Points beyond

\pm 3\sigma

are not shown. Every model predicts the direction of most changes correctly, and accuracy rises once small changes are excluded.

Figure 5:

R^{2}

of each model as a function of reasoning level, from low to max. GLM-5.3 only has three reasoning levels, while all others have five. Error bars are

95\%

intervals from bootstrapping over the ideas.

R^{2}

rises with reasoning level for the three strongest models, but not for Kimi K3 and GLM-5.3.

Figure 6: From left to right, the model sees only the idea, the idea and its code diff, or the idea, its code diff, a data card describing the data, and the first

20

rows of the data. Error bars are

95\%

intervals from bootstrapping over the ideas. Removing the code diff lowers

R^{2}

for the three strongest models, while the data card helps only Opus 5.5.

Figure 7: Each point is one idea, plotted at its change in score when implemented by Fable 5.1 (x-axis) and by Kimi K3 (y-axis). Points on the dashed line scored identically under both implementations. The four circled ideas are examples where the implementer does and does not matter. Most ideas score nearly the same regardless of the implementer, with

58

66

within

0.25\sigma

Figure 8: Each cell shows how far one model’s predictions (rows) lie above its own line of best fit, relative to the other four models on the same ideas, averaged over the ideas written by each model (columns). The outlined diagonal shows each model predicting its own ideas, where self-preference would appear as positive values. All diagonal values are close to zero, so we find no self-preference.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Each dot is one idea at its real change in score, split by whether the model was prompted to improve or to worsen the score. The black line marks the median, and the numbers on the right show how often the score went up or down. Points beyond

\pm 3\sigma

and failed runs are not shown, but they are still counted.

		Change in score ( $\sigma$ )
Task	Idea	Real	Opus 5.5	Kimi K3
Spooky author identification Fable 5.1, asked to worsen	Remove English stop words from the word-level features (word TF-IDF block and the naive Bayes count features)	$-0.26$	$-0.34$	$-0.05$
Toxic comment classification Kimi K3, asked to improve	Tune per-label A/B rank-blend weights on a 10% holdout instead of fixed 0.5/0.5	$-0.01$	$-0.02$	$-0.05$
Random acts of pizza Fable 5.1, asked to improve	Add a giver_username_if_known indicator column	$+2.75$	$+2.96$	$-0.54$
Tabular playground Kimi K3, asked to improve	Refit LightGBM on 100% of the training rows at the early-stopped iteration before predicting the test set	$+0.05$	$+0.03$	$+0.02$
Leaf classification Fable 5.1, asked to worsen	Label smoothing (eps=0.1) in the two-branch NN cross-entropy loss	$-1.16$	$-0.31$	$+0.02$

Appendix

Table 1: Each row is one idea, where we specify the model that wrote it and whether it was asked to improve or worsen the score. In the dog breed identification and denoising dirty documents examples, the ideas were intended to worsen the score but improved it, whereas Opus 5.5 predicted a decrease. The full text of the dog breed idea is shown below the table.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?
Apr 12, 2026Udari Madhushani Sehwag, Elaine Lau, Haniyeh Ehsani Oskouie +14AI-Assisted Scientific Research Scientific Reasoning in Language Models
Scale AI · University of California, Los Angeles · Human Frontier Collective, Scale AI +2
RPRA: Predicting an LLM-Judge for Efficient but Performant Inference
Apr 14, 2026Dylan R. Ashley, Gaël Le Lan, Changsheng Zhao +7Language Model Cascading Language Model Performance Prediction
Meta Platforms, Inc., Menlo Park, United States of America · The Swiss AI Lab IDSIA (USI-SUPSI), Lugano, Switzerland · Università della Svizzera italiana, Lugano, Switzerland +2
Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation
Apr 6, 2026Srujan P Mule, Aniketh Garikaparthi, Manasi PatwardhanScientific Idea Evaluation Large Language Model-Guided Scientific Discovery
IISER Pune · TCS Research

Apr 12, 2026J/K move · Enter open · S save

SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?

Udari Madhushani Sehwag, Elaine Lau, Haniyeh Ehsani Oskouie, Shayan Shabihi+13

Scale AI · University of California, Los Angeles · Human Frontier Collective, Scale AI+2

AI-Assisted Scientific Research Scientific Reasoning in Language Models LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Accelerating scientific discovery requires the identification of which experiments would yield the best outcomes before committing resources to costly physical validation. While existing benchmarks evaluate LLMs on scientific knowledge and reasoning, their ability to predict experimental outcomes - a task where AI could significantly exceed human capabilities - remains largely underexplored. We introduce SciPredict, a benchmark comprising 405 tasks derived from recent empirical studies in 33 specialized sub-fields of physics, biology, and chemistry. SciPredict addresses two critical questions: (a) can LLMs predict the outcome of scientific experiments with sufficient accuracy? and (b) can such predictions be reliably used in the scientific research process? Evaluations reveal fundamental limitations on both fronts. Model accuracies are 14-26% and human expert performance is

\approx

20%. Although some frontier models exceed human performance model accuracy is still far below what would enable reliable experimental guidance. Even within the limited performance, models fail to distinguish reliable predictions from unreliable ones, achieving only

\approx

20% accuracy regardless of their confidence or whether they judge outcomes as predictable without physical experimentation. Human experts, in contrast, demonstrate strong calibration: their accuracy increases from

\approx

5% to

\approx

80% as they deem outcomes more predictable without conducting the experiment. SciPredict establishes a rigorous framework demonstrating that superhuman performance in experimental science requires not just better predictions, but better awareness of prediction reliability. For reproducibility all our data and code are provided at https://github.com/scaleapi/scipredict

Abstract

LLM Evaluation

Explore similar work

SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?

RPRA: Predicting an LLM-Judge for Efficient but Performant Inference

Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation