Evolving Executable Pipeline Programs for AutoML with Language Models
Authors: Sofoklis Kitharidis, Cor J. Veenman, Jan N. van Rijn, Thomas Bäck, Niki van Stein
Organizations: Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands · Netherlands Organization for Applied Scientific Research (TNO), The Netherlands
Automated machine learning (AutoML) systems search for pipelines within a space of preprocessing operators, learners, and hyper-parameters specified in advance: they can select and tune known components, but cannot produce structure outside that space. We present LACE, an AutoML framework that instead searches over complete executable pipeline programs: an evolutionary loop maintains a population of scikit-learn-compatible Python classes, and a large language model acts as the variation operator. To our knowledge, LACE is the first to formulate general tabular pipeline AutoML this way, evaluated on standardized OpenML tasks under a leakage-controlled protocol that withholds dataset identity from the generator. Because every candidate is ordinary Python, the returned pipeline and the search that produced it can be inspected and edited directly, rather than only through a framework's model objects. On 68 OpenML classification tasks, LACE with GPT-5.4-mini significantly outperforms auto-sklearn, H2O, and a fixed XGBoost baseline, with no detectable difference against AutoGluon, the strongest search-based system evaluated, while covering the full benchmark. Newer tabular foundation models are more accurate on the subset of tasks they support, but apply a fixed pretrained predictor rather than returning an editable task-specific program. LACE's contribution is therefore not raw accuracy but a search space defined by code: complete coverage, pipelines practitioners can reuse directly, and a component set extended by editing the prompt rather than the framework.
Figures & tables
Figure 1: Overview of LACE. The generator receives a coarse, non-identifying task summary and proposes executable pipelines; the isolated evaluator has full task access and returns inner-validation scores or errors. After the budget is exhausted, the selected pipeline is refit and evaluated once per official test fold.
Figure 2: Benchmark comparison, coloured by method family. Top: normalized closeness to the task-best score (higher is better; mean and one standard deviation) on (a) 45 common tasks and (b) all 68 with failure-aware penalties; bar labels show coverage. Bottom: Demšar-style mean task-level ranks (lower is better), with the Nemenyi critical difference at α=0.05 , on (c) 45 common tasks and (d) all 68. Pairwise tests are in Section I of the supplementary material.
Figure 3: Normalized best-so-far progress by generation, shown as the median and interquartile range across runs. Runs without a valid initial candidate or improvement are excluded.
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Baseline
Version
Main configuration
AutoGluon-Tabular
1.4.0
high_quality preset, num_gpus=0 , dynamic_stacking=False , and num_stack_levels=1 ; task budget divided across remaining pre-defined splits. Search space restricted to GBM, CAT, XGB, RF, XT, KNN, and LR; neural and foundation-model families ( NN_TORCH , FASTAI , REALMLP , TABM , TABPFNV2 , TABICL , MITRA ) excluded. Each split records the fitted model names and the run fails if any excluded family appears.
auto-sklearn
0.15.0
time_left_for_this_task set to the per-split budget; per-model time limit set to half the per-split budget (minimum 30 seconds).
H2O AutoML
3.46.0.10
Allowed algorithms GLM, DRF, GBM, and StackedEnsemble, with DeepLearning excluded; a 20% inner-validation split (seeded by repetition) within each official training fold is used for model selection.
TabPFN v2.5
tabpfn 6.0.6
create_default_for_version(V2_5) on GPU; full official folds with no row or feature subsampling; ignore_pretraining_limits=False ; external wall-clock timeout.
TabPFN v3
tabpfn 8.0.3
OSS default classifier with n_estimators=8 , device=cuda , fit_mode=fit_preprocessors , memory_saving_mode=auto , ignore_pretraining_limits=False ; full official folds, no external HPO.
TabICL
tabicl 2.1.1
TabICLClassifier with n_estimators=8 , device=cuda , batch_size=1 , kv_cache=False , offload_mode=auto , and random_state set to the repetition seed; full official OpenML folds with no manual row or feature subsampling and no external HPO; external wall-clock timeout matched to the corresponding LACE task repetition. Remaining constructor options use the package defaults.
Appendix
Table 1: Baseline configurations used in the OpenML experiments.
Figure 4: Backbone screening on OpenML task 190146. Boxplots and individual points summarize the best inner-validation fitness obtained over five repetitions. Percentages below the candidate language-model names report the fraction of generated candidates assigned non-finite fitness. An earlier harness configuration, AutoGluon, and TabPFN are shown only as contextual references and were not candidates in the backbone selection.
Figure 5: Backbone screening on OpenML task 359970, using the same protocol and notation as Figure 4 .
Figure 6: Backbone screening on OpenML task 359974, using the same protocol and notation as Figure 4 .
Backbone
Finals
Preproc.
Ensemble
Custom
Flags
DeepSeek
340
100.0%
72.6%
3.2%
7
GPT-5.4-mini
340
100.0%
70.2%
2.9%
2
Qwen3
340
99.7%
77.1%
15.9%
15
Overall
1,020
99.9%
73.3%
7.4%
24
Appendix
Table 2: Overview of the 1,020 traceable final pipelines. Preproc. is the share with at least one detected preprocessing construction beyond the evaluator’s fixed preprocessing; Ensemble is the share combining at least two resolved base learners; Custom is the share whose estimator logic is hand-written rather than assigned to a recognized library family; Flags counts programs marked by at least one audit detector before manual classification.
DeepSeek
GPT-5.4-mini
Qwen3
Overall
Family
Cand.
Final
Cand.
Final
Cand.
Final
Cand.
Final
Gradient boosting
70.8
75.3
51.7
66.7
56.3
66.2
59.6
69.4
Forest / extra trees
35.5
37.1
46.0
64.3
49.9
56.2
43.1
52.5
Linear
58.5
55.3
60.4
44.8
74.4
67.4
62.8
55.8
SVM
21.1
19.1
21.7
18.0
16.6
15.9
20.3
17.7
k-NN
4.3
4.1
2.1
2.9
4.9
5.9
3.5
4.3
Appendix
Table 3: Estimator-family occurrence among valid candidates (Cand.) and final pipelines (Final), in percent. Denominators: DeepSeek 25,396 and 340; GPT-5.4-mini 29,691 and 340; Qwen3 15,911 and 340; overall 70,998 and 1,020.
Category
DeepSeek
GPT
Qwen3
Overall
Any preprocessing
100.0
100.0
99.7
99.9
Scaling
100.0
86.7
98.5
95.1
Feature selection
77.4
73.5
55.3
68.7
Dimensionality red.
27.1
3.8
8.8
13.2
Nonlinear transforms
85.0
54.9
59.7
66.5
Class imbalance
23.2
59.6
67.9
50.2
Appendix
Table 4: Added preprocessing among final pipelines, in percent of each backbone’s finals.
DeepSeek
GPT
Qwen3
Overall
Finals analyzed
340
340
340
1,020
Internal CV constructs
3
0
4
7
Ensemble of four
4
2
7
13
Fit at prediction time
0
0
4
4
Any flag
7
2
15
24
Confirmed violations
7
2
13
22
Appendix
Table 5: Audit flags among final pipelines by type and backbone, with the outcome of manual classification, and cross-fitting recorded descriptively. The internal hyperparameter search, nested AutoML, and estimator-switching detectors flagged no final and are omitted.
Method
Full rank (68) ↓
Full dev. (68) ↓
Common rank (45) ↓
Common dev. (45) ↓
Frac. wins ↑
Coverage
TabICL
3.28
18.7%
2.97
15.4%
14.75
67/68
TabPFN v3
3.91
26.1%
3.32
20.0%
18.25
62/68
LACE (GPT-5.4-mini)
4.50
32.3%
4.71
32.8%
3.0
68/68
AutoGluon-Tabular
4.59
31.9%
5.49
38.8%
14.0
68/68
LACE (DeepSeek)
5.76
41.1%
5.76
38.8%
0.0
68/68
LACE (Qwen3)
5.99
45.5%
5.88
40.7%
2.75
68/68
Appendix
Table 6: Aggregate performance and coverage. Common statistics use the 45 tasks completed by all methods; full statistics use all 68 under the failure-aware convention of the method-comparison figure of the main paper, penalizing missing results with the tied average of the remaining worst ranks and normalized deviation 1. A task is valid when at least three of five repetitions return a finite score. Fractional wins split tied task wins equally among the tied methods. Methods are ordered by full-benchmark rank.
Comparison
Shared tasks
W–T–L
Median diff.
Mean diff.
Raw p
Holm p
AutoGluon
68
33–0–35
−0.001
−0.268
0.66
0.66
auto-sklearn
68
50–0–18
+0.234
+0.381
0.000442
0.00133
XGBoost
68
48–0–20
+0.370
+1.233
4.97×10−5
0.000298
H2O AutoML
68
61–0–7
+1.024
+3.241
3.92×10−10
2.75×10−9
TabPFN v2.5
45
16–0–29
−0.144
−0.422
0.0997
0.199
TabPFN v3
62
15–0–47
−0.336
−1.019
9.02×10−5
0.000451
Appendix
Table 7: Paired Wilcoxon signed-rank comparisons between LACE with GPT-5.4-mini and the external baselines and alternative LACE backbones. W–T–L gives task-level wins, exact ties, and losses for GPT-5.4-mini. Differences are GPT-5.4-mini minus the comparison method, in percentage points of the metric designated for each task. p -values are Holm-adjusted within two families: the seven baseline comparisons and the two backbone comparisons.
Figure 7: Pairwise task-level win rates. Each cell gives the share of comparable tasks on which the row method achieves a higher score than the column method, with the underlying counts below it; a task is comparable when both methods return a valid result, and denominators therefore vary with coverage. Exact ties remain in the denominator, so complementary cells sum to 100% only for pairs without ties; ties occur in ten pairs and never exceed two tasks. Methods are ordered by failure-aware mean rank, and tick-label colour groups them by family as in the main paper’s method-comparison figure. These are raw win counts without significance testing; the tested comparisons involving LACE (GPT-5.4-mini) are in Table 7 .
Figure 8: Failure-aware mean task-level rank by (a) number of rows and (b) number of features. Lower is better; each column averages the within-task ranks of all ten methods over the tasks in that bin, with missing methods penalized as in the method-comparison figure of the main paper. Column labels give the number of tasks per bin; an asterisk marks bins with fewer than ten tasks, whose values should be read with caution. Row-label colour groups methods by family.
Backbone
Planned runs
First-shot valid
Final valid
Invalid recovered
Recovery rate
Qwen3
340
128/340 ( 37.6% )
340/340 ( 100.0% )
212/212
100%
DeepSeek
340
226/340 ( 66.5% )
340/340 ( 100.0% )
114/114
100%
GPT-5.4-mini
340
271/340 ( 79.7% )
340/340 ( 100.0% )
69/69
100%
Appendix
Table 8: Validity of the first zero-shot candidate and the final selected pipeline. A recovered invalid first shot is a run whose first candidate has non-finite fitness but whose final selected pipeline is valid. A first-shot record is present for every planned run.
Backbone
n
Median [IQR]
95% CI
Final − first shot, invalid assigned 0
Qwen3
67
80.00 [14.35, 90.98]
[69.00, 86.38]
DeepSeek
67
6.07 [1.68, 26.14]
[4.15, 11.44]
GPT-5.4-mini
67
7.02 [2.55, 14.02]
[4.36, 9.63]
Overall
201
11.33 [3.54, 76.94]
[8.99, 18.27]
Final − first shot, valid first shots only
Appendix
Table 9: Task-level improvement in inner-validation fitness, in accuracy percentage points, over the 67 accuracy-scored tasks. Differences are computed within repetition and then median-aggregated within each backbone–task unit; n is the number of units with at least three paired repetitions. The first block assigns an observed invalid first shot a failure-aware score of 0 ; the second restricts the comparison to valid first shots; the third compares the final pipeline with the best valid member of the initial population. Confidence intervals are percentile-bootstrap intervals for the median over backbone–task units.
Figure 10: Held-out official-test improvement from the valid literal first program to the returned LACE pipeline over the 67 accuracy-scored tasks. Each point is one backbone–task unit with at least three finite paired official replays; differences are computed within repetition and then median-aggregated within each unit. Invalid first programs and official-evaluation timeouts remain missing, and no artificial score of zero is assigned. Consequently, each backbone is evaluated on a validity-conditioned subset that differs in size and composition: 62/67 task units for Qwen3, 65/67 for DeepSeek, and 66/67 for GPT-5.4-mini. The distributions therefore quantify within-backbone improvement from first to final and should not be interpreted as a like-for-like comparison between backbones. The axis extends below zero; three task units have negative differences, demonstrating that held-out improvement is not sign-constrained by elitist selection. Diamonds denote medians, crosses denote means, and thick horizontal lines show percentile-bootstrap 95% confidence intervals. Task 360114 is excluded because its official evaluation recorded the AUC.
Backbone
0
1
2
3
4
5
6
7
8
DeepSeek
25.0
25.0
16.7
16.7
16.7
25.0
25.0
25.0
25.0
GPT-5.4-mini
0.0
8.3
8.3
8.3
8.3
8.3
8.3
8.3
8.3
Qwen3
75.0
58.3
50.0
50.0
50.0
50.0
50.0
50.0
58.3
Overall
25.0
33.3
25.0
25.0
25.0
25.0
25.0
25.0
25.0
Appendix
Table 11: Run-level median invalid-candidate share per generation, in percent.
Backbone
Pairs
Improving (%)
Degrading (%)
P(valid∣inv.)
P(valid∣val.)
DeepSeek
23,993
21.0 (21.0)
71.3 (72.3)
48.7
76.6
GPT-5.4-mini
28,150
23.0 (22.6)
73.2 (73.8)
60.1
88.4
Qwen3
14,557
22.1 (21.6)
73.7 (74.5)
23.9
49.9
Overall
66,700
22.1 (21.9)
72.6 (73.6)
34.9
72.2
Appendix
Table 14: Parent–offspring outcomes among valid parent–offspring pairs. Improving and degrading columns give pooled percentages, with run-level medians in parentheses; the final two columns give pooled offspring-validity rates conditional on parent validity.
Task
Default
Passive
Explicit
TabPFN
deepseek-v4-flash with TabPFN v2.5
2073
62.28 (0.58)
62.53 (0.61)
62.61 (0.50)
61.31 (0.24)
146818
87.36 (0.38)
86.84 (0.50)
87.48 (0.22)
86.75 (0.22)
168350
90.68 (0.48)
90.64 (0.56)
92.27 (0.18)
91.77 (0.13)
359983
87.43 (0.02)
87.40 (0.09)
87.40 (0.08)
86.68 (0.04)
359990
94.73 (0.15)
94.85 (0.20)
94.68 (0.40)
—
Appendix
Table 16: Accuracy in percent, averaged over five repetitions, with standard deviation across repetitions in parentheses. Default is the environment of the main experiments; passive installs TabPFN without mentioning it to the generator; and explicit announces it in the prompt. TabPFN v2.5 returns no valid score on task 359990, whose size exceeds its input limit.
Figure 11: Official-test accuracy distributions for OpenML tasks 359984, 167120, 359959, and 211986.
Figure 12: Official-test accuracy distributions for OpenML tasks 2073, 359969, 189354, and 189356.
Figure 13: Official-test accuracy distributions for OpenML tasks 359954, 359974, 168910, and 359985.
Figure 14: Official-test accuracy distributions for OpenML tasks 211979, 10090, 359970, and 359993.
Figure 15: Official-test accuracy distributions for OpenML tasks 359973, 168757, 190412, and 359955.
Figure 16: Official-test accuracy distributions for OpenML tasks 359967, 190410, 168784, and 168911.
Figure 17: Official-test accuracy distributions for OpenML tasks 359992, 190146, 359977, and 359962.
Figure 18: Official-test accuracy distributions for OpenML tasks 190411, 359994, 146818, and 359983.
Figure 19: Official-test accuracy distributions for OpenML tasks 189355, 359956, 190392, and 359964.
Figure 20: Official-test accuracy distributions for OpenML tasks 359991, 359976, 359982, and 168350.
Figure 21: Official-test accuracy distributions for OpenML tasks 359958, 359953, 359981, and 359963.
Figure 22: Official-test accuracy distributions for OpenML tasks 190137, 359990, 359957, and 359979.
Figure 23: Official-test accuracy distributions for OpenML tasks 7593, 359972, 360113, and 189922.
Figure 24: Official-test accuracy distributions for OpenML tasks 359968, 359971, 359980, and 359966.
Figure 25: Official-test accuracy distributions for OpenML tasks 3945, 359961, 146820, and 168909.
Figure 26: Official-test accuracy distributions for OpenML tasks 359975, 168868, 359965, and 359989.
Figure 27: Official-test accuracy distributions for OpenML tasks 359960, 360112, and 359987.
Figure 28: Official-test ROC-AUC distributions for OpenML task 360114.
Automated machine learning (AutoML) is reshaping data-driven science and industrial practice, and as large language models are introduced into AutoML, pipeline reliability becomes as important as automation efficiency. However, existing AutoML still struggles to realize instant feedback and adaptive optimization during execution, so once a run drifts into a suboptimal or failed state, it lacks a process-level correction mechanism. The fundamental pathology lies in its one-way pipeline: intermediate failures are typically terminated or bypassed, while fixed paradigms often strengthen model generation but leave ensemble decisions static, weakening both execution reliability and the controlled use of structural diversity. This indicates that LLM-driven AutoML needs a closed-loop ability for trial-correction-improvement together with evidence-based use of model diversity. To this end, we propose SAGE-Loop, a reliable closed-loop, self-adaptive, LLM-driven AutoML framework that performs multi-round generation and validation for trial-and-repair, and adaptively selects ensemble strategies in both supervised and unsupervised tasks, thereby unifying how to generate with how to use models. Across 20 public datasets, SAGE-Loop consistently improves performance and stability on classification, regression, and clustering tasks. Additional results further show its ability to recover from execution failures and maintain robust pipeline behavior.
The Rashomon effect describes the existence of multiple near-optimal models that achieve comparable performance while offering fundamentally different explanations. This creates a critical vulnerability in AutoML: x-hacking, the selective post-hoc choice of a model based on its explanation rather than predictive merit. No existing AutoML framework exposes this risk. We introduce ARSA ML, an open-source Python framework that quantifies Rashomon set structure and predictive multiplicity within AutoML pipelines. Using ARSA ML, we benchmark AutoGluon and H2O across 28 binary classification datasets, and conduct a post-hoc x-hacking analysis revealing a consistent structural asymmetry: AutoGluon produces larger, diverse sets with stable explanations, while H2O generates compact sets with markedly higher prediction divergence and explanation instability -- making H2O users considerably more exposed to x-hacking. This gap persists across all evaluated metrics and epsilon thresholds, pointing to a fundamental difference in each framework's model-building strategy. ARSA ML is available at https://pypi.org/project/arsa-ml/ .
Katarzyna Woźnica, Katarzyna Rogalska, Zuzanna Sieńko +1
Warsaw University of Technology · Systems Research Institute, Polish Academy of Sciences · Eskisehir Technical University
The purpose of our paper is to develop a unified multi-agent architecture that automates end-to-end machine learning (ML) pipeline generation from datasets and natural-language (NL) goals, improving efficiency, robustness and explainability. A five-agent system is proposed to handle profiling, intent parsing, microservice recommendation, Directed Acyclic Graph (DAG) construction and execution. It integrates code-grounded Retrieval-Augmented Generation (RAG) for microservice understanding, an explainable hybrid recommender combining multiple criteria, a self-healing mechanism using Large Language Model (LLM)-based error interpretation and adaptive learning from execution history. The approach is evaluated on 150 ML tasks across diverse scenarios. The system achieves an 84.7% end-to-end pipeline success rate, outperforming baseline methods. It demonstrates improved robustness through self-healing and reduces workflow development time compared to manual construction. The study introduces a novel integration of code-grounded RAG, explainable recommendation, self-healing execution and adaptive learning within a single architecture, showing that tightly coupled intelligent components can outperform isolated solutions.