Evolving Executable Pipeline Programs for AutoML with Language Models

Sofoklis Kitharidis; Cor J. Veenman; Jan N. van Rijn; Thomas Bäck; Niki van Stein

cs.LGAug 17, 2026

Evolving Executable Pipeline Programs for AutoML with Language Models

Authors: Sofoklis Kitharidis, Cor J. Veenman, Jan N. van Rijn, Thomas Bäck, Niki van Stein

Organizations: Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands · Netherlands Organization for Applied Scientific Research (TNO), The Netherlands

Abstract

Automated machine learning (AutoML) systems search for pipelines within a space of preprocessing operators, learners, and hyper-parameters specified in advance: they can select and tune known components, but cannot produce structure outside that space. We present LACE, an AutoML framework that instead searches over complete executable pipeline programs: an evolutionary loop maintains a population of scikit-learn-compatible Python classes, and a large language model acts as the variation operator. To our knowledge, LACE is the first to formulate general tabular pipeline AutoML this way, evaluated on standardized OpenML tasks under a leakage-controlled protocol that withholds dataset identity from the generator. Because every candidate is ordinary Python, the returned pipeline and the search that produced it can be inspected and edited directly, rather than only through a framework's model objects. On 68 OpenML classification tasks, LACE with GPT-5.4-mini significantly outperforms auto-sklearn, H2O, and a fixed XGBoost baseline, with no detectable difference against AutoGluon, the strongest search-based system evaluated, while covering the full benchmark. Newer tabular foundation models are more accurate on the subset of tasks they support, but apply a fixed pretrained predictor rather than returning an editable task-specific program. LACE's contribution is therefore not raw accuracy but a search space defined by code: complete coverage, pipelines practitioners can reuse directly, and a component set extended by editing the prompt rather than the framework.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of LACE. The generator receives a coarse, non-identifying task summary and proposes executable pipelines; the isolated evaluator has full task access and returns inner-validation scores or errors. After the budget is exhausted, the selected pipeline is refit and evaluated once per official test fold.

Figure 2: Benchmark comparison, coloured by method family. Top: normalized closeness to the task-best score (higher is better; mean and one standard deviation) on (a) 45 common tasks and (b) all 68 with failure-aware penalties; bar labels show coverage. Bottom: Demšar-style mean task-level ranks (lower is better), with the Nemenyi critical difference at

\alpha=0.05

, on (c) 45 common tasks and (d) all 68. Pairwise tests are in Section I of the supplementary material.

Figure 3: Normalized best-so-far progress by generation, shown as the median and interquartile range across runs. Runs without a valid initial candidate or improvement are excluded.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Baseline	Version	Main configuration
AutoGluon-Tabular	1.4.0	high_quality preset, num_gpus=0 , dynamic_stacking=False , and num_stack_levels=1 ; task budget divided across remaining pre-defined splits. Search space restricted to GBM, CAT, XGB, RF, XT, KNN, and LR; neural and foundation-model families ( NN_TORCH , FASTAI , REALMLP , TABM , TABPFNV2 , TABICL , MITRA ) excluded. Each split records the fitted model names and the run fails if any excluded family appears.
auto-sklearn	0.15.0	time_left_for_this_task set to the per-split budget; per-model time limit set to half the per-split budget (minimum 30 seconds).
H2O AutoML	3.46.0.10	Allowed algorithms GLM, DRF, GBM, and StackedEnsemble, with DeepLearning excluded; a 20% inner-validation split (seeded by repetition) within each official training fold is used for model selection.
TabPFN v2.5	tabpfn 6.0.6	create_default_for_version(V2_5) on GPU; full official folds with no row or feature subsampling; ignore_pretraining_limits=False ; external wall-clock timeout.
TabPFN v3	tabpfn 8.0.3	OSS default classifier with n_estimators=8 , device=cuda , fit_mode=fit_preprocessors , memory_saving_mode=auto , ignore_pretraining_limits=False ; full official folds, no external HPO.
TabICL	tabicl 2.1.1	TabICLClassifier with n_estimators=8 , device=cuda , batch_size=1 , kv_cache=False , offload_mode=auto , and random_state set to the repetition seed; full official OpenML folds with no manual row or feature subsampling and no external HPO; external wall-clock timeout matched to the corresponding LACE task repetition. Remaining constructor options use the package defaults.

Appendix

Table 1: Baseline configurations used in the OpenML experiments.

Figure 4: Backbone screening on OpenML task 190146. Boxplots and individual points summarize the best inner-validation fitness obtained over five repetitions. Percentages below the candidate language-model names report the fraction of generated candidates assigned non-finite fitness. An earlier harness configuration, AutoGluon, and TabPFN are shown only as contextual references and were not candidates in the backbone selection.

Figure 5: Backbone screening on OpenML task 359970, using the same protocol and notation as Figure 4 .

Figure 6: Backbone screening on OpenML task 359974, using the same protocol and notation as Figure 4 .

Backbone	Finals	Preproc.	Ensemble	Custom	Flags
DeepSeek	340	100.0%	72.6%	3.2%	7
GPT-5.4-mini	340	100.0%	70.2%	2.9%	2
Qwen3	340	99.7%	77.1%	15.9%	15
Overall	1,020	99.9%	73.3%	7.4%	24

Appendix

Table 2: Overview of the

1{,}020

traceable final pipelines. Preproc. is the share with at least one detected preprocessing construction beyond the evaluator’s fixed preprocessing; Ensemble is the share combining at least two resolved base learners; Custom is the share whose estimator logic is hand-written rather than assigned to a recognized library family; Flags counts programs marked by at least one audit detector before manual classification.

	DeepSeek		GPT-5.4-mini		Qwen3		Overall
Family	Cand.	Final	Cand.	Final	Cand.	Final	Cand.	Final
Gradient boosting	70.8	75.3	51.7	66.7	56.3	66.2	59.6	69.4
Forest / extra trees	35.5	37.1	46.0	64.3	49.9	56.2	43.1	52.5
Linear	58.5	55.3	60.4	44.8	74.4	67.4	62.8	55.8
SVM	21.1	19.1	21.7	18.0	16.6	15.9	20.3	17.7
k-NN	4.3	4.1	2.1	2.9	4.9	5.9	3.5	4.3

Appendix

Table 3: Estimator-family occurrence among valid candidates (Cand.) and final pipelines (Final), in percent. Denominators: DeepSeek 25,396 and 340; GPT-5.4-mini 29,691 and 340; Qwen3 15,911 and 340; overall 70,998 and 1,020.

Category	DeepSeek	GPT	Qwen3	Overall
Any preprocessing	100.0	100.0	99.7	99.9
Scaling	100.0	86.7	98.5	95.1
Feature selection	77.4	73.5	55.3	68.7
Dimensionality red.	27.1	3.8	8.8	13.2
Nonlinear transforms	85.0	54.9	59.7	66.5
Class imbalance	23.2	59.6	67.9	50.2

Appendix

Table 4: Added preprocessing among final pipelines, in percent of each backbone’s finals.

	DeepSeek	GPT	Qwen3	Overall
Finals analyzed	340	340	340	1,020
Internal CV constructs	3	0	4	7
Ensemble of four	4	2	7	13
Fit at prediction time	0	0	4	4
Any flag	7	2	15	24
Confirmed violations	7	2	13	22

Appendix

Table 5: Audit flags among final pipelines by type and backbone, with the outcome of manual classification, and cross-fitting recorded descriptively. The internal hyperparameter search, nested AutoML, and estimator-switching detectors flagged no final and are omitted.

Method	Full rank (68) $\downarrow$	Full dev. (68) $\downarrow$	Common rank (45) $\downarrow$	Common dev. (45) $\downarrow$	Frac. wins $\uparrow$	Coverage
TabICL	3.28	18.7%	2.97	15.4%	14.75	67/68
TabPFN v3	3.91	26.1%	3.32	20.0%	18.25	62/68
LACE (GPT-5.4-mini)	4.50	32.3%	4.71	32.8%	3.0	68/68
AutoGluon-Tabular	4.59	31.9%	5.49	38.8%	14.0	68/68
LACE (DeepSeek)	5.76	41.1%	5.76	38.8%	0.0	68/68
LACE (Qwen3)	5.99	45.5%	5.88	40.7%	2.75	68/68

Appendix

Table 6: Aggregate performance and coverage. Common statistics use the 45 tasks completed by all methods; full statistics use all 68 under the failure-aware convention of the method-comparison figure of the main paper, penalizing missing results with the tied average of the remaining worst ranks and normalized deviation 1. A task is valid when at least three of five repetitions return a finite score. Fractional wins split tied task wins equally among the tied methods. Methods are ordered by full-benchmark rank.

Comparison	Shared tasks	W–T–L	Median diff.	Mean diff.	Raw $p$	Holm $p$
AutoGluon	68	33–0–35	$-0.001$	$-0.268$	$0.66$	$0.66$
auto-sklearn	68	50–0–18	$+0.234$	$+0.381$	$0.000442$	$0.00133$
XGBoost	68	48–0–20	$+0.370$	$+1.233$	$4.97\times 10^{-5}$	$0.000298$
H2O AutoML	68	61–0–7	$+1.024$	$+3.241$	$3.92\times 10^{-10}$	$2.75\times 10^{-9}$
TabPFN v2.5	45	16–0–29	$-0.144$	$-0.422$	$0.0997$	$0.199$
TabPFN v3	62	15–0–47	$-0.336$	$-1.019$	$9.02\times 10^{-5}$	$0.000451$

Appendix

Table 7: Paired Wilcoxon signed-rank comparisons between LACE with GPT-5.4-mini and the external baselines and alternative LACE backbones. W–T–L gives task-level wins, exact ties, and losses for GPT-5.4-mini. Differences are GPT-5.4-mini minus the comparison method, in percentage points of the metric designated for each task.

p

-values are Holm-adjusted within two families: the seven baseline comparisons and the two backbone comparisons.

Figure 7: Pairwise task-level win rates. Each cell gives the share of comparable tasks on which the row method achieves a higher score than the column method, with the underlying counts below it; a task is comparable when both methods return a valid result, and denominators therefore vary with coverage. Exact ties remain in the denominator, so complementary cells sum to 100% only for pairs without ties; ties occur in ten pairs and never exceed two tasks. Methods are ordered by failure-aware mean rank, and tick-label colour groups them by family as in the main paper’s method-comparison figure. These are raw win counts without significance testing; the tested comparisons involving LACE (GPT-5.4-mini) are in Table 7 .

Figure 8: Failure-aware mean task-level rank by (a) number of rows and (b) number of features. Lower is better; each column averages the within-task ranks of all ten methods over the tasks in that bin, with missing methods penalized as in the method-comparison figure of the main paper. Column labels give the number of tasks per bin; an asterisk marks bins with fewer than ten tasks, whose values should be read with caution. Row-label colour groups methods by family.

Backbone	Planned runs	First-shot valid	Final valid	Invalid recovered	Recovery rate
Qwen3	340	$128/340$ ( $37.6\%$ )	$340/340$ ( $100.0\%$ )	$212/212$	$100\%$
DeepSeek	340	$226/340$ ( $66.5\%$ )	$340/340$ ( $100.0\%$ )	$114/114$	$100\%$
GPT-5.4-mini	340	$271/340$ ( $79.7\%$ )	$340/340$ ( $100.0\%$ )	$69/69$	$100\%$

Appendix

Table 8: Validity of the first zero-shot candidate and the final selected pipeline. A recovered invalid first shot is a run whose first candidate has non-finite fitness but whose final selected pipeline is valid. A first-shot record is present for every planned run.

Final $-$ first shot, invalid assigned $0$
Backbone	$n$	Median [IQR]	95% CI
Qwen3	67	80.00 [14.35, 90.98]	[69.00, 86.38]
DeepSeek	67	6.07 [1.68, 26.14]	[4.15, 11.44]
GPT-5.4-mini	67	7.02 [2.55, 14.02]	[4.36, 9.63]
Overall	201	11.33 [3.54, 76.94]	[8.99, 18.27]
Final $-$ first shot, valid first shots only

Appendix

Table 9: Task-level improvement in inner-validation fitness, in accuracy percentage points, over the 67 accuracy-scored tasks. Differences are computed within repetition and then median-aggregated within each backbone–task unit;

n

is the number of units with at least three paired repetitions. The first block assigns an observed invalid first shot a failure-aware score of

0

; the second restricts the comparison to valid first shots; the third compares the final pipeline with the best valid member of the initial population. Confidence intervals are percentile-bootstrap intervals for the median over backbone–task units.

Figure 10: Held-out official-test improvement from the valid literal first program to the returned LACE pipeline over the 67 accuracy-scored tasks. Each point is one backbone–task unit with at least three finite paired official replays; differences are computed within repetition and then median-aggregated within each unit. Invalid first programs and official-evaluation timeouts remain missing, and no artificial score of zero is assigned. Consequently, each backbone is evaluated on a validity-conditioned subset that differs in size and composition: 62/67 task units for Qwen3, 65/67 for DeepSeek, and 66/67 for GPT-5.4-mini. The distributions therefore quantify within-backbone improvement from first to final and should not be interpreted as a like-for-like comparison between backbones. The axis extends below zero; three task units have negative differences, demonstrating that held-out improvement is not sign-constrained by elitist selection. Diamonds denote medians, crosses denote means, and thick horizontal lines show percentile-bootstrap 95% confidence intervals. Task 360114 is excluded because its official evaluation recorded the AUC.

Backbone	0	1	2	3	4	5	6	7	8
DeepSeek	25.0	25.0	16.7	16.7	16.7	25.0	25.0	25.0	25.0
GPT-5.4-mini	0.0	8.3	8.3	8.3	8.3	8.3	8.3	8.3	8.3
Qwen3	75.0	58.3	50.0	50.0	50.0	50.0	50.0	50.0	58.3
Overall	25.0	33.3	25.0	25.0	25.0	25.0	25.0	25.0	25.0

Appendix

Table 11: Run-level median invalid-candidate share per generation, in percent.

Backbone	Pairs	Improving (%)	Degrading (%)	$P(\text{valid}\mid\text{inv.})$	$P(\text{valid}\mid\text{val.})$
DeepSeek	23,993	21.0 (21.0)	71.3 (72.3)	48.7	76.6
GPT-5.4-mini	28,150	23.0 (22.6)	73.2 (73.8)	60.1	88.4
Qwen3	14,557	22.1 (21.6)	73.7 (74.5)	23.9	49.9
Overall	66,700	22.1 (21.9)	72.6 (73.6)	34.9	72.2

Appendix

Table 14: Parent–offspring outcomes among valid parent–offspring pairs. Improving and degrading columns give pooled percentages, with run-level medians in parentheses; the final two columns give pooled offspring-validity rates conditional on parent validity.

deepseek-v4-flash with TabPFN v2.5
Task	Default	Passive	Explicit	TabPFN
2073	62.28 (0.58)	62.53 (0.61)	62.61 (0.50)	61.31 (0.24)
146818	87.36 (0.38)	86.84 (0.50)	87.48 (0.22)	86.75 (0.22)
168350	90.68 (0.48)	90.64 (0.56)	92.27 (0.18)	91.77 (0.13)
359983	87.43 (0.02)	87.40 (0.09)	87.40 (0.08)	86.68 (0.04)
359990	94.73 (0.15)	94.85 (0.20)	94.68 (0.40)	—

Appendix

Table 16: Accuracy in percent, averaged over five repetitions, with standard deviation across repetitions in parentheses. Default is the environment of the main experiments; passive installs TabPFN without mentioning it to the generator; and explicit announces it in the prompt. TabPFN v2.5 returns no valid score on task 359990, whose size exceeds its input limit.

Figure 11: Official-test accuracy distributions for OpenML tasks 359984, 167120, 359959, and 211986.

Figure 12: Official-test accuracy distributions for OpenML tasks 2073, 359969, 189354, and 189356.

Figure 13: Official-test accuracy distributions for OpenML tasks 359954, 359974, 168910, and 359985.

Figure 14: Official-test accuracy distributions for OpenML tasks 211979, 10090, 359970, and 359993.

Figure 15: Official-test accuracy distributions for OpenML tasks 359973, 168757, 190412, and 359955.

Figure 16: Official-test accuracy distributions for OpenML tasks 359967, 190410, 168784, and 168911.

Figure 17: Official-test accuracy distributions for OpenML tasks 359992, 190146, 359977, and 359962.

Figure 18: Official-test accuracy distributions for OpenML tasks 190411, 359994, 146818, and 359983.

Figure 19: Official-test accuracy distributions for OpenML tasks 189355, 359956, 190392, and 359964.

Figure 20: Official-test accuracy distributions for OpenML tasks 359991, 359976, 359982, and 168350.

Figure 21: Official-test accuracy distributions for OpenML tasks 359958, 359953, 359981, and 359963.

Figure 22: Official-test accuracy distributions for OpenML tasks 190137, 359990, 359957, and 359979.

Figure 23: Official-test accuracy distributions for OpenML tasks 7593, 359972, 360113, and 189922.

Figure 24: Official-test accuracy distributions for OpenML tasks 359968, 359971, 359980, and 359966.

Figure 25: Official-test accuracy distributions for OpenML tasks 3945, 359961, 146820, and 168909.

Figure 26: Official-test accuracy distributions for OpenML tasks 359975, 168868, 359965, and 359989.

Figure 27: Official-test accuracy distributions for OpenML tasks 359960, 360112, and 359987.

Figure 28: Official-test ROC-AUC distributions for OpenML task 360114.

Automated Machine Learning

Automated machine learning (AutoML) builds and tunes data-to-model pipelines with minimal manual effort, balancing predictive quality, compute, and deployment constraints.

See all 96 papers

SAGE-Loop: Reliable Closed-Loop LLM-Driven AutoML with Trial-and-Correction and Adaptive Ensembling
Sep 11, 2026Junquan Gu, Shibo Cui, Xiangfeng Luo +1LLM Self-Correction Ensemble Learning
Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packages
Sep 29, 2026Katarzyna Woźnica, Katarzyna Rogalska, Zuzanna Sieńko +1Automated Machine Learning Explainability Evaluation
Warsaw University of Technology · Systems Research Institute, Polish Academy of Sciences · Eskisehir Technical University
Think it, Run it: Autonomous ML pipeline generation via self-healing multi-agent AI
Apr 29, 2026Adela Bara, Gabriela Dobrita, Simona-Vasilica OpreaAutomated Machine Learning Multi-Agent LLMs
Bucharest University of Economic Studies, Department of Economic Informatics and Cybernetics, no.6 Piața Romană, Bucharest, 010374, Romania

Sep 11, 2026J/K move · Enter open · S save

SAGE-Loop: Reliable Closed-Loop LLM-Driven AutoML with Trial-and-Correction and Adaptive Ensembling

Junquan Gu, Shibo Cui, Xiangfeng Luo, Hang Yu

LLM Self-Correction Ensemble Learning Large Language Model-Guided Optimization LLM Reliability Automated Machine Learning

Open paper AskPDF arXiv

Automated machine learning (AutoML) is reshaping data-driven science and industrial practice, and as large language models are introduced into AutoML, pipeline reliability becomes as important as automation efficiency. However, existing AutoML still struggles to realize instant feedback and adaptive optimization during execution, so once a run drifts into a suboptimal or failed state, it lacks a process-level correction mechanism. The fundamental pathology lies in its one-way pipeline: intermediate failures are typically terminated or bypassed, while fixed paradigms often strengthen model generation but leave ensemble decisions static, weakening both execution reliability and the controlled use of structural diversity. This indicates that LLM-driven AutoML needs a closed-loop ability for trial-correction-improvement together with evidence-based use of model diversity. To this end, we propose SAGE-Loop, a reliable closed-loop, self-adaptive, LLM-driven AutoML framework that performs multi-round generation and validation for trial-and-repair, and adaptively selects ensemble strategies in both supervised and unsupervised tasks, thereby unifying how to generate with how to use models. Across 20 public datasets, SAGE-Loop consistently improves performance and stability on classification, regression, and clustering tasks. Additional results further show its ability to recover from execution failures and maintain robust pipeline behavior.

Abstract

Automated Machine Learning

Explore similar work

SAGE-Loop: Reliable Closed-Loop LLM-Driven AutoML with Trial-and-Correction and Adaptive Ensembling

Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packages

Think it, Run it: Autonomous ML pipeline generation via self-healing multi-agent AI