EvoForest: A Novel Machine-Learning Paradigm via Open-Ended Evolution of Computational Graphs

Kamer Ali Yuksel; Hassan Sawaf

cs.AIMar 26, 2026

EvoForest: A Novel Machine-Learning Paradigm via Open-Ended Evolution of Computational Graphs

Authors: Kamer Ali Yuksel, Hassan Sawaf

Organizations: aiXplain, Inc., San Jose, CA, USA

Abstract

Modern machine learning is still largely organized around a single recipe: choose a parameterized model family and optimize its weights. Although highly successful, this paradigm is too narrow for many structured prediction problems, where the main bottleneck is not parameter fitting but discovering what should be computed from the data. Success often depends on identifying the right transformations, statistics, invariances, interaction structures, temporal summaries, gates, or nonlinear compositions, especially when objectives are non-differentiable, evaluation is cross-validation-based, interpretability matters, or continual adaptation is required. We present EvoForest, a hybrid neuro-symbolic system for end-to-end open-ended evolution of computation. Rather than merely generating features, EvoForest jointly evolves reusable computational structure, callable function families, and trainable low-dimensional continuous components inside a shared directed acyclic graph. Intermediate nodes store alternative implementations, callable nodes encode reusable transformation families such as projections, gates, and activations, output nodes define candidate predictive computations, and persistent global parameters can be refined by gradient descent. For each graph configuration, EvoForest evaluates the discovered computation and uses a lightweight Ridge-based readout to score the resulting representation against a non-differentiable cross-validation target. The evaluator also produces structured feedback that guides future LLM-driven mutations. In the 2025 ADIA Lab Structural Break Challenge, EvoForest reached 94.13% ROC-AUC after 600 evolution steps, exceeding the publicly reported winning score of 90.14% under the same evaluation protocol.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: EvoForest architecture. EvoForest DAG (left), two-phase evaluation pipeline (center), LLM-guided scientist/engineer mutation loop (right), and asynchronous island model (bottom).

Entry / system	Category	Description / relevance	Score
EvoForest (ours)	Search-first	Multi-alternative DAG with reusable computations, callable families, trainable globals, configuration search, and diagnostic-guided LLM mutation.	94.13
alphabot	Challenge winner	Top public entry on the final leaderboard; strongest direct reference.	90.14
homely-rat	Industry entry	Weco AI using AIDE , an autonomous ML competition agent.	78.32

Appendix

Table 1: Benchmark comparison for the ADIA structural-break task. The public leaderboard provides the most direct external benchmark for this live competition. We also include the Weco AI leaderboard entry as a baseline because they publicly introduced AIDE , an autonomous competition agent; independent evaluation in MLE-bench characterizes AIDE as a Kaggle-specialized tree-search scaffold.

Panel of Figure 2 — Figure 2: Optimization and search-space diagnostics. (a) Best-so-far ROC–AUC. (b) Graph complexity over time. (c) Per-island scores. (d) Configuration count.

Panel of Figure 3 — Figure 3: Structural and systems diagnostics. (a) Score vs. complexity. (b) Structural changes. (c) Acceptance rates. (d) Wall-clock time.

Panel of Figure 4 — Figure 4: Additional diagnostics. (a) Discoveries by island. (b) Best–mean gap. (c) Score statistics. (d) Score variance.

Version	Step	Island	AUC	$\Delta$ AUC	Phase / key innovation
v001	1	0-3	0.6120	—	Bootstrap. LLM generated seed; jump/local-MAD ratio and variance-jump interaction establish the initial pre/post contrast basis.
v005	8	1	0.6418	+0.0298	Bootstrap. First strong jump from skewness/kurtosis contrasts, jump-absolute features, and a soft-gated nonlinear pathway.
v009	18	2	0.6506	+0.0088	Bootstrap. Spectral-entropy pairing, FFT-slope fusion, and mean–std interaction improve coarse segment discrimination.
v014	42	0	0.6640	+0.0134	Early feature expansion. Spectral entropy difference, Gaussian KL divergence, and an early MLP-style gate enrich the feature family.
v018	74	2	0.6726	+0.0086	Early feature expansion. Multi-scale windowed variance ratio, refined pruning, and CUSUM-conditioned gating improve robustness.
v022	111	2	0.6789	+0.0063	Early feature expansion. Joint gating, mixed RBF-style similarity, and higher-order moment contrasts expand the candidate family.

Appendix

Table 2: Global-best milestones for the long-horizon ADIA run.

Comparison	Dims	Mean Acc.	W / T / L
MSRF full	1,410	0.808	—
MiniRocket	10,000	0.794	59 / 14 / 39
MiniRocket+MSRF	10,410	0.803	67 / 31 / 14 ∗
∗ vs. MiniRocket alone

Appendix

Table 3: MSRF on the 112-dataset UCR archive. W/T/L counts are relative to each baseline.

Neuro-Symbolic AI

Neuro-symbolic artificial intelligence combines neural learning with symbolic knowledge and reasoning to build more structured, interpretable, and reliable AI.

See all 100 papers

Sep 24, 2026

EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery

AI-driven scientific discovery accelerates research by autonomously developing solutions and designs. Large language model (LLM) agents support this process through iterative generation and evaluation. Yet these iterations alone do not ensure cumulative progress or establish which directions to pursue next. Costly evaluation further constrains the scope of exploration. Neural architecture discovery brings these challenges together, coupling open-ended design with resource-intensive experimentation. We introduce EvoTreeNAD, a genealogy-guided evolutionary algorithm that constructs trainable architectures without a supplied seed or a hand-specified search space. Starting from an empty root, it grows a persistent genealogy in which each new node represents a complete architecture. Top-percentile values computed from each node and its descendants guide lineage selection. Using the selected design history, an Idea Agent proposes a variant and a Code Agent implements it. Each evaluated variant becomes a child node, expanding the genealogy while providing evidence for subsequent lineage selection. Our theoretical analysis establishes the existence of stationary variation regimes as the genealogy grows. Under specified variation assumptions, sustained top-percentile family values quantify the probability of generating high-reward architectures in these regimes. EvoTreeNAD discovers architectures that outperform the compared NAS and NAD baselines, achieving CIFAR-10/100 test errors of

2.05{\pm}0.06\%

and

15.09{\pm}0.22\%

. On all six MedMNIST-v2 tasks, the discovered architectures surpass the strongest listed baselines. A controlled CIFAR-10 study further shows that EvoTreeNAD outperforms direct generation, best-of-

N

greedy continuation, and full-family-mean routing.

Lishan Yu, Derek Jiu, Qizhen Lan +1

Department of Health Data Science and AI McWilliams School of Biomedical Informatics University of Texas Health Science Center at Houston

Large Language Model-Guided NAS Evolutionary Optimization Neural Architecture Search

Ask PDF

Jul 2, 2026

Evolutionary Feature Engineering for Structured Data

Large language models are increasingly used as open-ended search operators in evolutionary optimization. We introduce Evolutionary Feature Engineering (EFE), a framework for using LLM-based evolution to discover preprocessing transformations for structured data. EFE represents transformations as Python programs with a standardized fit/transform interface, allowing them to be inserted directly into existing machine learning pipelines. During evolution, candidate programs are refined using dataset context, summary statistics, and downstream performance feedback on validation set. We instantiate EFE in two settings. For time-series forecasting, EFE-Time learns invertible, dataset-specific normalizations that improve off-the-shelf time-series foundation models. It reduces forecasting errors (MASE, WQL, MAE) 3% or more when averaged across datasets and improvements are as much as 19% on the COVID-Deaths dataset. Notably, these improvements occur with recent TSFMs such as Chronos-2. For tabular prediction, EFE-Tab evolves compact feature programs that add useful interpretable features and remove redundant ones, improving or matching existing LLM-based feature-engineering methods. We found EFE-Tab to be particularly effective on classical decision trees, where small sets of evolved features yield competitive accuracy while preserving interpretability. Overall, EFE demonstrates that LLM-based evolution can improve both accuracy and interpretability when automatically tackling structured data.

Ege Onur Taga, Yilin Zhuang, M. Emrullah Ildiz +4

University of Michigan · Google Research

Large Language Model-Guided Optimization Time Series Forecasting Tabular ML Automated Feature Engineering Large Language Model-Based Feature Engineering

Ask PDF

Jun 5, 2026

FunctionEvolve: Structure-Guided Symbolic Regression with LLMs

Symbolic regression aims to uncover explicit scientific laws from data. Recent methods use LLMs to guide mutation from background text, which is more directed than random genetic programming. However, exact symbolic recovery requires both semantic guidance and explicit structure, so that domain-informed search are carried out through valid symbolic representation. Current LLM-driven systems remain structure-blind: they select among opaque candidates, lack explicit mechanisms for local mutation, and rely on brittle coefficient fitting that can undervalue correct skeletons. We propose FunctionEvolve, an evolutionary framework using expression trees to organize the whole search: structural summaries promote diverse parent selection, local tree edits preserve useful subexpressions, and structure-aware fitting decomposes, constrains, and simplifies coefficients for more reliable scoring. It uses only elementary function families, without additional domain-specific rules limiting generalization. On the 129-task synthetic subset of LLM-SRBench, FunctionEvolve with Claude Opus 4.6 recovers 107 exact forms, reaching 82.9% SA@50, 4.5x above same-backbone baselines, and 55.8% SA@1, 3.6x above the strongest previously published top-1 result. Ablations show that structure-visible search is central to reliable recovery, with LLM-guided refinements and structure-aware coefficient optimization serving as essential proposal and scoring mechanisms. We also audit the benchmark and show that collinearity in its materials-science subset creates identifiability issues.

Zeyu Xia, Jun Zhu, Dong Yan

Bosch Center for Artificial Intelligence · Department of Computer Science and Technology, Tsinghua University · Department of Computer Science and Technology +1

Genetic Programming Large Language Model-Guided Scientific Discovery Symbolic Regression

Ask PDF

Abstract

Neuro-Symbolic AI

Explore similar work

EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery

Evolutionary Feature Engineering for Structured Data

FunctionEvolve: Structure-Guided Symbolic Regression with LLMs