TabDAG: Structured Evolutionary Search for Tabular Feature Engineering
Organizations: FusionBrain Lab · AXXX
Abstract
Feature engineering can improve tabular prediction, but useful transformations are compositional, data-dependent, and subject to leakage constraints. We present TabDAG, a framework that combines evolutionary search with a graph representation of feature-engineering programs. Nodes implement row-wise or data-fitted transformations; edges encode dependencies among raw variables, intermediate values, and final features. The representation supports interactions, statistical encodings, clustering- and neighborhood-based features, and multi-stage compositions in a single executable program. Mutations are structured edits constrained by a parent-specific schema and checked before evaluation. In our experiments, no structurally invalid program reached evaluation, although proposals could be rejected and valid graphs could still fail at runtime. Across ten regression and classification datasets and three tabular model families, evolved features improve predictive performance over raw inputs. TabDAG and its combination with OpenFE achieve the two best mean ranks among the compared methods; pairwise rank differences between feature-engineering methods do not survive multiple-testing correction. The LLM-based searches use the same backbone and candidate-evaluation budget. A controlled ablation compares free-form Python, whole-graph JSON, and constrained graph edits with the search engine, budget, and backbone held fixed. Constrained edits yield fewer failed evaluations than either rewrite baseline. The resulting graphs expose the transformations and dependencies selected by search, enabling feature-level analysis of the observed gains.
Figures & tables
| Reference | Baselines | Ours | |||||
| Dataset | Est. | Raw | OpenFE | LLM-FE | LLM-FE OF | TabDAG | TabDAG OF |
| Regression (R 2 ) | |||||||
| house | CatBoost | 0.6729 .0024 | 0.6821 .0033 ▲ | 0.6722 .0037 | 0.6802 .0038 ▲ | 0.6697 .0054 | 0.6829 .0028 ▲ |
| TabM | 0.6564 .0079 | 0.6747 .0060 ▲ | 0.6612 .0105 | 0.6692 .0127 ▲ | 0.6626 .0076 | 0.6706 .0095 ▲ | |
| TabPFN | 0.7496 .0018 | 0.7486 .0021 | 0.7474 .0019 ▼ | 0.7478 .0014 ▼ | 0.7515 .0011 ▲ | 0.7437 .0018 ▼ | |
| airfoil | CatBoost | 0.9460 .0047 | 0.9696 .0013 ▲ | 0.9710 .0015 ▲ | 0.9732 .0013 ▲ | 0.9736 .0007 ▲ | 0.9732 .0011 ▲ |
| selector | CatBoost | XGBoost | TabICL | TabPFN | RealMLP | TabM | as source |
|---|---|---|---|---|---|---|---|
| CatBoost | self | 3 | 4 | 4 | 5 | 4 | 20 |
| XGBoost | 2 | self | 4 | 4 | 7 | 4 | 21 |
| TabICL | 2 | 2 | self | 2 | 6 | 2 | 14 |
| TabPFN | 0 | 2 | 6 | self | 4 | 2 | 14 |
| RealMLP | 0 | 0 | 1 | 1 | self | 1 | 3 |
| TabM | 2 | 4 | 3 | 3 | 5 | self | 17 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Task (metric) | Train rows | Raw cols |
|---|---|---|---|
| house | regression (R 2 ) | 14,581 | 16 |
| airfoil | regression (R 2 ) | 501 | 5 |
| protein | regression (R 2 ) | 15,243 | 9 |
| concrete | regression (R 2 ) | 343 | 8 |
| miami | regression (R 2 ) | 4,592 | 15 |
| churn | binary (acc.) | 6,400 | 11 |
| Selector receiver | CatBoost | XGBoost | TabICL | TabPFN | RealMLP | TabM |
|---|---|---|---|---|---|---|
| CatBoost | self | 8/16 | 6/16 | 6/16 | 11/16 | 6/16 |
| XGBoost | 6/16 | self | 9/16 | 5/16 | 8/16 | 8/16 |
| TabICL | 3/16 | 6/16 | self | 6/16 | 7/16 | 6/16 |
| TabPFN | 1/16 | 4/16 | 5/16 | self | 5/16 | 5/16 |
| RealMLP | 2/16 | 4/16 | 5/16 | 4/16 | self | 3/16 |
| TabM | 2/16 | 9/16 | 4/16 | 2/16 | 8/16 | self |
| Selector receiver | CatBoost | XGBoost | TabICL | TabPFN | RealMLP | TabM |
|---|---|---|---|---|---|---|
| CatBoost | self | 3/8 | 4/8 | 4/8 | 5/8 | 4/8 |
| XGBoost | 2/8 | self | 4/8 | 4/8 | 7/8 | 4/8 |
| TabICL | 2/8 | 2/8 | self | 2/8 | 6/8 | 2/8 |
| TabPFN | 0/8 | 2/8 | 6/8 | self | 4/8 | 2/8 |
| RealMLP | 0/8 | 0/8 | 1/8 | 1/8 | self | 1/8 |
| TabM | 2/8 | 4/8 | 3/8 | 3/8 | 5/8 | self |
| Selector | Top-5 mean better | Rank 1 better | Cells |
|---|---|---|---|
| CatBoost | 20 | 28 | 48 |
| XGBoost | 16 | 32 | 48 |
| TabICL | 14 | 34 | 48 |
| TabPFN | 21 | 27 | 48 |
| RealMLP | 8 | 40 | 48 |
| TabM | 23 | 25 | 48 |
| Dataset | Metric | Raw | Semantic | Anonymous | Shuffled | Holm | Holm | ||
| A. Ten paired evaluator refits | |||||||||
| Gallstone | AUC | .8744 .0068 | .8819 .0061 | .8717 .0151 | .8753 .0072 | .1699 | .1699 | ||
| 5G Energy | MAPE | .1067 .0020 | .1026 .0020 | .1076 .0011 | .1041 .0016 | .1699 | |||
| SDSS-17 | log loss | .0740 .0004 | .0662 .0005 | .0652 .0006 | .0696 .0003 | ||||
| B. One evaluator fit (descriptive extension) | |||||||||
| HIGGS-small | AUC | .8054 | .8136 | .8084 | .8074 | – | – | ||
| Dataset | CatBoost | XGBoost | TabICL | TabPFN | RealMLP | TabM |
|---|---|---|---|---|---|---|
| Binary (AUC) | ||||||
| Adult | 0.9282 .0002 | 0.9279 .0000 | 0.9283 .0020 | 0.9196 .0004 | 0.9125 .0015 | 0.9170 .0009 |
| Churn | 0.8666 .0006 | 0.8590 .0000 | 0.8670 .0004 | 0.8708 .0003 | 0.8605 .0042 | 0.8647 .0026 |
| HIGGS-small | 0.8140 .0005 | 0.8118 .0000 | 0.8227 .0004 | 0.8256 .0003 | 0.8141 .0010 | 0.8310 .0008 |
| Regression (R 2 ) | ||||||
| House | 0.6737 .0045 | 0.6542 .0000 | 0.7249 .0015 | 0.7506 .0017 | 0.6539 .0072 | 0.6676 .0054 |
| Arm | priced cells | total ($) | $/cell | to best ($) |
|---|---|---|---|---|
| LLM-FE | 24 | 44.70 | 1.86 | 25.80 |
| Python | 21 | 102.17 | 4.87 | 51.99 |
| JSON | 24 | 145.53 | 6.06 | 74.03 |
| TabDAG | 17 | 107.28 | 6.31 | 60.56 |
| total | 86 | 399.69 | 212.39 |
| dataset | est. | LLM-FE | Python | JSON | TabDAG |
|---|---|---|---|---|---|
| house | CatBoost | 1.81 (0.81) | 4.77 (4.35) | 6.08 (5.39) | 5.99 (3.74) |
| TabM | 2.00 (1.28) | 5.41 (4.88) | 6.31 (3.69) | 5.87 (0.60) | |
| TabPFN | 1.92 (0.64) | 5.73 (2.09) | 6.07 (0.72) | 6.40 (6.32) | |
| airfoil | CatBoost | 1.89 (0.76) | 4.50 (0.99) | 5.68 (5.46) | 7.14 (6.93) |
| TabM | 2.04 (1.12) | 4.59 (4.54) | 5.65 (2.36) | 6.52 (3.63) | |
| TabPFN | 1.83 (0.93) | 4.42 (3.68) | 5.89 (2.92) | 6.10 (1.86) |
| Reference | Evolved representations | ||||
| Dataset | Est. | Raw | JSON | TabDAG | Python |
| Regression (R 2 ) | |||||
| house | CatBoost | 0.6729 .0024 | 0.6654 .0034 ▼ | 0.6713 .0046 ▼ | 0.6642 .0026 ▼ |
| TabM | 0.6564 .0079 | 0.6547 .0045 | 0.6603 .0121 | 0.6567 .0156 | |
| TabPFN | 0.7496 .0018 | 0.7478 .0017 ▼ | 0.7515 .0011 ▲ | 0.7498 .0013 | |
| airfoil | CatBoost | 0.9460 .0047 | 0.9717 .0007 ▲ | 0.9729 .0010 ▲ | 0.9694 .0010 ▲ |
| JSON | TabDAG (ladder) | TabDAG sequential | ||||
|---|---|---|---|---|---|---|
| mean sd | median | mean sd | median | mean sd | median | |
| cells | 29 | 30 | 20 | |||
| nodes per selected program | 3.00 | 3.00 | 3.00 | |||
| graph depth (levels) | 1.00 | 1.00 | 1.50 | |||
| nodes fed by another node | 0.00 | 0.00 | 0.11 | |||
| aggregate nodes (share) | 0.50 | 0.19 | 0.25 | |||
| adult | airfoil | churn | concrete | higgs | house | miami | otto | protein | students | |
| 16/16 | 14/16 | 16/16 | 13/16 | 12/16 | 11/16 | 11/16 | 6/16 | 16/16 | 15/16 | |
| no evaluation feedback | ||||||||||
| seed | 0.8732 [-1.5pt] .0024 | 0.9129 [-1.5pt] .0115 | 0.8637 [-1.5pt] .0044 | 0.9201 [-1.5pt] .0198 | 0.7263 [-1.5pt] .0033 | 0.6617 [-1.5pt] .0186 | 0.9248 [-1.5pt] .0090 | 0.8044 [-1.5pt] .0007 | 0.6102 [-1.5pt] .0123 | 0.7728 [-1.5pt] .0065 |
| 1 draw | 0.8729 [-1.5pt] .0004 | 0.9348 [-1.5pt] .0069 | 0.8335 [-1.5pt] .0659 | 0.9151 [-1.5pt] .0025 | 0.7325 [-1.5pt] .0011 | 0.6393 [-1.5pt] .0735 | 0.8402 [-1.5pt] .2713 | 0.8054 [-1.5pt] .0006 | 0.6410 [-1.5pt] .0035 | 0.7796 [-1.5pt] .0022 |
| best of 16 | 0.8740 [-1.5pt] .0003 | 0.9404 [-1.5pt] .0004 | 0.8657 [-1.5pt] .0004 | 0.9210 [-1.5pt] .0018 | 0.7342 [-1.5pt] .0003 | 0.6738 [-1.5pt] .0025 | 0.9255 [-1.5pt] .0011 | 0.8059 [-1.5pt] .0001 | 0.6485 [-1.5pt] .0018 | 0.7826 [-1.5pt] .0004 |
| searched, 100 evaluations | ||||||||||
| tier | what it covers |
|---|---|
| Decode | Unknown or missing fields, wrong types, values outside an enumeration, references to a node that does not exist, fewer nodes than the minimum. Not expressible. |
| Transcription | Duplicate ids uniquified; input_cols extended with literal column reads found in the code that resolve to a real column; dependencies inferred from inputs that name an earlier node’s output; output_types re-synchronized after an output_cols edit; an over-long rationale truncated. Each is logged. |
| Validation | Node cap, the graph invariants above, the declared structural intent, a Python parse and compile of every node body, and the requirement that transform returns. Failure discards the child before anything runs. |
| Execution | Determinism, batch purity, own-target exclusion, declared output types, target round-trip. Failure marks the candidate invalid. |
| Processor | |
| model | AMD EPYC 9474F, 48 cores |
| sockets / cores / threads | 2 / 48 / 2 (192 logical) |
| clock | 1.50 GHz base, 4.11 GHz max |
| cache | 3 MiB L1d, 3 MiB L1i, 96 MiB L2, |
| 512 MiB L3 (16 instances) | |
| NUMA | 2 nodes (CPUs 0–47/96–143, |
| component | version | recorded |
| Python | 3.12.13 | |
| PyTorch | 2.13.0 (CUDA 13) | |
| NumPy | 2.5.1 | |
| pandas | 2.2.2 | |
| scikit-learn | 1.9.0 | |
| SciPy | 1.18.0 |
Automated Feature Engineering
Automated Feature Engineering creates, transforms, and selects predictive inputs with less manual design across data types and applications.
See all 57 papers