Closing the gap on tabular data with Fourier and Implicit Categorical Features

Marius Dragoi; Florin Gogianu; Elena Burceanu

cs.LGFeb 26, 2026

Closing the gap on tabular data with Fourier and Implicit Categorical Features

Authors: Marius Dragoi, Florin Gogianu, Elena Burceanu

Organizations: Bitdefender, Romania

Abstract

While Deep Learning has demonstrated impressive results in applications on various data types, it continues to lag behind tree-based methods when applied to tabular data, often referred to as the last "unconquered castle" for neural networks. We hypothesize that a significant advantage of tree-based methods lies in their intrinsic capability to model and exploit non-linear interactions induced by features with categorical characteristics. In contrast, neural-based methods exhibit biases toward uniform numerical processing of features and smooth solutions, making it challenging for them to effectively leverage such patterns. We address this performance gap by using statistical-based feature processing techniques to identify features that are strongly correlated with the target once discretized. We further mitigate the bias of deep models for overly-smooth solutions, a bias that does not align with the inherent properties of the data, using Learned Fourier. We show that our proposed feature preprocessing significantly boosts the performance of deep learning models and enables them to achieve a performance that closely matches or surpasses XGBoost on a comprehensive tabular data benchmark.

Figures & tables

Figure 1: Method overview. We select either (A) ICF or (B) LFF for data preprocessing in each run. In ICF, we add zero-padding for features with a lower number of bins to allow concatenation.

Figure 2: Performance by budget across 15 random simulations of hyperparameter optimization with 150 random seeds for each baseline.

Figure 3: Performance plot on the top eight runs (approx. the 95th percentile for most tasks).

Figure 4: Heatmap of the top eight runs (corresponding to the 95th percentile for most tasks) ResNet+F|C and xgb . We observe runs where ResNet+F|C achieves a performance ”spike”, while xgb has a more consistent performance landscape

Figure 5: Heatmap of the top eight runs (corresponding to the 95th percentile for most tasks) for the base model ResNet (RN) and ResNet+F|C (RN-F $|$ C). We additionally show the separated runs of ResNet+F|C into ResNet+F (RN-F) and ResNet+C (RN-C)

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Regression num	Regression cat	Classif num	Classif cat
xgb	0.7558	0.7770	0.8298	0.7872
xgb with ICF	0.7502	0.7751	0.8271	0.7838

Appendix

Table 1: Impact of categorical encoding for xgb

Parameter	Range
eta	logUniform [1e-05, 0.7]
gamma	logUniform [1e-08, 7]
max depth	uniformInt [3, 11]
subsample	uniformInt [0.5, 1]
lambda	logUniform [1, 4]
alpha	logUniform [1e-08, 1e2]

Appendix

Table 2: xgb parameter space

Parameter	Range
depth	uniformInt [2, 8]
width	choice [128, 256, 512, 1024]
activation	choice [ReLU, LeakyReLU]
batch normalization	choice [True, False]
dropout	choice [0.0, 0.5, 0.6, 0.7, 0.8, 0.9]

Appendix

Table 3: mlp parameter space

Parameter	Range
num block	uniformInt [1,3]
num linear	uniformInt [1,3]
use norm	choice [True, False]
norm type	choice [batch, layer]
use do	choice [True, False]
do prob	choice [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]

Appendix

Table 4: ResNet parameter space

Parameter	Range
opt name	AdamW
learning rate	uniform [0.001, 0.1]
eps	uniform [1e-08, 1e-04]
weight decay	uniform [0.0001, 0.6]
scheduler	CosineAnnealingWarmRestart
T 0	choice [10, 20, 30, 50, 75, 100]

Appendix

Table 5: Optimizer parameter space

Parameter	Range
chi thresh	uniform [1e-50, 1e-03]
ANOVA thresh	uniform [1e-30, 1e-03]
mi thresh	uniform [0.75, 1.50]
min cardinality	choice [0, 10, 100]
max cardinality	choice [300, 500, 1000, 1500, 5000]

Appendix

Table 6: ICF identification tests

Tabular Prediction

Tabular prediction uses structured rows and columns to forecast labels, values, risks, and outcomes from data.

See all 111 papers

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks
Jun 1, 2026Andrej Tschalzev, Nick Erickson, Yuyang Wang +4Tabular ML Automated Feature Engineering
University of Mannheim · Prior Labs · AWS +3
LassoFlexNet: Flexible Neural Architecture for Tabular Data
Mar 21, 2026Kry Yik Chau Lui, Cheng Chi, Kishore Basu +1Tabular ML Tabular Representation Learning
RBC Borealis · RBC
DynaTab: Dynamic Feature Ordering as Neural Rewiring for High-Dimensional Tabular Data
May 5, 2026Al Zadid Sultan Bin Habib, Gianfranco Doretto, Donald A. AdjerohDynamic Neural Networks Tabular ML
Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506, USA · Scientific Computing and Imaging Institute & Department of Biomedical Informatics, The University of Utah, Salt Lake City, UT 84112, USA

Jun 1, 2026J/K move · Enter open · S save

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

Andrej Tschalzev, Nick Erickson, Yuyang Wang, Huzefa Rangwala+3

University of Mannheim · Prior Labs · AWS+3

Tabular ML Automated Feature Engineering Tabular Prediction

Open paper AskPDF arXiv

Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remains a critical yet underexplored component of real-world modeling pipelines that is entirely absent from modern benchmarks, which creates an unquantified evaluation gap. In this work, we introduce TabPrep, a lightweight preprocessing pipeline composed of feature generators that are carefully designed to target three specific structural data patterns. We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance. Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models, often surpassing gains achieved by model-centric innovations alone. TabPrep outperforms previous automated feature engineering approaches in performance, efficiency, and applicability across datasets, enabling integration into large-scale benchmarks. By releasing TabPrep (see https://github.com/atschalz/tabprep), we enable researchers to integrate feature engineering into their benchmarking setup, filling a longstanding gap in tabular evaluations.

Abstract

Tabular Prediction

Explore similar work

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

LassoFlexNet: Flexible Neural Architecture for Tabular Data

DynaTab: Dynamic Feature Ordering as Neural Rewiring for High-Dimensional Tabular Data