Do Tabular Foundation Models Still Need Feature Engineering?

Yifan WU; Pinjun Dong; Jiran Tao; Binyan Jiang

cs.LGAug 14, 2026

Do Tabular Foundation Models Still Need Feature Engineering?

Authors: Yifan WU, Pinjun Dong, Jiran Tao, Binyan Jiang

Organizations: Department of Data Science, The Hong Kong Polytechnic University, Hong Kong

Abstract

Feature engineering has long been a cornerstone of tabular machine learning. Tabular foundation models (TFMs) are pretrained on a wide range of tabular datasets and applied via in-context learning. Their rise raises a natural question: does manual feature construction still matter as these models become more capable? To answer this, we perform a controlled study across several versions of two major TFM families, testing a wide range of existing feature engineering techniques on benchmark datasets from TabArena. We find a consistent pattern: feature engineering gains are concentrated in earlier model generations and become negligible for the strongest models. These results suggest that stronger TFMs depend less on explicitly engineered input representations. In a complementary experiment, however, adding in-context information from related datasets still improves performance. Our findings indicate a shift in the source of performance gains for stronger TFMs: re-representing existing inputs becomes less effective, while providing additional task-relevant context remains beneficial.

Figures & tables

Dataset	Samples	Features	Classes
Classification
Blood transfusion	748	4	2
Diabetes	768	8	2
Anneal	898	38	5
Credit-g	1,000	20	2
Maternal health	1,014	6	3

Table 1: The 13 TabArena datasets used in this study, ordered by sample count within each task.

Family	Count	Methods	Role
Numeric transforms	5	Standard; log; sqrt; Box–Cox; min–max	Transform numeric inputs
Categorical encoding	5	One-hot; ordinal; frequency; target; crosses	Encode categorical inputs
Binning	2	Equal-width; equal-frequency	Model threshold effects
Interactions	2	Numeric interactions; group aggregation	Add higher-order structure
Feature selection	4	Variance filter; collinearity filter; PCA; supervised selection	Remove redundant features
TabPrep	6	Full pipeline; no-groupby; no-RSFC; no-arithmetic; no-categorical-interaction; no-OOF-target-encoding	Combine feature transformations

Table 2: The 26 feature engineering conditions grouped by family. Applicability depends on dataset column types; the identity baseline is not shown.

Benchmark	Basis for alignment	Source data	Residual mismatch	Difficulty
Airfoil noise	Both include NACA 0012 aerofoil measurements and represent closely related aeroacoustic prediction problems ( 5 ; 4 ) .	A separate University of Bristol aeroacoustic and PIV campaign covering pre-stall, stall, and post-stall conditions.	Differences in facility, operating conditions, flow regime, and measurement protocol.	Lower
Concrete strength	Both relate mixture composition and curing conditions to compressive strength ( 23 ; 15 ) .	A literature-assembled collection of geopolymer-concrete mixtures.	Differences in binder chemistry, precursor and activator composition, curing protocols, and cross-study curation.	Moderate
Maternal health	Six physiological predictors can be aligned between the two Bangladesh maternal-health datasets ( 1 ; 16 ) .	A later rural clinical dataset containing additional historical and physiological variables.	Differences in patient population and label construction; the source labels missing the target mid-risk class.	Higher

Table 3: Source–target compatibility in context augmentation. Difficulty is residual mismatch after predictor alignment.

Figure 1: Feature engineering response across model generations. Per dataset and version, gains are medians over 30 paired splits; gray crosses and hollow circles show their median and maximum across interventions. Orange and blue marks are the respective across-dataset medians. Positive favors feature engineering; the maximum is oracle, not validation-selected. Units are macro-F1 points (classification) and relative RMSE reduction (regression).

Figure 2: Across-dataset median effects for all 26 interventions and 12 estimators; positive favors feature engineering. Classification uses macro-F1 points (color limits $\pm 2$ ); regression uses relative RMSE reduction ( $\pm 10\%$ ). Open circles mark positive effects on at least half of applicable datasets; rules separate intervention families.

	Classification (F1 pp)		Regression (relative RMSE %)
Family	PFN v3	ICL v2	PFN v3	ICL v2
Numeric transform	$+0.00$ (9/35)	$+0.00$ (1/35)	$-0.04$ (6/25)	$+0.00$ (18/25)
Categorical encoding	$+0.00$ (2/25)	$+0.00$ (4/25)	$-0.23$ (5/13)	$-0.10$ (3/13)
Binning	$-0.88$ (1/14)	$-0.96$ (2/14)	$-6.70$ (0/10)	$-7.40$ (0/10)
Interactions	$+0.00$ (1/11)	$+0.00$ (0/11)	$-0.55$ (2/8)	$-0.01$ (3/8)
Selection	$+0.00$ (5/31)	$+0.00$ (6/31)	$-0.93$ (0/20)	$-0.83$ (0/20)

Table 4: Feature engineering family effects for the strongest evaluated model variants. Each cell reports the median direction normalized effect across applicable (dataset, intervention) cells, followed by the number of positive cells over the number applicable. Positive values favor feature engineering.

Panel	$D/G$	Exact test; statistic; raw $p$ ; median contraction (declining)
TabPFN classification	8/5	Page; $L=389$ ; 0.0174; 1.095 (5/8)
TabPFN regression	5/4	Page; $L=139$ ; 0.0167; 1.590 (4/5)
TabICL classification	8/2	Sign-flip; $T=0.412$ ; 0.0234; 0.429 (6/8)

Table 5: Exact dataset-level tests for declining best observed feature engineering headroom. Raw

p

is one-sided. Joint adjustment gives

q_{\mathrm{BH}}=0.0234

and

p_{\mathrm{Holm}}=0.0502

for every comparison. Contraction is the median first-to-last difference; parentheses count declines across datasets. Units are macro-F1 points (classification) and relative RMSE points (regression).

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	PFN v2	PFN v2.5	PFN v2.6	PFN v3	ICL v2
Identity	$0.9737\mathbin{\pm}0.0046$	$0.9726\mathbin{\pm}0.0048$	$0.9751\mathbin{\pm}0.0048$	$0.9784\mathbin{\pm}0.0036$	$0.9756\mathbin{\pm}0.0039$
Standard scaling	$0.9731\mathbin{\pm}0.0050$	$0.9728\mathbin{\pm}0.0050$	$0.9731\mathbin{\pm}0.0042$	$0.9782\mathbin{\pm}0.0035$	$0.9756\mathbin{\pm}0.0039$
Log transform	$0.9730\mathbin{\pm}0.0046$	$0.9725\mathbin{\pm}0.0051$	$0.9722\mathbin{\pm}0.0046$	$0.9781\mathbin{\pm}0.0038$	$0.9763\mathbin{\pm}0.0040$
Square-root transform	$0.9730\mathbin{\pm}0.0047$	$0.9727\mathbin{\pm}0.0050$	$0.9752\mathbin{\pm}0.0046$	$0.9778\mathbin{\pm}0.0039$	$0.9760\mathbin{\pm}0.0040$
Box–Cox	$0.9730\mathbin{\pm}0.0047$	$0.9726\mathbin{\pm}0.0050$	$0.9721\mathbin{\pm}0.0048$	$0.9785\mathbin{\pm}0.0034$	$0.9759\mathbin{\pm}0.0038$
Min–max scaling	$0.9733\mathbin{\pm}0.0047$	$0.9728\mathbin{\pm}0.0050$	$0.9733\mathbin{\pm}0.0049$	$0.9782\mathbin{\pm}0.0036$	$0.9756\mathbin{\pm}0.0039$

Appendix

Table 6: Complete raw metric results for Airfoil noise, metric r2. Values are mean

\mathbin{\pm}