From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning

Manish Kumar; Anton Frederik Thielmann; Christoph Weisser; Benjamin Säfken

cs.LGApr 7, 2026

From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning

Authors: Manish Kumar, Anton Frederik Thielmann, Christoph Weisser, Benjamin Säfken

Organizations: BASF · Clausthal University of Technology · Amazon Music · Bielefeld School of Business, Hochschule Bielefeld (HSBI)

Abstract

Numerical preprocessing remains a critical component of tabular deep learning, as the representation of continuous features can strongly affect downstream performance. We systematically study spline-based numerical encodings, including B-splines, M-splines, and integrated splines (I-splines), under uniform, quantile-based, target-aware, and learnable-knot placement. For the learnable variants, we adopt a differentiable knot parameterization that enables stable end-to-end optimization of knot locations jointly with the backbone. We evaluate these encodings on a diverse collection of public regression and classification datasets using MLP, ResNet, and FT-Transformer backbones, and compare them against common numerical preprocessing baselines. Our results show that the effectiveness of numerical encoding depends strongly on the task, encoding size, and backbone. For classification, piecewise-linear encoding (PLE) is the most robust choice overall, while spline-based encodings remain competitive. For regression, no single encoding dominates, with performance depending on the spline family and knot-placement strategy, and larger gains generally observed for MLP and ResNet than for FT-Transformer. Learnable-knot variants can be optimized stably but may substantially increase training cost. Overall, numerical encodings should therefore be assessed jointly in terms of predictive performance and computational overhead. The implementation is publicly available at https://github.com/mkumar73/tdl-numerical-encodings/.

Figures & tables

Figure 1: The figure illustrates how a single numerical value $x_{i,j}$ , shown by the red dashed line, is represented by different schemes. Std and MinMax keep one scalar value per feature, whereas PLE and B-/M-/I-splines expand the value into coordinates $(e_{1},\ldots,e_{m_{j}})$ by evaluating bin ramps or basis functions. The markers show the evaluated values and the bars show the resulting encoded coordinates. B-splines provide smooth local bases, M-splines nonnegative local bases, and I-splines cumulative monotone bases. For the PLE and spline panels, we set the output size to $m_{j}=6$ for illustration. Details of the construction are provided in Appendix E .

Panel of Figure 2 — Figure 2: Regression critical difference diagrams. CD diagrams aggregated over all backbones for output sizes $m\in\{7,15,30\}$ . Lower average rank indicates better overall performance. Methods connected by a horizontal bar are not significantly different under the Nemenyi test. Preprocessing abbreviations are given in Table 3 , and detailed regression results for the corresponding output sizes are provided in Tables 11 , 12 , and 13 .

Panel of Figure 3 — Figure 3: Critical difference diagrams for classification. The diagrams are aggregated over all backbones for output sizes $m\in\{7,15,30\}$ . Lower average rank indicates better overall performance. Methods connected by a horizontal bar are not significantly different under the Nemenyi test. Preprocessing abbreviations are given in Table 3 , and detailed classification results for the corresponding output sizes are reported in Tables 14 , 15 , and 16 .

Figure 4: Average regression performance across backbones and output sizes. Each heatmap cell shows the mean test NRMSE ( $\downarrow$ ) across all regression datasets for a given backbone, preprocessing method, and output size $m\in\{7,15,30\}$ . Preprocessing abbreviations are given in Table 3 , and detailed results are provided in Tables 11 , 12 , and 13 .

Figure 5: Average classification performance across backbones and output sizes. Each heatmap cell shows the mean test AUC ( $\uparrow$ ) across all classification datasets for a given backbone, preprocessing method, and output size $m\in\{7,15,30\}$ . For multiclass datasets, AUC is computed as weighted one-vs-rest ROC-AUC. Higher values indicate better performance. Preprocessing abbreviations are given in Table 3 , and detailed results are provided in Tables 14 , 15 , and 16 .

Panel of Figure 6 — Figure 6: PLE and cubic B-spline fits on simple synthetic examples with the same basis budget ( $m=10$ ). (a) Regression example with a smooth nonlinear target, fitted using Ridge regression. The B-spline fit is smoother and tracks the target more closely, whereas the PLE fit shows piecewise-linear changes at the bin boundaries. (b) Classification example with a piecewise-constant class-probability function, fitted using logistic regression. The PLE fit better captures the sharp transitions and flat regions, whereas the B-spline fit changes more smoothly across the boundaries. This figure is intended as an illustration of the different behaviors of the two encodings rather than as a benchmark result.

Preprocessing variant	Transformation cost	Forward	Backward
Std	$O(d\,B)$	–	–
MinMax	$O(d\,B)$	–	–
PLE	$O(d\,B\,m)$	–	–
Fixed knots	$O(d\,B\,m\,p)$	–	–
Learnable knots	–	$O(d\,B\,m\,p)+O(d\,n_{\text{int}})$	$O(d\,B\,m\,p)+O(d\,B\,n_{\text{int}})$

Table 1: Asymptotic time complexity per batch. Here,

d

denotes the number of numerical features,

B

the batch size,

m

the per-feature output size,

p

the spline degree, and

n_{\text{int}}

the number of learnable internal knot parameters. Fixed knots refer to the B-, M-, and I-spline variants with uniform, quantile, and target-aware knot placement, while learnable knots refer to the gradient-based variants. Fixed preprocessing methods incur only transformation cost, whereas learnable-knot variants add forward and backward overhead during joint training with the backbone. Preprocessing abbreviations are given in Table 3 .

Figure 7: SGEMM efficiency case study: total GPU wall-clock time over 5-fold cross-validation. Comparison of Std, MinMax, PLE, fixed-knot spline variants (BS-U and IS-U), and learnable-knot spline variants (BS-Grad-U, MS-Grad-U, and IS-Grad-U) across basis budgets $m\in\{7,15,30\}$ for MLP, ResNet, and FT-Transformer. Values denote total end-to-end training time across all five folds. For learnable-knot variants, timings include the initial warm-up phase followed by joint optimization of knot parameters and backbone weights.

Factor	Setting	Start with	Guideline
Backbone	MLP or ResNet-style models	Explicit encodings	These models benefit most from numerical encodings because their input representation is relatively simple. Use PLE for classification and B-splines for regression, starting with small output sizes such as $m\approx 7$ – $10$ .
Backbone	Feature-tokenizing or high-capacity models	Std or MinMax	For models such as FT-Transformer, TabM, TabTransformer, or SAINT, simple preprocessing is a strong first choice. Add PLE or fixed-knot splines only if the simple baseline is insufficient.
Task	Classification	PLE	PLE is the strongest default for classification, with robust rankings across output sizes and backbones. It should usually be tried before more expensive spline variants.
Task	Regression	B-splines	Start with B-splines using uniform or quantile knot placement. With larger budgets, include target-aware knots and I-splines in the sweep.
Budget	Learnable-knot variants	BS-Grad-U	Among learnable-knot variants, BS-Grad-U provides the most favorable tradeoff between efficiency and performance. MS-Grad-U and IS-Grad-U should be treated as high-cost options.
Caution	M-splines	Not first choice	M-splines are not recommended as the default. In our experiments, both fixed and learnable M-spline variants were less attractive due to stability and efficiency concerns.

Table 2: Practical recommendations for selecting numerical preprocessing methods based on task, backbone, and computational budget.

Figure 8: Sensitivity to basis resolution on a synthetic regression task. Test NRMSE (mean $\pm$ std over 5 seeds) for PLE and spline-based encodings as the number of bins or basis functions varies over $\{5,10,15,20,25,30,35,40,45,50\}$ . Results are grouped by knot-selection strategy. Dotted horizontal lines show the Std, MinMax, and $\mathrm{PLE}_{\mathrm{adp}}^{50}$ baselines. Preprocessing abbreviations are given in Table 3 . All results use an MLP backbone.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Method	Description	Target-aware	Learnable-knot
Baseline	Std	Standardization (z-score)	–	–
	MinMax	min-max scaling to $[0,1]$	–	–
	PLE	Piecewise Linear Encoding	$\checkmark$	–
	$\mathrm{PLE}_{\mathrm{adp}}^{50}$	Adaptive PLE with $n_{\mathrm{bins}}\in[5,50]$ selected by tree splits (Table 8 )	$\checkmark$	–
B-Spline	BS-U	Uniform knot placement	–	–
	BS-Q	Quantile-based knot placement	–	–

Appendix

Table 3: Preprocessing abbreviations used throughout the paper. The table summarizes the naming convention for baseline, spline-based, target-aware, and learnable-knot variants.

Dataset	Abbr.	#cat	#num	Train	Val	Test	Ratio	Reference / OpenML ID
Regression
Abalone	AB	1	7	3008	334	835	–	Nash et al. (1994)
California Housing	CA	1	8	14861	1651	4128	–	OpenML: 45028
CPU Small	CPU	0	12	5899	655	1638	–	OpenML: –
Diamonds	DI	3	6	38837	4315	10788	–	OpenML: 44979
House Sales	HS	0	18	15562	1729	4322	–	OpenML: 42092

Appendix

Table 4: Benchmark datasets used in the experiments. For each dataset, we report the abbreviation, the numbers of categorical (

\#\mathrm{cat}

) and numerical (

\#\mathrm{num}

) features, and the average train, validation, and test split sizes over 5-fold cross-validation. Ratio denotes the dominant-class percentage for classification and multiclass datasets. The last column gives the OpenML dataset ID or the corresponding UCI citation.

Metric	Regression	Classification	Total
Number of datasets	13	12	25
Total samples	255,489	255,928	511,417
Avg. samples per dataset	19,653	21,327	20,456
Avg. features per dataset	10.5	13.0	11.7
Min. features	5	6	5
Max. features	19	23	23

Appendix

Table 5: Benchmark dataset summary. Aggregate statistics of the benchmark suite, including the number of datasets, total samples, average samples per dataset, and feature counts, reported separately for regression and classification datasets and for the full collection.

Symbol	Meaning
$n$	Number of training samples in the current fold
$d$	Number of numerical features
$\mathbf{x}^{(i)}$	Input vector for sample $i$
$\mathbf{x}^{(i)}_{\mathrm{num}}$	Numerical feature vector for sample $i$
$\mathbf{x}^{(i)}_{\mathrm{cat}}$	Categorical feature vector for sample $i$
$x_{i,j}$	Scalar value of numerical feature $j$ for sample $i$

Appendix

Table 6: Main notation used in the methodology.

Model	Architecture	Configuration
MLP	3-layer MLP	Hidden dims $[256,128,64]$ ; ReLU activations; dropout $0.3$ .
ResNet	Residual MLP blocks ( Gorishniy et al. (2021) )	Block: Linear $\rightarrow$ BN $\rightarrow$ ReLU $\rightarrow$ Dropout $\rightarrow$ Linear $\rightarrow$ BN + skip; $d_{\text{model}}=256$ ; $n_{\text{blocks}}=3$ ; $d_{\text{hidden\_factor}}=2.0$ ; dropout $0.3$ ; batch normalization.
FTT
(FT-Transformer)	Feature Tokenizer + Transformer ( Gorishniy et al. (2021) )	$d_{\text{token}}=192$ ; $n_{\text{blocks}}=3$ ; $n_{\text{heads}}=8$ ; attention dropout $0.2$ ; FFN dropout $0.1$ ; residual dropout $0.0$ ; $ffn\_factor=4/3$ ; ReGLU activations.
Shared training setup (all models)
AdamW with backbone learning rate $\eta_{\theta}=10^{-4}$ and weight decay $10^{-5}$ , batch size $512$ , and a maximum of $200$ epochs. Early stopping patience is set to $15$ , and ReduceLROnPlateau uses patience $10$ with factor $0.1$ . For FT-Transformer, weight decay is excluded from feature token embeddings, layer normalization parameters, the [CLS] token, and bias terms.

Appendix

Table 7: Backbone architectures and training configuration. We report the hyperparameters for the MLP, ResNet, and FT-Transformer backbones, along with the shared optimization strategy. Additional settings specific to gradient-based knot optimization are listed separately.

Method	Variant	Component	Non-adaptive (fixed)	Adaptive (range)
PLE	CART-based	Output size	$m=\{7,15,30\}$	$min\_bins=5$ $max\_bins=50$
PLE	CART-based	Tree regularization	$min\_samples\_leaf=1$ $min\_samples\_split=2$	$min\_samples\_leaf=25$ $min\_samples\_split=2$
Splines	CART-based	Output size	$m=\{7,15,30\}$	–
Splines	CART-based	Tree / knot constraints	$max\_depth=6$ $min\_knot\_spacing=0.01$	–
Splines	LightGBM-based	Output size	$m=\{7,15,30\}$	–
Splines	LightGBM-based	GBDT hyperparameters	$n\_estimators=100$ $max\_depth=3$ $learning\_rate=0.1$	–

Appendix

Table 8: Target-aware configuration details and output-size settings. We report the configurations for target-aware PLE binning and target-aware spline knot placement using CART and LightGBM. Adaptive output size is used only for PLE in the ablation study; spline encodings use fixed output sizes

m\in\{7,15,30\}

\kappa_{j,\ell}\leftarrow\frac{\ell}{K_{j}+1},\hskip 17.00024pt\ell=1,\ldots,K_{j}.

Appendix

Algorithm 1 Spline-Based Numerical Encoding Pipeline

\bm{\kappa}_{j}^{(0)}=(\kappa^{(0)}_{j,1},\ldots,\kappa^{(0)}_{j,K_{j}})

Appendix

Algorithm 2 Learnable-knot optimization for spline feature expansion

Preprocessing Method	Avg. rank, $m=7$	Avg. rank, $m=15$	Avg. rank, $m=30$
Std	9.29	8.96	8.46
MinMax	11.36	10.67	10.05
PLE	8.50	7.36	5.91
BS-U	7.29	7.82	7.97
BS-Q	5.65	7.08	8.42
BS-CART	5.91	7.46	9.73

Appendix

Table 9: Average rank (lower is better) of each preprocessing method across all backbones on the regression benchmark, for basis resolutions of 7, 15 and 30 bins/basis functions. The best (lowest) rank in each column is in bold.

Preprocessing Method	Avg. rank, $m=7$	Avg. rank, $m=15$	Avg. rank, $m=30$
Std	11.33	11.21	10.21
MinMax	12.53	12.10	11.25
PLE	5.01	3.96	3.33
BS-U	8.15	7.64	8.56
BS-Q	6.12	7.78	8.64
BS-CART	5.25	5.78	8.00

Appendix

Table 10: Average rank (lower is better) of each preprocessing method across all backbones on the classification benchmark, for basis resolutions of 7, 15 and 30 bins/basis functions. The best (lowest) rank in each column is in bold.

Figure 9: Synthetic regression dataset used in the ablation study. Scatter plot of the informative feature $x_{0}$ against the target variable $y$ , illustrating the heterogeneous structure induced by the synthetic target function.

Panel of Figure 10 — Figure 10: Effect of knot learning rate on knot relocation during training. Learned knot trajectories for the informative feature under the learnable-knot B-spline variant BS-Grad-U, shown for increasing knot learning rates $\eta_{a}\in\{0.00001,0.00005,0.0001,0.0002\}$ . All panels use the same synthetic regression setup and training configuration, differing only in the knot learning rate. Smaller values of $\eta_{a}$ lead to limited knot movement, whereas larger values produce more pronounced relocation during training. Details of BS-Grad-U and the appendix setup are provided in Appendix K.1 .

Tabular Representation Learning

Tabular representation learning builds useful embeddings of structured rows, columns, and tables for prediction, retrieval, clustering, and generation.

See all 53 papers

Aug 10, 2026

Tabular Numeric Stretch Transformation

Tabular data presents unique challenges for deep learning due to its heterogeneous nature, where numeric features exhibit diverse distributions, scales, and statistical properties. Although recent advances have improved how models learn from tabular data, how numeric data are transformed into model-friendly representations remains comparatively underexplored. We introduce the stretch transformation framework, which formulates numeric feature preprocessing as an optimization problem to make the target function smoother and thus more learnable. Our framework has two variants: (1) unsupervised stretch, which uniformly redistributes feature density via minimax optimization, and (2) supervised stretch, which optimizes target-aware numeric feature transformations from the perspective of target-function smoothness by minimizing the target function's Dirichlet energy in the transformed space. Our theoretical analysis further connects this framework to several popular transformations: unsupervised stretch is closely related to Piecewise Linear Encoding through a shared piecewise-linear geometry and approaches the empirical CDF transformation as the number of bins grows, while supervised stretch becomes closely related to target encoding in the fine-binning limit. Comprehensive experiments on 38 datasets from the TALENT benchmark demonstrate that supervised stretch consistently outperforms all baselines. These results show that explicitly optimizing for target function smoothness is a powerful and underexplored strategy for tabular deep learning.

Zihao Ye, Juyong Kim, Johnna Sundberg +2

Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213, USA

Tabular ML Automated Feature Engineering Tabular Prediction

Ask PDF

Oct 9, 2026

Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping

While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight how differences in optimal stopping times across uncertainty regions can limit the effectiveness of global early stopping. The code is available at https://github.com/yandex-research/tabular-regression.

Danil Rudenko, Georgii Bulgakov, Nikolay Kartashev +3

Yandex Research · BRAInLab · HSE University

Tabular Regression Quantile Regression Heteroscedastic Regression Robust Loss Functions Early Stopping

Ask PDF

Jun 8, 2026

TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders

Tabular encoders are usually evaluated inside task-specific end-to-end pipelines, so models from different training paradigms are difficult to compare directly even when they operate on similar tabular signals. We introduce TRL-Bench, a multi-granular tabular representation learning (TRL) benchmark that standardizes cross-paradigm representation-level evaluation: each encoder exports row-, column-, or table embeddings through its supported wrapper, and shared lightweight heads probe them across three suites: TRL-CTbench (column/table), TRL-Rbench (row), and TRL-DLTE (compositional Data-Lake Table Enrichment spanning all three granularities). To support this standardized setting, we release curated benchmark assets and task reformulations, including 50 OpenML tables with 123 verified targets, 16 row-pair linkage rewrites, and a 47,772-table DLTE lake derived from 1,379 parent tables. Across 20 models and 16 tasks, TRL-Bench shows that once downstream conditions are standardized, encoder quality is capability-specific rather than captured by a single leaderboard. In TRL-CTbench, generic text encoders often lead on tasks with strong surface-text signal, while tabular specialists win where their pretraining objective aligns with the task. In TRL-Rbench, within-table prediction and cross-table linkage favor different training regimes, with atomic linkage performance correlating strongly with the row-matching stage of DLTE pipelines. In TRL-DLTE, the strongest pipelines combine capability-matched specialists rather than reuse a single encoder, and top end-to-end quality depends on non-additive compositional fit rather than per-stage marginal rank alone. TRL-Bench provides a common protocol for measuring reusable signal in exported tabular representations under shared downstream conditions. Code and data: https://github.com/LOGO-CUHKSZ/TRL-Bench

Wei Pang, Xiangru Jian, Hehan Li +10

The Chinese University of Hong Kong, Shenzhen · University of Waterloo · The University of Hong Kong +2

Representation Learning Tabular Representation Learning Tabular Prediction

Ask PDF

Abstract

Tabular Representation Learning

Explore similar work

Tabular Numeric Stretch Transformation

Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping

TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders