Revisiting Chebyshev Polynomial and Anisotropic RBF Models for Tabular Regression

Luciano Gerber; Huw Lloyd

cs.LGFeb 25, 2026

Revisiting Chebyshev Polynomial and Anisotropic RBF Models for Tabular Regression

Authors: Luciano Gerber, Huw Lloyd

Organizations: Department of Computing and Mathematics, Manchester Metropolitan University, Dalton Building, Chester Street, Manchester M1 5GD, United Kingdom

Abstract

Smooth-basis models such as Chebyshev polynomial regressors and radial basis function (RBF) networks are well established in numerical analysis. Their continuously differentiable prediction surfaces suit surrogate optimisation, sensitivity analysis, and other settings where the response varies gradually with inputs. Despite these properties, smooth models seldom appear in tabular regression, where tree ensembles dominate. We ask whether they can compete, benchmarking models across 55 regression datasets organised by application domain. We develop an anisotropic RBF network with data-driven centre placement and gradient-based width optimisation, a ridge-regularised Chebyshev polynomial regressor, and a smooth-tree hybrid (Chebyshev model tree); all three are released as scikit-learn-compatible packages. We benchmark these against tree ensembles, a pre-trained transformer, and standard baselines, evaluating accuracy alongside generalisation behaviour. The transformer ranks first on accuracy across a majority of datasets, but its GPU dependence, inference latency, and dataset-size limits constrain deployment in the CPU-based settings common across applied science and industry. Among CPU-viable models, smooth models and tree ensembles are statistically tied on accuracy, but the former tend to exhibit tighter generalisation gaps. We recommend routinely including smooth-basis models in the candidate pool, particularly when downstream use benefits from tighter generalisation and gradually varying predictions.

Figures & tables

Panel of Figure 1 — Figure 1: Mean rank for predictive accuracy ( $\bar{R}^{2}$ ); parentheses show rank-1 wins / rank-2 places. Bars show the interquartile range of per-dataset ranks; CD bars indicate the Nemenyi critical difference ( section 3.6 ): models whose mean ranks differ by less than CD are statistically indistinguishable. Full numerical results in tables 12 and 11 (appendix).

Figure 2: Distribution of

\bar{R}^{2}

across 55 datasets per model (ordered by mean rank). Notched boxes show median and IQR; dots are individual datasets. The competitive cluster has largely overlapping distributions, with the main separation occurring in the lower tail.

	S1 (13)		S2 (10)		S3 (16)		S4 (16)
Model	Rank	$\bar{R}^{2}$	Rank	$\bar{R}^{2}$	Rank	$\bar{R}^{2}$	Rank	$\bar{R}^{2}$
tabpfn	1.38	0.886	3.60	0.502	1.25	0.715	1.25	0.851
erbf	3.38	0.860	5.10	0.450	3.81	0.651	4.19	0.808
chebytree	4.00	0.845	4.40	0.456	4.69	0.625	4.00	0.812
xgb	4.23	0.820	5.40	0.459	4.94	0.645	3.81	0.821
chebypoly	4.46	0.774	4.50	0.457	4.44	0.627	5.81	0.788

Table 1: Per-stratum mean rank and

\bar{R}^{2}

(all nine models). Model order as in Figure 1(a) . Within each stratum, the best CPU-viable model per column is in bold and the second underlined ; rank and

\bar{R}^{2}

are emphasised independently. TabPFN is shown for reference but excluded from the emphasis.

Model	Continuous (42)	Non-continuous (13)	All (55)
erbf	2.88	4.15	3.18
chebytree	3.50	3.08	3.40
xgb	3.36	4.23	3.56
chebypoly	4.19	3.23	3.96
ebm	4.29	3.77	4.16
rf	4.17	4.54	4.25

Table 2: Mean rank by target type (CPU-viable models, excluding tabpfn ). Non-continuous targets are intrinsically discrete (ordinal ratings, integer counts, or quantised low-resolution measurements; see text for the criterion). Best in bold per column among competitive models, second underlined .

Figure 3: Mean rank for generalisation gap (

R^{2}_{\mathrm{train}}-R^{2}_{\mathrm{test}}

); parentheses show rank-1 wins / rank-2 places; bars show the interquartile range. Lower rank = smaller gap. Smooth models ( chebypoly , erbf ), ebm , and the hybrid chebytree cluster at the top; xgb ranks last among competitive models. ridge ’s top position reflects underfitting.

Panel of Figure 4 — Figure 4: Pareto trade-off diagrams. Each point represents one model at its mean $\bar{R}^{2}$ plotted against (a) mean generalisation gap and (b) mean total time per dataset including tuning (log scale). Step lines trace the Pareto front; arrows on axes indicate the direction of improvement.

Model	Mean gap	Median CoV
ridge	0.009	0.021
chebypoly	0.025	0.034
chebytree	0.030	0.062
ebm	0.031	0.060
erbf	0.033	0.065
dt	0.044	0.095

Table 3: Generalisation gap and median prediction CoV per model, each averaged across the 29 datasets. Models ordered by gap magnitude. Lower values indicate tighter generalisation and more stable predictions, respectively.

Dataset	S	$n$	$d$	xgb	chebytree	chebypoly	ridge	dt
medical_charges	S4	163K	3	0.978	0.980	0.979	0.825	0.978
diamonds	S4	54K	6	0.946	0.945	0.943	0.926	0.942
superconduct	S3	21K	79	0.910	0.900	0.837	0.733	0.825
friedman1_d100	S1	2K	100	0.926	0.878	0.905	0.710	0.676
geo_music	S2	1K	117	0.647	0.670	0.758	0.756	0.473
qsar_tid_11	S3	6K	1024	0.611	0.634	0.523	0.538	0.265

Table 4: Scalability check:

\bar{R}^{2}

per dataset for scalable models on full-scale data (no feature selection or subsampling). Datasets ordered by mean

\bar{R}^{2}

across models. Best per dataset in bold .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Parameter	Range / choices	Scale
Ridge (20 trials)	alpha	$[10^{-3},\;10^{3}]$	Log
Decision Tree (25 trials)	max_depth	$[1,\;20]$	Int
	min_samples_leaf	$[0.005,\;0.1]$	Uniform
	min_samples_split	$[0.01,\;0.1]$	Uniform
Random Forest (25 trials)	n_estimators	$[50,\;500]$	Int
	max_depth	$[3,\;20]$	Int

Appendix

Table 5: Hyperparameter search spaces and fixed settings for each model. “Log” indicates log-uniform sampling; “Cat” indicates categorical. Trial budgets are given in parentheses after each model name. Greyed rows show fixed (non-tuned) settings.

Package	Version	Role
Model implementations
tabpfn	6.3.1 (v2.5 weights)	TabPFN regressor
xgboost	3.1.3	XGBoost
scikit-learn	1.6.1	Ridge, Decision Tree, Random Forest
erbf	0.1.0	ERBF regressor
poly_basis_ml	0.2.0	ChebyPoly, ChebyTree regressors

Appendix

Table 6: Software versions used in all experiments.

Dataset	$R^{2}$ (mean $\pm$ std)	Gap (mean $\pm$ std)
california_housing	$0.769\pm 0.008$	$0.037\pm 0.003$
concrete_strength	$0.850\pm 0.012$	$0.047\pm 0.010$
esol	$0.910\pm 0.004$	$0.017\pm 0.003$
power_plant	$0.938\pm 0.001$	$0.010\pm 0.001$
superconduct	$0.853\pm 0.005$	$0.023\pm 0.006$

Appendix

Table 7: ERBF convergence stability across five random seeds. Values are mean

\pm

standard deviation over seeds.

Dataset	Stratum	$d$	Main effects	+Interactions
concrete_strength	S1	8	15.5	127.9
power_plant	S1	4	3.7	24.3
airfoil_noise	S1	5	6.7	3.7
energy_efficiency_heating	S1	8	0.9	12.2
friedman1	S1	5	12.6	1.7
cpu_act	S1	10	0.7	0.5

Appendix

Table 8: Ratio of monomial to Chebyshev condition number (linear scale) for 21 datasets at degree 5 with top-10 MI-selected features. Values

{>}1

indicate Chebyshev advantage; values

{<}1

indicate monomial advantage. Datasets ordered by stratum.

Dataset	Lip/Ridge	Lip/Var	KM/Ridge	KM/Var	Range
california_housing	0.768	0.763	0.745	0.751	0.023
concrete_strength	0.849	0.850	0.871	0.849	0.022
esol	0.908	0.896	0.900	0.898	0.012
power_plant	0.938	0.938	0.938	0.940	0.002
superconduct	0.855	0.847	0.833	0.837	0.022

Appendix

Table 9: ERBF mean test

R^{2}

across all four initialisation combinations (2 centre placement strategies

\times

2 width initialisation strategies), averaged over 3 random seeds. Lip = Lipschitz-guided centre placement; KM =

K

-means centre placement; Ridge = local ridge width initialisation; Var = local variance width initialisation. Range = max minus min across the four configurations.

Dataset	$n$	$d$	$d^{\prime}$	Source	Data URL
S1 – Engineering/Simulation (13 datasets)
airfoil_noise	1 503	5		UCI	uci.edu/dataset/291
Ailerons †	13 750	33	20	OpenML	openml.org/d/44137
concrete_strength	1 030	8		UCI	uci.edu/dataset/165
cpu_act †	8 192	21		OpenML	openml.org/d/44132
elevators †	16 599	16		OpenML	openml.org/d/44134

Appendix

Table 10: All 55 benchmark datasets grouped by stratum.

n

= samples,

d

= features,

n^{\prime}

and

d^{\prime}

= effective counts after preprocessing (shown only where they differ from

n

d

). For the eight datasets that undergo per-fold feature selection,

d^{\prime}

is the median effective feature count across the five outer folds, as the selected set varies by fold; for all other datasets

d^{\prime}

reflects only deterministic quality filtering (removal of high-missingness and quasi-constant features). Datasets with non-continuous targets are marked † (13 total; see table 2 ). Data URLs are hyperlinked in the electronic version.

Model	Top-2	Mean Rank	Rank IQR	$\bar{R}^{2}$ median	$\bar{R}^{2}$ mean $\pm$ std
tabpfn	41 / 6	1.71	1–1	0.836	$\mathbf{0.761\pm 0.231}$
erbf	4 / 8	4.05	3–6	0.791	$\underline{0.710\pm 0.245}$
chebytree	2 / 5	4.27	3–5	0.785	$0.700\pm 0.251$
xgb	1 / 14	4.53	2–6	0.771	$0.704\pm 0.245$
chebypoly	1 / 7	4.85	4–6	0.764	$0.678\pm 0.247$
ebm	3 / 5	5.05	4–6	0.748	$0.672\pm 0.242$

Appendix

Table 11: Predictive accuracy–all models (55 datasets, with tabpfn ). Models ordered by mean rank on

\bar{R}^{2}

. Top-2 = rank-1 / rank-2 counts. Best in bold , second underlined .

Model	Top-2	Mean Rank	Rank IQR	$\bar{R}^{2}$ median	$\bar{R}^{2}$ mean $\pm$ std
erbf	12 / 13	3.18	2–5	0.791	$\mathbf{0.710\pm 0.245}$
chebytree	6 / 12	3.40	2–4	0.785	$0.700\pm 0.251$
xgb	15 / 8	3.56	1–5	0.771	$\underline{0.704\pm 0.245}$
chebypoly	6 / 6	3.96	3–5	0.764	$0.678\pm 0.247$
ebm	6 / 7	4.16	3–5	0.748	$0.672\pm 0.242$
rf	8 / 5	4.25	3–6	0.739	$0.691\pm 0.239$

Appendix

Table 12: Predictive accuracy–CPU-viable models (55 datasets, without tabpfn ). Models ranked by mean rank on

\bar{R}^{2}

. Top-2 = rank-1 / rank-2 counts. Best in bold , second underlined .

Model	Top-2	Mean Rank	Rank IQR	Gap median	Gap mean $\pm$ std
ridge	40 / 7	1.82	1–2	0.003	$0.011\pm 0.016$
chebypoly	2 / 16	3.47	2–4	0.014	$\mathbf{0.029\pm 0.040}$
ebm	2 / 6	4.40	3–6	0.020	$\underline{0.035\pm 0.044}$
erbf	2 / 6	4.47	3–6	0.018	$0.039\pm 0.054$
chebytree	2 / 7	4.82	3–6	0.029	$0.040\pm 0.053$
rf	2 / 4	5.80	4–7	0.024	$0.062\pm 0.097$

Appendix

Table 13: Generalisation gap rankings (55 datasets, all nine models). Lower gap = better generalisation. Models ordered by mean gap rank. Top-2 = rank-1 / rank-2 counts. Best in bold , second underlined (among competitive models; ridge ’s low gap reflects underfitting).

Threshold	Pairs	Smooth wins gap (%)
0.005	67	86.6
0.010	108	86.1
0.020	167	86.8
0.050	252	83.7
0.100	307	82.7

Appendix

Table 14: Sensitivity of the matched-accuracy gap analysis to the accuracy-equivalence threshold

|\Delta\bar{R}^{2}|

. “Pairs” is the number of smooth–tree model-dataset pairs meeting the threshold; “Smooth wins gap” is the percentage where the smooth model has the smaller generalisation gap.

Model	Tune (s)	Train (s)	Train (ms/1K)	Predict (ms/1K)
tabpfn	2	43.47	3 000	3 609
ridge	20	0.01	0	0
dt	30	0.26	0	1
chebypoly	40	0.34	0	12
chebytree	61	0.47	0	20
xgb	182	0.99	0	9

Appendix

Table 15: Computational cost (mean across 55 datasets). Tune time includes all Optuna trials for one outer fold. Predict time is per 1 000 instances. Train/1K is train time normalised per 1 000 instances, averaged across datasets; actual scaling is model-dependent. Models ordered by tune time. Fastest CPU-viable competitive model in bold , second underlined (tune time).

Dataset		$n$	$d$	ridge	dt	rf	xgb	ebm	erbf	chebypoly	chebytree
concrete_strength	/trial	1 030	8	0.3	0.2	10.2	1.2	16.7	2.2	0.2	0.8
	train			0.0	0.0	6.5	0.5	5.9	1.0	0.0	0.1
california_housing	/trial	20 640	8	0.3	0.8	92.4	4.9	33.5	25.6	0.7	2.4
	train			0.0	0.4	48.3	1.2	12.3	18.1	0.2	1.1
superconduct	/trial	21 263	79	1.7	4.1	312.6	27.0	175.0	83.2	4.5	7.1
	train			0.1	2.2	217.8	8.8	61.4	51.9	2.6	3.9

Appendix

Table 16: Time per tuning trial and training time (seconds) on three datasets spanning the size range (CPU models only; tabpfn requires no per-trial tuning). Trial counts: ridge 20, dt 25, rf 25, xgb 50, ebm 15, erbf 30, chebypoly 30, chebytree 30.

Model	Type	Parameter	Median (IQR)
ridge	HP	alpha	5.8 (1.2, 18.3)
dt	HP	max_depth	12 (10, 13)
	HP	min_samples_leaf	0.007 (0.007, 0.014)
	HP	min_samples_split	0.019 (0.014, 0.035)
	Fit	actual depth	9 (8, 11)
	Fit	leaves	72 (34, 87)

Appendix

Table 17: Distribution of winning configurations across 55 datasets. Tuned hyperparameters are selected by Optuna; fitted properties are emergent characteristics of the resulting model. Numeric values are median (IQR) of per-dataset fold-averaged best parameters. Categorical values show the majority choice with its frequency. Search spaces are given in table 5 .

Dataset	tabpfn	erbf	chebytree	xgb	ebm	chebypoly	rf	dt	ridge
S1: Engineering/Simulation
Ailerons	0.793	0.760	0.779	0.771	0.765	0.781	0.756	0.719	0.756
airfoil_noise	0.977	0.921	0.831	0.923	0.557	0.728	0.828	0.725	0.503
concrete_strength	0.945	0.885	0.865	0.917	0.907	0.892	0.880	0.809	0.597
cpu_act	0.988	0.983	0.982	0.982	0.981	0.983	0.976	0.968	0.736
elevators	0.929	0.862	0.886	0.873	0.873	0.890	0.689	0.594	0.812

Appendix

Table 18: Per-dataset adjusted

R^{2}

for all nine models, grouped by stratum. Overall best per dataset in bold ; best CPU-viable model underlined (shown only when different from overall best). † Ordinal (discrete integer) target.

Dataset	erbf	chebytree	xgb	ebm	chebypoly	rf	dt	ridge
S1: Engineering/Simulation
Ailerons	5	2	3	4	1	7	8	6
airfoil_noise	2	3	1	7	5	4	6	8
concrete_strength	4	6	1	2	3	5	7	8
cpu_act	2	3	4	5	1	6	7	8
elevators	5	2	3	4	1	7	8	6

Appendix

Table 19: Per-dataset rank of the eight CPU-viable models by adjusted

R^{2}

(rank 1 = best on that dataset; ties share the lower rank). Best per dataset in bold . † Ordinal (discrete integer) target. TabPFN is excluded (analysed separately). Mean ranks per model are reported in the main text.

Dataset	Unique $y$	$n$	ERBF	XGB	Best other	Model
pmlb_1028_SWD	4	1 000	0.413	0.394	0.422	ebm
pmlb_1029_LEV	5	1 000	0.550	0.504	0.554	chebypoly
wine_quality	7	6 497	0.388	0.440	0.393	rf
pmlb_1030_ERA	9	1 000	0.347	0.328	0.368	chebytree
analcatdata_supreme	10	4 052	0.977	0.978	0.978	chebypoly
pol	11	15 000	0.169	0.161	0.168	rf

Appendix

Table 20: Adjusted

R^{2}

on the nine datasets with discrete or ordinal targets, ordered by number of unique target values. ERBF and XGBoost are shown alongside the best-performing model among the remaining six (named in the rightmost column). All results use Optuna-tuned hyperparameters from the main benchmark. Bike_Sharing_Demand is an integer count with many distinct values (869); it is the highest-cardinality of the thirteen non-continuous datasets ( table 2 ) and is included here as a contrast to the low-resolution ordinal and count targets.

Levels	ERBF $R^{2}$	XGB $R^{2}$	DT $R^{2}$
3	0.845	0.849	0.779
5	0.929	0.915	0.850
7	0.954	0.940	0.879
10	0.971	0.959	0.907
15	0.980	0.970	0.921
21	0.984	0.975	0.932

Appendix

Table 21: Synthetic experiment: effect of target rounding on model accuracy. A smooth function is discretised to varying numbers of equally spaced levels. Default hyperparameters; 5-fold CV.

Tabular Regression

Tabular regression predicts continuous outcomes from structured rows of features for tasks such as pricing, forecasting, and scientific measurement.

See all 35 papers

Apr 5, 2026

Spectral Path Regression: Directional Chebyshev Harmonics for Interpretable Tabular Learning

Classical approximation bases such as Chebyshev polynomials provide principled and interpretable representations, but their multivariate tensor-product constructions scale exponentially with dimension and impose axis-aligned structure that is poorly matched to real tabular data. We address this by replacing tensorised oscillations with directional harmonic modes of the form

\cos(\mathbf{m}^{\top}\arccos(\mathbf{x}))

, which organise multivariate structure by direction in angular space rather than by coordinate index. This representation yields a discrete spectral regression model in which complexity is controlled by selecting a small number of structured frequency vectors (spectral paths), and training reduces to a single closed-form ridge solve with no iterative optimisation. Experiments on standard continuous-feature tabular regression benchmarks show that the resulting models achieve accuracy competitive with strong nonlinear baselines while remaining compact, computationally efficient, and explicitly interpretable through analytic expressions of learned feature interactions.

Milo Coombs

Tabular Regression Spectral Methods Frequency-Domain Feature Learning

Ask PDF

Aug 18, 2026

TabNSM: Neural Sparse Mixer for Tabular Regression

Large-scale, high-dimensional tabular regression remains challenging: tree-based models are robust but lack end-to-end representation learning, while deep models enable flexible feature learning but often incur costly interaction modeling and sensitivity to noisy or redundant features. We propose TabNSM, a scalable regression framework that extends our earlier sparse-attention and mixer architectures. At its core, the Adaptive Sparse Interaction Module (ASIM) integrates foreground feature discovery, sparse local interaction encoding, and Feature-Token Mixing, providing near-linear complexity under fixed sparse configurations. For regression, TabNSM introduces three complementary components: a Multi-Stage Regression Head for progressive prediction refinement; GridLoss, an ordinal-aware soft-binning objective that incorporates target structure into representation learning; and RISE (Reweighted Instance Sampling by Error), a difficulty-aware sampling strategy based on loss-quantile bins. Across nine real-world regression benchmarks, TabNSM delivers strong predictive performance and practical scalability, with particularly consistent gains on high-dimensional and heterogeneous datasets. These results demonstrate that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.

Ali Eslamian, Qiang Cheng

Department of Computer Science University of Kentucky Lexington, KY 40506 · Department of Computer Science Institute for Biomedical Informatics University of Kentucky Lexington, KY 40506

Tabular Regression Tabular Representation Learning Feature Interaction Modeling Sparse Neural Networks

Ask PDF

Oct 9, 2026

TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy

Tabular foundation models (TFMs) are increasingly popular for their strong predictive performance. Many also provide posterior predictive distributions (PPDs), whose reliability remains less studied. We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency. We then introduce TabReliBench, a large classification and regression benchmark for these comparisons and dataset-level analyses. We find that TFMs' rankings change between TabArena and our benchmark and across reliability metrics. Regression TFMs usually achieve near-nominal marginal coverage and retain narrow intervals, but this favorable marginal behavior does not imply valid conditional coverage. Dataset-level analyses show that classification difficulty often accompanies poor calibration, whereas regression predictive performance is not a reliable indicator of calibration. Ensembling improves predictive performance but does not consistently improve calibration of PPDs; TFMs also benefit little from conformal post-processing, which does not reliably resolve subgroup gaps. Reliable TFM selection therefore requires multidimensional evaluation beyond aggregate rankings. Code is available at https://github.com/Tianqi-Zhao/TabReliBench.

Tianqi Zhao, Jia Guo, Yan Shuo Tan +1

Renmin University of China · Cerebras Systems, Inc. · National University of Singapore +1

Tabular Foundation Models Probability Calibration Regression Prediction Intervals Multicalibration

Ask PDF

Abstract

Tabular Regression

Explore similar work

Spectral Path Regression: Directional Chebyshev Harmonics for Interpretable Tabular Learning

TabNSM: Neural Sparse Mixer for Tabular Regression

TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy