SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data

Bastian Schäfer; Dominika Matus; Lennart Purucker; Ricardo Knauer; Erik Rodner; Frank Hutter; Anton Sokolov; Arnaud Liefooghe

cs.LGOct 9, 2026

SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data

Authors: Bastian Schäfer, Dominika Matus, Lennart Purucker, Ricardo Knauer, Erik Rodner, Frank Hutter, Anton Sokolov, Arnaud Liefooghe

Organizations: Université du Littoral-Côte-d’Opale · University of Freiburg · Prior Labs · HTW Berlin ·

Abstract

Selecting a subset of features (columns) from a tabular dataset is a ubiquitous challenge in real-world applications. Industry experts seek to minimize model training costs, climate scientists require an interpretable subset of factors to understand extreme weather events, and clinicians need to choose among diagnostic tests due to financial and time constraints. When looking at the scientific literature to determine which feature selection method to use, practitioners across domains face an unrepresentative landscape of benchmarking. Despite extensive research, existing feature selection benchmarks provide limited guidance because they rely on outdated models, ad hoc evaluation protocols, or artificially generated data that do not reflect real-world problems. To close the gap between scientific benchmarking and real-world applications, we introduce SelectArena, a modern benchmark for feature selection. We compare 15 widely used feature selection methods on 71 manually curated real-world tabular datasets, covering classification and regression problems, using standardized evaluation protocols and state-of-the-art tabular (foundation) models. Our results reveal clear trade-offs across predictive performance, runtime, stability, and validity. Tree-based methods, CART and RFImportance, along with F-Test are Pareto-optimal when taking into account predictive performance and runtime. F-Test, information-theory-, and distance-based methods lead on stability and validity, with F-Test emerging as a balanced choice across our metrics. Search-based methods are slow, unstable and achieve low performance. SelectArena turns these findings into practical guidance and, as an extensible platform, supports future feature selection research.

Figures & tables

Figure 1: SelectArena at a glance. We present Elo ratings of the $15$ feature selection methods ordered left to right from worst- to best-performing method. Elo rating summarizes how often a method beats another on the same dataset, budget, and downstream model, with random selection anchored at $1000$ (Appendix D.2 ). Error bars are $95\%$ confidence intervals from $1000$ cluster bootstrap resamples over the $71$ datasets. The left bar of each pair is shaded by stability (how much the selected subset changes across resamples) and the right bar by validity (how well injected noise features are rejected), where darker is better for both.

Benchmark	# Meth- ods	# Data- sets	Domain	SotA models	Perf	Stab	Val
Kohavi and John (1997)	3 /4	5 /14
Jain and Zongker (1997)	8 /15	0 /1
Reunanen (2003)	2 /2	2 /7
Guyon et al. (2007a)	1 /3	0 /5
Saeys et al. (2008)	4 /4	6 /6
Brown et al. (2012)	7 /9	4 /15

Table 1: Comparison of feature selection benchmarks. We present former benchmarks in comparison to SelectArena. # Methods : how many out of the total number of benchmark methods correspond to our first three curation criteria (not a variant of a method and applicable across tabular classification and regression tasks). # Datasets : how many out of the total number of benchmark datasets satisfy our curation criteria for high-quality tabular data (Section 3.2 ). Domain : visualizes the dataset domain: (synthetic), (environmental), (biomedical), (technological), (scientific), (financial) or (not applicable). SotA models : whether the benchmark includes state-of-the-art downstream models, such as tuned gradient-boosted trees or pretrained tabular foundation models ( Erickson et al., 2025 ) , with the options (yes) or (no). The final three columns ( Perf , Stab , Val ) indicate whether the benchmark reports an evaluation of the predictive performance, stability, or validity.

Name	Deter-	Model	Subgroup	Cita-	Implemen-
	minism	coupling		titons	tation
CART ( Breiman et al., 1984 )			Tree	4	scikit-learn
ElasticNet ( Zou and Hastie, 2005 )			Regularization	3	scikit-learn
F-test ( st1989anova )			Correlation	6	scikit-learn
GainRatio ( Quinlan, 1986 )			Info theory	3	Us
LaplacianScore ( He et al., 2005 )			Distance	5	Us

Table 2: Benchmarked feature selection methods. We report all included methods in SelectArena. Determinism: whether a method yields the same subset for the same data across repeated evaluations (also referred to as consistency in Dembinsky et al. (2026) ). The options are (yes), (no), or (model-dependent). Model coupling : whether feature selection can be performed in conjunction with the downstream model

M

(related to portability in Dembinsky et al. (2026) ). Subgroup : we specify which subgroup a method belongs to (Appendix A ). Citations : reflects a method’s popularity and impact within the relevant feature selection literature, measured as the frequency of occurrence across relevant survey and benchmark works (Section 3.2 ). For implementation details, please refer to Appendix B.4 .

Figure 2: Dataset curation process. The reason and number of datasets excluded in each step of our dataset curation pipeline are shown, along with the remaining number of datasets after each step.

Panel of Figure 3 — Figure 3: Method and dataset characteristics. SelectArena represents a diverse collection of widely adopted feature selection methods as well as high-quality real-world datasets.

Panel of Figure 4 — Figure 4: Predictive performance. Both panels report improvement over Random (%, higher is better), averaged over three seeds, five cardinalities, and 71 datasets. ( Left ): Improvement averaged over downstream models against feature selection time (s) per 1,000 features with Pareto front marked in orange. We exclude LaplacianScore as it performs worse than Random. ( Right ): Improvement for every model–method interaction.

Figure 5: Stability. Nogueira score ( Nogueira and Brown, 2016 ) across datasets. We use the events per variable, i.e. , the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty ( Harrell, 2015 ; Heinze et al., 2018 ) .

Figure 6: Validity. Selection precision across datasets at a noise level of 50%. Other noise levels can be found in Appendix D . Like for the stability, we use the events per variable, i.e. , the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty ( Harrell, 2015 ; Heinze et al., 2018 ) .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Taxonomy of feature selection methods for SelectArena. We complement the traditional structural perspective ( Guyon and Elisseeff, 2003 ) with a functional categorization into search and non-search approaches ( Kohavi and John, 1997 ) . Search subgroups include forward and backward procedures, non-search subgroups are based on Duch (2006) and extended to include regularization-based methods ( Tibshirani, 1996 ; Zou and Hastie, 2005 ) .

Jain and Zongker (1997)	Bolón-Canedo et al. (2013)	Li et al. (2018)
Kohavi and John (1997)	Wang et al. (2013)	Bommert et al. (2020)
Guyon and Elisseeff (2003)	Chandrashekar and Sahin (2014)	Kou et al. (2020)
Reunanen (2003)	Khalid et al. (2014)	Covert et al. (2021)
Duch (2006)	Tang et al. (2014)	Hopf and Reifenrath (2021)
Guyon et al. (2007a)	Jović et al. (2015)	Bouchlaghem et al. (2022)
Saeys et al. (2007)	Xue et al. (2015)	Li et al. (2022)

Appendix

Table B.1: Reviewed survey and benchmark works. The 33 works used to determine the popularity of each feature selection method, ordered by year of publication.

Feature Selection Method Name	Year	# appearances	Decision
AACO	2024	1	Citations
ACO 1	2004	1	Citations
ACO 2	2010	1	Citations
ACO 3	2012	1	Citations
ACO 4	2014	1	Citations
AISCBR	2011	1	Citations

Appendix

Table B.2: Overview of all investigated feature selection methods. We list all methods with their source links, provide the year, and the number of appearances across the relevant literature (Section 3.2 ). In the Decision column, we use Yes if the method is included in SelectArena or list the reason for exclusion ( Citations if there are fewer than three citations, Variation if the method is a variation of another method, Wrong Domain if the method is not suitable for tabular data, and Not task-agnostic if the method is not suitable for all tasks). We additionally mark some methods with Yes, future work to indicate that their implementations are ongoing. In all cases, these implementations are missing differential entropy logic.

Agarwal et al. (2022)	Kira and Rendell (1992)	Robnik-Šikonja and Kononenko (2003)
Bakhmach et al. (2026)	Kohavi and John (1997)	Ruiz-Ramírez et al. (2023)
Battiti (1994)	Koller and Sahami (1996)	Saeys et al. (2008)
Belaid et al. (2023)	Kononenko et al. (1997)	Schütz et al. (2025)
Bennasar et al. (2015)	Kou et al. (2020)	Sithakoul et al. (2024)
Bolón-Canedo et al. (2013)	Lee and Lee (2006)	Smyth (2004)
Bommert et al. (2020)	Li et al. (2012)	Stiglic et al. (2012)

Appendix

Table C.1: Reviewed dataset sources. The 73 articles screened for real-world tabular datasets, in alphabetical order.

Dataset	# Features	# Samples	# Classes	Task	License
				type
allstate-claims-severity ( Ferguson et al., 2016 )	130	188318	/	Regression	Public
ancestry-study ( Ruiz-Ramírez et al., 2023 )	104	635	10	Multiclass	Public
anneal ( Unknown, 1990 )	38	898	5	Multiclass	CC-BY-4
aps-failure ( IDA2016Challenge, 2016 )	169	76000	2	Binary	CC-BY-4
audiology-diagnosis ( Bareiss et al., 1990 )	68	199	3	Multiclass	/

Appendix

Table C.2: Overview of selected datasets. We list all datasets included in SelectArena, the number of features, samples and classes, as well as the task type and license for the usage of the datasets.

Cardinality idx $j$	Min $k$	Max $k$	Linear ( $p\leq 100$ ) min–max $k$	Exp. ( $p>100$ ) min–max $k$	Mean $k$
1	2	16	2–16	3–6	4.7
2	4	31	4–31	5–29	11.1
3	5	150	5–47	11–150	27.4
4	7	791	7–62	23–791	87.3
5	9	4190	9–78	48–4190	344.5

Appendix

Table C.3: Number of selected features

k

across the 71 benchmark datasets, per cardinality index

j

Method	Elo rating	95% CI
RFImportance	1305.8	[1259.6, 1356.2]
CART	1262.4	[1214.3, 1314.2]
ElasticNet	1249.4	[1203.0, 1301.6]
MI	1245.8	[1207.2, 1290.1]
F-Test	1226.7	[1184.1, 1272.2]
Lasso	1224.1	[1180.0, 1274.8]

Appendix

Table D.1: Elo rating. We report the score per method along with 95% confidence intervals from 1000 cluster bootstrap resamples at the dataset level. Higher is better.

Name	Improvement over random (%)	Time per 1k features (s)
RFImportance	26.23	45.19
MI	23.80	51.24
CART	23.78	15.71
F-Test	22.76	0.79
ElasticNet	21.88	2400.69
mRMR	21.67	56.50

Appendix

Table D.2: Predictive performance vs. runtime. We report the mean improvement over random and mean runtime per 1k features for all benchmarked feature selection methods.

Panel of Figure 17 — Figure D.1: Critical difference diagrams. Average ranks per dataset with critical distance $=2.546$ .

Figure D.2: Impact of tuning. Normalized scores per method per model, averaged over datasets and budgets. Tuned LGBM (striped blue) outperforms the default (solid blue) by a clear margin.

	Timeout		Fallback		Improvement over Random
Method	$n$	%	$n$	%	Fallback	No fallback	$\Delta$ (pp)
SBE	832	46.9	825	46.5	10.01	28.06	$+18.05$
SFS	795	44.8	778	43.8	16.96	$-1.99$	$-18.95$
LOCO	417	23.5	134	7.5	11.58	10.11	$-1.47$
(R)ReliefF	301	17.0	0	0.0	19.23	19.22	$-0.01$
Lasso	71	4.0	0	0.0	20.95	20.95	$+0.00$

Appendix

Table D.3: Impact of random fallback on run completion and improvement over Random . A run is a unique combination of dataset, budget, and downstream model; each method has 1,775 runs. Timeout counts runs exceeding the one-hour limit. Fallback counts runs completed or reduced through random selection after a timeout. Improvement over Random ( Apppendix D.1 ) normalizes performance across tasks with different error metrics. We report the mean for the default pipeline ( Fallback ) and the ablated one ( No fallback ), with

\Delta=\text{No fallback}-\text{Fallback}

Figure D.3: Impact of random fallback on Elo ranking. New Elo ratings are calculated based on the ablation study’s results for a single seed and are presented in orange. SBE is shown in striped red to indicate that excluding random fallback has a different effect on it than other methods. Default Elo ratings are presented in blue. Numbers on the bars represent rank.

Figure D.4: Stability. Nogueira score ( Nogueira and Brown, 2016 ) across datasets. We use the events per variable, i.e. , the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty ( Harrell, 2015 ; Heinze et al., 2018 ) . The top-3 methods per EPV are highlighted.

Figure D.5: Thresholded stability. We display all methods within a 3% performance threshold of the best-performing method.

Method	Stability	Validity (50%)	Validity (75%)	Validity (100%)
F-Test	0.679027	0.922444	0.885531	0.848069
LaplacianScore	0.656319	0.875447	0.837405	0.797492
MI	0.653849	0.879216	0.831729	0.785338
GainRatio	0.648343	0.909595	0.867829	0.826548
(R)ReliefF	0.602695	0.836929	0.775536	0.740539
mRMR	0.586207	0.749956	0.698682	0.654502

Appendix

Table D.4: Mean validity and stability. We report the average scores across all datasets. Higher is better.

Figure D.6: Validity at 50% added noise. Selection precision across datasets at a noise level of 50%. Like for the stability, we use the events per variable, i.e. , the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty ( Harrell, 2015 ; Heinze et al., 2018 ) . The top-3 methods per EPV are highlighted.

Figure D.7: Thresholded validity at 50% added noise. We display all methods within a 3% performance threshold of the best-performing method at a noise level of 50%.

Figure D.8: Validity at 75% added noise. Selection precision across datasets at a noise level of 75%. Like for the stability, we use the events per variable, i.e. , the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty ( Harrell, 2015 ; Heinze et al., 2018 ) . The top-3 methods per EPV are highlighted.

Figure D.9: Thresholded validity at 75% added noise. We display all methods within a 3% performance threshold of the best-performing method at a noise level of 75%.

Figure D.10: Validity at 100% added noise. Selection precision across datasets at a noise level of 100%. Like for the stability, we use the events per variable, i.e. , the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty ( Harrell, 2015 ; Heinze et al., 2018 ) . The top-3 methods per EPV are highlighted.

Figure D.11: Thresholded validity at 100% added noise. We display all methods within a 3% performance threshold of the best-performing method at a noise level of 100%.

Dataset	Budget idx $j$	CART	ElasticNet	F-test	GainRatio	LaplacianScore	Lasso	LOCO	MarkovBlanket	MI	mRMR	(R)ReliefF	Random	SBE	RFImportance	SFS
log-loss ( $\downarrow$ )
horse_colic survival	1	.778 $\pm$ .020	.762 $\pm$ .037	.759 $\pm$ .028	.834 $\pm$ .065	.879 $\pm$ .027	.762 $\pm$ .037	.784 $\pm$ .075	.779 $\pm$ .031	.823 $\pm$ .066	.807 $\pm$ .021	.809 $\pm$ .042	.863 $\pm$ .073	.821 $\pm$ .081	.760 $\pm$ .037	.894 $\pm$ .012
	2	.753 $\pm$ .046	.739 $\pm$ .043	.748 $\pm$ .031	.805 $\pm$ .034	.889 $\pm$ .029	.738 $\pm$ .036	.763 $\pm$ .051	.761 $\pm$ .042	.796 $\pm$ .063	.790 $\pm$ .026	.767 $\pm$ .045	.802 $\pm$ .042	.774 $\pm$ .043	.739 $\pm$ .059	.887 $\pm$ .011
	3	.746 $\pm$ .037	.742 $\pm$ .033	.728 $\pm$ .049	.773 $\pm$ .013	.886 $\pm$ .022	.737 $\pm$ .038	.760 $\pm$ .054	.743 $\pm$ .043	.777 $\pm$ .038	.764 $\pm$ .015	.749 $\pm$ .035	.776 $\pm$ .083	.748 $\pm$ .049	.727 $\pm$ .057	.886 $\pm$ .016
	4	.730 $\pm$ .046	.719 $\pm$ .043	.723 $\pm$ .045	.731 $\pm$ .048	.793 $\pm$ .008	.720 $\pm$ .046	.757 $\pm$ .051	.730 $\pm$ .054	.746 $\pm$ .036	.732 $\pm$ .035	.743 $\pm$ .037	.759 $\pm$ .048	.737 $\pm$ .050	.718 $\pm$ .056	.852 $\pm$ .016
	5	.725 $\pm$ .042	.725 $\pm$ .035	.719 $\pm$ .043	.725 $\pm$ .042	.752 $\pm$ .021	.722 $\pm$ .039	.759 $\pm$ .044	.731 $\pm$ .041	.743 $\pm$ .034	.716 $\pm$ .050	.735 $\pm$ .038	.751 $\pm$ .039	.737 $\pm$ .050	.725 $\pm$ .047	.770 $\pm$ .030

Appendix

Table D.5: Per-cell results with downstream model LGBM . Tasks are grouped by metric. Each cell is the mean over three seeds, with the sample standard deviation as a subscript.

Dataset	Budget idx $j$	CART	ElasticNet	F-test	GainRatio	LaplacianScore	Lasso	LOCO	MarkovBlanket	MI	mRMR	(R)ReliefF	Random	SBE	RFImportance	SFS
log-loss ( $\downarrow$ )
horse_colic survival	1	.767 $\pm$ .068	.737 $\pm$ .075	.741 $\pm$ .061	.804 $\pm$ .050	.836 $\pm$ .053	.737 $\pm$ .075	.796 $\pm$ .070	.770 $\pm$ .030	.762 $\pm$ .088	.786 $\pm$ .061	.791 $\pm$ .050	.838 $\pm$ .068	.826 $\pm$ .119	.765 $\pm$ .067	.865 $\pm$ .040
	2	.763 $\pm$ .070	.791 $\pm$ .093	.766 $\pm$ .049	.769 $\pm$ .044	.861 $\pm$ .106	.769 $\pm$ .066	.790 $\pm$ .068	.731 $\pm$ .054	.803 $\pm$ .100	.763 $\pm$ .040	.774 $\pm$ .047	.841 $\pm$ .112	.808 $\pm$ .105	.720 $\pm$ .050	.865 $\pm$ .085
	3	.786 $\pm$ .059	.814 $\pm$ .085	.796 $\pm$ .016	.826 $\pm$ .048	.874 $\pm$ .080	.797 $\pm$ .085	.811 $\pm$ .121	.783 $\pm$ .045	.817 $\pm$ .108	.778 $\pm$ .041	.827 $\pm$ .030	.771 $\pm$ .047	.795 $\pm$ .023	.787 $\pm$ .018	.901 $\pm$ .085
	4	.868 $\pm$ .080	.878 $\pm$ .078	.870 $\pm$ .032	.857 $\pm$ .053	.812 $\pm$ .053	.878 $\pm$ .065	.889 $\pm$ .184	.843 $\pm$ .066	.840 $\pm$ .018	.848 $\pm$ .024	.867 $\pm$ .013	.889 $\pm$ .037	.803 $\pm$ .017	.859 $\pm$ .014	.949 $\pm$ .109
	5	.899 $\pm$ .033	.904 $\pm$ .048	.920 $\pm$ .034	.876 $\pm$ .018	.855 $\pm$ .042	.908 $\pm$ .069	.929 $\pm$ .142	.882 $\pm$ .017	.896 $\pm$ .047	.871 $\pm$ .041	.906 $\pm$ .037	.893 $\pm$ .107	.816 $\pm$ .015	.904 $\pm$ .019	.970 $\pm$ .100

Appendix

Table D.6: Per-cell results with downstream model Linear . Tasks are grouped by metric. Each cell is the mean over three seeds, with the sample standard deviation as a subscript.

Dataset	Budget idx $j$	CART	ElasticNet	F-test	GainRatio	LaplacianScore	Lasso	LOCO	MarkovBlanket	MI	mRMR	(R)ReliefF	Random	SBE	RFImportance	SFS
log-loss ( $\downarrow$ )
horse_colic survival	1	1.02 $\pm$ .11	.939 $\pm$ .184	.927 $\pm$ .080	.982 $\pm$ .177	1.12 $\pm$ .27	.945 $\pm$ .177	.961 $\pm$ .101	.862 $\pm$ .049	.915 $\pm$ .123	1.02 $\pm$ .11	1.06 $\pm$ .27	1.32 $\pm$ .60	1.17 $\pm$ .29	.913 $\pm$ .021	1.44 $\pm$ .16
	2	.796 $\pm$ .047	.754 $\pm$ .063	.769 $\pm$ .035	.794 $\pm$ .113	.947 $\pm$ .151	.741 $\pm$ .027	.816 $\pm$ .100	.727 $\pm$ .011	.803 $\pm$ .135	.773 $\pm$ .031	.780 $\pm$ .043	.847 $\pm$ .121	.854 $\pm$ .160	.747 $\pm$ .008	.903 $\pm$ .157
	3	.751 $\pm$ .026	.754 $\pm$ .043	.722 $\pm$ .026	.751 $\pm$ .081	.818 $\pm$ .107	.751 $\pm$ .030	.753 $\pm$ .093	.728 $\pm$ .024	.762 $\pm$ .099	.728 $\pm$ .044	.762 $\pm$ .003	.740 $\pm$ .037	.764 $\pm$ .016	.734 $\pm$ .014	.812 $\pm$ .166
	4	.722 $\pm$ .012	.724 $\pm$ .038	.722 $\pm$ .024	.708 $\pm$ .015	.725 $\pm$ .061	.723 $\pm$ .036	.741 $\pm$ .093	.719 $\pm$ .021	.712 $\pm$ .046	.726 $\pm$ .023	.721 $\pm$ .018	.765 $\pm$ .052	.699 $\pm$ .027	.722 $\pm$ .030	.783 $\pm$ .068
	5	.717 $\pm$ .015	.712 $\pm$ .027	.723 $\pm$ .030	.721 $\pm$ .013	.716 $\pm$ .032	.731 $\pm$ .051	.744 $\pm$ .095	.712 $\pm$ .026	.718 $\pm$ .029	.718 $\pm$ .022	.718 $\pm$ .027	.743 $\pm$ .082	.709 $\pm$ .046	.721 $\pm$ .031	.772 $\pm$ .075

Appendix

Table D.7: Per-cell results with downstream model RF . Tasks are grouped by metric. Each cell is the mean over three seeds, with the sample standard deviation as a subscript.

Dataset	Budget idx $j$	CART	ElasticNet	F-test	GainRatio	LaplacianScore	Lasso	LOCO	MarkovBlanket	MI	mRMR	(R)ReliefF	Random	SBE	RFImportance	SFS
log-loss ( $\downarrow$ )
horse_colic survival	1	.803 $\pm$ .087	.743 $\pm$ .093	.740 $\pm$ .074	.814 $\pm$ .022	.858 $\pm$ .076	.743 $\pm$ .093	.809 $\pm$ .115	.799 $\pm$ .026	.785 $\pm$ .115	.789 $\pm$ .059	.800 $\pm$ .072	.850 $\pm$ .095	.845 $\pm$ .129	.799 $\pm$ .031	.877 $\pm$ .047
	2	.783 $\pm$ .026	.757 $\pm$ .095	.752 $\pm$ .033	.765 $\pm$ .058	.852 $\pm$ .058	.774 $\pm$ .103	.843 $\pm$ .043	.749 $\pm$ .037	.788 $\pm$ .048	.767 $\pm$ .053	.744 $\pm$ .065	.805 $\pm$ .061	.813 $\pm$ .111	.780 $\pm$ .126	.826 $\pm$ .071
	3	.802 $\pm$ .056	.753 $\pm$ .077	.730 $\pm$ .032	.765 $\pm$ .033	.806 $\pm$ .042	.803 $\pm$ .131	.796 $\pm$ .054	.729 $\pm$ .034	.759 $\pm$ .105	.761 $\pm$ .056	.826 $\pm$ .097	.767 $\pm$ .015	.800 $\pm$ .019	.752 $\pm$ .051	.830 $\pm$ .065
	4	.778 $\pm$ .006	.750 $\pm$ .021	.778 $\pm$ .041	.740 $\pm$ .034	.775 $\pm$ .087	.774 $\pm$ .053	.780 $\pm$ .075	.743 $\pm$ .053	.748 $\pm$ .043	.745 $\pm$ .044	.764 $\pm$ .027	.801 $\pm$ .033	.750 $\pm$ .019	.746 $\pm$ .020	.800 $\pm$ .034
	5	.780 $\pm$ .030	.765 $\pm$ .041	.763 $\pm$ .095	.754 $\pm$ .013	.736 $\pm$ .022	.766 $\pm$ .077	.767 $\pm$ .073	.794 $\pm$ .064	.740 $\pm$ .026	.733 $\pm$ .035	.755 $\pm$ .050	.764 $\pm$ .062	.733 $\pm$ .038	.752 $\pm$ .033	.783 $\pm$ .091

Appendix

Table D.8: Per-cell results with downstream model RealMLP . Tasks are grouped by metric. Each cell is the mean over three seeds, with the sample standard deviation as a subscript.

Dataset	Budget idx $j$	CART	ElasticNet	F-test	GainRatio	LaplacianScore	Lasso	LOCO	MarkovBlanket	MI	mRMR	(R)ReliefF	Random	SBE	RFImportance	SFS
log-loss ( $\downarrow$ )
horse_colic survival	1	.770 $\pm$ .066	.743 $\pm$ .100	.740 $\pm$ .062	.790 $\pm$ .059	.834 $\pm$ .071	.743 $\pm$ .100	.781 $\pm$ .102	.764 $\pm$ .024	.750 $\pm$ .076	.781 $\pm$ .034	.786 $\pm$ .052	.798 $\pm$ .093	.815 $\pm$ .098	.760 $\pm$ .074	.833 $\pm$ .038
	2	.731 $\pm$ .031	.720 $\pm$ .087	.736 $\pm$ .056	.744 $\pm$ .029	.804 $\pm$ .079	.714 $\pm$ .047	.743 $\pm$ .033	.704 $\pm$ .034	.744 $\pm$ .077	.733 $\pm$ .015	.742 $\pm$ .068	.754 $\pm$ .072	.772 $\pm$ .082	.712 $\pm$ .057	.791 $\pm$ .058
	3	.713 $\pm$ .034	.724 $\pm$ .066	.723 $\pm$ .065	.723 $\pm$ .041	.769 $\pm$ .063	.716 $\pm$ .058	.715 $\pm$ .039	.693 $\pm$ .035	.729 $\pm$ .080	.707 $\pm$ .020	.725 $\pm$ .033	.713 $\pm$ .043	.744 $\pm$ .054	.705 $\pm$ .025	.760 $\pm$ .072
	4	.709 $\pm$ .008	.702 $\pm$ .032	.714 $\pm$ .062	.689 $\pm$ .015	.698 $\pm$ .040	.702 $\pm$ .040	.708 $\pm$ .075	.690 $\pm$ .019	.692 $\pm$ .031	.703 $\pm$ .037	.688 $\pm$ .009	.740 $\pm$ .037	.678 $\pm$ .018	.696 $\pm$ .028	.756 $\pm$ .055
	5	.700 $\pm$ .025	.709 $\pm$ .037	.713 $\pm$ .067	.699 $\pm$ .026	.682 $\pm$ .014	.709 $\pm$ .046	.708 $\pm$ .074	.685 $\pm$ .024	.695 $\pm$ .018	.702 $\pm$ .034	.685 $\pm$ .023	.709 $\pm$ .062	.685 $\pm$ .028	.695 $\pm$ .029	.726 $\pm$ .061

Appendix

Table D.9: Per-cell results with downstream model TabICLv2 . Tasks are grouped by metric. Each cell is the mean over three seeds, with the sample standard deviation as a subscript.

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 194 papers

Apr 20, 2026

FSEVAL: Feature Selection Evaluation Toolbox and Dashboard

Feature selection is a fundamental machine learning and data mining task, involved with discriminating redundant features from informative ones. It is an attempt to address the curse of dimensionality by removing the redundant features, while unlike dimensionality reduction methods, preserving explainability. Feature selection is conducted in both supervised and unsupervised settings, with different evaluation metrics employed to determine which feature selection algorithm is the best. In this paper, we propose FSEVAL, a feature selection evaluation toolbox accompanied with a visualization dashboard, with the goal to make it easy to comprehensively evaluate feature selection algorithms. FSEVAL aims to provide a standardized, unified, evaluation and visualization toolbox to help the researchers working in the field, conduct extensive and comprehensive evaluation of feature selection algorithms with ease.

Muhammad Rajabinasab, Arthur Zimek

Department of Mathematics and Computer Science University of Southern Denmark Odense, Denmark

Feature Selection Automated Evaluation

Ask PDF

Jun 5, 2026

Bias in Filter Feature Selection Evaluation: A Meta-Analysis of Datasets, Baselines, and Experimental Design Choices

Background: Since 1990 many feature selection methods have been proposed across heterogeneous applications. To validate the usefulness of a new method, it needs to be compared against at least one baseline method from the existing literature on a feature selection task using at least one dataset. Recent developments in tabular Deep Learning (DL) and data valuation in Machine Learning (ML) suggest that the evaluation of new methods, algorithms, and models may be consciously or unconsciously biased. We hypothesise that a similar trend exists in feature selection (FS), particularly in filter feature selection (FFS). The aim of this study is therefore to examine FFS studies to identify factors that influence the evaluation and that might consist entry point for biases in order to recommend stronger principles for FFS evaluation. Methods: We analyse a sample of 28 high profile FFS studies published between 1994 and 2025. The analysis provides reflections on how to examine FFS studies, highlights lessons learned throughout the process, and gives five evidence-based recommendations for future FFS evaluation. Results: Multivariate Linear Regression analysis achieved a score of

R^2=0.33

. It means that 33% of the variance in the performance of new methods against chosen baselines (win rate) is explained by the number of datasets (#Datasets), the number of baselines (#Baselines), and the number of new methods (#NewMethods). Discussion:

R^2=0.33

is considered medium explanation; which is promising given that this is the first such study. The medium explanation result is due to the fact that win rate is influenced by additional factors such as the maturity of the feature selection domain, the type of datasets and baselines, and the simplicity of the regression model used to explain the relationship.

Malick Ebiele, Malika Bendechache, Rob Brennan

1*Computer Science, University College Dublin, Dublin, Dublin, Ireland. · 3ADAPT Centre, Dublin, Dublin, Ireland. · 2Computer Science, University of Galway, Galway, Galway, Ireland.

Feature Selection

Ask PDF

Jun 1, 2026

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remains a critical yet underexplored component of real-world modeling pipelines that is entirely absent from modern benchmarks, which creates an unquantified evaluation gap. In this work, we introduce TabPrep, a lightweight preprocessing pipeline composed of feature generators that are carefully designed to target three specific structural data patterns. We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance. Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models, often surpassing gains achieved by model-centric innovations alone. TabPrep outperforms previous automated feature engineering approaches in performance, efficiency, and applicability across datasets, enabling integration into large-scale benchmarks. By releasing TabPrep (see https://github.com/atschalz/tabprep), we enable researchers to integrate feature engineering into their benchmarking setup, filling a longstanding gap in tabular evaluations.

Andrej Tschalzev, Nick Erickson, Yuyang Wang +4

University of Mannheim · Prior Labs · AWS +3

Tabular ML Automated Feature Engineering Tabular Prediction

Ask PDF

Abstract

Feature Selection

Explore similar work

FSEVAL: Feature Selection Evaluation Toolbox and Dashboard

Bias in Filter Feature Selection Evaluation: A Meta-Analysis of Datasets, Baselines, and Experimental Design Choices

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks