KGroups: A Versatile Univariate Max-Relevance Min-Redundancy Feature Selection Algorithm for High-dimensional Biological Data

Malick Ebiele; Malika Bendechache; Rob Brennan

cs.LGMar 30, 2026

KGroups: A Versatile Univariate Max-Relevance Min-Redundancy Feature Selection Algorithm for High-dimensional Biological Data

Authors: Malick Ebiele, Malika Bendechache, Rob Brennan

Organizations: Computer Science, University College Dublin, Dublin, Dublin, Ireland. · ADAPT Centre, Dublin, Dublin, Ireland. · Computer Science, University of Galway, Galway, Galway, Ireland.

Abstract

This paper proposes a new univariate filter feature selection (FFS) algorithm called KGroups. The majority of work in the literature focuses on investigating the relevance or redundancy estimations of feature selection (FS) methods. This has shown promising results and a real improvement of FFS methods' predictive performance. However, limited efforts have been made to investigate alternative FFS algorithms. This raises the following question: how much of the FFS methods' predictive performance depends on the selection algorithm rather than the relevance or the redundancy estimations? The majority of FFS methods fall into two categories: relevance maximisation (Max-Rel, also known as KBest) or simultaneous relevance maximisation and redundancy minimisation (mRMR). KBest is a univariate FFS algorithm that employs sorting (descending) for selection. mRMR is a multivariate FFS algorithm that employs an incremental search algorithm for selection. In this paper, we propose a new univariate mRMR called KGroups that employs clustering for selection. Extensive experiments on 14 high-dimensional biological benchmark datasets showed that KGroups achieves similar predictive performance compared to multivariate mRMR while being up to 821 times faster. KGroups is parameterisable, which leaves room for further predictive performance improvement through hyperparameter finetuning, unlike mRMR and KBest. KGroups outperforms KBest.

Figures & tables

Type	Predic-tive Perfor-mance	Compu-tational Cost	Genera-lisation to other models	Example Methods	Short Description
Filters	Low	Low	High	SelectKBest; SelectPercentile	- Perform feature selection at pre-processing without relying on ML model - Use statistics, information theory, or geometry to measure relevance and redundancy - Model-agnostic approach
Wrappers	High	High	Low	Forward Selection; Backward Selection; Recursive Selection; Exhaustive Selection	- Integrate variable selection into training process - Evaluate subsets based on predictive performance - Can use SHAP values, permutation importance - Assigned values can be negative
Embedded	Medium	Medium	Medium	Gini and Information Gain Importance	- Use ML models with built-in feature selection - Tied to specific models (e.g. tree-based models) - Assign importance values during training - Non-relevant features often assigned zero

Table 1: Comparative analysis of the primary types of feature selection methods (adapted from Jiao et al. (2023) )

Figure 1: Overview of FFS methods on a tabular data as found in the literature.

Name	#Instances	#Features	#Classes
ALLAML	72	7129	2
CLL_SUB_111	111	11340	3
Carcinom	174	9182	11
GLI_85	85	22283	2
GLIOMA	50	4434	4
Prostate_GE	102	5966	2

Table 2: The list of benchmark datasets used in our experiments.

		Actual
		Positive (P)	Negative (N)
Predicted	Positive (P)	True Positive (TP)	False Positive (FP)
Predicted	Negative (N)	False Negative (FN)	True Negative (TN)

Table 3: Confusion matrix.

		Gini importance		Mutual info.		F-value
Dataset	All feat.	KBest	KGroups	KBest	KGroups	KBest	KGroups
ALLAML	95.91	100.00*	100.00*	98.57	100.00*	98.57	100.00*
CLL_SUB_111	73.82	88.26*	88.22	84.66	84.66	71.15	79.21
Carcinom	95.47	94.82	96.54*	95.38	95.39	92.52	77.01
GLIOMA	78.00	98.00*	96.00	90.00	94.00	88.00	90.00
GLI_85	84.62	95.29	98.82*	96.47	98.82*	95.29	97.65

Table 4: Best overall configurations of KBest and KGroups across classifiers. The highest accuracy per relevance estimator excluding ties (Win) is in bold . Tied highest accuracy scores (Draw) are underlined . The star ( ∗ ) indicates best configuration per dataset.

{}^{\text{a}}

indicates abnormally low accuracy.

		Gini importance		Mutual info.		F-value
Dataset	All feat.	mRMR	KGroups	mRMR	KGroups	mRMR	KGroups
ALLAML	95.91	100.00*	100.00*	100.00*	100.00*	100.00*	100.00*
CLL_SUB_111	73.82	91.86*	88.22	90.04	84.66	85.57	79.21
Carcinom	95.47	95.98	96.54	96.55*	95.39	95.97	77.01
GLIOMA	78.00	94.00	96.00*	90.00	94.00	88.00	90.00
GLI_85	84.62	96.47	98.82*	97.65	98.82*	97.65	97.65

Table 5: Best overall configurations of mRMR and KGroups across classifiers. The highest accuracy per relevance estimator excluding ties (Win) is in bold . Tied highest accuracy scores (Draw) are underlined . The star ( ∗ ) indicates best configuration per dataset.

{}^{\text{a}}

indicates abnormally low accuracy.

Figure 2: Distribution of number of features selected by KBest, mRMR, and KGroups using Gini importance relevance estimator and KNeighbors, GaussianNB, RandomForest, XGB, MLP, and LinearSVC classification models.

Figure 3: Distribution of number of features selected by KBest, mRMR, and KGroups using mutual information relevance estimator and KNeighbors, GaussianNB, RandomForest, XGB, MLP, and LinearSVC classification models.

Figure 4: Distribution of number of features selected by KBest, mRMR, and KGroups using F-value relevance estimator and KNeighbors, GaussianNB, RandomForest, XGB, MLP, and LinearSVC classification models.

		Gini import.		Mutual info.		F-value
Dataset	All feat.	KBest	KG	KBest	KG	KBest	KG
ALLAML	87.88	98.36	99.56*	97.43	99.33	96.98	98.33
	$\pm$ 7.31	$\pm$ 1.37	$\pm$ 1.09	$\pm$ 2.21	$\pm$ 1.12	$\pm$ 2.03	$\pm$ 1.41
CLL_SUB_111	65.39	85.28	85.99*	78.96	80.00	65.62	73.36
	$\pm$ 6.38	$\pm$ 2.50	$\pm$ 1.68	$\pm$ 4.41	$\pm$ 2.64	$\pm$ 4.58	$\pm$ 4.19
Carcinom	85.13	86.88	90.41*	87.26	89.82	81.93	69.05

Table 6: Average

\pm

standard deviation of the best configurations of KBest and KGroups (KG) across classifiers. The highest average accuracy per relevance estimator excluding ties (Win) is in bold . Tied highest average accuracy scores (Draw) are underlined . The star ( ∗ ) indicates best configuration per dataset.

{}^{\text{a}}

indicates abnormally low accuracy.

		Gini import.		Mutual info.		F-value
Dataset	All feat.	mRMR	KG	mRMR	KG	mRMR	KG
ALLAML	87.88	98.16	99.56*	99.10	99.33	98.13	98.33
	$\pm$ 7.31	$\pm$ 1.45	$\pm$ 1.09	$\pm$ 0.70	$\pm$ 1.12	$\pm$ 2.11	$\pm$ 1.41
CLL_SUB_111	65.39	88.12*	85.99	85.14	80.00	81.05	73.36
	$\pm$ 6.38	$\pm$ 2.75	$\pm$ 1.68	$\pm$ 3.35	$\pm$ 2.64	$\pm$ 5.67	$\pm$ 4.19
Carcinom	85.13	90.41*	90.41*	90.22	89.82	89.82	69.05

Table 7: Average

\pm

standard deviation of the best configurations of mRMR and KGroups (KG) across classifiers. The highest average accuracy per relevance estimator excluding ties (Win) is in bold . Tied highest average accuracy scores (Draw) are underlined . The star ( ∗ ) indicates best configuration per dataset.

{}^{\text{a}}

indicates abnormally low accuracy.

Figure 5: Boxplots of the execution time in seconds of KBest, mRMR, and KGroups.

	Gini import.			Mutual info.				F-value
Dataset	KG	RFCQ	KB	KG	MID	MIQ	KB	KG	FCD	FCQ	KB
ALLAML	180	5,952	186	186	52,448	52,432	192	184	5,878	5,864	176
CLL_SUB_111	234	9,600	275	253	94,996	94,983	271	233	9,434	9,425	245
Carcinom	552	8,307	917	752	96,504	94,879	746	28	8,004	7,998	660
GLIOMA	196	3,757	188	204	31,009	30,729	206	189	3,695	3,691	193
GLI_85	193	18,635	239	204	168,077	167,806	237	203	18,261	18,257	191

Table 8: Execution time in seconds of the three FFS algorithms, KGroups (KG, on the left), KBest (KB, on the right), and mRMR (everything in between) per relevance estimator. The lowest execution time per relevance estimator is in bold . The last letter indicates the variant of mRMR: D for Difference and Q for Quotient. The other letters indicate the relevance and redundancy estimators e.g. RFCQ for RandomForest Pearson Correlation Quotient, MID for mutual information (for both relevance and redundancy) Difference, and FCQ for F-value Pearson Correlation Quotient.

Figure 6: Impact of $\alpha$ and the smoothing variable on the bins size for $k=20$ ; example of mutual information relevance estimator on the ALLAML dataset.

Bioinformatics

See all 41 papers

Advancing Interaction-Sensitive Feature Selection: Novel Relief-Based Algorithms, Expanded Comparisons, and Recommendations for Biomedical Data Mining
Aug 28, 2026Kia Kazemi-Nia, Harsh Bandhey, Philip J. Freda +1Feature Selection Feature Interaction Modeling
Cedars-Sinai Health Sciences University, Los Angeles, 90048, CA, USA
Scalable unsupervised feature selection via weight stability
Jun 6, 2025Xudong Zhang, Renato Cordeiro de AmorimUnsupervised Learning Clustering
School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe, UK.
Beyond False Discovery Rate: A Stepdown Group SLOPE Approach for Grouped Variable Selection
Mar 1, 2026Xuelin Zhang, Jingxuan Liang, Xinyue Liu +2Feature Selection FDR Control
College of Informatics, Huazhong Agricultural University, Wuhan, 430070, Hubei Province, China.

Aug 28, 2026J/K move · Enter open · S save

Advancing Interaction-Sensitive Feature Selection: Novel Relief-Based Algorithms, Expanded Comparisons, and Recommendations for Biomedical Data Mining

Kia Kazemi-Nia, Harsh Bandhey, Philip J. Freda, Ryan J. Urbanowicz

Cedars-Sinai Health Sciences University, Los Angeles, 90048, CA, USA

Feature Selection Feature Interaction Modeling

Open paper AskPDF arXiv

As a precursor to high-dimensional biomedical data modeling, reliable feature selection can reduce computational expense, improve modeling performance, and yield simpler, more interpretable models. However, most filter-based feature selection methods struggle to detect feature interactions, while wrapper or embedded feature selection methods are computationally expensive. Relief-based algorithms (RBAs) are filter methods that are sensitive to feature interactions while mitigating these other limitations. This study (1) refactors, optimizes, and expands the scikit-rebate Python package with existing and newly proposed RBA variants and (2) conducts rigorous RBA benchmark comparisons across diverse genomic simulations. We expand scikit-rebate to include SWRF*, mu-Relief, and 5 novel RBA variants implementing alternative strategies for neighbor selection and feature scoring. All RBAs were evaluated to compare predictive feature ranking and runtime across simulated genomic datasets varying in sample size, number of features, heritability, and underlying association type (e.g. main effects and interactions). All RBAs, except mu-Relief, were proficient in detecting 2-way interactions in noisy data. RBAs utilizing 'far' scoring were best at detecting 2-way interactions - with MultiSWRFDB* top-performing - but were far less sensitive to main effects. SWRF, MultiSWRF, MultiSURF, and MultiSWRFDB yielded top performance across main effect and 2-way interaction datasets with MultiSWRFDB performing best when also considering 3-way interactions. Refactoring of scikit-rebate resulted in 10 to 35-fold reductions in RBA runtimes. The newly introduced RBAs were among the strongest performing, and by robustly retaining both main effects and 2-way epistatic interactions, these algorithms preserve predictive signals for downstream modeling.

Abstract

Bioinformatics

Explore similar work

Advancing Interaction-Sensitive Feature Selection: Novel Relief-Based Algorithms, Expanded Comparisons, and Recommendations for Biomedical Data Mining

Scalable unsupervised feature selection via weight stability

Beyond False Discovery Rate: A Stepdown Group SLOPE Approach for Grouped Variable Selection