Feature selection in ultra-high-dimensional, noisy datasets remains a critical challenge due to exponential search space growth, where conventional binary metaheuristics often suffer from premature convergence. To address this, EMG-Hybrid is introduced as a novel three-level hierarchical optimization architecture designed around decoupled operational planes, feedback-driven scheduling, and isolated state refinement. At Level 1, a Dual-Engine Controller monitors population Shannon entropy and empirical fitness velocity, functioning as an architectural master-switch that dynamically routes execution between global exploration and local intensification. This control loop conditions the Level 2 active optimization plane, where an Adaptive Black Hole Gravity (ABHG) execution layer performs aggressive feature manifold pruning via a distance-decoupled penalty kernel driven by fitness similarities and Hamming metrics. To structurally isolate macro-exploration from micro-exploitation, Level 3 encapsulates an autonomous Elite Memory Archive that preserves high-fidelity sub-schemas and hosts a discrete bitwise Differential Evolution (DE) micro-refinement sandbox for resolving complex parameter epistasis. Theoretically, uniform combinatorial bounds derived via Vapnik--Chervonenkis (VC) dimension criteria demonstrate that this multi-level restriction mitigates high-dimensional overfitting. Empirical results across manifolds scaling up to d>100,000 features, complemented by high-performance computing (HPC) profiling under lock-free atomic Compare-and-Swap (CAS) primitives, indicate strong performance gains over state-of-the-art baselines. The framework exhibits near-linear parallel speedups, sub-quadratic runtime scaling, and substantial dimensionality reduction, offering a scalable and structurally innovative paradigm for large-scale data mining.
Figures & tables
Figure 1: Three-level hierarchical architecture of the proposed EMG-Hybrid framework. Level 1 processes structural population diversity and elite fitness trajectories to broadcast the synchronized control channel Ct . Level 2 executes the mode-conditioned Unified Search Operator ( OEMG ) on the active population Xt , integrating parameterized BHO/BGWO search dynamics with elite-conditioned ABHG memory injections. Level 3 maintains the structurally protected Elite Memory Archive ( Et ) with an internal, isolated Differential Evolution (DE) refinement sandbox, acts as a read-only knowledge repository for Level 2, and closes the feedback loop back to the Level 1 decision plane.
Dataset
No. Attributes
No. Instances
Dimensionality
Noisy
breast cancer original
9
699
low
No
wine
13
178
low
No
statlog heart
13
270
low
No
zoo
16
101
low
No
vehicle
18
946
low
Medium
waveform
21
5000
low
Yes
Table 1: Datasets Information for 3-fold cross validation
Parameter
Value
Reference
Population Size
10
[ 11 ]
Maximum Iterations
100
[ 11 ]
Differential Evolution Params
F=0.5 (adaptive, scaled by phase), CR=0.9
[ 7 ] (adapted)
HHO Patience
5
[ 11 ]
GWO Patience
2
[ 11 ]
PSO Inertia Weights
wmax=0.9 , wmin=0.4
[ 15 ]
Table 2: Parameter Settings for the Proposed Model and Baseline Algorithms
Dataset
10-fold ( η tuned)
3-fold ( η tuned)
p -value
Effect size r
wine
0.040±0.005
0.062±0.007
0.120 (n.s.)
–
zoo
0.058±0.006
0.043±0.004
0.008
0.45
vehicle
0.267±0.012
0.211±0.010
0.004
0.52
diagnostic
0.050±0.005
0.045±0.004
0.031
0.38
ionosphere
0.055±0.006
0.052±0.005
0.067 (n.s.)
–
connectionist bench
0.042±0.004
0.035±0.003
0.009
0.44
Table 3: Comparison of best fitness value (mean ± std) between 10 - fold and 3 - fold cross - validation after tuning η separately for each fold type. Lower fitness is better. p -values are derived from the two-tailed Wilcoxon signed - rank test ( n=10 paired runs with identical random seeds). Effect size is computed as r=∣Z∣/n , where n denotes the total number of paired runs, for statistically significant differences.
Dataset
Average Accuracy
Average Feature Selection
Average Best Fitness
Stol=3
Stol=5
Stol=7
Stol=3
Stol=5
Stol=7
Stol=3
Stol=5
Stol=7
connectionist bench
0.962
0.989
0.978
15.87
14.93
15.43
0.061
0.035
0.046
darwin
0.941
0.949
0.97
101.07
96.7
64.17
0.069
0.062
0.036
toxicity
0.816
0.902
0.818
102.8
131.74
108.07
0.174
0.099
0.173
micro-mass
0.77
0.901
0.775
243.53
182.43
248.7
0.226
0.106
0.222
leukemia
0.92
0.917
0.918
1429.07
1271.87
969.47
0.093
0.093
0.087
Table 4: Performance Comparison for Different Stagnation Tolerance Stol Values
Dataset
Avg Accuracy
Avg Feature Selection
Avg Best Fitness
10−3
10−2
5×10−2
10−3
10−2
5×10−2
10−3
10−2
5×10−2
connectionist bench
0.981
0.989
0.98
14.53
14.93
14.17
0.042
0.035
0.042
darwin
0.967
0.949
0.961
58.47
96.7
67
0.039
0.062
0.046
toxicity
0.819
0.903
0.818
104.27
131.74
97.67
0.172
0.099
0.172
micro-mass
0.769
0.901
0.77
244.43
182.43
251.43
0.227
0.106
0.227
leukemia
0.928
0.917
0.915
925.77
1271.87
1069.53
0.093
0.093
0.091
Table 5: Impact of Improvement Threshold ( Δft ) on Optimization Performance.
Dataset
EMG-Hybrid
HHO
GWO
PSO
GA
ACO
Forward
breast cancer original
0.975
0.98
0.968
0.98
0.963
0.963
0.969
wine
0.993
0.978
0.973
0.982
0.982
0.974
0.966
statlog heart
0.959
0.840
0.827
0.847
0.846
0.839
0.822
zoo
0.99
0.99
0.977
0.984
0.963
0.980
0.951
vehicle
0.814
0.716
0.707
0.714
0.717
0.712
0.721
waveform
0.850
0.801
0.85
0.805
0.797
0.798
0.829
Table 6: Comparison of classification accuracy between the proposed model, traditional metaheuristics, and forward selection. Mean ranks and statistically significant p -values ( p<0.05 , bolded) highlight performance differences.
Dataset
EMG-Hybrid
HHO
GWO
PSO
GA
ACO
Forward
breast cancer original
4.07
5.43
4.8
3.1
2.97
4
3.67
wine
7.2
7.6
8.87
5.4
5.8
7.13
2.67
statlog heart
4.98
7.9
9.5
5.83
6.4
8.33
2.67
zoo
6.37
9.17
6.2
10.97
8.6
11.37
4.67
vehicle
7.83
9.73
12.4
8.9
8.73
12.03
5.33
waveform
11.47
12.87
15.43
13.03
11.37
14.23
11
Table 7: Comparison of feature selection between the proposed model, traditional metaheuristics, and forward selection. Mean ranks and statistically significant p -values ( p<0.05 , bolded) highlight performance differences.
Dataset
EMG-Hybrid
HHO
GWO
PSO
GA
ACO
Forward
breast cancer original
0.068
0.067
0.07
0.065
0.066
0.078
0.032
wine
0.062
0.078
0.093
0.058
0.061
0.079
0.034
statlog heart
0.076
0.165
0.229
0.182
0.188
0.209
0.178
zoo
0.043
0.065
0.096
0.083
0.087
0.089
0.049
vehicle
0.211
0.287
0.333
0.307
0.303
0.326
0.279
waveform
0.190
0.203
0.253
0.238
0.236
0.249
0.171
Table 8: Comparison of fitness value between the proposed model, traditional metaheuristics, and forward selection. Mean ranks and statistically significant p -values ( p<0.05 , bolded) highlight performance differences.
Figure 2: Dual-engine diversity and fitness dynamics of the EMG-Hybrid framework on the high-dimensional GCM dataset ( d=16,064 ), tracking the cooperative interaction between the Level 1 macro Λ -Engine and Level 2 micro M -Engine. Top Panel: Evolution of the binary population diversity index ( DIt ) where structural stagnation and genotypic homogenization are dynamically detected as the rolling mean hits the critical baseline floor ( τ=0.18 ). Reaching this threshold automatically triggers a macro-level switch event ( Vt at t∈{34,60,81} ), forcing the Level 1 controller to transition from the exploitative BGWO regime back into the exploratory BHHO schema phase to re-inject genetic variance. Bottom Panel: Evolutionary trajectory of the functional fitness velocity ( Δt ). When the fitness advancement drops below the stagnation tolerance ( ϵ=10−4 ) for a sustained window ( Stol ), the micro M -Engine flags localized phenotypic stagnation (shaded yellow regions). Following each macro entropy-driven switch event, a sharp velocity spike (marked as recovery ↑ ) demonstrates the successful escape from sub-optimal landscape traps, driving the active population configurations toward superior optimized feature manifolds.
Figure 3: Spatiotemporal feature survival frequency and cross-phase knowledge transfer visualization on ultra-high-dimensional space. The heatmap illustrates the evolution of the feature selection probability across the optimization horizon ( T=100 ). The red dashed line ( t=40 ) marks the deterministic Level 3 Elite Memory Archive injection event, triggering a clear structural separation between highly discriminative features (Indices 0–10 ) and discarded noisy features (Indices ≥20 ). The orange dotted line ( t=80 ) indicates the activation of the internal Level 3 Binary DE sub-kernel refinement layer, which isolates and stabilizes the boundaries of ambiguous feature schemas within the bounded elite sandbox.
Figure 4: Dynamic feature retention rates of the EMG-Hybrid framework under a highly corrupted adversarial synthetic landscape ( d=20,000 ). The trajectories illustrate the framework’s capacity to escape the univariable local optima trap at t=19 , executing aggressive top-down structural pruning of spurious ( Dspur ) and heavy-tailed noise ( Dnoise ) blocks, while monotonically maximizing the identification and retention of non-linear synergistic golden markers ( Dgold ) via adaptive Level 1 phase switching.
Figure 5: Computational efficiency and scalability of the EMG-Hybrid framework compared to five baseline metaheuristics (HHO, GWO, PSO, GA, ACO) over T=100 iterations. (A) Wall-clock scaling (log–log) showing sub-quadratic growth. (B) Iterations required to reach 99% of the global best fitness across increasing dimensionality. (C) Normalized runtime overhead relative to the baseline mean, with a maximum of 2.84× . (D) Per-iteration cost breakdown on the Leukemia dataset ( d=7,130 ), including Level 1 entropy tracking, Level 2 ABHG, and Level 3 DE components. (E) Overall speedup across seven dimensional settings, reaching up to 3.04× .
Figure 6: Granular high-performance computing (HPC) scalability and asynchronous profiling of the EMG-Hybrid framework on the Leukemia ( d=7,130 ) dataset. (A) Strong-scaling trajectories up to 32 workers, demonstrating a 7.2× speedup boundary for the lock-free Async Elite strategy against synchronous baselines under a fixed population constraint ( N=10 ). (B) Empirical race-condition probability profiles as a function of Elite Archive slot size k across multiple worker thresholds. (C) Comparative peak memory bandwidth pressure ( GB/s ) across expanding feature dimensions, highlighting the efficiency of lock-free reads. (D) Algorithmic convergence resilience under escalating archive update lags, confirming stability past the m=10 DE interval. (E) Schematic topology of the asynchronous, lock-free distributed worker layer regulated via atomic Compare-and-Swap ( CAS ) hardware primitives.
Automatic feature engineering is an effective approach for improving predictive performance in tabular learning. However, expand-and-reduce methods, such as OpenFE, become increasingly computationally expensive as the input dimensionality grows. This limitation arises primarily from the combinatorial explosion of candidate features generated through operator-feature combinations. To address this issue, we propose SCOPE-FE, a structured search space control framework that improves efficiency by reducing the candidate space prior to feature generation. SCOPE-FE jointly regulates two major sources of combinatorial growth: the operator space and feature-pair space. First, OperatorProbing estimates the dataset-specific utility of candidate operators and eliminates low-contribution operators in advance. Second, FeatureClustering employs spectral embedding and fuzzy c-means clustering to group structurally related features, thereby restricting candidate generation to relevant within-cluster combinations. In addition, we introduce ReliabilityScoring, which incorporates variance across subsamples to stabilize pruning decisions. Experiments on ten benchmark datasets demonstrate that SCOPE-FE substantially reduces feature engineering time while maintaining competitive predictive performance relative to existing baselines. The efficiency gains are particularly pronounced for high-dimensional datasets. These results indicate that structured control of the search space is an effective strategy for scalable automatic feature engineering. The code will be made publicly available upon acceptance.
Selecting the most relevant or informative features is a key issue in actual machine learning problems. Since an exhaustive search is not feasible even for a moderate number of features, an intelligent search strategy must be employed for finding an optimal subset, which implies considering how features interact with each other in promoting class separability. Balancing feature subset size and classification accuracy constitutes a multi-objective optimization challenge. Here we propose MOELIGA, a multi-objective genetic algorithm incorporating an evolutionary local improvement strategy that evolves subordinate populations to refine feature subsets. MOELIGA employs a crowding-based fitness sharing mechanism and a sigmoid transformation to enhance diversity and guide compactness, alongside a geometry-based objective promoting classifier independence. Experimental evaluation on 14 diverse datasets demonstrates MOELIGA's ability to identify smaller feature subsets with superior or comparable classification performance relative to 11 state-of-the-art methods. These findings suggest MOELIGA effectively addresses the accuracy-dimensionality trade-off, offering a robust and adaptable approach for multi-objective feature selection in complex, high-dimensional scenarios.
Leandro Vignolo, Matias Gerard
Research Institute for Signals, Systems and Computational Intelligence (sınc(i)), FICH–UNL/CONICET, Ciudad Universitaria UNL, Santa Fe, Argentina.
Gradient Boosted Decision Trees (GBDT), exemplified by LightGBM, spend a dominant fraction of training time -- typically 65-70% -- constructing per-feature histograms. Existing approaches such as random feature subsampling (feature_fraction) discard features without regard for their predictive utility. We propose EMA-based Feature Screening (EMA-FS), an algorithm-level optimization that maintains an exponential moving average (EMA) of per-feature split gains across boosting iterations and, after a short warmup, restricts histogram construction to the top-K features ranked by historical gain. Unlike random subsampling, EMA-FS is informed: it retains high-gain features while screening out low-gain ones. Operating at the per-tree level, it preserves full compatibility with LightGBM's histogram subtraction trick, requiring no changes to core routines. We evaluate EMA-FS on datasets spanning financial fraud detection, advertising click-through prediction, industrial quality control, and synthetic benchmarks, with feature dimensionalities from 29 to 968. On dense, moderate-to-high-dimensional data it achieves significant speedups: 2.61x on a 500-feature synthetic benchmark and 1.45x on the 432-feature IEEE-CIS Fraud dataset at 30% retention. At 70% retention it improves AUC by 0.11 points while delivering a 1.34x speedup. On extremely sparse data (Bosch, >90% missing) it yields no speedup, as LightGBM's sparse bin optimization already bypasses empty values. We further introduce Stochastic EMA-FS (S-EMA-FS), which replaces deterministic top-K selection with gain-weighted random sampling controlled by a concentration parameter beta, unifying deterministic EMA-FS (beta -> infinity) and random subsampling (beta = 0) in one framework. Both are implemented in ~120 lines of C++ across all six LightGBM tree learners and are fully backward-compatible.