GeNeX: Genetic Network eXperts framework for addressing Validation Overfitting
Authors: Emmanuel Pintelas, Ioannis E. Livieris
Organizations: Department of Statistics & Insurance Science, University of Piraeus, GR 18534, Greece · Department of Business & Organization Administration, University of Peloponnese, GR 24100, Greece
Excessive reliance on validation performance during model selection can lead to validation overfitting (VO), where models appear effective during development but fail at test time. This issue is further amplified in low-data regimes and under distribution shifts, where validation signals become unreliable. Although ensemble learning is widely used to improve robustness and generalization, most ensemble construction pipelines depend heavily on validation scores, leaving them vulnerable to VO and limiting their reliability in real-world deployment scenarios. To address this, we propose GeNeX (Genetic Network eXperts), a framework that mitigates validation overfitting at both model generation and ensemble construction stages. In the generation phase, GeNeX employs a dual-path strategy: gradient-based training is coupled with genetic model evolution. Offspring networks are created via crossover of trained parents, promoting structural diversity and weight-level regeneration without relying on validation feedback. This results in a candidate pool of robust, non-overfitted models. During ensemble construction, the candidate networks are clustered by prediction behavior to identify complementary model spaces. Within each cluster, multiple diverse experts are selected using criteria such as robustness and representativeness, and fused at the weight level to form compact prototype networks. The final ensemble aggregates these prototypes, with predictions optimized via Sequential Quadratic Programming for output-level synergy. To rigorously evaluate VO resilience, we introduce a VO-aware evaluation protocol that simulates realistic deployment scenarios by enforcing distributional divergence between training and test subsets.
Figures & tables
Fig. 1 : GeNeX overview. GenE generates a diverse pool M without validation monitoring via short supervised training and genetic crossover/mutation, encouraging broad weight-space exploration and limiting early validation dependence. ProtoNeX clusters models in behavior (prediction) space, elects complementary experts via multi-criteria selection, and fuses them into K compact prototypes. Instead of clustering data and training separate models per cluster, ProtoNeX clusters the models themselves and uses prototype fusion, promoting complementarity and behavioral diversity across the model space to enhance generalization.
Fig. 2 : Distribution-Shifted Train/Test partitioning based on the proposed JSD-guided clustering algorithm applied on Skin Cancer dataset. The goal of the method is to partition a dataset into training and test subsets such that the JSD between their distributions is maximized. This process generates distributionally shifted splits that simulate realistic and challenging learning scenarios. Such splits are useful for reliably benchmarking a model’s robustness to validation overfitting. Visual inspection reveals distinct data characteristics across the two subsets, emphasizing the diversity of the resulting partitions.
Frameworks
Description
Generation
Selection
Validation Monitoring
N
Criterion
K
Single (1-ep)
Single model just trained for 1 epoch.
✗
1
-
1
Aver-Ens (1-ep)
Averaging ensemble of random initialized models trained for 1 epochs.
3
3
Single ( N50 )
(1) A pool of N models is trained under diverse configurations using a standard train-validation development pipeline. (2) Validation performance is monitored to guide model development. (3) The top-validation model is selected as final single predictor
✓
50
Top-Val-based
1
Single ( N500 )
500
Inc-Ens ( N50,K3 )
Ensemble approach that generates N models as in (1) and (2). Incrementally selects K total models from the generated pool, each chosen to progressively enhance validation performance.
50
3
TABLE I : Summary classification of all utilized frameworks in the experimental comparison.
Frameworks
Skin Cancer
DeepFake
Plant Disease
Pneumonia
RS JSD = 1e-4
GS JSD = 0.360
RS JSD = 1e-4
GS JSD = 0.545
RS JSD = 1e-4
GS JSD = 0.434
RS JSD = 1e-4
GS JSD = 0.142
TP
VO
TP
VO
TP
VO
TP
VO
TP
VO
TP
VO
TP
VO
TP
VO
Single (1-ep)
696
<1
672
54
658
<1
585
19
769
<1
597
171
526
<1
533
120
Aver-Ens (1-ep)
702
<1
693
55
697
<1
605
16
847
<1
600
175
541
<1
645
113
Single ( N50 )
791
2
607
214
706
3
507
234
943
<1
533
380
914
<1
812
152
Single ( N500 )
802
8
577
248
715
9
445
298
946
2
540
395
929
<1
798
170
TABLE II : Performance results under the proposed VO-aware evaluation protocol: Ratio-constrained JSD-guided clustering algorithm was applied to generate highly divergent training-testing splits (target ratio r=0.3 ).
Fig. 3 : Visualization of Train–Val Overfitting (TO) and Val–Test Overfitting (VO) gaps for two parent networks A and B and their genetically evolved child models. The study is conducted on the most challenging VO-pruned dataset, GS-DeepFake, which exhibits the highest distributional shift (JSD = 0.545). We observe that despite minimal TO and high validation performance (epochs 12–18 for A), severe VO gaps can still occur, leading to unreliable selection. After genetic crossover (vertical line), child models demonstrate improved generalization and reduced overfitting.
Deep learning models may converge to suboptimal solutions despite strong validation accuracy, masking an optimization failure we term Trajectory Deviation. This is because as training proceeds, models can abandon high generalization states for specific data sub-populations, thus discarding previously learned latent features without triggering classical overfitting signals. To address this problem we introduce VISTA, an online self-distillation framework that enforces consistency along the optimization trajectory. Using a validation-informed Marginal Coverage score, VISTA identifies expert anchors, which are earlier model states that retain specialized competence over distinct data regions. A coverage-weighted ensemble of these anchors is integrated online during training, regularizing the loss landscape and preserving mastered knowledge. When evaluated across multiple benchmarks, VISTA demonstrates improved robustness and generalization over standard training and prior self-distillation methods, while a lightweight implementation reduces storage overhead by 90% without performance loss.
Eli Corn, Daphna Weinshall
Department of Computer Science, Hebrew University of Jerusalem, Jerusalem, Israel.
Unsupervised outlier detection is attractive because it eliminates the need for labeled data. Moreover, forming multi-model ensembles can improve detection robustness. However, composing an ensemble without labeled data is challenging. Naively composed ensembles can suffer from ensemble saturation, where redundant or unreliable detection models degrade performance and incur unnecessary computation. We propose MetaEns, an automatic unsupervised framework for selecting ensembles of outlier detection models. Using labeled meta-datasets, MetaEns learns a model that predicts marginal ensemble gains, estimating the expected improvement from adding a candidate model to a partially constructed ensemble. At test time, this learned signal is combined with a submodular-inspired proxy objective that enforces diminishing returns through diversity-aware discounting and family-level risk regularization, thereby enabling greedy sequential selection with adaptive early stopping. As a result, MetaEns constructs compact, high-quality ensembles without access to ground-truth labels. Experiments on 39 real-world datasets show that MetaEns consistently outperforms state-of-the-art unsupervised selectors and ensemble baselines, achieving higher average precision while using fewer models.
Hong-Phuc Phan, Tuan-Anh Vu, Tung Kieu +3
Department of Software Engineering, FPT University, Vietnam · Department of Information Technology, Can Tho University of Technology, Vietnam · Department of Computer Science, Aalborg University, Denmark +2
Ensemble models achieve state-of-the-art performance on prediction tasks, but usually require aggregating a large number of weak learners. This can hinder deployment, interpretability, and downstream tasks such as robustness verification. Remedies to this issue fall into two main camps: pruning, which discards redundant learners, and compression, which generates new ones from scratch. We introduce PACE, a framework that interleaves these paradigms in a two-phase strategy. First, new learners are actively generated via a theoretically grounded procedure to enhance the diversity of the initial ensemble. When no more relevant learners can be found, a second phase of pruning is performed on this enriched ensemble. During both operations, PACE allows fine control on the faithfulness to the original ensemble. Experiments show that our method outperforms prior pruning and compression methods while offering principled control of faithfulness guarantees.
Fabian Akkerman, Julien Ferry, Théo Guyard +1
Industrial Engineering and Management Science University of Twente, Netherlands · SCALE-AI Chair and CIRRELT Polytechnique Montréal, Canada