Ensemble Learning

Follow this topic Ask about this topic

Ensemble learning studies how to combine predictions from multiple models, from Random Forests and boosting to stacking, voting, and mixtures of experts. Methods may average outputs, learn weights from validation data, or select and route among specialists for each example, with applications ranging from tabular prediction and medical imaging to weather forecasting and language models. Combining models can reduce errors and quantify uncertainty, but gains depend on whether their mistakes are complementary rather than correlated. Current work also weighs accuracy against calibration, computational cost, distribution shifts, and leakage-safe evaluation, while developing theoretical guarantees for aggregation and boosting.

Latest papers

278 papers25 papers in the last four weeks, down 24% on the four weeks before. 0.1% of all new papers.

Apr 12, 2026

Exact Finite-Sample Variance Decomposition of Subagging: A Spectral Filtering Perspective

Standard resampling ratios (e.g.,

α\approx 0.632

) are widely used as default baselines in ensemble learning for three decades. However, how these ratios interact with a base learner's intrinsic functional complexity in finite samples lacks a exact mathematical characterization. We leverage the Hoeffding-ANOVA decomposition to derive the first exact, finite-sample variance decomposition for subagging, applicable to any symmetric base learner without requiring asymptotic limits or smoothness assumptions. We establish that subagging operates as a deterministic low-pass spectral filter: it preserves low-order structural signals while attenuating

c

-th order interaction variance by a geometric factor approaching

α^c

. This decoupling reveals why default baselines often under-regularize high-capacity interpolators, which instead require smaller

α

to exponentially suppress spurious high-order noise. To operationalize these insights, we propose a complexity-guided adaptive subsampling algorithm, empirically demonstrating that dynamically calibrating

α

to the learner's complexity spectrum consistently improves generalization over static baselines.

Ye Su, Mingrui Ye, Yining Wang +2

Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences Shenzhen 518055, China · School of Mathematics Renmin University of China Beijing 100872, China · College of Information Science and Technology Beijing University of Chemical Technology Beijing 100029, China +1

Ensemble Learning Statistical Learning Theory Adaptive Sampling

Apr 10, 2026

A Hybrid Intelligent Framework for Uncertainty-Aware Condition Monitoring of Industrial Systems

Hybrid approaches that combine data-driven learning with physics-based insight have shown promise for improving the reliability of industrial condition monitoring. This work develops a hybrid condition monitoring framework that integrates primary sensor measurements, lagged temporal features, and physics-informed residuals derived from nominal surrogate models. Two hybrid integration strategies are examined. The first is a feature-level fusion approach that augments the input space with residual and temporal information. The second is a model-level ensemble approach in which machine learning classifiers trained on different feature types are combined at the decision level. Both hybrid approaches of the condition monitoring framework are evaluated on a continuous stirred-tank reactor (CSTR) benchmark using several machine learning models and ensemble configurations. Both feature-level and model-level hybridization improve diagnostic accuracy relative to single-source baselines, with the best model-level ensemble achieving a 2.9% improvement over the best baseline ensemble. To assess predictive reliability, conformal prediction is applied to quantify coverage, prediction-set size, and abstention behavior. The results show that hybrid integration enhances uncertainty management, producing smaller and well-calibrated prediction sets at matched coverage levels. These findings demonstrate that lightweight physics-informed residuals, temporal augmentation, and ensemble learning can be combined effectively to improve both accuracy and decision reliability in nonlinear industrial systems.

Maryam Ahang, Todd Charter, Masoud Jalayer +1

University of Victoria, Victoria, Canada · Aalto University, Espoo, Finland

Fault Diagnosis Physics-Informed ML Ensemble Learning

Apr 9, 2026

Ensembles at Any Cost? Accuracy-Energy Trade-offs in Recommender Systems

Ensemble methods are frequently used in recommender systems to improve accuracy by combining multiple models. Recent work reports sizable performance gains, but most studies still optimize primarily for accuracy and robustness rather than for energy efficiency. This paper measures accuracy energy trade offs of ensemble techniques relative to strong single models. We run 93 controlled experiments in two pipelines: 1. explicit rating prediction with Surprise (RMSE) and 2. implicit feedback ranking with LensKit (NDCG@10). We evaluate four datasets ranging from 100,000 to 7.8 million interactions (MovieLens 100K, MovieLens 1M, ModCloth, Anime). We compare four ensemble strategies (Average, Weighted, Stacking or Rank Fusion, Top Performers) against baselines and optimized single models. Whole system energy is measured with EMERS using a smart plug and converted to CO2 equivalents. Across settings, ensembles improve accuracy by 0.3% to 5.7% while increasing energy by 19% to 2,549%. On MovieLens 1M, a Top Performers ensemble improves RMSE by 0.96% at an 18.8% energy overhead over SVD++. On MovieLens 100K, an averaging ensemble improves NDCG@10 by 5.7% with 103% additional energy. On Anime, a Surprise Top Performers ensemble improves RMSE by 1.2% but consumes 2,005% more energy (0.21 vs. 0.01 Wh), increasing emissions from 2.6 to 53.8 mg CO2 equivalents, and LensKit ensembles fail due to memory limits. Overall, selective ensembles are more energy efficient than exhaustive averaging,

Jannik Nitschke, Lukas Wegmeth, Joeran Beel

University of Siegen, Siegen, Germany

Recommender Systems Ensemble Learning Energy-Efficient ML

Apr 9, 2026

Plug-and-Play Logit Fusion for Heterogeneous Pathology Foundation Models

Pathology foundation models (FMs) have become central to computational histopathology, offering strong transfer performance across a wide range of diagnostic and prognostic tasks. The rapid proliferation of pathology foundation models creates a model-selection bottleneck: no single model is uniformly best, yet exhaustively adapting and validating many candidates for each downstream endpoint is prohibitively expensive. We address this challenge with a lightweight and novel model fusion strategy, LogitProd, which treats independently trained FM-based predictors as fixed experts and learns sample-adaptive fusion weights over their slide-level outputs. The fusion operates purely on logits, requiring no encoder retraining and no feature-space alignment across heterogeneous backbones. We further provide a theoretical analysis showing that the optimal weighted product fusion is guaranteed to perform at least as well as the best individual expert under the training objective. We systematically evaluate LogitProd on 22 benchmarks spanning WSI-level classification, tile-level classification, gene mutation prediction, and discrete-time survival modeling. LogitProd ranks first on 20/22 tasks and improves the average performance across all tasks by ~3% over the strongest single expert. LogitProd enables practitioners to upgrade heterogeneous FM-based pipelines in a plug-and-play manner, achieving multi-expert gains with

\sim

12

\times

lower training cost than feature-fusion alternatives.

Gexin Huang, Anqi Li, Yusheng Tan +4

University of British Columbia, Vancouver, BC, Canada · Vector Institute, Toronto, ON, Canada · Washington University in St. Louis, St. Louis, MO, USA

Computational Pathology Ensemble Learning Data Fusion Pathology Foundation Models Histopathology Image Classification

Apr 9, 2026

ViSAGE @ NTIRE 2026 Challenge on Video Saliency Prediction

In this report, we present our champion solution for the NTIRE 2026 Challenge on Video Saliency Prediction held in conjunction with CVPR 2026. To exploit complementary inductive biases for video saliency, we propose Video Saliency with Adaptive Gated Experts (ViSAGE), a multi-expert ensemble framework. Each specialized decoder performs adaptive gating and modulation to refine spatio-temporal features. The complementary predictions from different experts are then fused at inference. ViSAGE thereby aggregates diverse inductive biases to capture complex spatio-temporal saliency cues in videos. On the Private Test set, ViSAGE ranked first on two out of four evaluation metrics, and outperformed most competing solutions on the other two metrics, demonstrating its effectiveness and generalization ability. Our code has been released at https://github.com/iLearn-Lab/CVPRW26-ViSAGE.

Kun Wang, Yupeng Hu, Zhiran Li +3

Shandong University · Harbin Institute of Technology (Shenzhen) · City University of Hong Kong +1

Video Saliency Prediction Visual Saliency Mixture of Experts Ensemble Learning

Apr 8, 2026

AdaBoost Does Not Always Cycle: A Computer-Assisted Counterexample

We give a computer-assisted counterexample to the open question, posed by Rudin, Schapire, and Daubechies in COLT 2012, of whether exhaustive AdaBoost always converges to a finite cycle. The construction is based on a block-product gadget whose two factors share an exact period-2 orbit for their 5-step branch maps, but whose linearized return maps have dominant eigenvalues with an irrational logarithmic ratio. This irrationality forces the burst-winner sequence to have an irrational asymptotic frequency, precluding eventual periodicity. All assertions are certified by exact rational arithmetic. This work was developed in collaboration with GPT-5.4 Pro and Claude Opus 4.6.

Erik Y. Wang

Department of Computer Science, Stanford University

Ensemble Learning Optimization Convergence Analysis

Apr 6, 2026

Quantifying how AI Panels improve precision

AI in applications like screening job applicants had become widespread, and may contribute to unemployment especially among the young. Biases in the AIs may become baked into the job selection process, but even in their absence, reliance on a single AI is problematic. In this paper we derive a simple formula to estimate, or at least place an upper bound on, the precision of such approaches for data resembling realistic CVs:

P(q) \approx \frac{ρn^b + q(1-ρ)}{1 + (n^b - 1)ρ}

where

b \approx q^* + 0.8 (1 - ρ)

and

q^*

is

q

clipped to

[0.07, 0.22]

where

P(q)

is the precision of the top

q

quantile selected by a panel of

n

AIs and

ρ

is their average pairwise correlation. This equation provides a basis for considering how many AIs should be used in a Panel, depending on the importance of the decision. A quantitative discussion of the merits of using a diverse panel of AIs to support decision-making in such areas will move away from dangerous reliance on single AI systems and encourage a balanced assessment of the extent to which diversity needs to be built into the AI parts of the socioeconomic systems that are so important for our future.

Nicholas CL Beale

AI-Assisted Decision Making AI and Labor Ensemble Learning

Apr 6, 2026

StableTTA: Improving Vision Model Performance by Training-free Test-Time Adaptation Methods

Ensemble methods improve predictive performance but often incur high memory and computational costs. We identify an aggregation instability induced by nonlinear projection and voting operations. To address both efficiency challenges and this inconsistency, we propose StableTTA, a training-free test-time adaptation method with two variants. StableTTA-I targets coherent-batch inference settings, where temporally or semantically adjacent observations are likely to belong to the same class. Examples include burst photography, video streams, robotics perception, and industrial inspection. Under coherent-batch inference, StableTTA-I substantially improves prediction consistency and accuracy through variance-aware logit aggregation. StableTTA-II establishes feature-level cropping, enabling efficient logit aggregation with a single forward pass on a single model backbone. Experiments on ImageNet-1K across 71 models demonstrate that StableTTA-I consistently improves prediction accuracy under coherent-batch inference, while StableTTA-II provides lightweight and architecture-agnostic accuracy improvements with minimal computational overhead. These results suggest that inference-time semantic coherence and aggregation stability provide useful perspectives for improving practical test-time adaptation systems.

Zheng Li, Jerry Cheng, Huanying Helen Gu

Department of Computer Science New York Institute of Technology

Test-Time Adaptation Ensemble Learning Efficient Neural Network Inference

Apr 4, 2026

Evaluation of Bagging Predictors with Kernel Density Estimation and Bagging Score

For a larger set of predictions of several differently trained machine learning models, known as bagging predictors, the mean of all predictions is taken by default. Nevertheless, this proceeding can deviate from the actual ground truth in certain parameter regions. An approach is presented to determine a representative y_BS from such a set of predictions using Kernel Density Estimation (KDE) in nonlinear regression with Neural Networks (NN) which simultaneously provides an associated quality criterion beta_BS, called Bagging Score (BS), that reflects the confidence of the obtained ensemble prediction. It is shown that working with the new approach better predictions can be made than working with the common use of mean or median. In addition to this, the used method is contrasted to several approaches of nonlinear regression from the literatur, resulting in a top ranking in each of the calculated error values without using any optimization or feature selection technique.

Philipp Seitz, Jan Schmitt, Andreas Schiffler

Institute of Digital Engineering, Technical University of Applied Sciences Würzburg-Schweinfurt, Schweinfurt, Bavaria, Germany

Ensemble Learning Uncertainty Quantification

Apr 4, 2026

HEDGE: Heterogeneous Ensemble for Detection of AI-GEnerated Images in the Wild

Robust detection of AI-generated images in the wild remains challenging due to the rapid evolution of generative models and varied real-world distortions. We argue that relying on a single training regime, resolution, or backbone is insufficient to handle all conditions, and that structured heterogeneity across these dimensions is essential for robust detection. To this end, we propose HEDGE, a Heterogeneous Ensemble for Detection of AI-GEnerated images, that introduces complementary detection routes along three axes: diverse training data with strong augmentation, multi-scale feature extraction, and backbone heterogeneity. Specifically, Route~~A progressively constructs DINOv3-based detectors through staged data expansion and augmentation escalation, Route~~B incorporates a higher-resolution branch for fine-grained forensic cues, and Route~C adds a MetaCLIP2-based branch for backbone diversity. All outputs are fused via logit-space weighted averaging, refined by a lightweight dual-gating mechanism that handles branch-level outliers and majority-dominated fusion errors. HEDGE achieves 4th place in the NTIRE 2026 Robust AI-Generated Image Detection in the Wild Challenge and attains state-of-the-art performance with strong robustness on multiple AIGC image detection benchmarks.

Fei Wu, Dagong Lu, Mufeng Yao +2

Shanghai Jiao Tong University · INTSIG Information

AI-Generated Image Detection Ensemble Learning

Apr 3, 2026

Isomorphic Functionalities between Ant Colony and Ensemble Learning: Part III -- Gradient Descent, Neural Plasticity, and the Emergence of Deep Intelligence

In Parts I and II of this series, we established isomorphisms between ant colony decision-making and two major families of ensemble learning: random forests (parallel, variance reduction) and boosting (sequential, bias reduction). Here we complete the trilogy by demonstrating that the fundamental learning algorithm underlying deep neural networks -- stochastic gradient descent -- is mathematically isomorphic to the generational learning dynamics of ant colonies. We prove that pheromone evolution across generations follows the same update equations as weight evolution during gradient descent, with evaporation rates corresponding to learning rates, colony fitness corresponding to negative loss, and recruitment waves corresponding to backpropagation passes. We further show that neural plasticity mechanisms -- long-term potentiation, long-term depression, synaptic pruning, and neurogenesis -- have direct analogs in colony-level adaptation: trail reinforcement, evaporation, abandonment, and new trail formation. Comprehensive simulations confirm that ant colonies trained on environmental tasks exhibit learning curves indistinguishable from neural networks trained on analogous problems. This final isomorphism reveals that all three major paradigms of machine learning -- parallel ensembles, sequential ensembles, and gradient-based deep learning -- have direct analogs in the collective intelligence of social insects, suggesting a unified theory of learning that transcends substrate. The ant colony, we conclude, is not merely analogous to learning algorithms; it is a living embodiment of the fundamental principles of learning itself.

Ernest Fokoué, Gregory Babbitt, Yuval Levental

School of Mathematics and Statistics, Rochester Institute of Technology · Gosnell School of Life Sciences, Rochester Institute of Technology · Center for Imaging Science, Rochester Institute of Technology

Ensemble Learning Stochastic Gradient Descent

Apr 1, 2026

Evolutionary Multi-Objective Fusion of Deepfake Speech Detectors

While deepfake speech detectors built on large self-supervised learning (SSL) models achieve high accuracy, employing standard ensemble fusion to further enhance robustness often results in oversized systems with diminishing returns. To address this, we propose an evolutionary multi-objective score fusion framework that jointly minimizes detection error and system complexity. We explore two encodings optimized by NSGA-II: binary-coded detector selection for score averaging and a real-valued scheme that optimizes detector weights for a weighted sum. Experiments on the ASVspoof 5 dataset with 36 SSL-based detectors show that the obtained Pareto fronts outperform simple averaging and logistic regression baselines. The real-valued variant achieves 2.37% EER (0.0684 minDCF) and identifies configurations that match state-of-the-art performance while significantly reducing system complexity, requiring only half the parameters. Our method also provides a diverse set of trade-off solutions, enabling deployment choices that balance accuracy and computational cost.

Vojtěch Staněk, Martin Perešíni, Lukáš Sekanina +2

Faculty of Information Technology, Brno University of Technology, Božetěchova 1/2, 612 00 Brno, Czech Republic

Audio Deepfake Detection Ensemble Learning ASVspoof

Mar 30, 2026

Optimized Weighted Voting System for Brain Tumor Classification Using MRI Images

The accurate classification of brain tumors from MRI scans is essential for effective diagnosis and treatment planning. This paper presents a weighted ensemble learning approach that combines deep learning and traditional machine learning models to improve classification performance. The proposed system integrates multiple classifiers, including ResNet101, DenseNet121, Xception, CNN-MRI, and ResNet50 with edge-enhanced images, SVM, and KNN with HOG features. A weighted voting mechanism assigns higher influence to models with better individual accuracy, ensuring robust decision-making. Image processing techniques such as Balance Contrast Enhancement, K-means clustering, and Canny edge detection are applied to enhance feature extraction. Experimental evaluations on the Figshare and Kaggle MRI datasets demonstrate that the proposed method achieves state-of-the-art accuracy, outperforming existing models. These findings highlight the potential of ensemble-based learning for improving brain tumor classification, offering a reliable and scalable framework for medical image analysis.

Ha Anh Vu

Artificial Intelligence Department, FPT University, Ho Chi Minh City, Vietnam

Brain Tumor Classification Ensemble Learning

Mar 30, 2026

FedDES: Graph-Based Dynamic Ensemble Selection for Personalized Federated Learning

Statistical heterogeneity in Federated Learning (FL) often leads to negative transfer, where a single global model fails to serve diverse client distributions. Personalized federated learning (pFL) aims to address this by tailoring models to individual clients. However, under most existing pFL approaches, clients integrate peer client contributions uniformly, which ignores the reality that not all peers are likely to be equally beneficial. Additionally, the potential for personalization at the instance level remains largely unexplored, even though the reliability of different peer models often varies across individual samples within the same client. We introduce FedDES (Federated Dynamic Ensemble Selection), a decentralized pFL framework that achieves instance-level personalization through dynamic ensemble selection. Central to our approach is a Graph Neural Network (GNN) meta-learner trained on a heterogeneous graph modeling interactions between data samples and candidate classifiers. For each test query, the GNN dynamically selects and weights peer client models, forming an ensemble of the most competent classifiers while effectively suppressing contributions from those that are irrelevant or potentially harmful for performance. Experiments on CIFAR-10 and real-world ICU healthcare data demonstrate that FedDES outperforms state-of-the-art pFL baselines in non-IID settings, offering robust protection against negative transfer.

Brianna Mueller, W. Nick Street

University of Iowa, Iowa City, Iowa

Federated Learning Non-IID Federated Learning Ensemble Learning

Mar 26, 2026

An Explainable Ensemble Learning Framework for Crop Classification with Optimized Feature Pyramids and Deep Networks

Agriculture is increasingly challenged by climate change, soil degradation, and resource depletion, and hence requires advanced data-driven crop classification and recommendation solutions. This work presents an explainable ensemble learning paradigm that fuses optimized feature pyramids, deep networks, self-attention mechanisms, and residual networks for bolstering crop suitability predictions based on soil characteristics (e.g., pH, nitrogen, potassium) and climatic conditions (e.g., temperature, rainfall). With a dataset comprising 3,867 instances and 29 features from the Ethiopian Agricultural Transformation Agency and NASA, the paradigm leverages preprocessing methods such as label encoding, outlier removal using IQR, normalization through StandardScaler, and SMOTE for balancing classes. A range of machine learning models such as Logistic Regression, K-Nearest Neighbors, Support Vector Machines, Decision Trees, Random Forest, Gradient Boosting, and a new Relative Error Support Vector Machine are compared, with hyperparameter tuning through Grid Search and cross-validation. The suggested "Final Ensemble" meta-ensemble design outperforms with 98.80% accuracy, precision, recall, and F1-score, compared to individual models such as K-Nearest Neighbors (95.56% accuracy). Explainable AI methods, such as SHAP and permutation importance, offer actionable insights, highlighting critical features such as soil pH, nitrogen, and zinc. The paradigm addresses the gap between intricate ML models and actionable agricultural decision-making, fostering sustainability and trust in AI-powered recommendations

Syed Rayhan Masud, SK Muktadir Hossain, Md. Ridoy Sarkar +3

Department of Computer Science & Engineering, American International University-Bangladesh, Dhaka, Bangladesh · EliteLab.AI, Queens,NY, U.S.A · Department of Computer Science & Engineering, Begum Rokeya University, Rangpur,Bangladesh

Ensemble Learning Precision Agriculture

Mar 25, 2026

On the Use of Bagging for Local Intrinsic Dimensionality Estimation

The theory of Local Intrinsic Dimensionality (LID) has become a valuable tool for characterizing local complexity within and across data manifolds, supporting a range of data mining and machine learning tasks. Accurate LID estimation requires samples drawn from small neighborhoods around each query to avoid biases from nonlocal effects and potential manifold mixing, yet limited data within such neighborhoods tends to cause high estimation variance. As a variance reduction strategy, we propose an ensemble approach that uses subbagging to preserve the local distribution of nearest neighbor (NN) distances. The main challenge is that the uniform reduction in total sample size within each subsample increases the proximity threshold for finding a fixed number k of NNs around the query. As a result, in the specific context of LID estimation, the sampling rate has an additional, complex interplay with the neighborhood size, where both combined determine the sample size as well as the locality and resolution considered for estimation. We analyze both theoretically and experimentally how the choice of the sampling rate and the k-NN size used for LID estimation, alongside the ensemble size, affects performance, enabling informed prior selection of these hyper-parameters depending on application-based preferences. Our results indicate that within broad and well-characterized regions of the hyper-parameters space, using a bagged estimator will most often significantly reduce variance as well as the mean squared error when compared to the corresponding non-bagged baseline, with controllable impact on bias. We additionally propose and evaluate different ways of combining bagging with neighborhood smoothing for substantial further improvements on LID estimation performance.

Kristóf Péter, Ricardo J. G. B. Campello, James Bailey +1

New Jersey Institute of Technology Newark, New Jersey, United States

Intrinsic Dimensionality Variance Reduction Ensemble Learning

Mar 25, 2026

KANEL: Kolmogorov-Arnold Network Ensemble Learning Enables Early Hit Enrichment in High-Throughput Virtual Screening

Machine learning models of chemical bioactivity are increasingly used for prioritizing a small number of compounds in virtual screening libraries for experimental follow-up. In these applications, assessing model accuracy by early hit enrichment such as Positive Predicted Value (PPV) calculated for top N hits (PPV@N) is more appropriate and actionable than traditional global metrics such as AUC. We present KANEL, an ensemble workflow that combines interpretable Kolmogorov-Arnold Networks (KANs) with XGBoost, random forest, and multilayer perceptron models trained on complementary molecular representations (LillyMol descriptors, RDKit-derived descriptors, and Morgan fingerprints).

Pavel Koptev, Nikita Krainov, Konstantin Malkov +1

AI1 Technologies · KAN Technologies, LLC · UNC-Chapel Hill

Virtual Screening Molecular Property Prediction Ensemble Learning

Mar 25, 2026

Isomorphic Functionalities between Ant Colony and Ensemble Learning: Part II-On the Strength of Weak Learnability and the Boosting Paradigm

In Part I of this series, we established a rigorous mathematical isomorphism between ant colony decision-making and random forest learning, demonstrating that variance reduction through decorrelation is a universal principle shared by biological and computational ensembles. Here we turn to the complementary mechanism: bias reduction through adaptive weighting. Just as boosting algorithms sequentially focus on difficult instances, ant colonies dynamically amplify successful foraging paths through pheromone-mediated recruitment. We prove that these processes are mathematically isomorphic, establishing that the fundamental theorem of weak learnability has a direct analog in colony decision-making. We develop a formal mapping between AdaBoost's adaptive reweighting and ant recruitment dynamics, show that the margin theory of boosting corresponds to the stability of quorum decisions, and demonstrate through comprehensive simulation that ant colonies implementing adaptive recruitment achieve the same bias-reduction benefits as boosting algorithms. This completes a unified theory of ensemble intelligence, revealing that both variance reduction (Part I) and bias reduction (Part II) are manifestations of the same underlying mathematical principles governing collective intelligence in biological and computational systems.

Ernest Fokoué, Gregory Babbitt, Yuval Levental

School of Mathematics and Statistics, Rochester Institute of Technology · Gosnell School of Life Sciences, Rochester Institute of Technology · Center for Imaging Science, Rochester Institute of Technology

Ensemble Learning Emergent Behavior in Multi-Agent Systems Statistical Learning Theory

Mar 24, 2026

Detection and Classification of (Pre)Cancerous Cells in Pap Smears: An Ensemble Strategy for the RIVA Cervical Cytology Challenge

Automated detection and classification of cervical cells in conventional Pap smear images can strengthen cervical cancer screening at scale by reducing manual workload, improving triage, and increasing consistency across readers. However, it is challenged by severe class imbalance and frequent nuclear overlap. We present our approach to the RIVA Cervical Cytology Challenge (ISBI 2026), which requires multi-class detection of eight Bethesda cell categories under these conditions. Using YOLOv11m as the base architecture, we systematically evaluate three strategies to improve detection performance: loss reweighting, data resampling and transfer learning. We build an ensemble by combining models trained under each strategy, promoting complementary detection behavior and combining them through Weighted Boxes Fusion (WBF). The ensemble achieves a mAP50-95 of 0.201 on the preliminary test set and 0.147 on the final test set, representing a 29% improvement over the best individual model on the final test set and demonstrating the effectiveness of combining complementary imbalance mitigation strategies.

Lautaro Kogan, María Victoria Ríos

Universidad de San Andrés, Buenos Aires, Argentina

Computational Pathology Class-Imbalanced Learning Ensemble Learning

Mar 24, 2026

Boost Like a (Var)Pro: Trust-Region Gradient Boosting via Variable Projection

Gradient boosting, a method of building additive ensembles from weak learners, has established itself as a practical and theoretically-motivated approach to approximate functions, especially using decision tree weak learners. Comparable methods for smooth parametric learners, such as neural networks, remain less developed in both training methodology and theory. To this end, we introduce VPBoost ({\bf V}ariable {\bf P}rojection {\bf Boost}ing), a gradient boosting algorithm for separable smooth approximators, i.e., models with a smooth nonlinear featurizer followed by a final linear mapping. VPBoost fuses variable projection, a training paradigm for separable models that enforces optimality of the linear weights, with a second-order weak learning strategy. The combination of second-order boosting, separable models, and variable projection give rise to a closed-form solution for the optimal linear weights and a natural interpretation of \VPBoost as a functional trust-region method. We thereby leverage trust-region theory to prove \VPBoost converges to a stationary point under mild geometric conditions and, under stronger assumptions, achieves a superlinear convergence rate. Comprehensive numerical experiments on synthetic data, image recognition, and scientific machine learning benchmarks demonstrate that \VPBoost learns an ensemble with improved evaluation metrics in comparison to gradient-descent-based boosting and attains competitive performance relative to an industry-standard decision tree boosting algorithm.

Abhijit Chowdhary, Elizabeth Newman, Deepanshu Verma

Department of Mathematics, Tufts University, Medford, MA 02155, USA · School of Mathematical and Statistical Sciences, Clemson University, Clemson, SC 29634

Ensemble Learning Trust-Region Optimization

Mar 24, 2026

REALITrees: Rashomon Ensemble Active Learning for Interpretable Trees

Active learning reduces labeling costs by selecting samples that maximize information gain. A dominant framework, Query-by-Committee (QBC), typically relies on perturbation-based diversity by inducing model disagreement through random feature subsetting or data blinding. While this approximates one notion of epistemic uncertainty, it sacrifices direct characterization of the plausible hypothesis space. We propose the complementary approach: Rashomon Ensembled Active Learning (REAL) which constructs a committee by exhaustively enumerating the Rashomon Set of all near-optimal models. To address functional redundancy within this set, we adopt a PAC-Bayesian framework using a Gibbs posterior to weight committee members by their empirical risk. Leveraging recent algorithmic advances, we exactly enumerate this set for the class of sparse decision trees. Across synthetic and established active learning baselines, REAL outperforms randomized ensembles, particularly in moderately noisy environments where it strategically leverages expanded model multiplicity to achieve faster convergence.

Simon D. Nguyen, Hayden McTavish, Kentaro Hoffman +2

Department of Statistics, University of Washington · Department of Computer Science, Duke University · Department of Sociology, University of Washington

PAC-Bayesian Theory Active Learning Ensemble Learning Decision Tree Learning

Mar 22, 2026

First-Mover Bias in Gradient Boosting Explanations: Mechanism, Detection, and Resolution

We identify first-mover bias -- path-dependent concentration of SHAP feature importance from sequential residual fitting in gradient boosting -- as a mechanistic contributor to attribution instability under multicollinearity. Scaling up a single model amplifies this effect: a Large Single Model matching our method's total tree count produces the poorest attribution reproducibility of any approach tested. We show that model independence largely neutralizes first-mover bias. Both DASH (Diversified Aggregation of SHAP) and simple seed-averaging (Stochastic Retrain) restore stability by breaking the sequential dependency chain. At rho=0.9, both achieve stability ~0.977, while Single Best degrades to 0.958 and LSM to 0.938. On Breast Cancer, DASH improves stability from 0.376 to 0.925 (+0.549), outperforming Stochastic Retrain by +0.063. Under nonlinear DGPs, the advantage emerges at rho>=0.7. DASH provides two diagnostic tools -- the Feature Stability Index and Importance-Stability Plot -- that detect first-mover bias without ground truth. A crossed ANOVA with formal F-statistics confirms the mechanism: DASH shifts variance from model-dominated (40.6%) to data-dominated (73.6%). Software at https://github.com/DrakeCaraker/dash-shap

Drake Caraker, Bryan Arnold, David Rhoads

Independent Researchers

SHAP Feature Attribution Gradient-Boosted Decision Trees Ensemble Learning

Mar 22, 2026

Deep Attention-based Sequential Ensemble Learning for BLE-Based Indoor Localization in Care Facilities

Indoor localization systems in care facilities enable optimization of staff allocation, workload management, and quality of care delivery. Traditional machine learning approaches to Bluetooth Low Energy (BLE)-based localization treat each temporal measurement as an independent observation, fundamentally limiting their performance. To address this limitation, this paper introduces Deep Attention-based Sequential Ensemble Learning (DASEL), a novel framework that reconceptualizes indoor localization as a sequential learning problem. The framework integrates frequency-based feature engineering, bidirectional GRU networks with attention mechanisms, multi-directional sliding windows, and confidence-weighted temporal smoothing to capture human movement trajectories. Evaluated on real-world data from a care facility using 4-fold temporal cross-validation, DASEL achieves a macro F1 score of 0.4438, representing a 53.1% improvement over the best traditional baseline (0.2898).

Minh Triet Pham, Quynh Chi Dang, Le Nhat Tan

School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia · Faculty of Applied Science, Ho Chi Minh City University of Technology, Vietnam National University - Ho Chi Minh City, Ho Chi Minh City, Vietnam

Wireless Localization Ensemble Learning Sequence Modeling

Mar 21, 2026

Multi-RF Fusion with Multi-GNN Blending for Molecular Property Prediction

Multi-RF Fusion achieves a test ROC-AUC of 0.8476 +/- 0.0002 on ogbg-molhiv (10 seeds), placing #1 on the OGB leaderboard ahead of HyperFusion (0.8475 +/- 0.0003). The core of the method is a rank-averaged ensemble of 12 Random Forest models trained on concatenated molecular fingerprints (FCFP, ECFP, MACCS, atom pairs -- 4,263 dimensions total), blended with deep-ensembled GNN predictions at 12% weight. Two findings drive the result: (1) setting max_features to 0.20 instead of the default sqrt(d) gives a +0.008 AUC gain on this scaffold split, and (2) averaging GNN predictions across 10 seeds before blending with the RF eliminates GNN seed variance entirely, dropping the final standard deviation from 0.0008 to 0.0002. No external data or pre-training is used.

Zacharie Bugaud

Molecular Property Prediction Ensemble Learning Random Forest

Mar 20, 2026

Multi-Stage Fine-Tuning of Pathology Foundation Models with Head-Diverse Ensembling for White Blood Cell Classification

The classification of white blood cells (WBCs) from peripheral blood smears is critical for the diagnosis of leukemia. However, automated approaches still struggle due to challenges including class imbalance, domain shift, and morphological continuum confusion, where adjacent maturation stages exhibit subtle, overlapping features. We present a multi-stage fine-tuning methodology for 13-class WBC classification in the WBCBench 2026 Challenge (ISBI 2026). Our best-performing model is a fine-tuned DINOBloom-base, on which we train multiple classifier head families (linear, cosine, and multilayer perceptron (MLP)). The cosine head performed best on the mature granulocyte boundary (Band neutrophil (BNE) F1 = 0.470), the linear head on more immature granulocyte classes (Metamyelocyte (MMY) F1 = 0.585), and the MLP head on the most immature granulocyte (Promyelocyte (PMY) F1 = 0.733), revealing class-specific specialization. Based on this specialization, we construct a head-diverse ensemble, where the MLP head acts as the primary predictor, and its predictions within the four predefined confusion pairs are replaced only when two other head families agree. We further show that cases consistently misclassified by all models are substantially enriched for probable labeling errors or inherent morphological ambiguity.

Antony Gitau, Martin Paulson, Bjørn-Jostein Singstad +3

University of South-Eastern Norway · Vestfold Hospital Trust

White Blood Cell Classification Fine-Grained Image Classification Pathology Foundation Models Ensemble Learning

Mar 20, 2026

AgenticRS-EnsNAS: Ensemble-Decoupled Self-Evolving Architecture Search

Neural Architecture Search (NAS) deployment in industrial production systems faces a fundamental validation bottleneck: verifying a single candidate architecture pi requires evaluating the deployed ensemble of M models, incurring prohibitive O(M) computational cost per candidate. This cost barrier severely limits architecture iteration frequency in real-world applications where ensembles (M=50-200) are standard for robustness. This work introduces Ensemble-Decoupled Architecture Search, a framework that leverages ensemble theory to predict system-level performance from single-learner evaluation. We establish the Ensemble-Decoupled Theory with a sufficient condition for monotonic ensemble improvement under homogeneity assumptions: a candidate architecture pi yields lower ensemble error than the current baseline if rho(pi) < rho(pi_old) - (M / (M - 1)) * (Delta E(pi) / sigma^2(pi)), where Delta E, rho, and sigma^2 are estimable from lightweight dual-learner training. This decouples architecture search from full ensemble training, reducing per-candidate search cost from O(M) to O(1) while maintaining O(M) deployment cost only for validated winners. We unify solution strategies across pipeline continuity: (1) closed-form optimization for tractable continuous pi (exemplified by feature bagging in CTR prediction), (2) constrained differentiable optimization for intractable continuous pi, and (3) LLM-driven search with iterative monotonic acceptance for discrete pi. The framework reveals two orthogonal improvement mechanisms -- base diversity gain and accuracy gain -- providing actionable design principles for industrial-scale NAS. All theoretical derivations are rigorous with detailed proofs deferred to the appendix. Comprehensive empirical validation will be included in the journal extension of this work.

Yun Chen, Moyu Zhang, Jinxin Hu +2

Alibaba Group, Beijing, China

Neural Architecture Search Ensemble Learning

Mar 20, 2026

Ensembles-based Feature Guided Analysis

Recent Deep Neural Networks (DNN) applications ask for techniques that can explain their behavior. Existing solutions, such as Feature Guided Analysis (FGA), extract rules on their internal behaviors, e.g., by providing explanations related to neurons activation. Results from the literature show that these rules have considerable precision (i.e., they correctly predict certain classes of features), but the recall (i.e., the number of situations these rule apply) is more limited. To mitigate this problem, this paper presents Ensembles-based Feature Guided Analysis (EFGA). EFGA combines rules extracted by FGA into ensembles. Ensembles aggregate different rules to increase their applicability depending on an aggregation criterion, a policy that dictates how to combine rules into ensembles. Although our solution is extensible, and different aggregation criteria can be developed by users, in this work, we considered three different aggregation criteria. We evaluated how the choice of the criterion influences the effectiveness of EFGA on two benchmarks (i.e., the MNIST and LSC datasets), and found that different aggregation criteria offer alternative trade-offs between precision and recall. We then compare EFGA with FGA. For this experiment, we selected an aggregation criterion that provides a reasonable trade-off between precision and recall. Our results show that EFGA has higher train recall (+28.51% on MNIST, +33.15% on LSC), and test recall (+25.76% on MNIST, +30.81% on LSC) than FGA, with a negligible reduction on the test precision (-0.89% on MNIST, -0.69% on LSC).

Federico Formica, Stefano Gregis, Andrea Rota +3

McMaster University, Canada · University of Bergamo, Italy

Neural Network Interpretability Feature Attribution Ensemble Learning

Mar 20, 2026

Decorrelation, Diversity, and Emergent Intelligence: The Isomorphism Between Social Insect Colonies and Ensemble Machine Learning

Social insect colonies and ensemble machine learning methods represent two of the most successful examples of decentralized information processing in nature and computation respectively. Here we develop a rigorous mathematical framework demonstrating that ant colony decision-making and random forest learning are isomorphic under a common formalism of stochastic ensemble intelligence. We show that the mechanisms by which genetically identical ants achieve functional differentiation -- through stochastic response to local cues and positive feedback -- map precisely onto the bootstrap aggregation and random feature subsampling that decorrelate decision trees. Using tools from Bayesian inference, multi-armed bandit theory, and statistical learning theory, we prove that both systems implement identical variance reduction strategies through decorrelation of identical units. We derive explicit mappings between ant recruitment rates and tree weightings, pheromone trail reinforcement and out-of-bag error estimation, and quorum sensing and prediction averaging. This isomorphism suggests that collective intelligence, whether biological or artificial, emerges from a universal principle: \textbf{randomized identical agents + diversity-enforcing mechanisms

\rightarrow

emergent optimality}.

Ernest Fokoué, Gregory Babbitt, Yuval Levental

School of Mathematics and Statistics, Rochester Institute of Technology · Gosnell School of Life Sciences, Rochester Institute of Technology · Center for Imaging Science, Rochester Institute of Technology

Shared Decision-Making Ensemble Learning Emergent Behavior in Multi-Agent Systems Random Forest

Mar 20, 2026

Data-driven ensemble prediction of the global ocean

Data-driven models have advanced deterministic ocean forecasting, but extending machine learning to probabilistic global ocean prediction remains an open challenge. Here we introduce FuXi-ONS, the first machine-learning ensemble forecasting system for the global ocean, providing 5-day forecasts on a global 1° grid up to 365 days for sea-surface temperature, sea-surface height, subsurface temperature, salinity and ocean currents. Rather than relying on repeated integration of computationally expensive numerical models, FuXi-ONS learns physically structured perturbations and incorporates an atmospheric encoding module to stabilize long-range forecasts. Evaluated against GLORYS12 reanalysis, FuXi-ONS improves both ensemble-mean skill and probabilistic forecast quality relative to deterministic and noise-perturbed baselines, and shows competitive performance against established seasonal forecast references for SST and Niño3.4 variability, while running orders of magnitude faster than conventional ensemble systems. These results provide a strong example of machine learning advancing a core problem in ocean science, and establish a practical path toward efficient probabilistic ocean forecasting and climate risk assessment.

Qiusheng Huang, Xiaohui Zhong, Anboyu Guo +3

Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · Shanghai Academy of Artificial Intelligence for Science, Shanghai, China. +2

Probabilistic Forecasting Spatiotemporal Forecasting Climate Science Ensemble Learning

Mar 20, 2026

When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines

Multi-agent LLM pipelines produce contradictory evidence on whether team diversity improves output quality: heterogeneous Mixture-of-Agents teams outperform single models, yet homogeneous Self-MoA teams consistently win under synthesis-based aggregation. We propose a resolution by identifying the selection bottleneck -- a crossover threshold in aggregation quality that determines whether diversity helps or hurts. Under this model, we obtain a closed-form crossover threshold

s^*

(Proposition 1) that separates the regimes where diversity helps and hurts. In a targeted experiment spanning 42 tasks across 7 categories (

N=210

), a diverse team with judge-based selection achieves a win rate of 0.810 against a single-model baseline, while a homogeneous team scores 0.512 -- near chance (Glass's

Δ= 2.07

). Judge-based selection outperforms MoA-style synthesis by

Δ_{\mathrm{WR}} = +0.631

-- the synthesis approach is preferred over the baseline in zero of 42 tasks by the judge panel. A decoupled evaluation with independent judges confirms all directional findings (Spearman

ρ= 0.90

). Exploratory evidence suggests that including a weaker model improves performance while reducing cost (

p < 10^{-4}

, not pre-registered). Our results suggest that selector quality may be a more impactful design lever than generator diversity in single-round generate-then-select pipelines.

Artem Maryanskyy, Dmitry Budnikov, Alibek T. Kaliyev

Uber Technologies, Sunnyvale, CA 94085, USA · Laboratories of Electrical, Thermal Technologies and Energy Saving, Federal Scientific Agroengineering Center VIM, 109428 Moscow, Russia · Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA

Multi-Agent LLM Systems Multi-LLM Collaboration Ensemble Diversity Ensemble Learning LLM Agent Evaluation