Ensemble Learning

Ask about this topic

Overview

Ensemble learning studies how to combine predictions from multiple models, from Random Forests and boosting to stacking, voting, and mixtures of experts. Methods may average outputs, learn weights from validation data, or select and route among specialists for each example, with applications ranging from tabular prediction and medical imaging to weather forecasting and language models. Combining models can reduce errors and quantify uncertainty, but gains depend on whether their mistakes are complementary rather than correlated. Current work also weighs accuracy against calibration, computational cost, distribution shifts, and leakage-safe evaluation, while developing theoretical guarantees for aggregation and boosting.

Latest papers

CardsList

278 papers25 papers in the last four weeks, down 24% on the four weeks before. 0.1% of all new papers.

May 13, 2026

Reducing Bias and Variance: Generative Semantic Guidance and Bi-Layer Ensemble for Image Clustering

Image clustering aims to partition unlabeled image datasets into distinct groups. A core aspect of this task is constructing and leveraging prior knowledge to guide the clustering process. Recent approaches introduce semantic descriptions as prior information, most of which typically relying on matching-based techniques with predefined vocabularies. However, the limited matching space restricts their adaptability to downstream clustering tasks. Moreover, these methods primarily focus on reducing bias to improve performance, frequently overlooking the importance of variance reduction. To address these limitations, we propose GSEC (Image Clustering based on Generative Semantic Guidance and Bi-Layer Ensemble), a framework designed to reduce bias through generative semantic guidance and mitigate variance via ensemble learning. Our method employs Multimodal Large Language Models to generate semantic descriptions and derive image embeddings via weighted averaging. Additionally, a bi-layer ensemble strategy integrates cross-modal information through BatchEnsemble in the inner layer and aligns outputs via an alignment mechanism in the outer layer. Comparative experiments demonstrate that GSEC outperforms 18 state-of-the-art methods across six benchmark datasets, while further analysis confirms its effectiveness in simultaneously reducing both bias and variance. The code is available at https://github.com/2017LI/GSEC.git.

Feijiang Li, Zhenxiong Li, Jieting Wang +3

Institute of Big Data Science and Industry, Shanxi University · Key Laboratory of Evolutionary Science Intelligence of Shanxi Province · School of Artificial Intelligence, Shanxi University

Ensemble LearningClusteringImage ClusteringVariance Reduction
AskPDF
May 12, 2026

Pitfalls of Unlabeled Disagreement-Based Drift Detection in Streaming Tree Ensembles

Detecting concept drift in high-speed data streams remains challenging, particularly when models must operate on unlabeled data and avoid false alarms caused by benign shifts. While disagreement-based uncertainty has shown promise in neural networks, its adaptation to ensembles of incremental decision trees (IDTs) remains largely unexplored. We investigate this approach by constructing batch-specific disagreement measures via label flipping in ensemble members and evaluating their effectiveness for drift detection in tabular data streams. Our experiments show that, although this method performs well in ensembles of multi-layer perceptrons (MLPs), it consistently underperforms loss-based detectors when applied to IDTs. We attribute this behavior to the intrinsic rigidity of IDTs: learning primarily through structural expansion, with limited parameter adaptation, restricts model plasticity and prevents disagreement from reliably reflecting learning potential. Recent work on restructuring IDTs using their intrinsic decomposition into non-overlapping rules offers a promising direction for improving adaptability.

Lara Sá Neves, Afonso Lourenço, Lizy K. John +1

GECAD, ISEP, Polytechnic of Porto, Portugal · The University of Texas at Austin, USA

Concept DriftEnsemble Learning
AskPDF
May 12, 2026

Multi-Narrow Transformation as a Single-Model Ensemble: Boundary Conditions, Mechanisms, and Failure Modes

Single-model ensembles (SMEs) have attracted attention as a way to approximate some of the benefits of deep ensembles within a single network. However, under an approximately matched parameter budget, it remains unclear whether model capacity should be concentrated in a single wide pathway or redistributed into many narrow and independent members. We investigate this question through the Multi-Narrow (MN) transformation, which converts a baseline CNN into an SME of narrow, path-wise independent branches while approximately preserving the dominant parameter budget. We systematically compare Single-Wide and Multi-Narrow configurations across different training-data regimes, architectures, and datasets. The results show that the effectiveness of MN is strongly data-dependent: weakly partitioned or baseline-wide models are preferable in data-rich settings, whereas highly partitioned MN models consistently outperform the baseline in low-data settings. This tendency is reproduced across multiple CNN architectures and image-classification datasets, suggesting that it is not specific to a single benchmark or model family. Analysis of internal representations shows that high-MN models learn more diverse and less redundant path-wise features. In low-data regimes, this diversity is broadly utilized and improves generalization, whereas in data-rich regimes, training becomes imbalanced and prediction is dominated by a small subset of paths. These findings clarify when and why Multi-Narrow transformation is effective, and provide practical guidance for allocating model capacity between width and member multiplicity under a limited budget.

Tatsuhito Hasegawa, Taisei Tanaka

Graduate School of Engineering, University of Fukui, Bunkyo, Fukui, 9108507, Fukui, Japan

Neural Network GeneralizationEnsemble LearningConvolutional Neural Networks
AskPDF
May 11, 2026

Physics-Informed Teacher-Student Ensemble Learning for Traffic State Estimation with a Varying Speed Limit Scenario

Physics-informed deep learning (PIDL) neural networks have shown their capability as a useful instrument for transportation practitioners in utilizing the underlying relationship between the state variables for traffic state estimation (TSE). Another efficient traffic management approach is implementing varying speed limits (VSLs) on transportation corridors to control traffic and mitigate congestion. However, the existing training architecture of PIDL in the literature cannot accommodate the changing traffic characteristics on a freeway with VSL. To tackle this challenge, we propose a novel framework integrating teacher-student ensemble training with PIDL neural networks for TSE under VSL scenarios. The physics of flow conservation law is encoded locally in the teacher models by PIDL, and the student model uses a multi-layer perceptron classifier (MLP) to identify traffic characteristics and selects the ensemble member of PIDL neural networks for TSE. This integrated framework provides a natural solution for capturing the heterogeneity of VSL and accurately addressing the TSE problem. The case study results validate the proposed ensemble approach, demonstrating its superior performance in TSE compared to other popular baseline methods, as indicated by relative L2 error.

Archie J. Huang, Dongdong Wang, Shaurya Agarwal +3

Department of Building, Civil and Environmental Engineering, Concordia University, Montreal, Canada · Urban Artificial Intelligence Laboratory, University of Florida, Gainesville, USA · Department of Civil, Environmental and Construction Engineering, University of Central Florida, Orlando, USA

Ensemble LearningTraffic Flow EstimationPhysics-Informed MLTeacher-Student Learning
AskPDF
May 10, 2026

Efficient Ensemble Selection from Binary and Pairwise Feedback

Organizations increasingly deploy multiple AI systems across task domains, but selecting a small, high-performing ensemble can require costly model calls, benchmark runs, and human evaluation. We study this selection problem as a distributional variant of multiwinner voting: tasks are drawn from an unknown domain distribution, each task induces feedback over candidate experts, and a committee's value on a task is determined by its best-performing member. We analyze both binary feedback, for tasks with correct/incorrect outcomes, and pairwise feedback, for tasks where candidate outputs are compared by preference. In the binary setting, the induced objective is coverage. We give exhaustive-elicitation baselines and matching worst-case query lower bounds, and we design a failure-conditioned greedy algorithm that preserves the standard (1−1/e)(1-1/e) guarantee while obtaining instance-dependent query savings. In the pairwise setting, we study θθ-winning committees. We show that full-information optimization admits a PTAS but no EPTAS under Gap-ETH, and that the objective is monotone but not submodular. This motivates a weighted ordinal coverage relaxation, which is submodular and supports a failure-conditioned greedy oracle under pairwise feedback. We then convert this oracle back into θθ-type guarantees through finite-family auditing or a minimax wrapper. We also provide small-scale LLM experiments illustrating the predicted query savings and the role of complementarity in committee selection.

Tzeh Yuan Neoh, Nicholas Teh, Je Qin Chooi +2

Harvard University, USA · University of Oxford, UK

Pairwise ComparisonPairwise Preference EvaluationEnsemble LearningPreference Elicitation
AskPDF
May 9, 2026

When Can Human-AI Teams Outperform Individuals? Tight Bounds with Impossibility Guarantees

Human-AI teams fail to outperform their best member in 70% of studies, yet no theory specifies when complementarity is achievable. We derive tight bounds for the broad class of confidence-based aggregation rules by integrating signal detection theory with information-theoretic analysis, yielding four results: (1) a complementarity theorem (teams outperform individuals iff error correlation ρHM<ρ∗ρ_{HM} < ρ^*, with ρ∗≈aρ^* \approx a in the symmetric near-chance regime); (2) minimax bounds showing gains scale as Θ(Δd)Θ(\sqrt{Δd}) with metacognitive sensitivity difference; (3) an impossibility result proving no confidence-based aggregation rule achieves complementarity when ρHM≥ρ∗ρ_{HM} \geq ρ^*; and (4) multi-class generalization ρK∗≈ρ∗/K−1ρ^*_K \approx ρ^*/\sqrt{K-1}. Predictions match observed team accuracy (R=0.94R = 0.94 on ImageNet-16H, R=0.91R = 0.91 on CIFAR-10H) and the multi-class threshold scaling holds on human data (R=0.93R = 0.93, K=16K = 16), with robustness under non-Gaussian distributions. The framework explains why complementarity is rare and provides actionable design formulas; results apply to aggregation, not to interactive deliberation that generates novel answers.

Dongxin Guo, Jikun Wu, Siu-Ming Yiu

The University of Hong Kong, Hong Kong, China · Stellaris AI Limited, Hong Kong, China · Brain Investing Limited, Hong Kong, China

Human-AI ComplementarityEnsemble LearningHuman-AI Decision MakingHuman-AI Collaboration
AskPDF
May 8, 2026

PLACO: A Multi-Stage Framework for Cost-Effective Performance in Human-AI Teams

Human-AI teams play a pivotal role in improving overall system performance when neither the human nor the model can achieve such performance on their own. With the advent of powerful and accessible Generative AI models, several mundane tasks have morphed into Human-AI team tasks. From writing essays to developing advanced algorithms, humans have found that using AI assistance has led to an accelerated work pace like never before. In classification tasks, where the final output is a single hard label, it is crucial to address the combination of human and model output. Prior work elegantly solves this problem using Bayes rule, using the assumption that human and model output are conditionally independent given the ground truth. Specifically, it discusses a combination method to combine a single deterministic labeler (the human) and a probabilistic labeler (the classifier model) using the model's instance-level and the human's class-level calibrated probabilities.

Pranavkumar Mallela, Vinay Kumar, Shashi Shekhar Jha +1

Indian Institute of Technology Ropar

Ensemble LearningHuman-AI Decision MakingHuman-AI Collaboration
AskPDF
May 7, 2026

Hedging Memory Horizons for Non-Stationary Prediction via Online Aggregation

We study online prediction under distribution shift, where inputs arrive chronologically and outcomes are revealed only after prediction. In this setting, predictors must remain stable in quiet regimes yet adapt when regimes shift, and the right adaptation memory is unknown in advance. We propose MELO (Memory-hedged Exponentially Weighted Least-Squares Online aggregation), a model-agnostic method that hedges across adaptation scales: it wraps any non-anticipating base-predictor pool with exponentially weighted least-squares (EWLS) adaptation experts at multiple forgetting factors, and aggregates raw and EWLS-adapted forecasts with MLpol, a parameter-free online aggregation rule. Under boundedness conditions, we establish deterministic oracle inequalities showing that it competes with both the best raw predictor and the best bounded, time-varying affine combinations of the base predictions, up to a path-length-dependent tracking cost and a sublinear aggregation overhead. We evaluate MELO on French national electricity-load forecasting through the COVID-19 lockdown using no regime indicators, lockdown dates, or policy covariates. MELO reduces overall RMSE by 34.7% relative to base-only MLpol and achieves lower overall RMSE than a TabICL reference supplied with an external COVID policy-response covariate. Moreover, MELO requires only lightweight per-step recursive updates without model retraining.

Yutong Wang, Yannig Goude, Qiwei Yao

Department of Statistics, LSE · Électricité de France R&D

Ensemble LearningTime Series ForecastingNon-Stationary Time Series ForecastingElectricity Load ForecastingAdaptive LearningTemporal Distribution Shift
AskPDF
May 7, 2026

PACE: Prune-And-Compress Ensemble Models

Ensemble models achieve state-of-the-art performance on prediction tasks, but usually require aggregating a large number of weak learners. This can hinder deployment, interpretability, and downstream tasks such as robustness verification. Remedies to this issue fall into two main camps: pruning, which discards redundant learners, and compression, which generates new ones from scratch. We introduce PACE, a framework that interleaves these paradigms in a two-phase strategy. First, new learners are actively generated via a theoretically grounded procedure to enhance the diversity of the initial ensemble. When no more relevant learners can be found, a second phase of pruning is performed on this enriched ensemble. During both operations, PACE allows fine control on the faithfulness to the original ensemble. Experiments show that our method outperforms prior pruning and compression methods while offering principled control of faithfulness guarantees.

Fabian Akkerman, Julien Ferry, Théo Guyard +1

Industrial Engineering and Management Science University of Twente, Netherlands · SCALE-AI Chair and CIRRELT Polytechnique Montréal, Canada

Model CompressionEnsemble Learning
AskPDF
May 7, 2026

BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification

Large language models (LLMs) have recently been adapted to tabular prediction by serializing structured features into natural language, but their performance in low-data regimes remains limited compared to gradient-boosted decision trees (GBDTs). In this work, we revisit the boosting paradigm, traditionally associated with tree ensembles, and ask whether it can be applied as a general training principle for LLM fine-tuning. We propose BoostLLM, a framework that transforms parameter-efficient fine-tuning into a multi-round residual optimization process by training sequential PEFT adapters as weak learners. To incorporate tabular inductive bias, BoostLLM integrates decision-tree paths as a second input view alongside raw features; analysis reveals that the path view acts as a structured teacher in early training steps before the model shifts toward feature-driven representations. Empirically, BoostLLM achieves consistent improvements over standard fine-tuning across multiple LLM backbones and datasets, matching or surpassing XGBoost across a wide range of shot counts and outperforming GPT-4o-based methods with a 4B model. We further show that the framework scales: pairing with stronger tree models and extended boosting horizons yields additional gains under appropriate stabilization. These results suggest that boosting can serve as a general training principle for LLM fine-tuning, particularly in low-data regimes for structured data.

Yi-Siang Wang, Kuan-Yu Chen, Yu-Chen Den +1

SinoPac Holdings

Ensemble LearningLLM Fine-TuningTabular Classification
AskPDF
May 7, 2026

Enabling Federated Inference via Unsupervised Consensus Embedding

Cooperative inference across independently deployed machine learning models is increasingly desirable in distributed environments, as there is a growing need to leverage multiple models while keeping their data and model parameters private. However, existing cooperative frameworks typically rely on sharing input data, model parameters, or a common encoder, which limits their applicability in privacy-sensitive or cross-organizational settings. To address this challenge, we propose Consensus Embedding-based Federated Inference (CE-FI), a framework that enables pretrained models to cooperate at inference time without sharing model parameters or raw inputs and without assuming a common encoder. CE-FI introduces two components: a Consensus Embedding (CE) layer that maps heterogeneous intermediate representations into a common embedding space, and a Cooperative Output (CO) layer that produces predictions from these embeddings. Both layers are trained using shared unlabeled data only, so the cooperative stage does not require additional labeled data. Experiments on image classification benchmarks -- CIFAR-10 and CIFAR-100 -- under diverse non-IID conditions show that CE-FI consistently outperforms solo inference and performs comparably to conventional methods that require stronger sharing assumptions. Additional evaluations on text and time-series tasks indicate applicability beyond image classification, although performance depends on the ensemble strategy. Further analysis identifies representation alignment as the primary bottleneck.

Yui Hashimoto, Takayuki Nishio, Yuichi Kitagawa +1

School of Engineering, Institute of Science Tokyo, Tokyo 152-8550, Japan · Research & Development Group, Hitachi Ltd., Tokyo 185-8601, Japan

Ensemble LearningPrivacy-Preserving MLDistributed InferenceRepresentation AlignmentFederated Learning
AskPDF
May 6, 2026

Learned Neighbor Trust for Collaborative Deployment in Model-Agnostic Decentralized Learning

Many decentralized distillation methods are designed around training-time coordination, yet deploy each node in isolation even when more capable neighbors remain available at inference time. This is an incomplete objective for settings such as IoT, where devices are heterogeneous, data is scarce and skewed, and a node's strongest neighbors may far exceed its own local capacity. We study how nodes should train so that their predictions compose well at deployment, and how each node should learn whom to trust. Under a server-free, model-agnostic protocol where nodes exchange only queries and soft predictions, we propose Learned Neighbor Trust (LNTrust) wherein each node learns a compact trust function over its neighborhood from local validation evidence. This trust function gates auxiliary distillation during training and defines a deployment ensemble at inference, so that collaboration learned during training transfers directly to deployment. Across datasets and topologies, LNTrust improves deployed accuracy over the strongest output-only baseline by large margins while using significantly less communication than previous methods.

Michael Lanier, Luise Ge, Sastry Kompella +1

Department of Computer Science Washington University in St. Louis and Nexcepta

Ensemble LearningDecentralized LearningKnowledge Distillation
AskPDF
May 6, 2026

ITBoost: Information-Theoretic Trust for Robust Boosting

Gradient boosting remains a strong and widely used method for tabular data learning, but its performance often degrades when training labels are noisy. This behavior is largely related to the way boosting algorithms emphasize samples with large gradients, without explicitly accounting for whether such errors originate from informative hard cases or from unreliable labels. We address this issue by reconsidering how sample reliability is evaluated during boosting. Instead of relying on instantaneous error, we examine the evolution of each sample's residuals across iterations. Based on this insight, we propose Information-Theoretic Trust Boosting (ITBoost), which uses the Minimum Description Length principle to measure the complexity of residual trajectories. Samples whose residual patterns fluctuate in an irregular manner are treated as less trustworthy and are down-weighted during learning. Theoretically, we derive a tighter generalization bound for ITBoost under label noise. Empirical results on various tabular benchmarks indicate that ITBoost provides improved robustness in noisy environments over leading boosting and deep tabular models, while retaining best average performance on clean data.

Ye Su, Longlong Zhao, Diego Garcia-Gil +4

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Department of Software Engineering, Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI), University of Granada +1

Gradient-Boosted Decision TreesEnsemble LearningTabular MLLearning with Noisy Labels
AskPDF
May 2, 2026

Perturb and Correct: Post-Hoc Ensembles using Affine Redundancy

Models that are nearly indistinguishable on in-distribution data can behave very differently under distribution shift. We introduce Perturb-and-Correct (P&C), a post-hoc method for constructing epistemically diverse predictors from a single pretrained network. P&C applies random hidden-layer perturbations with a least-squares correction in the following affine layer, producing predictors that agree on calibration data while remaining free to disagree away from it. We analyze this mechanism through the post-correction residual that remains in the activation of the layer corrected using calibration data, yielding a leverage-based preservation bound and a covariance interpretation of ensemble disagreement. Empirically, P&C achieves a strong ID/OOD tradeoff across MuJoCo dynamics prediction and CIFAR-10 OOD detection, and scales to a pretrained ViT-B/16 on ImageNet-1K. In matched comparisons with Deep Ensembles, P&C substantially reduces construction and storage cost. Our findings highlight the potential in further exploiting overparameterization as a strength of deep learning models.

Eleanor Quint

Department of Computing University of Nebraska–Lincoln Lincoln, NE 68508, USA

Distribution Shift RobustnessEnsemble LearningOOD GeneralizationOOD Detection
AskPDF
Apr 29, 2026

Smart Ensemble Learning Framework for Predicting Groundwater Heavy Metal Pollution

Groundwater in the Densu Basin is increasingly threatened by heavy metal contamination, but conventional methods fail to capture the statistical complexity and spatial heterogeneity of pollution indicators. A key challenge is modelling the Heavy Metal Pollution Index (HPI), which is typically skewed and affected by correlated contaminants, leading to biased predictions without transformation. This study develops a predictive framework integrating response transformations with nested cross-validated ensemble machine learning. Three transformations (raw, log, and Gaussian copula) were applied to HPI and evaluated across six learners: support vector regression (SVM), kk-nearest neighbours (k-NN), CART, Elastic Net, kernel ridge regression, and a stacked Lasso ensemble. Raw-scale models produced deceptively high fits (Elastic Net and stacked ensemble R2≈1.0R^2 \approx 1.0), suggesting over-optimism. The log transformation stabilised variance (SVM: R2=0.93R^2 = 0.93, RMSE =0.18= 0.18; k-NN: R2=0.92R^2 = 0.92, RMSE =0.20= 0.20). The Gaussian copula gave the most reliable results: stacked ensemble R2=0.96R^2 = 0.96 (RMSE =0.19= 0.19), with other learners maintaining high accuracy. Copula-based models improved residuals and produced spatially plausible maps. DBSCAN clustering revealed Fe and Mn as primary HPI contributors, consistent with regional hydrogeochemistry. Limitations include reliance on random (not spatial) cross-validation and basin-specific scope. Future work should explore spatial validation and other geological settings. Overall, distribution-aware ensembles with clustering diagnostics offer robust, interpretable assessments of groundwater contamination.

T. Ansah-Narh, G. Y. Afrifa, J. B. Tandoh +6

Ensemble Learning
AskPDF
Apr 27, 2026

A Divergence-Based Method for Weighting and Averaging Model Predictions

This paper uses a minimum divergence framework to introduce a new way of calculating model weights that can be used to average probabilistic predictions from statistical and machine learning models. The method is general and can be applied regardless of whether the models under consideration are fit to data using frequentist, Bayesian, or some other fitting method. The proposed method is motivated in two different ways and is shown empirically to perform better than or on a par with standard model averaging methods, including model stacking and model averaging that relies on Akaike-style negative exponentiated model weighting, especially when the sample size is small. Our theoretical analysis explains why the method has a small-sample advantage.

Olav Benjamin Vassend

University of Inland Norway

Ensemble Learning
AskPDF
Apr 27, 2026

Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classification

Training reliable respiratory sound classification models remains challenging due to the limited size and subject diversity of datasets. Ensemble methods can improve robustness, but when base models are trained on identical data, models tend to overfit and produce highly correlated predictions, thereby reducing the effectiveness of ensembling. In this work, we investigate a meta-ensemble learning methodology that enhances prediction diversity by training base models on diverse data splits and combining their outputs through a trained meta-model. Specifically, we train base models on the ICBHI dataset using two data split settings: fixed 80-20% split and five-fold cross-validation split, under two data granularity settings: patient- and sample-level. The resulting diversity in base model predictions enables the meta-model to better generalize. Our approach achieves new state-of-the-art performance on the ICBHI benchmark, reaching a Score of 66.49% and showing improved generalization on two out-of-distribution datasets, indicating its potential applicability to real-world clinical data.

June-Woo Kim, Miika Toikkanen, Heejoon Koo +3

Department of Electronic Engineering, Wonkwang University, Republic of Korea · RSC LAB, MODULABS, Republic of Korea · AICU Global Inc., Republic of Korea +1

Ensemble LearningAudio ClassificationRespiratory Sound Classification
AskPDF
Apr 25, 2026

A Hierarchical Ensemble Inference Pipeline for Robust White Blood Cell Classification Under Domain Shifts

Automated white blood cell (WBC) classification is essential for scalable leukaemia screening. However, real-world deployment is challenged by domain shifts caused by staining protocols, scanner characteristics, and inter-laboratory variability, which often degrade model performance. The White Blood Cell Classification Challenge (WBCBench) at ISBI 2026 aims to advance robust WBC recognition, with a focus on accurately identifying blast cells and other clinically critical rare subtypes. We propose a memory-augmented, hierarchical ensemble pipeline for WBC classification under domain shifts, leveraging a feature bank and a DinoBloom backbone fine-tuned with LoRA. Our three-stage inference hierarchy combines k-nearest neighbors (kNN) retrieval at each level, reducing over-reliance on any single decision. Evaluated on the WBCBench dataset, our method ranks within the top ten by macro F1-score in the final testing phase.

Ruyi Dai, Tingkwong Ng, Hao Chen

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, China · Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology, Hong Kong, China · Division of Life Science, Hong Kong University of Science and Technology, Hong Kong, China +2

Hierarchical ClassificationEnsemble LearningDomain GeneralizationMedical Image ClassificationWhite Blood Cell Classification
AskPDF
Apr 24, 2026

Conformalized Super Learner

The Super Learner (SL) is a widely used ensemble method that combines point predictions from a library of learners based on their predictive performance. Interval predictions are of considerable practical interest because they allow uncertainty in predictions produced by an individual learner or an ensemble to be quantified. Several methods have been proposed for constructing interval predictions based on the SL, however, these approaches are typically justified using asymptotic arguments or rely on computationally intensive procedures such as the bootstrap. Conformal prediction (CP) is a machine learning framework for constructing prediction intervals with finite-sample and asymptotic coverage guarantees under mild conditions. We propose coupling CP with the SL through a natural construction that mirrors the original SL framework, using individual learner weights and combining learner-specific conformity scores via a weighted majority vote. We characterize the properties of the resulting SL-based prediction intervals for continuous outcomes. We cover settings under exchangeability, potential violations of exchangeability, and data-generating mechanisms exhibiting heteroscedasticity, sparsity, and other forms of distributional heterogeneity. A comprehensive simulation study shows that the conformalized SL achieves valid finite-sample coverage with competitive performance relative to the true data-generating mechanism. A central contribution of this work is an application to predicting creatinine levels using socio-demographic, biometric, and laboratory measurements. This example demonstrates the benefits of an ensemble with carefully selected learners designed to capture key aspects of complex regression functions, including non-linear effects, interactions, sparsity, heteroscedasticity, and robustness to outliers.

Zhanli Wu, Fabrizio Leisen, Miguel-Angel Luque-Fernandez +1

Department of Mathematics King’s College London London, UK · Department of Statistics and Operations Research University of Granada Granada, Spain · Department of Statistical Science University College London London, UK

Ensemble LearningConformal PredictionHeteroscedastic RegressionRegression Prediction Intervals
AskPDF
Apr 23, 2026

Promoting Simple Agents: Ensemble Methods for Event-Log Prediction

We compare lightweight automata-based models (n-grams) with neural architectures (LSTM, Transformer) for next-activity prediction in streaming event logs. Experiments on synthetic patterns and five real-world process mining datasets show that n-grams with appropriate context windows achieve comparable accuracy to neural models while requiring substantially fewer resources. Unlike windowed neural architectures, which show unstable performance patterns, n-grams provide stable and consistent accuracy. While we demonstrate that classical ensemble methods like voting improve n-gram performance, they require running many agents in parallel during inference, increasing memory consumption and latency. We propose an ensemble method, the promotion algorithm, that dynamically selects between two active models during inference, reducing overhead compared to classical voting schemes. On real-world datasets, these ensembles match or exceed the accuracy of non-windowed neural models with lower computational cost.

Benedikt Bollig, Matthias Függer, Thomas Nowak +1

Université Paris-Saclay, CNRS, ENS Paris-Saclay, LMF, Gif-sur-Yvette, France · Institut Universitaire de France, Paris, France · Direction Générale de l’Armement, Paris, France

Next-Action PredictionEnsemble LearningProcess MiningLanguage ModelingNext-Event Prediction
AskPDF
Apr 22, 2026

A Hierarchical Ensemble Pipeline for Anomaly Detection in ESA Satellite Telemetry

A hierarchical ensemble pipeline is introduced to address anomaly detection in multivariate telemetry data provided by European Space Agency (ESA). The method integrates shapelet-based and statistical feature extraction, per-channel modeling, intra-channel stacking, and a final cross-channel aggregation. The pipeline is trained and validated using time-series cross-validation and two-level masking strategies to prevent information leakage. Results on the European Space Agency Anomaly Detection Benchmark (ESA-ADB) challenge demonstrate strong generalization, highlighting the effectiveness of hierarchical modeling in detecting subtle anomalies in realistic satellite telemetry.

Lorenzo Riccardo Allegrini, Geremia Pompei

ContinualIST, Pisa, Italy · University of Pisa, Department of Computer Science, Pisa, Italy

Ensemble LearningTime-Series Anomaly DetectionAnomaly Detection
AskPDF
Apr 22, 2026

SDNGuardStack: An Explainable Ensemble Learning Framework for High-Accuracy Intrusion Detection in Software-Defined Networks

Software-Defined Networking (SDN) is another technology that has been developing in the last few years as a relevant technique to improve network programmability and administration. Nonetheless, its centralized design presents a major security issue, which requires effective intrusion detection systems. The SDN-specific machine learning-based intrusion detection system described in this paper is innovative because it is trained and tested on the InSDN dataset which models attack scenarios and realistic traffic patterns in SDN. Our approach incorporates a comprehensive preprocessing pipeline, feature selection via Mutual Information, and a novel ensemble learning model, SDNGuardStack, which combines multiple base learners to enhance detection accuracy and efficiency. In addition, we include explainable AI methods, including SHAP to add transparency to model predictions, which helps security analysts respond to incidents. The experiments prove that SDNGuard-Stack has an accuracy rate of 99.98% and a Cohen Kappa of 0.9998, surpassing other models, and at the same time being interpretable and practically executable. It is interesting to see such features like Flow ID, Bwd Header Len, and Src Port as the most important factors in the model predictions. The work is a step towards closing the gap between performance intrusion detection and realistic deployment in SDN, which will lead to the creation of secure and resilient network infrastructures.

Ashikuzzaman, Md. Saifuzzaman Abhi, Mahabubur Rahman +3

Dept. of CSE University of Barishal Barishal, Bangladesh · Dept. of CSE University of Scholars Dhaka, Bangladesh

Ensemble LearningNetwork Intrusion DetectionExplainable Intrusion Detection
AskPDF
Apr 20, 2026

Improving reproducibility by controlling random seed stability in machine learning based estimation via bagging

Predictions from machine learning algorithms can vary across random seeds, inducing instability in downstream debiased machine learning estimators. We formalize random seed stability via a concentration condition and prove that subbagging guarantees stability for any bounded-outcome regression algorithm. We introduce a new cross-fitting procedure, adaptive cross-bagging, which simultaneously eliminates seed dependence from both nuisance estimation and sample splitting in debiased machine learning. Numerical experiments confirm that the method achieves the targeted level of stability whereas alternatives do not. Our method incurs a small computational penalty relative to standard practice whereas alternative methods incur large penalties.

Nicholas Williams, Alejandro Schuler

Division of Biostatistics, University of California, Berkeley, California, U.S.A.

Cross-FittingDebiased MLEnsemble LearningAlgorithmic StabilityML Reproducibility
AskPDF
Apr 19, 2026

STRIKE: Additive Feature-Group-Aware Stacking Framework for Credit Default Prediction

Credit risk default prediction remains a cornerstone of risk management in the financial industry. The task involves estimating the likelihood that a borrower will fail to meet debt obligations, an objective critical for lending decisions, portfolio optimization, and regulatory compliance. Traditional machine learning models such as logistic regression and tree-based ensembles are widely adopted for their interpretability and strong empirical performance. However, modern credit datasets are high-dimensional, heterogeneous, and noisy, increasing overfitting risk in monolithic models and reducing robustness under distributional shift. We introduce STRIKE (Stacking via Targeted Representations of Isolated Knowledge Extractors), a feature-group-aware stacking framework for structured tabular credit risk data. Rather than training a single monolithic model on the complete dataset, STRIKE partitions the feature space into semantically coherent groups and trains independent learners within each group. This decomposition is motivated by an additive perspective on risk modeling, where distinct feature sources contribute complementary evidence that can be combined through a structured aggregation. The resulting group-specific predictions are integrated through a meta-learner that aggregates signals while maintaining robustness and modularity. We evaluate STRIKE on three real-world datasets spanning corporate bankruptcy and consumer lending scenarios. Across all settings, STRIKE consistently outperforms strong tree-based baselines and conventional stacking approaches in terms of AUC-ROC. Ablation studies confirm that performance gains stem from meaningful feature decomposition rather than increased model complexity. Our findings demonstrate that STRIKE is a stable, scalable, and interpretable framework for credit risk default prediction tasks.

Swattik Maiti, Ritik Pratap Singh, Fardina Fathmiul Alam

Science Academy, University of Maryland, College Park MD, USA · Department of Computer Science, University of Maryland, College Park MD, USA

Ensemble LearningTabular Prediction
AskPDF
Apr 16, 2026

Collaborative Filtering Through Weighted Similarities of User and Item Embeddings

In recent years, neural networks and other complex models have dominated recommender systems, often setting new benchmarks for state-of-the-art performance. Yet, despite these advancements, award-winning research has demonstrated that traditional matrix factorization methods can remain competitive, offering simplicity and reduced computational overhead. Hybrid models, which combine matrix factorization with newer techniques, are increasingly employed to harness the strengths of multiple approaches. This paper proposes a novel ensemble method that unifies user-item and item-item recommendations through a weighted similarity framework to deliver top-N recommendations. Our approach is distinctive in its use of shared user and item embeddings for both recommendation strategies, simplifying the architecture and enhancing computational efficiency. Extensive experiments across multiple datasets show that our method achieves competitive performance and is robust in varying scenarios that favor either user-item or item-item recommendations. Additionally, by eliminating the need for embedding-specific fine-tuning, our model allows for the seamless reuse of hyperparameters from the base algorithm without sacrificing performance. This results in a method that is both efficient and easy to implement. Our open-source implementation is available at https://github.com/UFSCar-LaSID/weighted-sims-recommender.

Pedro R. Pires, Rafael T. Sereicikas, Gregorio F. Azevedo +1

Federal University of São Carlos

Ensemble LearningRecommender SystemsCollaborative Filtering
AskPDF
Apr 15, 2026

MyoVision: A Mobile Research Tool and NEATBoost-Attention Ensemble Framework for Real Time Chicken Breast Myopathy Detection

Woody Breast (WB) and Spaghetti Meat (SM) myopathies significantly impact poultry meat quality, yet current detection methods rely either on subjective manual evaluation or costly laboratory-grade imaging systems. We address the problem of low-cost, non-destructive multi-class myopathy classification using consumer smartphones. MyoVision is introduced as a mobile transillumination imaging framework in which 14-bit RAW images are captured and structural texture descriptors indicative of internal tissue abnormalities are extracted. To classify three categories (Normal, Woody Breast, Spaghetti Meat), we propose a NEATBoost-Attention Ensemble model, which is a neuroevolution-optimized weighted fusion of LightGBM and attention-based MLP models. Hyperparameters are automatically discovered using NeuroEvolution of Augmenting Topologies (NEAT), eliminating manual tuning and enabling architecture diversity for small tabular datasets. On a dataset of 336 fillets collected from a commercial processing facility, our method achieves 82.4% test accuracy (F1 = 0.83), outperforming conventional machine learning and deep learning baselines and matching performance reported by hyperspectral imaging systems costing orders of magnitude more. Beyond classification performance, MyoVision establishes a reproducible mobile RGB-D acquisition pipeline for multimodal meat quality research, demonstrating that consumer-grade imaging can support scalable internal tissue assessment.

Chaitanya Pallerla, Siavash Mahmoudi, Dongyi Wang

Department of Biological and Agricultural Engineering, University of Arkansas, Fayetteville, AR · Department of Food Science, University of Arkansas, Fayetteville, AR

Ensemble LearningComputational Imaging
AskPDF
Apr 15, 2026

Minimax Optimality and Spectral Routing for Majority-Vote Ensembles under Markov Dependence

Majority-vote ensembles achieve variance reduction by averaging over diverse, approximately independent base learners. When training data exhibits Markov dependence, as in time-series forecasting, reinforcement learning (RL) replay buffers, and spatial grids, this classical guarantee degrades in ways that existing theory does not fully quantify. We provide a minimax characterization of this phenomenon for discrete classification in a fixed-dimensional Markov setting, together with an adaptive algorithm that matches the rate on a graph-regular subclass. We first establish an information-theoretic lower bound for stationary, reversible, geometrically ergodic chains in fixed ambient dimension, showing that no measurable estimator can achieve excess classification risk better than Ω(\Tmix/n)Ω(\sqrt{\Tmix/n}). We then prove that, on the AR(1) witness subclass underlying the lower-bound construction, dependence-agnostic uniform bagging is provably suboptimal with excess risk bounded below by Ω(\Tmix/n)Ω(\Tmix/\sqrt{n}), exhibiting a \Tmix\sqrt{\Tmix} algorithmic gap. Finally, we propose adaptive spectral routing, which partitions the training data via the empirical Fiedler eigenvector of a dependency graph and achieves the minimax rate O(\Tmix/n)\mathcal{O}(\sqrt{\Tmix/n}) up to a lower-order geometric cut term on a graph-regular subclass, without knowledge of \Tmix\Tmix. Experiments on synthetic Markov chains, 2D spatial grids, the 128-dataset UCR archive, and Atari DQN ensembles validate the theoretical predictions. Consequences for deep RL target variance, scalability via Nyström approximation, and bounded non-stationarity are developed as supporting material in the appendix.

Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University

Ensemble LearningMajority VotingClassificationMinimax Estimation
AskPDF
Apr 14, 2026

Shapley Value-Guided Adaptive Ensemble Learning for Explainable Financial Fraud Detection with U.S. Regulatory Compliance Validation

Financial crime costs U.S. institutions over $32 billion each year. Although AI tools for fraud detection have become more advanced, their use in real-world systems still faces a major obstacle: many of these models operate as black boxes that cannot provide the transparent, auditable explanations required by regulations such as OCC Bulletin 2011-12 and Federal Reserve SR 11-7. This study makes three main contributions. First, it offers a thorough evaluation of explanation quality across faithfulness (sufficiency and comprehensiveness at k=5, 10, and 15) and stability (Kendall's W across 30 bootstrap samples). XGBoost paired with TreeExplainer achieves near-perfect stability (W=0.9912), while LSTM with DeepExplainer shows weak results (W=0.4962). Second, the paper introduces the SHAP-Guided Adaptive Ensemble (SGAE), which dynamically adjusts per-transaction ensemble weights based on SHAP attribution agreement, achieving the highest AUC-ROC among all tested models (0.8837 held-out; 0.9245 cross-validation). Third, a complete three-architecture evaluation of LSTM, Transformer, and GNN-GraphSAGE on the full 590,540-transaction IEEE-CIS dataset is provided, with GNN-GraphSAGE achieving AUC-ROC 0.9248 and F1=0.6013. All results are mapped directly to OCC, SR 11-7, and BSA-AML regulatory compliance requirements.

Mohammad Nasir Uddin, Md Munna Aziz

Visual Data Analyst and Applied AI Researcher, Westcliff University, Irvine, CA, USA · Data Analyst and Applied AI Researcher, Westcliff University, Irvine, CA, USA

Financial Fraud DetectionEnsemble LearningExplainability EvaluationSHAP Feature AttributionAlgorithmic Auditing
AskPDF
Apr 13, 2026

Training-Free Model Ensemble for Single-Image Super-Resolution via Strong-Branch Compensation

Single-image super-resolution has progressed from deep convolutional baselines to stronger Transformer and state-space architectures, yet the corresponding performance gains typically come with higher training cost, longer engineering iteration, and heavier deployment burden. In many practical settings, multiple pretrained models with partially complementary behaviors are already available, and the binding constraint is no longer architectural capacity but how effectively their outputs can be combined without additional training. Rather than pursuing further architectural redesign, this paper proposes a training-free output-level ensemble framework. A dual-branch pipeline is constructed in which a Hybrid attention network with TLC inference provides stable main reconstruction, while a MambaIRv2 branch with geometric self-ensemble supplies strong compensation for high-frequency detail recovery. The two branches process the same low-resolution input independently and are fused in the image space via a lightweight weighted combination, without updating any model parameters or introducing an additional trainable module. As our solution to the NTIRE 2026 Image Super-Resolution (×4\times 4) Challenge, the proposed design consistently improves over the base branch and slightly exceeds the pure strong branch in PSNR at the best operating point under a unified DIV2K bicubic ×4\times 4 evaluation protocol. Ablation studies confirm that output-level compensation provides a low-overhead and practically accessible upgrade path for existing super-resolution systems.

Gengjia Chang, Xining Ge, Weijun Yuan +4

Hefei University of Technology · Hangzhou Dianzi University · Jinan University +2

Image Super-ResolutionEnsemble Learning
AskPDF
Apr 13, 2026

Exact Certification of Neural Networks and Partition Aggregation Ensembles against Label Poisoning

Label-flipping attacks, which corrupt training labels to induce misclassifications at inference, remain a major threat to supervised learning models. This drives the need for robustness certificates that provide formal guarantees about a model's robustness under adversarially corrupted labels. Existing certification frameworks rely on ensemble techniques such as smoothing or partition-aggregation, but treat the corresponding base classifiers as black boxes, yielding overly conservative guarantees. We introduce EnsembleCert, the first certification framework for partition-aggregation ensembles that utilizes white-box knowledge of the base classifiers. Concretely, EnsembleCert yields tighter guarantees than black-box approaches by aggregating per-partition white-box certificates to compute ensemble-level guarantees in polynomial time. To extract white-box knowledge from the base classifiers efficiently, we develop ScaLabelCert, a method that leverages the equivalence between sufficiently wide neural networks and kernel methods using the neural tangent kernel. ScaLabelCert yields the first exact, polynomial-time calculable certificate for neural networks against label-flipping attacks. EnsembleCert is either on par, or significantly outperforms the existing partition-based black box certificates. Exemplary, on CIFAR-10, our method can certify upto +26.5% more label flips in median over the test set compared to the existing black-box approach while requiring 100 times fewer partitions, thus, challenging the prevailing notion that heavy partitioning is a necessity for strong certified robustness.

Ajinkya Mohgaonkar, Lukas Gosch, Mahalakshmi Sabanayagam +2

School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Data Science Institute, Technical University of Munich · Munich Center for Machine Learning, Germany +1

Certified RobustnessNeural Network VerificationAdversarial AttacksEnsemble Learning
AskPDF