Evaluation of Bagging Predictors with Kernel Density Estimation and Bagging Score

Philipp Seitz; Jan Schmitt; Andreas Schiffler

cs.LGApr 4, 2026

Evaluation of Bagging Predictors with Kernel Density Estimation and Bagging Score

Authors: Philipp Seitz, Jan Schmitt, Andreas Schiffler

Organizations: Institute of Digital Engineering, Technical University of Applied Sciences Würzburg-Schweinfurt, Schweinfurt, Bavaria, Germany

Abstract

For a larger set of predictions of several differently trained machine learning models, known as bagging predictors, the mean of all predictions is taken by default. Nevertheless, this proceeding can deviate from the actual ground truth in certain parameter regions. An approach is presented to determine a representative y_BS from such a set of predictions using Kernel Density Estimation (KDE) in nonlinear regression with Neural Networks (NN) which simultaneously provides an associated quality criterion beta_BS, called Bagging Score (BS), that reflects the confidence of the obtained ensemble prediction. It is shown that working with the new approach better predictions can be made than working with the common use of mean or median. In addition to this, the used method is contrasted to several approaches of nonlinear regression from the literatur, resulting in a top ranking in each of the calculated error values without using any optimization or feature selection technique.

Figures & tables

Panel of Figure 1 — Figure 1. Example for expected and unexpected usual asymmetric cumulative distribution when evaluating bagging predictors where mean and median are likely to deviate from the actual ground truth.

Panel of Figure 2 — Figure 2. Use case of determining the ensemble prediction and its Bagging Score by two examples of a synthetic function $f_{GT}(x)$ . Top: Example for good normal distribution in prediction set. Bottom: Example for shifted distribution.

Method	$R^{2}$	$RMSE$	$MAPE$	$MAE$
$\tilde{y}_{MEAN}$	0.921	5.64	16.6	4.49
$\tilde{y}_{MEDIAN}$	0.928	4.89	12.6	3.66
$\tilde{y}_{BS}$	0.934	4.52	10.8	3.30

Table 1. Comparison to MEAN and MEDIAN. The best results are highlighted in bold.

Year	Ref.	Model	$R^{2}$	$RMSE$	$MAPE$	$MAE$
1998	( 20 )	ANN with manual optimization.	0.91	–	–	–
2010	( 21 )	Multiple additive regression tree with manual optimization.	0.95	4.95	13.89	–
2013	( 22 )	Wavelet gradient boosted ANN with manual optimization.	0.95	5.75	–	4.83
2013	( 23 )	ANN without optimization.	0.93	6.33	15.3	4.41
2014	( 24 )	Ensemble stacking without tuning.	–	5.08	11.97	3.52
2014	( 25 )	Genetic weighted pyramid operation tree with manual optimization.	–	6.379	16.1	4.79

Table 2. Comparison to the literature with Concrete dataset. The best results are highlighted in bold.

Ensemble Learning

Ensemble learning combines multiple predictors to improve accuracy, robustness, calibration, or uncertainty estimates.

See all 272 papers

A Kernel Nonconformity Score for Multivariate Conformal Prediction
Apr 23, 2026Louis Meyer, Wenkai XuConformal Prediction Kernel Methods
University of Warwick
On the Use of Bagging for Local Intrinsic Dimensionality Estimation
Mar 25, 2026Kristóf Péter, Ricardo J. G. B. Campello, James Bailey +1Intrinsic Dimensionality Variance Reduction
New Jersey Institute of Technology Newark, New Jersey, United States
Robust Bayes-Assisted Conformal Prediction
Jul 5, 2026Kianoosh Ashouritaklimi, Stefano Cortinovis, François CaronDistribution Shift Robustness Conformal Prediction
Department of Statistics, University of Oxford

Apr 23, 2026J/K move · Enter open · S save

A Kernel Nonconformity Score for Multivariate Conformal Prediction

Louis Meyer, Wenkai Xu

University of Warwick

Conformal Prediction Kernel Methods

Open paper AskPDF arXiv

Multivariate conformal prediction requires nonconformity scores that compress residual vectors into scalars while preserving certain implicit geometric structure of the residual distribution. We introduce a Multivariate Kernel Score (MKS) that produces prediction regions that explicitly adapt to this geometry. We show that the proposed score resembles the Gaussian process posterior variance, unifying Bayesian uncertainty quantification with the coverage guarantees of frequentist-type. Moreover, the MKS can be decomposed into an anisotropic Maximum Mean Discrepancy (MMD) that interpolates between kernel density estimation and covariance-weighted distance. We prove finite-sample coverage guarantees and establish convergence rates that depend on the effective rank of the kernel-based covariance operator rather than the ambient dimension, enabling dimension-free adaptation. On regression tasks, the MKS reduces the volume of prediction region significantly, compared to ellipsoidal baselines while maintaining nominal coverage, with larger gains at higher dimensions and tighter coverage levels.

Abstract

Ensemble Learning

Explore similar work

A Kernel Nonconformity Score for Multivariate Conformal Prediction

On the Use of Bagging for Local Intrinsic Dimensionality Estimation

Robust Bayes-Assisted Conformal Prediction