Exact Finite-Sample Variance Decomposition of Subagging: A Spectral Filtering Perspective

Ye Su; Mingrui Ye; Yining Wang; Jipeng Guo; Yong Liu

cs.LGApr 12, 2026

Exact Finite-Sample Variance Decomposition of Subagging: A Spectral Filtering Perspective

Authors: Ye Su, Mingrui Ye, Yining Wang, Jipeng Guo, Yong Liu

Organizations: Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences Shenzhen 518055, China · School of Mathematics Renmin University of China Beijing 100872, China · College of Information Science and Technology Beijing University of Chemical Technology Beijing 100029, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing 100872, China

Abstract

Standard resampling ratios (e.g., $α\approx 0.632$ ) are widely used as default baselines in ensemble learning for three decades. However, how these ratios interact with a base learner's intrinsic functional complexity in finite samples lacks a exact mathematical characterization. We leverage the Hoeffding-ANOVA decomposition to derive the first exact, finite-sample variance decomposition for subagging, applicable to any symmetric base learner without requiring asymptotic limits or smoothness assumptions. We establish that subagging operates as a deterministic low-pass spectral filter: it preserves low-order structural signals while attenuating $c$ -th order interaction variance by a geometric factor approaching $α^c$ . This decoupling reveals why default baselines often under-regularize high-capacity interpolators, which instead require smaller $α$ to exponentially suppress spurious high-order noise. To operationalize these insights, we propose a complexity-guided adaptive subsampling algorithm, empirically demonstrating that dynamically calibrating $α$ to the learner's complexity spectrum consistently improves generalization over static baselines.

Figures & tables

Figure 1: Structural shift of the optimal subsampling ratio

\alpha^{*}

across nine evaluated datasets. Blue markers represent the low-complexity regime (shallow trees), clustering near the classical heuristic

\alpha\approx 0.632

(dotted line) to preserve structural signals. Red markers represent the high-complexity regime (unpruned deep trees), demonstrating a systematic shift toward smaller subsampling ratios. This empirical shift corroborates the comparative statics in Theorem 5.5 , highlighting the necessity of stringent geometric filtering for high-capacity interpolators.

Datasets	Metric	Regime 1 (Low-Complexity)			Regime 2 (High-Complexity)
Datasets	Metric	0.632	0.8	CGAS	0.632	0.8	CGAS
yprop_4_1	MSE	0.9672 ∗	0.9686 ∗	0.9670	1.0192 ∗	1.0427 ∗	0.9923
yprop_4_1	MAE	0.6605	0.6610 ∗	0.6605	0.7851 ∗	0.7936 ∗	0.7745
wine_quality	MSE	0.7231 ∗	0.7276 ∗	0.7221	0.9326 ∗	0.9821 ∗	0.8843
wine_quality	MAE	0.6671 ∗	0.6695 ∗	0.6666	0.7663 ∗	0.7859 ∗	0.7457
superconduct	MSE	0.2881 ∗	0.2908 ∗	0.2876	0.8628 ∗	0.9650 ∗	0.7468

Table 1: Out-of-sample MSE and MAE across various datasets under two distinct functional complexity regimes. The best performance metrics in each regime are highlighted in bold .

Datasets	Metric	Regime 1 (Low-Complexity)			Regime 2 (High-Complexity)
Datasets	Metric	0.632	0.8	CGAS	0.632	0.8	CGAS
yprop_4_1	MSE	0.9649 ∗	0.9660 ∗	0.9647	1.3692 ∗	1.5465 ∗	1.0295
yprop_4_1	MAE	0.6622 ∗	0.6647 ∗	0.6618	0.9137 ∗	0.9724 ∗	0.7900
wine_quality	MSE	0.6449 ∗	0.6415	0.6420	1.1593 ∗	1,2971 ∗	0.9195
wine_quality	MAE	0.6297 ∗	0.6267	0.6267	0.8543 ∗	0.9031 ∗	0.7608
superconduct	MSE	0.1449 ∗	0.1360 ∗	0.1304	0.9728 ∗	1.0731 ∗	0.7637

Table 2: Out-of-sample MSE and MAE for KNN across various datasets under two distinct functional complexity regimes. The best performance metrics in each regime are highlighted in bold .

Figure 2: Relative error reduction in out-of-sample MSE (left) and MAE (right) achieved by the theoretically calibrated

\text{RF}^{*}

(bootstrap with max_samples=

\alpha^{*}

) over the standard RF baseline (default bootstrap). The grey bars indicate marginal yet strictly non-negative improvements under the low-complexity regime. The red bars highlight substantial and consistent performance gains under the high-complexity regime, empirically validating that restricting the sampling fraction dynamically suppresses high-order interpolation noise even within a standard bootstrap framework.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition	Domain/Space
1. Sets and Spaces
$\mathcal{X},\mathcal{Y}$	Feature input space and target continuous output space	-
$\mathcal{Z}$	The measurable product space $\mathcal{X}\times\mathcal{Y}$	-
$\mathcal{D}_{n}$	The finite training dataset $\{Z_{1},\dots,Z_{n}\}$	$\mathcal{Z}^{n}$
$[n],[k]$	The integer index sets $\{1,2,\dots,n\}$ and $\{1,2,\dots,k\}$	$\mathbb{Z}^{+}$
$\binom{[n]}{k},\binom{[n]}{c}$	The collection of all $k$ -element or $c$ -element subsets drawn from $[n]$	$2^{[n]}$

Appendix

Table 3: Summary of Global Symbols and Notations

Ensemble Learning

Ensemble learning combines multiple predictors to improve accuracy, robustness, calibration, or uncertainty estimates.

See all 272 papers

Feature Bagging Provides Stability
Jul 29, 2026Yuheng Ma, Qiang SunEnsemble Learning Algorithmic Stability
School of Statistics, East China Normal University · University of Toronto · MBZUAI
The Interplay Between Interpolation and Aggregation in Regression: Optimal Sample Complexity
May 28, 2026Mikael Møller Høgsgaard, Kasper Green Larsen, Liang-Yu ZouStatistical Learning Theory
Department of Computer Science, Aarhus University, Aarhus, Denmark · Department of Statistics, University of Oxford, Oxford, United Kingdom
Useful to Whom? The Easy-versus-Coverage Boundary Moves with the Learner
Sep 22, 2026Yangze Liu, Xiao-Long Yin, Zhongyi HanTraining Data Selection Coreset Selection
Shandong University · Nanjing University

Jul 29, 2026J/K move · Enter open · S save

Feature Bagging Provides Stability

Yuheng Ma, Qiang Sun

School of Statistics, East China Normal University · University of Toronto · MBZUAI

Ensemble Learning Algorithmic Stability Statistical Learning Theory

Open paper AskPDF arXiv

We study feature bagging through the lens of algorithmic stability. Feature bagging is an ensemble strategy that aggregates base learners trained on randomly subsampled feature subsets, possibly in a data-dependent manner. We introduce feature instability (FI), the feature-axis analogue of instance instability (II), which measures sensitivity to removing a single feature. Smaller values of II or FI correspond to stronger stability, and our experiments show that FI captures generalization-relevant information complementary to II. Within this framework, we analyze feature bagging in both a parametric linear model and a model-free setting inspired by recursive feature subsampling in random forests. In both settings, we establish formal guarantees showing that feature bagging improves the relevant stability relative to its non-bagged counterpart, with larger improvements under more aggressive subsampling. We further show that a modest number of bagging rounds is sufficient to approach the infinite-bagging stability level.

Abstract

Ensemble Learning

Explore similar work

Feature Bagging Provides Stability

The Interplay Between Interpolation and Aggregation in Regression: Optimal Sample Complexity

Useful to Whom? The Easy-versus-Coverage Boundary Moves with the Learner