LOCO-AdaMP: Built-in LOCO Inference for Adaptive Minipatch Ensembles with Enhanced Prediction

Yinan Cheng; Lili Zheng

stat.MLSep 28, 2026

LOCO-AdaMP: Built-in LOCO Inference for Adaptive Minipatch Ensembles with Enhanced Prediction

Authors: Yinan Cheng, Lili Zheng

Organizations: Department of Statistics University of Illinois Urbana-Champaign

Abstract

As black-box machine learning models become increasingly common, extracting interpretations with uncertainty quantification has become a critical challenge. One popular type of interpretation is leave-one-covariate-out (LOCO) feature importance, while prior LOCO inference methods often require data-splitting or model-refitting. A recent ensemble framework, LOCO-MP, addresses these challenges using minipatches that subsample both observations and features, but massive feature subsampling can hurt prediction in high-dimensional sparse settings. Motivated by this limitation, we consider minipatch ensembles with adaptive feature sampling guided by LOCO importance, and propose LOCO-AdaMP, which enables free LOCO inference for the resulting adaptive minipatch ensemble. We show that LOCO-AdaMP yields substantially improved predictive models while retaining asymptotically valid feature importance inference without data-splitting, despite the complex dependence between the adaptive sampling distribution and the LOCO importance statistics. Our analysis relies on a careful leave-two-out perturbation bound for the iteratively updated sampling probabilities together with the stability of LOCO scores induced by observation subsampling. Empirical results on synthetic and real datasets demonstrate advantages of LOCO-AdaMP over existing methods in predictive performance, inferential power, and stability. Overall, LOCO-AdaMP provides a flexible ensemble framework (agnostic to base models) that delivers both strong predictive performance and asymptotically valid, powerful feature importance inference for regression.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the LOCO metric and LOCO-Split.

Panel of Figure 2 — Figure 2: LOCO-MP utilizes uniform feature sampling to yield minipatches, while LOCO-AdaMP utilizes non-uniform feature sampling to yield minipatches. $\mathcal{U}_{n}^{[N]}$ and $\mathcal{U}_{m}^{[M]}$ represent uniform distributions, and $Q^{(t)}$ represents the non-uniform feature sampling distribution. Formal definitions of these sampling distributions are included in Section C.1 in the appendix.

Figure 3: Prediction errors over iterations in the low-dimensional (

M=50,N=200

, top row) and high-dimensional (

M=500,N=200

, bottom row) settings, averaged over

100

simulation replicates. We use the same simulation setup as in Section 4.2.1 .

t^{*}:=\sup\left\{0<t\leq\max_{l}\tilde{\Delta}_{l}:\frac{m\max_{l}\omega_{l}(t)}{\sum_{l=1}^{M}\omega_{l}(t)}\leq\delta\right\}

(3)

Algorithm 3 Update Sampling Probabilities Using Feature Importance

Figure 4: Inference targets

\Delta_{j}^{(t)}

of LOCO-AdaMP over iterations

t

in the low-dimensional setting (

M=50

) for default linear model setup. Feature 1 and feature 5 are signal features with SNR

=2.5

and SNR

=1

respectively, while feature 6 is a noise feature. Results are averaged over

100

simulation replicates. We use the same simulation setup as in Section 4.2.1 . Figures with y-axis on a log-scale and results under correlated linear model, nonlinear model and high-dimensional settings are included in the Appendix D.3.2 .

Panel of Figure 5 — Figure 5: Illustrative figure for the LOCO-LOO feature importance scores in LOCO-MP and LOCO-AdaMP. Different from LOCO-MP, the LOO predictor also depends on the held-out sample through the sampling distribution $Q^{(T)}$ .

Figure 6: Illustration of the iterative updates for $\boldsymbol{q}^{(T)}$ and $\boldsymbol{q}^{(\backslash i,T)}$ .

Figure 7: Coverage rates of

90\%

confidence intervals for the targets in the low-dimensional setting

(M=50)

over

100

replicates. The first and third rows use the default setting with independent features, whereas the second uses the setting with one correlated feature pair. The blue and orange lines represent a signal feature (feature 5) and a noise feature (feature 6) respectively. The true target value is approximated via Monte Carlo method with

10,000

test observations; hence the reported coverage rates slightly underestimate the true coverage. To address the underestimation effect due to Monte Carlo randomness, adjusted coverage rates are reported in Figure 22 . Results for the high-dimensional setting (

M=500

) are qualitatively similar and can be found in Figures 15 and 23 .

Figure 8: Prediction error comparison between our LOCO-AdaMP and baseline LOCO methods; the top row is the low-dimensional setting (

M=50,N=200

) and the bottom row is the high-dimensional setting (

M=500,N=200

). “Full Data” denotes the corresponding base learner trained on the entire data set and is shown as reference for what can be achieved if LOCO inference is not required. The first and third columns correspond to the default independent features, whereas the second considers one correlated feature pair.

Figure 9: 90% confidence intervals for the first 8 features in the low-dimensional setting (

M=50,N=200

). The first

5

M=500

) are qualitatively similar and are presented in Figure 16 .

Figure 10: Rejection rates of one-sided hypothesis tests (

\alpha=0.1

) for feature 5 in the low-dimensional setting

(M=50,N=200)

, averaged over 100 replicates. The x-axis represents values for

\beta_{5}

. LOCO-AdaMP quickly attains higher rejection rates when

\beta_{5}

increases above zero. The first and third rows use the default setting with independent features, whereas the second uses the setting with one correlated feature pair. Results for the high-dimensional setting (

M=500

) are qualitatively similar and are presented in Figure 17 .

Panel of Figure 11 — Figure 11: Comparison of predictive and inferential performance between LOCO-AdaMP and the baseline methods on the ROSMAP data. The left panel compares the predictive performance of the models constructed by different LOCO methods. The right panel compares the predictive usefulness of the top significant features identified by each method by training a separate random forest using the selected features and evaluating it on the test data. LOCO-MP and LOCO-AdaMP use decision trees as their base learners, whereas the other methods use random forests.

Figure 12: Histograms of the feature selection frequencies (selected if declared as significant with level 0.1) across the 10 random splits for each method. X-axis represents the number of times a feature is selected as significant. Y-axis shows the number of features corresponding to each selection count.

Figure 13: Histograms of the feature selection frequencies across the 10 random splits for LOCO-AdaMP and LOCO-MP. Only features selected by at least one split are included.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

\sum_{\ell>k}\tilde{\Delta}_{(\ell)}>\left(\frac{m}{\delta}-k\right)\tilde{\Delta}_{(k+1)}.

(9)

Appendix

Algorithm 5 Solving ( 3 ) for the Threshold

$\mu_{I,F}(X^{*};\boldsymbol{Z})$	$H(\boldsymbol{Z}_{I,F})(X^{*}_{F})$
$\mu^{(t)}_{k}(X^{*};\boldsymbol{Z})$	$H(\boldsymbol{Z}_{I_{k}^{(t)},F_{k}^{(t)}})(X^{*}_{F_{k}^{(t)}})$
$\mu^{(t)}(X^{*};\boldsymbol{Z})$	$\frac{1}{K}\sum_{k=1}^{K}\mu_{k}^{(t)}(X^{*};\boldsymbol{Z})$
$\mu_{\backslash j}^{(t)}(X^{*};\boldsymbol{Z})$	$\frac{1}{K}\sum_{k=1}^{K}\mu_{\tilde{I}_{k}^{(t)},\tilde{F}_{k}^{(t)}}(X^{*};\boldsymbol{Z})$
$\mu_{-i}^{(t)}(X^{*};\boldsymbol{Z})$	$\frac{1}{\sum_{k=1}^{K}\mathbb{I}(i\notin I_{k}^{(t)})}\sum_{k=1}^{K}\mathbb{I}(i\notin I_{k}^{(t)})\mu_{k}^{(t)}(X^{*};\boldsymbol{Z})$
$\mu_{-i}^{(t)-j}(X^{*};\boldsymbol{Z})$	$\frac{1}{\sum_{k=1}^{K}\mathbb{I}(i\notin I_{k}^{(t)})\mathbb{I}(j\notin F_{k}^{(t)})}\sum_{k=1}^{K}\mathbb{I}(i\notin I_{k}^{(t)})\mathbb{I}(j\notin F_{k}^{(t)})\mu_{k}(X^{*};\boldsymbol{Z})$
$\mu^{}(X^{};Q,\boldsymbol{Z})$	$\sum_{I\subset[N],F\subset[M]}\frac{1}{{N\choose n}}\mathbb{I}(\|I\|=n)Q(F)\mu_{I,F}(X^{*};\boldsymbol{Z})$

Appendix

Table 1: List of notations for predictions. In the fourth row,

\tilde{I_{k}}^{(t)}

’s are i.i.d. randomly sampled indices with the same uniform distribution as

I_{k}^{(t)}

’s and

\tilde{F_{k}}^{(t)}

’s are i.i.d. randomly sampled indices with distribution

Q_{\backslash j}^{(t)}

. When the epoch

t

is clear from context, the superscript

t

can be omitted.

$h_{j}(Z^{*},\boldsymbol{Z},Q)$	$\ell(Z^{},\mu^{}(X^{};Q_{\backslash j},\boldsymbol{Z}))-\ell(Z^{},\mu^{}(X^{};Q,\boldsymbol{Z}))$
$h_{j}^{(t)}(Z^{*},\boldsymbol{Z})$	$\ell(Z^{},\mu^{(t)}_{\backslash j}(X^{};\boldsymbol{Z}))-\ell(Z^{},\mu^{(t)}(X^{},\boldsymbol{Z}))$
$h_{j}(Z^{*},\boldsymbol{Z})$	$h_{j}(Z^{},\boldsymbol{Z},Q^{(T)}(\boldsymbol{Z}))$ where $Q^{*(T)}(\boldsymbol{Z})$ is the final-iteration feature sampling mass function if applying Algorithm 6 on $\boldsymbol{Z}$ .
$h_{j,N}(Z^{*})$	$\mathbb{E}_{\boldsymbol{Z}}(h_{j}(Z^{},\boldsymbol{Z})\|Z^{})$ where $Z^{*}$ is independent from $\boldsymbol{Z}$ , $\boldsymbol{Z}=(Z_{1},\dots,Z_{N})$ with $Z_{i}\overset{i.i.d.}{\sim}\mathcal{P}$ . We sometimes omit the subscript $N$ when it is clear from the context.
$\widetilde{h}_{j}(Z^{*},\boldsymbol{Z})$	$h_{j}(Z^{},\boldsymbol{Z})-h_{j,N}(Z^{})$
$\hat{h}_{j}^{(t)}(Z_{i},\boldsymbol{Z}_{\backslash i,:})$	$\hat{\Delta}^{(t)}_{j}(Z_{i})$
$\hat{h}_{j}^{(t)}(Z_{i},\boldsymbol{Z}_{\backslash i,:},Q)$	The LOCO-LOO score $\hat{\Delta}^{(t)}_{j}(Z_{i})$ obtained by running Algorithms 1 and 2 sequentially, enforcing feature sampling distribution $Q$

Appendix

Table 2: List of notations for feature importance functions.

Figure 14: Empirical assessment for Assumption 5 in the high-dimensional setting

M=500

. The x-axis represents the largest sampling probability ever assigned to a feature identified as least important in one iteration

t>=2

. The y-axis represents the frequency across 100 simulation replicates and all iterations

t>=2

Figure 15: Coverage rates of

90\%

confidence intervals for the targets in the high-dimensional setting

(M=500)

over

100

10,000

Figure 16: 90% confidence intervals for the first 8 features in the high-dimensional setting (

M=500,N=200

). The first

5

features are signal features with decreasing strengths. The first and third rows use the default setting with independent features, whereas the second uses the setting with one correlated feature pair. For the second column, decision trees are used for LOCO-MP and LOCO-AdaMP, and random forests are used for LOCO-Split. Unlike the low-dimensional results (Figure 9 ), LOCO-AdaMP’s interval magnitudes are larger than LOCO-Split when the base model is ridge/kernel ridge, perhaps due to that overly large shrinkage of ridge penalty in high-dimensional settings.

Figure 17: Rejection rates of one-sided hypothesis tests (

\alpha=0.1

) for feature 5 in the high-dimensional setting

(M=500,N=200)

, averaged over 100 replicates. The x-axis represents values for

\beta_{5}

. LOCO-AdaMP quickly attains higher rejection rates when

\beta_{5}

increases above zero. The first and third rows use the default setting with independent features, whereas the second uses the setting with one correlated feature pair. LOCO-AdaMP still attains the highest rejection rate as

\beta_{5}

increases beyond zero.

Figure 18: Inference targets

\Delta_{j}^{(t)}

of LOCO-AdaMP over iterations

t

in the low-dimensional setting (

M=50

) for correlated linear model and nonlinear model. Feature 1 and feature 5 are signal features with SNR

=2.5

and SNR

=1

respectively, while feature 6 is a noise feature. Results are averaged over

100

simulation replicates.

Figure 19: Inference targets

\Delta_{j}^{(t)}

of LOCO-AdaMP over iterations

t

in the high-dimensional setting (

M=500

). Feature 1 and feature 5 are signal features with SNR

=2.5

and SNR

=1

respectively, while feature 6 is a noise feature. Results are averaged over

100

simulation replicates.

Figure 20: Log-scaled inference targets

\log\Delta_{j}^{(t)}

of LOCO-AdaMP over iterations

t

in the low-dimensional setting (

M=50

). Feature 1 and feature 5 are signal features with SNR

=2.5

and SNR

=1

respectively, while feature 6 is a noise feature. Results are averaged over

100

simulation replicates. The y-axis uses a symmetric logarithmic scale with a linear region within

[-10^{-3},10^{-3}]

Figure 21: Log-scaled inference targets

\log\Delta_{j}^{(t)}

of LOCO-AdaMP over iterations

t

in the high-dimensional setting (

M=500

). Feature 1 and feature 5 are signal features with SNR

=2.5

and SNR

=1

respectively, while feature 6 is a noise feature. Results are averaged over

100

simulation replicates. The y-axis uses a symmetric logarithmic scale with a linear region within

[-10^{-3},10^{-3}]

Figure 22: Adjusted coverage rates of

90\%

confidence intervals for the targets in the low-dimensional setting

(M=50)

over

100

Figure 23: Adjusted coverage rates of

90\%

confidence intervals for the targets in the high-dimensional setting

(M=500)

over

100

Figure 24: Coverage rates of

90\%

confidence intervals for the targets in the low-dimensional setting

(M=50)

over

100

10,000

test observations; hence the reported coverage rates slightly underestimate the true coverage.

Figure 25: Coverage rates of

90\%

confidence intervals for the targets in the high-dimensional setting

(M=500)

over

100

10,000

test observations; hence the reported coverage rates slightly underestimate the true coverage.

Figure 26: Prediction error comparison between our LOCO-AdaMP and baseline LOCO methods; the top row is the low-dimensional setting (

M=50,N=200

) and the bottom row is the high-dimensional setting (

M=500,N=200

). “Full Data” denotes the corresponding base learner trained on the entire data set and is shown as reference for what can be achieved without requiring LOCO inference. The first and third columns correspond to the default independent features, whereas the second considers one correlated feature pair.

Figure 27: 90% confidence intervals for the first 8 features in the low-dimensional setting (

M=50,N=200

). The first

5

Figure 28: 90% confidence intervals for the first 8 features in the high-dimensional setting (

M=500,N=200

). The first

5

Figure 29: Rejection rates of one-sided hypothesis tests (

\alpha=0.1

) for feature 5 in the low-dimensional setting

(M=50,N=200)

, averaged over 100 replicates. The x-axis represents values for

\beta_{5}

. LOCO-AdaMP quickly attains higher rejection rates when

\beta_{5}

increases above zero. The first and third rows use the default setting with independent features, whereas the second uses the setting with one correlated feature pair.

Figure 30: Rejection rates of one-sided hypothesis tests (

\alpha=0.1

) for feature 5 in the high-dimensional setting

(M=500,N=200)

, averaged over 100 replicates. The x-axis represents values for

\beta_{5}

. LOCO-AdaMP quickly attains higher rejection rates when

\beta_{5}

increases above zero. The first and third rows use the default setting with independent features, whereas the second uses the setting with one correlated feature pair.

Ensemble Learning

Ensemble learning combines multiple predictors to improve accuracy, robustness, calibration, or uncertainty estimates.

See all 278 papers

Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling
Sep 21, 2026Xuhui Liu, Lili ZhengAdaptive Sampling Feature Selection
Department of Statistics, University of Illinois Urbana-Champaign
Cluster LOCO: Feature Importance For Interpreting Clusters
Jun 12, 2026Claire M. He, Genevera I. AllenClustering
Department of Statistics · Columbia University · New York, NY 10027, USA
Prediction-Powered Conditional Inference
Mar 5, 2026Yang Sui, Jin Zhou, Hua Zhou +1Prediction-Powered Inference Confidence Region Estimation
University of California, Los Angeles

Sep 21, 2026J/K move · Enter open · S save

Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling

Xuhui Liu, Lili Zheng

Department of Statistics, University of Illinois Urbana-Champaign

Adaptive Sampling Feature Selection

Open paper AskPDF arXiv

Black-box machine learning models increasingly deliver strong predictions, but extracting useful information from them, such as a set of important features, remains challenging. Existing model-agnostic methods primarily estimate feature importance or conduct inference on it rather than directly selecting features, whereas many feature selection methods are model-specific or rely on the model-X assumption. We introduce LOCO-guided Adaptive Minipatch Sampling (LAMPS), a model-agnostic ensemble framework that uses any black-box regression algorithm as its base learner to select features important for predicting the response. The base learner need only produce predictions and need not perform feature selection itself. LAMPS operates within a minipatch ensemble framework that subsamples both observations and features, allowing leave-one-covariate-out (LOCO) feature importance scores to be easily computed. It adaptively concentrates minipatch sampling on features with high LOCO scores while maintaining exploration. The resulting sampling probabilities rapidly separate signal from noise features after a few iterations, enabling selection through simple thresholding. We establish that LAMPS achieves exact feature selection in high-dimensional settings, provided that the base predictive models are sufficiently well trained on average. Extensive experiments on synthetic and real data show that LAMPS outperforms state-of-the-art feature selection methods, with particularly strong performance in the presence of correlated features.

Abstract

Ensemble Learning

Explore similar work

Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling

Cluster LOCO: Feature Importance For Interpreting Clusters

Prediction-Powered Conditional Inference