Scale-invariant Optimal Sampling for Rare-events Data with Sparse Models

Jing Wang; HaiYing Wang; Qiang Zhang; Hao Helen Zhang

stat.MLAug 23, 2026

Scale-invariant Optimal Sampling for Rare-events Data with Sparse Models

Authors: Jing Wang, HaiYing Wang, Qiang Zhang, Hao Helen Zhang

Organizations: Department of Statistics, University of Connecticut, Storrs, CT 06269, USA · Vickie and Jack Farber Vision Research Center, Wills Eye Hospital Thomas Jefferson University, 840 Walnut St, Philadelphia, PA 19107, USA · Department of Mathematics, University of Arizona, Tucson, AZ 85721, USA

Abstract

Subsampling is effective in tackling computational challenges for massive data with rare events. Overly aggressive subsampling may adversely affect estimation efficiency, and optimal subsampling is essential to mitigate the information loss. However, existing optimal subsampling probabilities depend on data scales, and some scaling transformations may result in inefficient subsamples. This problem is more significant when there are inactive features, because their influence on the subsampling probabilities can be arbitrarily magnified by inappropriate scaling transformations. We tackle this challenge and introduce a scale-invariant optimal subsampling function in the context of sparse models, where inactive features are commonly assumed. Instead of focusing on estimating model parameters, we define an optimal subsampling function to minimize the prediction error, using adaptive lasso to outline the estimation procedure and study its theoretical guarantee. We first introduce the adaptive lasso estimator for rare-events data and establish its oracle properties, thereby validating the use of subsampling. Then we derive a scale-invariant optimal subsampling function that minimizes the prediction error of the inverse probability weighted (IPW) adaptive lasso. Finally, we present an estimator based on the maximum sampled conditional likelihood (MSCL) to further improve the estimation efficiency. We conduct numerical experiments using both simulated and real-world data sets to demonstrate the performance of the proposed methods.

Figures & tables

Panel of Figure 1 — Figure 1: Prediction errors with different scale transformations of the same model with (a) non-sparse parameter $(-1,-1,-0.01,-0.01,-0.01,-0.01)^{\mathrm{T}}$ and (b) sparse parameter $(-1,0,0,0,0,0)^{\mathrm{T}}$ .

Panel of Figure 2 — Figure 2: Contributions of individual variables to sampling probabilities. The bar charts in the left panel present contributions with the original scale of $x_{(6)}$ ; those in the right panel present contributions after re-scaling $x_{(6)}$ .

Panel of Figure 3 — Figure 3: Bar charts for contributions of individual variables to the P-optimal probabilities. The left panel presents the contributions with the original scale and the right panel presents the contributions after re-scaling.

Panel of Figure 4 — Figure 4: eMSE for different true parameters with different sampling rates.

Panel of Figure 5 — Figure 5: eMPSE of estimated probability with different sampling rates.

$\rho$	first-stage	Uni	A-OS	L-OS	P-OS
0.0025	13.27(0.34)	2.84(0.02)	2.97(0.02)	2.96(0.02)	2.96(0.02)
0.005	12.46(0.32)	2.94(0.02)	3.04(0.03)	3.05(0.03)	3.06(0.03)
0.0075	12.76(0.33)	2.97(0.01)	3.04(0.02)	3.03(0.02)	3.03(0.02)
0.01	12.81(0.34)	2.96(0.01)	3.03(0.02)	3.02(0.01)	3.02(0.01)

Table 1: Mean number of selected variables in Case C.

$\rho$	Uni	A-OS	L-OS	P-OS
0.0025	0.168(0.017)	0.086(0.013)	0.088(0.013)	0.084(0.013)
0.005	0.100(0.013)	0.068(0.011)	0.066(0.011)	0.066(0.011)
0.0075	0.066(0.011)	0.046(0.009)	0.048(0.010)	0.046(0.009)
0.01	0.068(0.011)	0.052(0.010)	0.054(0.010)	0.054(0.010)

Table 2: Rates of excluding active variables (false negative rate) in Case C.

Case	Uni	A-OS	L-OS	P-OS	A-lasso (full)	Lasso (full)
A	0.29	1.09	0.91	1.06	129.62	112.97
B	0.31	1.23	1.20	1.27	129.89	122.40
C	0.31	1.02	0.93	1.00	130.33	121.29

Table 3: Mean computational time (seconds).

$\rho$	Las	SIS-3	SIS-6	SIS-13	SIS-20	Las-mle
0.0025	0.025	0.729	0.023	0.034	0.047	0.030
0.0050	0.017	0.728	0.014	0.023	0.031	0.020
0.0075	0.014	0.726	0.012	0.016	0.025	0.016

Table 4: Table of eMSEs with different pilot estimators for Case C.

Panel of Figure 6 — Figure 6: Empirical median AUCs for two real data sets.

Panel of Figure 7 — Figure 7: Empirical median AUC for the IRIS® data set.

Figure 8: Frequencies of selected variables.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

{\hat{\bm{\theta}}}_{\mathrm{pl}}:=\mathop{\arg\max}_{{\bm{\theta}}}\left\{\sum_{i=1}^{N_{\mathrm{pl}}^{*}}[y_{i}^{\mathrm{pl}}g(\bm{x}_{i}^{\mathrm{pl}};{\bm{\theta}})-\log\{1+e^{g(\bm{x}_{i}^{\mathrm{pl}};{\bm{\theta}})+l}\}]-\lambda_{\mathrm{pl}}\sum_{j=1}^{p}|\beta_{(j)}|\right\},

(25)

Appendix

Algorithm 3 Subsampling adaptive lasso algorithm

	Case A (five active variables)
$\rho$	first-stage	Uni	A-OS	L-OS	P-OS
0.0025	14.87(0.30)	4.99(0.02)	5.02(0.02)	5.03(0.02)	5.01(0.02)
0.005	14.68(0.28)	5.05(0.02)	5.07(0.02)	5.06(0.02)	5.08(0.02)
0.0075	14.20(0.27)	5.09(0.03)	5.08(0.02)	5.08(0.02)	5.09(0.03)
0.01	14.46(0.30)	5.13(0.02)	5.13(0.02)	5.13(0.02)	5.12(0.03)

Appendix

Table 5: Mean number of selected variables in Case A and Case B.

	Case A
$\rho$	Uni	A-OS	L-OS	P-OS
0.0025	0.094(0.013)	0.076(0.012)	0.068(0.011)	0.070(0.012)
0.005	0.052(0.010)	0.048(0.010)	0.050(0.010)	0.044(0.010)
0.0075	0.054(0.010)	0.052(0.010)	0.052(0.010)	0.052(0.010)
0.01	0.040(0.009)	0.036(0.008)	0.036(0.008)	0.036(0.008)

Appendix

Table 6: Rates of excluding active variables (false negative rate) in Case A and Case B.

	Case A
$\rho$	Uni	A-OS	L-OS	P-OS
0.0025	0.856(0.016)	0.870(0.015)	0.868(0.015)	0.878(0.015)
0.005	0.884(0.014)	0.872(0.015)	0.880(0.015)	0.872(0.015)
0.0075	0.858(0.016)	0.868(0.015)	0.868(0.015)	0.868(0.015)
0.01	0.854(0.016)	0.866(0.015)	0.862(0.015)	0.868(0.015)

Appendix

Table 7: Rates of selecting the true model.

Panel of Figure 9 — Figure 9: Empirical median squared error of estimated probability for different parameters with different sampling rates. The same pilot sample size is $N_{\mathrm{pl}}=500$ .

Panel of Figure 10 — Figure 10: Empirical median squared error of estimated probability for different parameters with different sampling rates. The same pilot sample size is $N_{\mathrm{pl}}=500$ .

	A			B			C
$\rho$	sUni	P-OS	sP-OS	sUni	P-OS	sP-OS	sUni	P-OS	sP-OS
0.0025	0.848	0.878	0.862	0.850	0.876	0.862	0.828	0.884	0.880
0.005	0.878	0.872	0.850	0.890	0.902	0.890	0.882	0.882	0.880
0.0075	0.850	0.868	0.846	0.850	0.848	0.842	0.906	0.914	0.910
0.01	0.840	0.868	0.844	0.868	0.872	0.862	0.900	0.910	0.904

Appendix

Table 8: Rates of selecting true models.

	A			B			C
$\rho$	sUni	P-OS	sP-OS	sUni	P-OS	sP-OS	sUni	P-OS	sP-OS
0.0025	0.088	0.070	0.068	0.106	0.070	0.070	0.164	0.084	0.084
0.005	0.056	0.044	0.044	0.046	0.036	0.036	0.100	0.066	0.066
0.0075	0.052	0.052	0.052	0.062	0.044	0.044	0.062	0.046	0.046
0.01	0.040	0.036	0.036	0.056	0.050	0.050	0.068	0.054	0.054

Appendix

Table 9: Rates of excluding active variables (false negative rate).

Rare-Event Estimation

Rare-event estimation measures probabilities of failures, extremes, or transitions that ordinary sampling may miss.

See all 28 papers

Robust Importance Sampling for Rare Events via Constrained Gaussian Mixtures
Oct 5, 2026Paweł Lorek, Rafał Nowak, Rafał Topolnicki +2Gaussian Mixture Models Importance Sampling
University of Wrocław · Tooploox · TRAILS University of Warsaw +3
Scalable Counterfactual Risk Estimation for Rare Events in Longitudinal Data
Jun 1, 2026Xiaohui Yin, Avijit Mitra, Ying Zhou +2Causal Effect Estimation Survival Analysis
University of Massachusetts Amherst Amherst, Massachusetts, USA · University of Massachusetts Lowell Lowell, Massachusetts, USA
Nearly Optimal Subdata Selection
Apr 27, 2026Min Yang, Wei Zheng, John Stufken +3Data Selection Training Data Selection
Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago · Department of Business Analytics and Statistics, University of Tennessee · Department of Statistics, George Mason University[stat.ME] +3

Oct 5, 2026J/K move · Enter open · S save

Robust Importance Sampling for Rare Events via Constrained Gaussian Mixtures

Paweł Lorek, Rafał Nowak, Rafał Topolnicki, Tomasz Trzciński+1

University of Wrocław · Tooploox · TRAILS University of Warsaw+3

Gaussian Mixture Models Importance Sampling Variance Reduction Rare-Event Estimation

Open paper AskPDF arXiv

We study estimating rare-event probabilities

I = \mathbb{P}(g(\mathbf{X}) > γ)

with

\mathbf{X} \sim \mathcal{N}(\boldsymbolμ, \boldsymbolΣ)

and general

g : \mathbb{R}^d \to \mathbb{R}

. We address this problem through importance sampling, and propose a framework that substantially improves efficiency and robustness over baselines such as crude Monte Carlo, adaptive cross-entropy, variational-inference-based methods (including reverse- and forward-KL approaches), as well as Safe-ICE, Subset Simulation, and Sequential Monte Carlo, drawing on ideas from both rare-event estimation and cross-entropy optimization. The key contribution has two parts: first, we separate the problem into coverage, to overcome the cold-start barrier, and fitting, to refine proposals once a meaningful signal is available; second, we constrain the final GMM proposal so that it has finite importance-sampling variance (since coverage alone is not sufficient -- without safeguards, importance sampling may still suffer from infinite variance). Together, these ingredients yield expressive proposals; finite variance does not by itself guarantee practical stability at a fixed sampling budget. Extensive experiments demonstrate substantial variance reduction, strong robustness across diverse benchmarks, and favorable cost--efficiency trade-offs, with the proposed approach often outperforming these baselines, particularly in high-dimensional and multimodal settings where competing methods frequently become unstable or fail. Our code is available at https://github.com/lorek/robust-cfi-is.

Abstract

Rare-Event Estimation

Explore similar work

Robust Importance Sampling for Rare Events via Constrained Gaussian Mixtures

Scalable Counterfactual Risk Estimation for Rare Events in Longitudinal Data

Nearly Optimal Subdata Selection