Scalable Learning of Multivariate Distributions via Coresets

Zeyu Ding; Katja Ickstadt; Nadja Klein; Alexander Munteanu; Simon Omlor

cs.LGMar 20, 2026

Scalable Learning of Multivariate Distributions via Coresets

Authors: Zeyu Ding, Katja Ickstadt, Nadja Klein, Alexander Munteanu, Simon Omlor

Abstract

Efficient and scalable non-parametric or semi-parametric regression analysis and density estimation are of crucial importance to the fields of statistics and machine learning. However, available methods are limited in their ability to handle large-scale data. We address this issue by developing a novel coreset construction for multivariate conditional transformation models (MCTMs) to enhance their scalability and training efficiency. To the best of our knowledge, these are the first coresets for semi-parametric distributional models. Our approach yields substantial data reduction via importance sampling. It ensures with high probability that the log-likelihood remains within multiplicative error bounds of $(1\pm\varepsilon)$ and thereby maintains statistical model accuracy. Compared to conventional full-parametric models, where coresets have been incorporated before, our semi-parametric approach exhibits enhanced adaptability, particularly in scenarios where complex distributions and non-linear relationships are present, but not fully understood. To address numerical problems associated with normalizing logarithmic terms, we follow a geometric approximation based on the convex hull of input data. This ensures feasible, stable, and accurate inference in scenarios involving large amounts of data. Numerical experiments demonstrate substantially improved computational efficiency when handling large and complex datasets, thus laying the foundation for a broad range of applications within the statistics and machine learning communities.

Figures & tables

	Method	$\ell_{2}$ err.	$\lambda$ err.	LR	Imp.(%)
DGP 1	$\ell_{2}$ -hull	$2.56\pm 0.7$	$0.44\pm 0.1$	$\mathbf{\,1.54\pm 0.2}$	$\mathbf{12.8}$
	$\ell_{2}$ -only	$\mathbf{2.54\pm 0.6}$	$0.51\pm 0.1$	$1.65\pm 0.3$	$1.6$
	uniform	$4.91\pm 4.7$	$\mathbf{\,0.29\pm 0.2}$	$1.94\pm 1.1$	baseline
DGP 2	$\ell_{2}$ -hull	$\mathbf{1.76\pm 0.8}$	$\mathbf{0.09\pm 0.1}$	$\mathbf{\,1.03\pm 0.0}$	$\mathbf{49.8}$
	$\ell_{2}$ -only	$2.18\pm 0.8$	$0.10\pm 0.1$	$1.05\pm 0.0$	$38.8$
	uniform	$3.08\pm 0.9$	$0.11\pm 0.1$	$1.21\pm 0.4$	baseline

Table 1: Performance comparison of coreset methods (coreset size = 30). Results show mean

\pm

std over 10 runs. Relative improvement: avg % improvement across metrics. Best values highlighted in bold .

Panel of Figure 1 — Figure 1: Coreset performance comparison on stock-return data. Top row: 10 stocks; bottom row: 20 stocks. Shaded bands indicate $\pm 1$ standard deviation, and solid lines show the averages over multiple repetitions.

Size	Method	Param L2	Lambda L2	LR
$k=50$	l2-hull	$\mathbf{23.4\pm 9.8}$	$\mathbf{18.2\pm 12.2}$	$\mathbf{11.4\pm 6.2}$
	l2-only	$40.7\pm 24.2$	$37.0\pm 26.6$	$71.8\pm 8.7$
	ridge-lss	$30.8\pm 21.5$	$25.6\pm 24.3$	$51.7\pm 45.7$
	root-l2	$64.6\pm 10.4$	$63.2\pm 10.7$	$122.0\pm 96.1$
	uniform	$52.6\pm 10.6$	$50.6\pm 10.9$	$84.8\pm 90.5$
$k=200$	l2-hull	$\mathbf{14.0\pm 3.1}$	$\mathbf{7.4\pm 5.4}$	$\mathbf{1.42\pm 0.13}$

Table 2: Covertype data performance for different coreset sizes. Results: mean

\pm

std over 5 trials. For

\ell_{2}

distances, lower is better. For log-likelihood Ratio, closer to 1 is better. Best values highlighted in bold .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

\hat{\theta}_{\text{coreset}}=\operatorname{argmax}_{\theta}\sum_{(Y_{j},w_{j})\in\mathcal{C}}w_{j}\cdot\log f_{\theta}(Y_{j})

Appendix

Algorithm 1 Hybrid coreset construction for MCTMs

Panel of Figure 2 — Figure 2: Coreset visualization for basic probability distributions. Each row shows a different sampling method: Uniform (top), $\ell_{2}$ Sensitivity (middle), and $\ell_{2}$ -Hull (bottom).

Panel of Figure 3 — Figure 3: Coreset visualization for complex probability distributions. Each column shows a different sampling method: Uniform (top), $\ell_{2}$ Sensitivity (middle), and $\ell_{2}$ -Hull (bottom).

Panel of Figure 4 — Figure 4: Coreset visualization for geometric dependency structures. Each column shows a different sampling method: Uniform (top), $\ell_{2}$ Sensitivity (middle), and $\ell_{2}$ -Hull (bottom).

Panel of Figure 5 — Figure 5: Coreset visualization for additional dependency structures. Each column shows a different sampling method: Uniform (top), $\ell_{2}$ Sensitivity (middle), and $\ell_{2}$ -Hull (bottom).

Panel of Figure 6 — Figure 6: Coreset visualization for functional dependency structures. Each column shows a different sampling method: Uniform (top), $\ell_{2}$ Sensitivity (middle), and $\ell_{2}$ -Hull (bottom).

Panel of Figure 7 — Figure 7: Convergence of the likelihood ratio, parameter error, and $\lambda$ error as coreset size increases. First row: Bivariate mixture data of two Gaussian distributions with different means and variances. Second row: Non-linear correlation. Third row: Bimodal clusters with two distinct clusters with opposing correlation structure.

Panel of Figure 8 — Figure 8: Convergence of the likelihood ratio, parameter error, and $\lambda$ error as coreset size increases. First row: circular-dependency. Data points are distributed along circular trajectories, demonstrating a circular dependency structure between variables. Second row: copula complex. A copula construct with tail dependence and non-linear correlation is used. Third row: heteroscedastic distribution. The conditional variance of the data varies with the level of the independent variable, reflecting heteroscedasticity.

Data Gen. Process	Method	Param. $\ell_{2}$ dist.	$\lambda$ error	Likelihood ratio	Rel. Impr.(%)	Total time(s)
Bivariate normal	$\ell_{2}$ -hull	$2.56\pm 0.76$	$0.44\pm 0.16$	$1.54\pm 0.29$	12.8	$0.21\pm 0.03$
	$\ell_{2}$ -only	$\mathbf{2.54\pm 0.62}$	$0.51\pm 0.13$	$1.65\pm 0.33$	1.6	$0.20\pm 0.02$
	uniform	$4.91\pm 4.74$	$\mathbf{0.29\pm 0.26}$	$1.94\pm 1.13$	baseline	$\mathbf{0.13\pm 0.03}$
Non-linear correlation	$\ell_{2}$ -hull	$\mathbf{1.76\pm 0.88}$	$\mathbf{0.09\pm 0.09}$	$\mathbf{1.03\pm 0.01}$	49.8	$0.19\pm 0.02$
	$\ell_{2}$ -only	$2.18\pm 0.82$	$0.10\pm 0.10$	$1.05\pm 0.02$	38.8	$0.19\pm 0.01$
	uniform	$3.08\pm 0.91$	$0.11\pm 0.07$	$1.21\pm 0.46$	baseline	$\mathbf{0.12\pm 0.01}$

Appendix

Table 3: Performance comparison of different coreset methods for various data generation processes (coreset Size = 30)

Data Gen. Process	Method	Param. $\ell_{2}$ dist.	$\lambda$ error	Likelihood ratio	Rel. Impr.(%)	Total time(s)
Bivariate normal	$\ell_{2}$ -hull	$\mathbf{1.80\pm 0.42}$	$0.30\pm 0.07$	$1.32\pm 0.11$	0	$0.23\pm 0.02$
	$\ell_{2}$ -only	$2.01\pm 0.36$	$0.39\pm 0.09$	$1.48\pm 0.17$	0	$0.24\pm 0.04$
	uniform	$1.98\pm 0.66$	$\mathbf{0.11\pm 0.06}$	$\mathbf{1.19\pm 0.08}$	baseline	$\mathbf{0.13\pm 0.02}$
Non-linear correlation	$\ell_{2}$ -hull	$\mathbf{1.05\pm 0.42}$	$\mathbf{0.03\pm 0.03}$	$\mathbf{1.01\pm 0.00}$	52.6	$0.20\pm 0.02$
	$\ell_{2}$ -only	$1.20\pm 0.53$	$\mathbf{0.03\pm 0.03}$	$\mathbf{1.01\pm 0.01}$	38.8	$0.22\pm 0.02$
	uniform	$1.79\pm 0.79$	$0.08\pm 0.05$	$1.02\pm 0.01$	baseline	$\mathbf{0.11\pm 0.02}$

Appendix

Table 4: Performance comparison of different coreset methods for various data generation processes (coreset Size = 100)

Panel of Figure 9 — Figure 9: Computation time for 9 simulation distributions

Panel of Figure 10 — Figure 10: Predicted marginal density of $x$ for the normal distribution of different coreset size, $k=50,100,500$ . First Row: uniform subsampling. Second row: Only $\ell_{2}$ sampling. Third row: $\ell_{2}$ with $\varepsilon$ -kernel convex hull

Panel of Figure 11 — Figure 11: Predicted marginal density of $y$ of different coreset size, $k=50,100,500$ . First Row: uniform subsampling. Second row: Only $\ell_{2}$ sampling. Third row: $\ell_{2}$ with $\varepsilon$ -kernel convex hull

Panel of Figure 12 — Figure 12: Pairwise relationships between different sets of terrain variables from the Covertype dataset.

Panel of Figure 13 — Figure 13: Covertype dataset (10-dimensional, unconditional model) on $\ell_{2}$ -hull versus uniform sampling ( uniform ) compared with respect to four evaluation metrics: (a) log-likelihood ratio, (b) parameter-space $\ell_{2}$ distances, (c) $\ell_{2}$ distances dependent on parameter $\lambda$ , (d) Total computation time.

Coreset Size	Method	Param. $\ell_{2}$ dist.	$\lambda$ error	Log‐likelihood ratio	Rel. Impr. L2 / $\lambda$ / LL (%)	Total time (s)
$k=50$	$\ell_{2}$ ‐hull	$\mathbf{45.518\pm 1.643}$	$2.992\pm 0.530$	$1.683\pm 0.234$	12.0 / 0.0 / 57.5	$8.21\pm 2.30$
	$\ell_{2}$ ‐only	$45.784\pm 0.554$	$2.599\pm 0.629$	$\mathbf{1.351\pm 0.082}$	11.5 / 0.0 / 78.2	$8.45\pm 2.53$
	uniform	$51.745\pm 2.474$	$\mathbf{1.515\pm 0.303}$	$2.610\pm 0.101$	baseline	$\mathbf{7.98\pm 1.88}$
$k=100$	$\ell_{2}$ ‐hull	$\mathbf{39.888\pm 0.643}$	$1.613\pm 0.106$	$1.399\pm 0.147$	22.1 / 0.0 / 89.3	$9.93\pm 2.58$
	$\ell_{2}$ ‐only	$41.183\pm 1.503$	$1.404\pm 0.114$	$\mathbf{1.153\pm 0.044}$	19.6 / 0.0 / 96.1	$9.80\pm 2.91$
	uniform	$51.195\pm 2.616$	$\mathbf{0.913\pm 0.140}$	$4.926\pm 0.390$	baseline	$\mathbf{8.94\pm 2.10}$

Appendix

Table 5: Performance comparison on 10 stock return series for different coreset sizes (1985-2025)

Coreset Size	Method	Param. $\ell_{2}$ dist.	$\lambda$ error	Log‐likelihood ratio	Rel. Impr. L2 / $\lambda$ / LL (%)	Total time (s)
$k=50$	$\ell_{2}$ ‐hull	$54.254\pm 1.367$	$3.944\pm 0.563$	$5.208\pm 0.210$	3.6 / 0.0 / 0.0	$30.95\pm 5.06$
	$\ell_{2}$ ‐only	$\mathbf{53.922\pm 1.280}$	$3.920\pm 0.344$	$30.303\pm 0.426$	4.2 / 0.0 / 0.0	$31.10\pm 6.00$
	uniform	$56.263\pm 1.042$	$\mathbf{3.490\pm 0.260}$	$\mathbf{4.504\pm 0.200}$	baseline	$\mathbf{29.00\pm 7.55}$
$k=100$	$\ell_{2}$ ‐hull	$\mathbf{46.280\pm 1.887}$	$2.270\pm 0.138$	$\mathbf{1.675\pm 0.154}$	13.4 / 0.0 / 89.6	$35.56\pm 7.94$
	$\ell_{2}$ ‐only	$47.385\pm 0.788$	$2.304\pm 0.370$	$1.718\pm 0.141$	11.4 / 0.0 / 88.9	$\mathbf{34.95\pm 6.53}$
	uniform	$53.469\pm 2.072$	$\mathbf{2.006\pm 0.151}$	$7.518\pm 0.305$	baseline	$35.22\pm 6.65$

Appendix

Table 6: Performance comparison on 20 stock return series for different coreset sizes (1985-2025)

Ticker	Company Name
JNJ	Johnson & Johnson
PG	Procter & Gamble Co.
KO	The Coca-Cola Company
XOM	Exxon Mobil Corporation
WMT	Walmart Inc.
IBM	International Business Machines Corporation

Appendix

Table 7: List of 10 Selected Stocks

Ticker	Company Name
JNJ	Johnson & Johnson
PG	Procter & Gamble Co.
KO	The Coca-Cola Company
XOM	Exxon Mobil Corporation
WMT	Walmart Inc.
IBM	International Business Machines Corporation

Appendix

Table 8: List of 20 Selected Stocks

Coreset Selection

Coreset selection chooses compact subsets that preserve a dataset’s learning, evaluation, or optimization value while reducing cost.

See all 58 papers

EMS Coreset: An Efficient Expectation-Maximization Algorithm for Sinkhorn Coreset
Aug 17, 2026Haoyun Yin, Chuanhui Liu, Xiao WangCoreset Selection Optimal Transport
Department of Statistics, Purdue University West Lafayette, IN 47906, USA
Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation
Jun 24, 2026Luyang Fang, Haoran Lu, Yongkai Chen +2Hyperparameter Transfer Knowledge Distillation
Department of Statistics University of Georgia, Athens, GA, 30602, USA
Beyond Worst-Case Coreset Bounds for $k$ -Clustering via Determinantal Sampling
Sep 6, 2026Diptarka Chakraborty, Satyaki Mukherjee, Gaurav Vallabhdas Revankar +1Coreset Selection Clustering
School of Computing, National University of Singapore · Department of Mathematics, National University of Singapore · Department of Mathematics, Indian Institute of Technology Bombay

Aug 17, 2026J/K move · Enter open · S save

EMS Coreset: An Efficient Expectation-Maximization Algorithm for Sinkhorn Coreset

Haoyun Yin, Chuanhui Liu, Xiao Wang

Department of Statistics, Purdue University West Lafayette, IN 47906, USA

Coreset Selection Optimal Transport

Open paper AskPDF arXiv

Coresets distill large datasets into small, representative subsets for efficient downstream learning. Yet Optimal Transport (OT)-based selection typically requires intensive computation of transport plans, limiting scalability. We introduce a scalable Sinkhorn coreset method that permits closed-form updates of the entropically regularized OT coupling by allowing non-uniform coreset weights. This produces centroids that generalize k-means via soft assignments. We establish asymptotic consistency of the selected measure and Lipschitz stability to data perturbations, providing accuracy and robustness guarantees. Across synthetic and real-world benchmarks, the proposed method achieves competitive or improved approximation quality while substantially reducing runtime compared to Wasserstein- and standard Sinkhorn-based coreset selection, especially at large scale.

Abstract

Coreset Selection

Explore similar work

EMS Coreset: An Efficient Expectation-Maximization Algorithm for Sinkhorn Coreset

Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation

Beyond Worst-Case Coreset Bounds for kkk-Clustering via Determinantal Sampling

Beyond Worst-Case Coreset Bounds for $k$ -Clustering via Determinantal Sampling