High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning

Filip Kovačević; Edwige Cyffers; Stefano Sarao Mannelli; Marco Mondelli

stat.MLOct 1, 2026

High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning

Authors: Filip Kovačević, Edwige Cyffers, Stefano Sarao Mannelli, Marco Mondelli

Organizations: Institute of Science and Technology Austria. · CNRS Dauphine PSL. · Computer Science and Engineering Department, Chalmers University of Technology and University of Gothenburg. · School of Computer Science and Applied Mathematics, University of the Witwatersrand.

Abstract

To commit to buying external data or participate in collaborative learning, one must decide whether the additional data will improve prediction enough to justify the cost. This comes with several challenges: (i) the decision often relies only on aggregated statistics available publicly, rather than individual-level data; (ii) covariate and model shifts can induce negative transfer, so the additional data deteriorates rather than improves performance; (iii) if the data is sensitive, its privatization requires the injection of noise, which can also offset the benefit of a larger sample size. In this paper, we model the problem of dataset selection through high-dimensional regression with multiple heterogeneous sources and a weighted ridge estimator. Our approach uses only summary statistics and it gives privacy guarantees either on labels only or jointly on features and labels, in terms of $ρ$ -zero-concentrated differential privacy. The main technical contribution is a deterministic equivalent of the test error, which captures the interactions between sample size, covariance structure, model shift, regularization and privacy noise. Our theory allows to optimize hyperparameters (weights and ridge regularizers) and, more broadly, to decide when private external datasets are useful without accessing the data itself but only relying on population-level quantities. This provides a theoretically tractable foundation for private transfer learning, which we support via experiments on both synthetic and real-world datasets.

Figures & tables

Figure 1: Dependence of target excess risk on covariance shift, model shift and privacy noise in a synthetic linear regression experiment with two clients. We take $x_{1}\sim{\mathcal{N}}(0,\Sigma_{1})$ and $x_{2}\sim{\mathcal{N}}(0,\Sigma_{2})$ , with $\Sigma_{1}$ and $\Sigma_{2}$ of the form $[K(\rho)]_{ij}=\rho^{|i-j|}$ . We set $\Sigma_{1}=K(\rho_{1})$ with $\rho_{1}=0.1$ and $\Sigma_{2}=a_{2}K(\rho_{2})$ , so that $\Tr(\Sigma_{2}\Sigma_{1}^{-1})=d$ . We take $d=80$ , $n_{1}=200$ and plot $\|\hat{\beta}-\beta\|_{\Sigma_{1}}^{2}$ as a function of $n_{2}$ . Solid lines show the deterministic equivalent prediction and dashed lines with crosses show the empirical mean over $400$ i.i.d. repetitions, with shaded $90\%$ confidence intervals. (a) Covariance shift: $\rho_{2}\in\{0.1,0.7,0.9\}$ , with model shift $r_{\delta}=0.5$ and privacy noise variance $\sigma_{\mathrm{DP}}^{2}=1$ . (b) Model shift: $r_{\delta}\in\{0,0.5,3\}$ , $\rho_{2}=0.7$ and $\sigma_{\mathrm{DP}}^{2}=1$ . (c) Privacy noise: $\sigma_{\mathrm{DP}}^{2}\in\{0,1,9\}$ , with $\rho_{2}=0.7$ and $r_{\delta}=0.5$ . The middle curve is the same in all three panels.

Panel of Figure 2 — Figure 2: Change in target risk relative to the target-only baseline as auxiliary data are added. (a) LSAC: Tier 3 is the target client and Tier 4 the auxiliary client. (b) Lianjia: District 8 is the target client and District 7 the auxiliary client. (c) Student Performance: Mousinho da Silveira is the target school and Gabriel Pereira the auxiliary school.

Panel of Figure 3 — Figure 3: Percentage change in target test risk relative to the target-only baseline under sequential auxiliary client selection. (a) LSAC: Tier 3 is the target client and Tiers 1, 2, 4, 5, and 6 are candidate auxiliary clients. (b) Lianjia: District 8 is the target client and Districts 1, 2, 4, 6, 7, 10, 11 and 13 are candidate auxiliary clients.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Change in target excess risk as auxiliary data are added in two-client semi-synthetic regression experiments. Lower values correspond to a better risk. Original covariates are retained, while responses are generated from planted linear models calibrated to each client. (a) LSAC: Tier 3 as the target client, and Tier 4 as the auxiliary client. (b) Lianjia: District 8 as the target client, and District 7 as the auxiliary client. (c) Student Performance: Mousinho da Silveira (MS) as the target school, and Gabriel Pereira (GP) as the auxiliary school. Solid lines show direction-aware deterministic equivalent predictions (DE) using the known generating parameters; dashed lines with crosses show empirical results (EMP).

Dataset	Method	1 Client	2 Clients	3 Clients
	Ours	$\mathbf{-2.41\pm 0.32}$	$\mathbf{-3.05\pm 0.33}$	$\mathbf{-3.05\pm 0.33}$
	Random	$-1.35\pm 0.18$	$-2.20\pm 0.26$	$-2.75\pm 0.29$
LSAC	Largest sample	$-1.32\pm 0.26$	$-2.81\pm 0.32$	$-2.88\pm 0.32$
	Best local risk	$-0.94\pm 0.20$	$-1.93\pm 0.25$	$-2.72\pm 0.30$
	Lower bound (oracle)	$-2.94\pm 0.31$	$-3.39\pm 0.32$	$-3.52\pm 0.33$
	Ours	$\mathbf{-2.15\pm 0.70}$	$\mathbf{-3.97\pm 0.83}$	$\mathbf{-4.40\pm 0.85}$

Appendix

Table 1: Change in target excess risk for auxiliary privacy noise variance

\sigma_{\mathrm{DP}}^{2}=0

Dataset	Method	1 Client	2 Clients	3 Clients
	Ours	$\mathbf{-1.56\pm 0.27}$	$\mathbf{-2.52\pm 0.32}$	$\mathbf{-2.84\pm 0.33}$
	Random	$-1.04\pm 0.15$	$-1.79\pm 0.23$	$-2.36\pm 0.27$
LSAC	Largest sample	$-0.95\pm 0.20$	$-2.07\pm 0.27$	$-2.25\pm 0.28$
	Best local risk	$-0.80\pm 0.19$	$-1.67\pm 0.26$	$-2.35\pm 0.30$
	Lower bound (oracle)	$-2.53\pm 0.29$	$-3.02\pm 0.32$	$-3.19\pm 0.33$
	Ours	$\mathbf{-1.95\pm 0.63}$	$\mathbf{-3.54\pm 0.81}$	$\mathbf{-4.08\pm 0.82}$

Appendix

Table 2: Change in target excess risk for auxiliary privacy noise variance

\sigma_{\mathrm{DP}}^{2}=1

Dataset	Method	1 Client	2 Clients	3 Clients
	Ours	$\mathbf{-1.02\pm 0.20}$	$\mathbf{-1.93\pm 0.29}$	$\mathbf{-2.31\pm 0.29}$
	Random	$-0.78\pm 0.12$	$-1.42\pm 0.18$	$-1.94\pm 0.23$
LSAC	Largest sample	$-0.69\pm 0.17$	$-1.45\pm 0.22$	$-1.66\pm 0.23$
	Best local risk	$-0.69\pm 0.18$	$-1.50\pm 0.26$	$-2.10\pm 0.26$
	Lower bound (oracle)	$-2.12\pm 0.24$	$-2.59\pm 0.28$	$-2.77\pm 0.30$
	Ours	$\mathbf{-1.87\pm 0.55}$	$\mathbf{-3.14\pm 0.77}$	$\mathbf{-3.67\pm 0.78}$

Appendix

Table 3: Change in target excess risk for auxiliary privacy noise variance

\sigma_{\mathrm{DP}}^{2}=4

Dataset	Method	1 Client	2 Clients	3 Clients
	Ours	$-0.60\pm 0.16$	$\mathbf{-1.49\pm 0.27}$	$-1.90\pm 0.28$
	Random	$-0.62\pm 0.10$	$-1.17\pm 0.17$	$-1.66\pm 0.22$
LSAC	Largest sample	$-0.52\pm 0.16$	$-1.05\pm 0.20$	$-1.32\pm 0.21$
	Best local risk	$\mathbf{-0.66\pm 0.19}$	$-1.28\pm 0.25$	$\mathbf{-1.93\pm 0.28}$
	Lower bound (oracle)	$-1.87\pm 0.24$	$-2.30\pm 0.27$	$-2.48\pm 0.29$
	Ours	$\mathbf{-1.40\pm 0.54}$	$-2.43\pm 0.67$	$\mathbf{-3.17\pm 0.68}$

Appendix

Table 4: Change in target excess risk for auxiliary privacy noise variance

\sigma_{\mathrm{DP}}^{2}=16

High-Dimensional Asymptotics

See all 19 papers

Differentially Private Mixing of Public Datasets Improves Private Learning
Oct 5, 2026Yufei Chen, Tejumade Afonja, Anvith Thudi +1Privacy-Preserving ML Differential Privacy
University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany
Towards Hyperparameter Transfer for Differentially Private Optimization
Oct 16, 2026Tianze Wang, Zhiqi Bu, Linjun ZhangNeural Network Optimization Hyperparameter Transfer
Department of Statistics, Rutgers University · Independent Researcher
Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Oct 16, 2026Yalda Shabanzadeh, Diyuan Wu, Marco MondelliTransfer Learning Ridge Regression
Institute of Science and Technology Austria (ISTA).

Oct 5, 2026J/K move · Enter open · S save

Differentially Private Mixing of Public Datasets Improves Private Learning

Yufei Chen, Tejumade Afonja, Anvith Thudi, Nicolas Papernot

University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany

Privacy-Preserving ML Differential Privacy Data Mixture Optimization

Open paper AskPDF arXiv

Many machine learning applications involve sensitive data and therefore require training under differential privacy (DP). However, DP training often degrades model utility. In some cases, first pre-training the model on "public" data before finetuning with DP on the sensitive data can reduce the drop in utility. However, the success of this depends on how relevant the selected public dataset is to the sensitive data. We introduce the first pipeline that privately learns the mixture of several public datasets to pretrain on for a given sensitive downstream task. Our key insight is that we can privately find the best mixture of multiple public datasets by privately learning a low-dimensional linear model. We tested our method on the NIH dataset for X-ray classification and the ENRON email dataset for language modeling. Applying our method to find tailored mixtures of X-ray datasets to pretrain on for diseases in the NIH ChestX-ray14 dataset, we improved macro AUC by up to 0.037 across privacy budgets compared to the baselines, with gains as large as +22.8% relative AUC on Cardiomegaly at

ε=1

. For DP training on the ENRON dataset, pre-training on our mixture of The Common Pile (a collection of public-domain text datasets) decreased test perplexity by 16% relative to the baseline mixtures.

Abstract

High-Dimensional Asymptotics

Explore similar work

Differentially Private Mixing of Public Datasets Improves Private Learning

Towards Hyperparameter Transfer for Differentially Private Optimization

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning