stat.MLOct 1, 2026

High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning

Authors: Filip Kovačević, Edwige Cyffers, Stefano Sarao Mannelli, Marco Mondelli

Organizations: Institute of Science and Technology Austria. · CNRS Dauphine PSL. · Computer Science and Engineering Department, Chalmers University of Technology and University of Gothenburg. · School of Computer Science and Applied Mathematics, University of the Witwatersrand.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

To commit to buying external data or participate in collaborative learning, one must decide whether the additional data will improve prediction enough to justify the cost. This comes with several challenges: (i) the decision often relies only on aggregated statistics available publicly, rather than individual-level data; (ii) covariate and model shifts can induce negative transfer, so the additional data deteriorates rather than improves performance; (iii) if the data is sensitive, its privatization requires the injection of noise, which can also offset the benefit of a larger sample size. In this paper, we model the problem of dataset selection through high-dimensional regression with multiple heterogeneous sources and a weighted ridge estimator. Our approach uses only summary statistics and it gives privacy guarantees either on labels only or jointly on features and labels, in terms of ρρ-zero-concentrated differential privacy. The main technical contribution is a deterministic equivalent of the test error, which captures the interactions between sample size, covariance structure, model shift, regularization and privacy noise. Our theory allows to optimize hyperparameters (weights and ridge regularizers) and, more broadly, to decide when private external datasets are useful without accessing the data itself but only relying on population-level quantities. This provides a theoretically tractable foundation for private transfer learning, which we support via experiments on both synthetic and real-world datasets.

Figures & tables

Figure 1: Dependence of target excess risk on covariance shift, model shift and privacy noise in a synthetic linear regression experiment with two clients. We take x1∼N(0,Σ1)x_{1}\sim{\mathcal{N}}(0,\Sigma_{1}) and x2∼N(0,Σ2)x_{2}\sim{\mathcal{N}}(0,\Sigma_{2}) , with Σ1\Sigma_{1} and Σ2\Sigma_{2} of the form [K(ρ)]ij=ρ∣i−j∣[K(\rho)]_{ij}=\rho^{|i-j|} . We set Σ1=K(ρ1)\Sigma_{1}=K(\rho_{1}) with ρ1=0.1\rho_{1}=0.1 and Σ2=a2K(ρ2)\Sigma_{2}=a_{2}K(\rho_{2}) , so that \Tr(Σ2Σ1−1)=d\Tr(\Sigma_{2}\Sigma_{1}^{-1})=d . We take d=80d=80 , n1=200n_{1}=200 and plot ∥β^−β∥Σ12\|\hat{\beta}-\beta\|_{\Sigma_{1}}^{2} as a function of n2n_{2} . Solid lines show the deterministic equivalent prediction and dashed lines with crosses show the empirical mean over 400400 i.i.d. repetitions, with shaded 90%90\% confidence intervals. (a) Covariance shift: ρ2∈{0.1,0.7,0.9}\rho_{2}\in\{0.1,0.7,0.9\} , with model shift rδ=0.5r_{\delta}=0.5 and privacy noise variance σDP2=1\sigma_{\mathrm{DP}}^{2}=1 . (b) Model shift: rδ∈{0,0.5,3}r_{\delta}\in\{0,0.5,3\} , ρ2=0.7\rho_{2}=0.7 and σDP2=1\sigma_{\mathrm{DP}}^{2}=1 . (c) Privacy noise: σDP2∈{0,1,9}\sigma_{\mathrm{DP}}^{2}\in\{0,1,9\} , with ρ2=0.7\rho_{2}=0.7 and rδ=0.5r_{\delta}=0.5 . The middle curve is the same in all three panels.
Figure 2: Change in target risk relative to the target-only baseline as auxiliary data are added. (a) LSAC: Tier 3 is the target client and Tier 4 the auxiliary client. (b) Lianjia: District 8 is the target client and District 7 the auxiliary client. (c) Student Performance: Mousinho da Silveira is the target school and Gabriel Pereira the auxiliary school.
Figure 3: Percentage change in target test risk relative to the target-only baseline under sequential auxiliary client selection. (a) LSAC: Tier 3 is the target client and Tiers 1, 2, 4, 5, and 6 are candidate auxiliary clients. (b) Lianjia: District 8 is the target client and Districts 1, 2, 4, 6, 7, 10, 11 and 13 are candidate auxiliary clients.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Change in target excess risk as auxiliary data are added in two-client semi-synthetic regression experiments. Lower values correspond to a better risk. Original covariates are retained, while responses are generated from planted linear models calibrated to each client. (a) LSAC: Tier 3 as the target client, and Tier 4 as the auxiliary client. (b) Lianjia: District 8 as the target client, and District 7 as the auxiliary client. (c) Student Performance: Mousinho da Silveira (MS) as the target school, and Gabriel Pereira (GP) as the auxiliary school. Solid lines show direction-aware deterministic equivalent predictions (DE) using the known generating parameters; dashed lines with crosses show empirical results (EMP).
DatasetMethod1 Client2 Clients3 Clients
 Ours−2.41±0.32\mathbf{-2.41\pm 0.32}−3.05±0.33\mathbf{-3.05\pm 0.33}−3.05±0.33\mathbf{-3.05\pm 0.33}
 Random−1.35±0.18-1.35\pm 0.18−2.20±0.26-2.20\pm 0.26−2.75±0.29-2.75\pm 0.29
LSACLargest sample−1.32±0.26-1.32\pm 0.26−2.81±0.32-2.81\pm 0.32−2.88±0.32-2.88\pm 0.32
 Best local risk−0.94±0.20-0.94\pm 0.20−1.93±0.25-1.93\pm 0.25−2.72±0.30-2.72\pm 0.30
 Lower bound (oracle)−2.94±0.31-2.94\pm 0.31−3.39±0.32-3.39\pm 0.32−3.52±0.33-3.52\pm 0.33
 Ours−2.15±0.70\mathbf{-2.15\pm 0.70}−3.97±0.83\mathbf{-3.97\pm 0.83}−4.40±0.85\mathbf{-4.40\pm 0.85}
Appendix
Table 1: Change in target excess risk for auxiliary privacy noise variance σDP2=0\sigma_{\mathrm{DP}}^{2}=0 .
DatasetMethod1 Client2 Clients3 Clients
 Ours−1.56±0.27\mathbf{-1.56\pm 0.27}−2.52±0.32\mathbf{-2.52\pm 0.32}−2.84±0.33\mathbf{-2.84\pm 0.33}
 Random−1.04±0.15-1.04\pm 0.15−1.79±0.23-1.79\pm 0.23−2.36±0.27-2.36\pm 0.27
LSACLargest sample−0.95±0.20-0.95\pm 0.20−2.07±0.27-2.07\pm 0.27−2.25±0.28-2.25\pm 0.28
 Best local risk−0.80±0.19-0.80\pm 0.19−1.67±0.26-1.67\pm 0.26−2.35±0.30-2.35\pm 0.30
 Lower bound (oracle)−2.53±0.29-2.53\pm 0.29−3.02±0.32-3.02\pm 0.32−3.19±0.33-3.19\pm 0.33
 Ours−1.95±0.63\mathbf{-1.95\pm 0.63}−3.54±0.81\mathbf{-3.54\pm 0.81}−4.08±0.82\mathbf{-4.08\pm 0.82}
Appendix
Table 2: Change in target excess risk for auxiliary privacy noise variance σDP2=1\sigma_{\mathrm{DP}}^{2}=1 .
DatasetMethod1 Client2 Clients3 Clients
 Ours−1.02±0.20\mathbf{-1.02\pm 0.20}−1.93±0.29\mathbf{-1.93\pm 0.29}−2.31±0.29\mathbf{-2.31\pm 0.29}
 Random−0.78±0.12-0.78\pm 0.12−1.42±0.18-1.42\pm 0.18−1.94±0.23-1.94\pm 0.23
LSACLargest sample−0.69±0.17-0.69\pm 0.17−1.45±0.22-1.45\pm 0.22−1.66±0.23-1.66\pm 0.23
 Best local risk−0.69±0.18-0.69\pm 0.18−1.50±0.26-1.50\pm 0.26−2.10±0.26-2.10\pm 0.26
 Lower bound (oracle)−2.12±0.24-2.12\pm 0.24−2.59±0.28-2.59\pm 0.28−2.77±0.30-2.77\pm 0.30
 Ours−1.87±0.55\mathbf{-1.87\pm 0.55}−3.14±0.77\mathbf{-3.14\pm 0.77}−3.67±0.78\mathbf{-3.67\pm 0.78}
Appendix
Table 3: Change in target excess risk for auxiliary privacy noise variance σDP2=4\sigma_{\mathrm{DP}}^{2}=4 .
DatasetMethod1 Client2 Clients3 Clients
 Ours−0.60±0.16-0.60\pm 0.16−1.49±0.27\mathbf{-1.49\pm 0.27}−1.90±0.28-1.90\pm 0.28
 Random−0.62±0.10-0.62\pm 0.10−1.17±0.17-1.17\pm 0.17−1.66±0.22-1.66\pm 0.22
LSACLargest sample−0.52±0.16-0.52\pm 0.16−1.05±0.20-1.05\pm 0.20−1.32±0.21-1.32\pm 0.21
 Best local risk−0.66±0.19\mathbf{-0.66\pm 0.19}−1.28±0.25-1.28\pm 0.25−1.93±0.28\mathbf{-1.93\pm 0.28}
 Lower bound (oracle)−1.87±0.24-1.87\pm 0.24−2.30±0.27-2.30\pm 0.27−2.48±0.29-2.48\pm 0.29
 Ours−1.40±0.54\mathbf{-1.40\pm 0.54}−2.43±0.67-2.43\pm 0.67−3.17±0.68\mathbf{-3.17\pm 0.68}
Appendix
Table 4: Change in target excess risk for auxiliary privacy noise variance σDP2=16\sigma_{\mathrm{DP}}^{2}=16 .
More in this topic

High-Dimensional Asymptotics

See all 19 papers

Explore similar work

CardsList
Oct 5, 2026

Differentially Private Mixing of Public Datasets Improves Private Learning

Many machine learning applications involve sensitive data and therefore require training under differential privacy (DP). However, DP training often degrades model utility. In some cases, first pre-training the model on "public" data before finetuning with DP on the sensitive data can reduce the drop in utility. However, the success of this depends on how relevant the selected public dataset is to the sensitive data. We introduce the first pipeline that privately learns the mixture of several public datasets to pretrain on for a given sensitive downstream task. Our key insight is that we can privately find the best mixture of multiple public datasets by privately learning a low-dimensional linear model. We tested our method on the NIH dataset for X-ray classification and the ENRON email dataset for language modeling. Applying our method to find tailored mixtures of X-ray datasets to pretrain on for diseases in the NIH ChestX-ray14 dataset, we improved macro AUC by up to 0.037 across privacy budgets compared to the baselines, with gains as large as +22.8% relative AUC on Cardiomegaly at ε=1ε=1. For DP training on the ENRON dataset, pre-training on our mixture of The Common Pile (a collection of public-domain text datasets) decreased test perplexity by 16% relative to the baseline mixtures.

Yufei Chen, Tejumade Afonja, Anvith Thudi +1

University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany

Privacy-Preserving MLDifferential PrivacyData Mixture Optimization
AskPDF
Oct 16, 2026

Towards Hyperparameter Transfer for Differentially Private Optimization

Scaling laws and hyperparameter tuning remain central challenges in training large-scale models, particularly under differential privacy (DP), where optimization is further complicated by noise injection and per-sample gradient clipping. While Maximal Update Parametrization (μμP) allows for zero-shot hyperparameter transfer in standard training, we show that it fails under DP constraints due to the distinct spectral properties of high-dimensional noise versus low-rank gradients. In this work, we introduce DP-μμP, a theoretically grounded framework that extends μμP to the privacy-preserving setting. By analyzing the interaction between gradient clipping, noise injection, and feature learning, we derive a Signal-to-Noise Ratio (SNR) condition that governs layer-wise learning rate scaling. Combining this with our proposed spectral scaling formulation, we provide specific scaling rules for DP-SGD, DP-Adam, and DP-Muon optimizers. Empirically, we demonstrate that DP-μμP enables effective zero-shot transfer of learning rates across model widths on diverse architectures (MLP, ViT, GPT-2), matching or exceeding the performance of individually tuned baselines while significantly reducing computational costs.

Tianze Wang, Zhiqi Bu, Linjun Zhang

Department of Statistics, Rutgers University · Independent Researcher

Neural Network OptimizationHyperparameter TransferLayer-Wise Learning Rate AdaptationDifferentially Private Stochastic Gradient Descent
AskPDF
Oct 16, 2026

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the L2L_2 distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.

Yalda Shabanzadeh, Diyuan Wu, Marco Mondelli

Institute of Science and Technology Austria (ISTA).

Transfer LearningRidge RegressionHigh-Dimensional Linear RegressionMinimax Estimation
AskPDF