Differentially Private Mixing of Public Datasets Improves Private Learning

Yufei Chen; Tejumade Afonja; Anvith Thudi; Nicolas Papernot

cs.LGOct 5, 2026

Differentially Private Mixing of Public Datasets Improves Private Learning

Authors: Yufei Chen, Tejumade Afonja, Anvith Thudi, Nicolas Papernot

Organizations: University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany

Abstract

Many machine learning applications involve sensitive data and therefore require training under differential privacy (DP). However, DP training often degrades model utility. In some cases, first pre-training the model on "public" data before finetuning with DP on the sensitive data can reduce the drop in utility. However, the success of this depends on how relevant the selected public dataset is to the sensitive data. We introduce the first pipeline that privately learns the mixture of several public datasets to pretrain on for a given sensitive downstream task. Our key insight is that we can privately find the best mixture of multiple public datasets by privately learning a low-dimensional linear model. We tested our method on the NIH dataset for X-ray classification and the ENRON email dataset for language modeling. Applying our method to find tailored mixtures of X-ray datasets to pretrain on for diseases in the NIH ChestX-ray14 dataset, we improved macro AUC by up to 0.037 across privacy budgets compared to the baselines, with gains as large as +22.8% relative AUC on Cardiomegaly at $ε=1$ . For DP training on the ENRON dataset, pre-training on our mixture of The Common Pile (a collection of public-domain text datasets) decreased test perplexity by 16% relative to the baseline mixtures.

Figures & tables

Figure 1: Overview of DP-MixMin. Proxy models trained on public data are combined with mixture weights

\lambda

learned with DP-SGD on a fraction

\rho

of the sensitive data (split A). We then pretrain on public data according to

\lambda

and DP fine-tune on the rest (split B).

Figure 2: DP-MixMin beats Uniform and Random Search at every $\varepsilon$ . Macro AUC on CheXpert/MIMIC-CXR $\rightarrow$ NIH ( $\uparrow$ ) and Enron test perplexity at $\rho=0.3$ ( $\downarrow$ ).

Figure 3: Per-task relative AUC gain over Random Init on NIH Chest X-ray at

\epsilon=1

. Bars show

\big(\mathrm{AUC(method)}-\mathrm{AUC(Random\ Init)}\big)/\mathrm{AUC(Random\ Init)}

as a percentage, with propagated standard deviation over 3–5 seeds. The macro AUC gain (unweighted mean across 14 tasks) is reported next to each subplot title; DP-MixMin roughly doubles, and in most cases more than doubles, the gain of both Uniform and Random Search on each public pretraining source. Full raw AUCs are reported in Tables 7 and 8 .

Method	$\rho$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$
DP-MixMin	$0.3$	$\mathbf{16.78\pm 0.55}$	$\mathbf{16.44\pm 0.95}$	$\mathbf{15.71\pm 0.77}$
DP-MixMin	$0.2$	$17.52\pm 1.55$	—	—
DP-MixMin	$0.1$	$17.97\pm 2.72$	—	—
Uniform	—	$19.88\pm 1.10$	$19.18\pm 1.01$	$18.24\pm 0.89$
Random Search	$0.3$	$20.46\pm 0.70$	$19.72\pm 0.66$	$18.72\pm 0.62$

Table 1: Enron test perplexity (mean

\pm

std) after DP finetuning at

\delta{=}10^{-5}

. DP-MixMin beats both Uniform and Random Search at every privacy budget. At

\epsilon{=}1

, the mixture quality (and stability) improves monotonically with the mixture weight finding fraction

\rho

Method	$\rho$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$
DP-MixMin	$0.1$	$\mathbf{12.22\pm 0.17}$	—	—
Uniform	—	$14.05\pm 0.17$	$14.04\pm 0.17$	$14.02\pm 0.17$

Table 2: OpenWebMath test perplexity (mean

\pm

std) after DP finetuning at

\delta{=}10^{-5}

. DP-MixMin lowers test perplexity by roughly

13\%

over uniform Common Pile pretraining at

\epsilon{=}1

. The uniform baseline is essentially flat in

\epsilon

on this domain, showing DP-MixMin at

\epsilon=1

is still better.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Proxy epochs	Atelectasis AUC
1	$0.744\pm 0.005$
2	$0.745\pm 0.005$
4 (main experiments)	$0.742\pm 0.006$
6	$0.749\pm 0.003$

Appendix

Table 3: DP-MixMin is insensitive to proxy training budget. NIH Atelectasis AUC (mean

\pm

std over 3 seeds) with CheXpert pretraining at

\epsilon=1

Figure 4: Excess error scales roughly as

\sqrt{k}

for large

k

. Average excess error of DP-MixMin over the 10 CIFAR-100 tasks as the number of public sources

k

increases (both axes log scale). Error bars and the shaded band show

\pm 1

standard deviation across tasks. The dashed red line is a

\sqrt{k}

reference; the fitted slope for

k\geq 32

0.49

, consistent with the predicted

\sqrt{k}

rate.

$k$	class 0	class 1	class 2	class 3	class 4	class 5	class 6	class 7	class 8	class 9	Avg
2	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.2}$	$0.1{\scriptstyle\pm 0.2}$	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.2}$	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.2}$	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.2}$	$\mathbf{0.1{\scriptstyle\pm 0.1}}$
4	$0.1{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.1{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.1{\scriptstyle\pm 0.0}$	$\mathbf{0.0{\scriptstyle\pm 0.0}}$
8	$1.1{\scriptstyle\pm 0.1}$	$1.4{\scriptstyle\pm 0.2}$	$1.3{\scriptstyle\pm 0.2}$	$3.7{\scriptstyle\pm 0.2}$	$3.3{\scriptstyle\pm 0.2}$	$2.1{\scriptstyle\pm 0.1}$	$2.3{\scriptstyle\pm 0.2}$	$2.8{\scriptstyle\pm 0.1}$	$4.1{\scriptstyle\pm 0.2}$	$2.1{\scriptstyle\pm 0.1}$	$\mathbf{2.4{\scriptstyle\pm 1.0}}$
16	$5.7{\scriptstyle\pm 0.4}$	$7.0{\scriptstyle\pm 0.2}$	$3.3{\scriptstyle\pm 0.4}$	$2.9{\scriptstyle\pm 0.3}$	$2.4{\scriptstyle\pm 0.4}$	$7.0{\scriptstyle\pm 0.6}$	$4.2{\scriptstyle\pm 0.3}$	$2.8{\scriptstyle\pm 0.4}$	$5.8{\scriptstyle\pm 0.5}$	$7.8{\scriptstyle\pm 0.5}$	$\mathbf{4.9{\scriptstyle\pm 2.0}}$
32	$2.8{\scriptstyle\pm 0.2}$	$6.7{\scriptstyle\pm 0.5}$	$3.4{\scriptstyle\pm 0.3}$	$2.0{\scriptstyle\pm 0.1}$	$1.9{\scriptstyle\pm 0.1}$	$3.4{\scriptstyle\pm 0.2}$	$3.7{\scriptstyle\pm 0.2}$	$2.7{\scriptstyle\pm 0.3}$	$5.8{\scriptstyle\pm 0.3}$	$5.3{\scriptstyle\pm 0.3}$	$\mathbf{3.8{\scriptstyle\pm 1.6}}$
64	$3.8{\scriptstyle\pm 0.2}$	$2.1{\scriptstyle\pm 0.3}$	$3.7{\scriptstyle\pm 0.2}$	$3.4{\scriptstyle\pm 0.3}$	$2.6{\scriptstyle\pm 0.3}$	$4.4{\scriptstyle\pm 0.3}$	$4.4{\scriptstyle\pm 0.2}$	$5.0{\scriptstyle\pm 0.5}$	$1.8{\scriptstyle\pm 0.1}$	$2.3{\scriptstyle\pm 0.3}$	$\mathbf{3.4{\scriptstyle\pm 1.1}}$

Appendix

Table 4: DP-MixMin excess error

\alpha=L(\bm{\lambda}_{\mathrm{DP}})-L(\bm{\lambda}^{\star})

across the number of public sources

k

(ImageNet-1k

\to

CIFAR-100,

\varepsilon{=}1

\delta{=}10^{-5}

), for all 10 CIFAR-100 one-vs-rest tasks. Mean

\pm

std over 5 seeds, in units of

10^{-3}

. The last column ( Avg ) averages over all 5 seeds and 10 tasks.

Class	E1	E2	E6
Atelectasis	$0.868\pm 0.015$	$0.986\pm 0.003$	$0.972\pm 0.014$
Consolidation	$0.886\pm 0.004$	$0.941\pm 0.035$	$0.944\pm 0.012$
Infiltration	$0.895\pm 0.018$	$0.873\pm 0.046$	$0.848\pm 0.023$
Pneumothorax	$0.952\pm 0.019$	$0.976\pm 0.003$	$0.975\pm 0.003$
Edema	$0.978\pm 0.003$	$0.965\pm 0.003$	$0.887\pm 0.029$
Emphysema	$0.986\pm 0.006$	$0.988\pm 0.002$	$0.970\pm 0.009$

Appendix

Table 5: Learned mixture is stable across proxy training budget. Cosine similarity of the DP-MixMin mixture (early-stop,

\varepsilon{=}1

) at proxy epochs

\{1,2,6\}

to the main

E{=}4

mixture, per NIH class (mean

\pm

std, 3 seeds).

E{=}4

is the self-reference (

1.000

Class	Sequential	Parallel	Non-private
Atelectasis	$0.740\pm 0.010$	$0.747\pm 0.005$	$0.746\pm 0.006$

Appendix

Table 6: Sequential and parallel composition perform similarly. NIH AUC (mean

\pm

std over 3 seeds), CheXpert pretraining, total

\epsilon=1

	DP-MixMin			Uniform			Random Search			No Pretraining
Class	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$
Atelectasis	0.740 $\pm$ 0.012	0.747 $\pm$ 0.005	0.747 $\pm$ 0.007	0.707 $\pm$ 0.013	0.724 $\pm$ 0.009	0.736 $\pm$ 0.008	0.724 $\pm$ 0.009	0.733 $\pm$ 0.011	0.739 $\pm$ 0.010	0.679 $\pm$ 0.006	0.696 $\pm$ 0.008	0.715 $\pm$ 0.006
Consolidation	0.742 $\pm$ 0.014	0.745 $\pm$ 0.017	0.748 $\pm$ 0.014	0.745 $\pm$ 0.015	0.749 $\pm$ 0.014	0.752 $\pm$ 0.013	0.734 $\pm$ 0.008	0.740 $\pm$ 0.011	0.742 $\pm$ 0.010	0.710 $\pm$ 0.006	0.716 $\pm$ 0.010	0.725 $\pm$ 0.010
Infiltration	0.671 $\pm$ 0.004	0.674 $\pm$ 0.005	0.678 $\pm$ 0.002	0.668 $\pm$ 0.004	0.670 $\pm$ 0.003	0.672 $\pm$ 0.003	0.664 $\pm$ 0.004	0.670 $\pm$ 0.003	0.675 $\pm$ 0.003	0.663 $\pm$ 0.004	0.672 $\pm$ 0.003	0.680 $\pm$ 0.002
Pneumothorax	0.780 $\pm$ 0.007	0.788 $\pm$ 0.002	0.798 $\pm$ 0.004	0.697 $\pm$ 0.008	0.728 $\pm$ 0.008	0.753 $\pm$ 0.009	0.674 $\pm$ 0.005	0.701 $\pm$ 0.009	0.726 $\pm$ 0.007	0.650 $\pm$ 0.009	0.684 $\pm$ 0.005	0.723 $\pm$ 0.005
Edema	0.829 $\pm$ 0.017	0.836 $\pm$ 0.014	0.845 $\pm$ 0.013	0.829 $\pm$ 0.006	0.835 $\pm$ 0.007	0.838 $\pm$ 0.008	0.844 $\pm$ 0.008	0.850 $\pm$ 0.008	0.857 $\pm$ 0.009	0.804 $\pm$ 0.012	0.816 $\pm$ 0.010	0.829 $\pm$ 0.010

Appendix

Table 7: Per-class final AUC on NIH ChestX-ray14 after transfer from CheXpert. Each entry is the mean

\pm

standard deviation of AUC over 3–5 seeds at

\delta=10^{-5}

. Bold indicates the best method per (class,

\epsilon

) cell. DP-MixMin achieves the best macro AUC across all three privacy budgets, with absolute gains of

+0.029

+0.037

over uniform pretraining.

	DP-MixMin			Uniform			Random Search			No Pretraining
Class	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$
Atelectasis	0.751 $\pm$ 0.010	0.749 $\pm$ 0.010	0.737 $\pm$ 0.000	0.728 $\pm$ 0.009	0.734 $\pm$ 0.008	0.740 $\pm$ 0.008	0.742 $\pm$ 0.006	0.749 $\pm$ 0.006	0.755 $\pm$ 0.005	0.679 $\pm$ 0.006	0.696 $\pm$ 0.008	0.715 $\pm$ 0.006
Consolidation	0.756 $\pm$ 0.012	0.758 $\pm$ 0.011	0.759 $\pm$ 0.015	0.748 $\pm$ 0.010	0.751 $\pm$ 0.011	0.753 $\pm$ 0.011	0.743 $\pm$ 0.012	0.746 $\pm$ 0.012	0.749 $\pm$ 0.012	0.710 $\pm$ 0.006	0.716 $\pm$ 0.010	0.725 $\pm$ 0.010
Infiltration	0.674 $\pm$ 0.008	0.679 $\pm$ 0.004	0.682 $\pm$ 0.005	0.667 $\pm$ 0.004	0.673 $\pm$ 0.005	0.677 $\pm$ 0.006	0.673 $\pm$ 0.006	0.676 $\pm$ 0.006	0.680 $\pm$ 0.003	0.663 $\pm$ 0.004	0.672 $\pm$ 0.003	0.680 $\pm$ 0.002
Pneumothorax	0.754 $\pm$ 0.013	0.770 $\pm$ 0.004	0.782 $\pm$ 0.005	0.719 $\pm$ 0.012	0.741 $\pm$ 0.015	0.754 $\pm$ 0.015	0.679 $\pm$ 0.015	0.707 $\pm$ 0.016	0.732 $\pm$ 0.016	0.650 $\pm$ 0.009	0.684 $\pm$ 0.005	0.723 $\pm$ 0.005
Edema	0.845 $\pm$ 0.003	0.849 $\pm$ 0.004	0.853 $\pm$ 0.005	0.839 $\pm$ 0.007	0.841 $\pm$ 0.008	0.845 $\pm$ 0.007	0.826 $\pm$ 0.011	0.834 $\pm$ 0.008	0.841 $\pm$ 0.011	0.804 $\pm$ 0.012	0.816 $\pm$ 0.010	0.829 $\pm$ 0.010

Appendix

Table 8: Per-class final AUC on NIH ChestX-ray14 after transfer from MIMIC-CXR. Each entry is the mean

\pm

standard deviation of AUC over 3–5 seeds at

\delta=10^{-5}

. Bold indicates the best method per (class,

\epsilon

) cell. DP-MixMin achieves the best macro AUC across all three privacy budgets, with absolute gains of

+0.024

+0.033

over uniform pretraining.

Figure 5: DP-MixMin beats both uniform-mixture and random-search Common Pile pretraining from scratch on Enron email completion across all three privacy budgets, with the largest gain at the tightest budget. Test-set perplexity after DP fine-tuning on Enron (

\delta{=}10^{-5}

) at

\epsilon\in\{1,2,5\}

. All three methods pretrain GPT-2 medium from random initialization on Common Pile and are then DP-finetuned on Enron under identical settings. DP-MixMin reduces test perplexity by roughly

16\%/14\%/14\%

over uniform and

18\%/17\%/16\%

over random-mixture pretraining at

\epsilon\in\{1,2,5\}

Figure 6: DP-MixMin consistently outperforms uniform Common Pile pretraining on Enron at

\epsilon{=}1

across all proxy-split fractions, with larger fractions yielding both lower perplexity and tighter cross-seed variance. Test-set perplexity after DP fine-tuning on Enron (

\delta{=}10^{-5}

) at

\epsilon{=}1

, varying the proxy fraction

\rho\in\{0.1,0.2,0.3\}

used by MixMin to estimate domain weights. The uniform baseline is independent of

\rho

and is shown for reference at all three values. DP-MixMin reduces test perplexity by

10\%/12\%/16\%

over uniform at

\rho\in\{0.1,0.2,0.3\}

, with

\rho{=}0.3

giving the most stable mixture (cross-seed

\sigma

drops from

2.72\to 0.45

NIH Idx	Class Name	# Pos.	% Pos.	Match	CheXpert/MIMIC-CXR Label
0	Atelectasis	11,559	10.31%	Yes	Atelectasis
1	Consolidation	4,667	4.16%	Yes	Consolidation
2	Infiltration	19,894	17.75%	Partial	Lung Opacity
3	Pneumothorax	5,302	4.73%	Yes	Pneumothorax
4	Edema	2,303	2.05%	Yes	Edema
5	Emphysema	2,516	2.24%	None	—

Appendix

Table 9: NIH ChestX-ray14 class distribution and correspondence to CheXpert/MIMIC-CXR labels. The Match column indicates whether each NIH class has a direct ( Yes ), Partial , or no ( None ) equivalent in CheXpert/MIMIC-CXR.

CheXpert $\to$ NIH ChestX-ray14
	$\epsilon=1$		$\epsilon=2$		$\epsilon=5$
	abs	rel	abs	rel	abs	rel
Group A ( $n=7$ )	$+0.040$	$+5.6\%$	$+0.030$	$+4.0\%$	$+0.029$	$+3.8\%$
Group B ( $n=7$ )	$+0.034$	$+6.0\%$	$+0.034$	$+5.7\%$	$+0.028$	$+4.7\%$
MIMIC-CXR $\to$ NIH ChestX-ray14
Group A ( $n=7$ )	$+0.029$	$+4.0\%$	$+0.020$	$+2.7\%$	$+0.015$	$+1.8\%$

Appendix

Table 10: Average AUC gains of DP-MixMin over Uniform pretraining, grouped by class type. Group A: well-represented classes with a direct CheXpert match (Atelectasis, Cardiomegaly, Effusion, Pneumonia, Pneumothorax, Consolidation, Edema). Group B: rare or unmatched classes (Infiltration, Mass, Nodule, Emphysema, Fibrosis, Pleural Thickening, Hernia); see Table 9 . Each cell is the within-group mean of per-class absolute (abs) and relative (rel) AUC gain of DP-MixMin over Uniform pretraining at

\delta=10^{-5}

CheXpert $\to$ NIH ChestX-ray14
		$\epsilon=0.2$		$\epsilon=0.5$		$\epsilon=1$		$\epsilon=2$		$\epsilon=5$
src	$n$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
CheXpert	$70$	$0.70{\scriptstyle\pm.08}$	$0.81{\scriptstyle\pm.14}$	$\mathbf{0.93}{\scriptstyle\pm.03}$	$\mathbf{0.37}{\scriptstyle\pm.09}$	$\mathbf{0.97}{\scriptstyle\pm.01}$	$\mathbf{0.24}{\scriptstyle\pm.06}$	$\mathbf{0.98}{\scriptstyle\pm.01}$	$\mathbf{0.19}{\scriptstyle\pm.05}$	$\mathbf{0.99}{\scriptstyle\pm.01}$	$\mathbf{0.16}{\scriptstyle\pm.04}$
MIMIC-CXR $\to$ NIH ChestX-ray14
MIMIC-CXR	$70$	$0.58{\scriptstyle\pm.17}$	$1.04{\scriptstyle\pm.30}$	$\mathbf{0.82}{\scriptstyle\pm.09}$	$\mathbf{0.57}{\scriptstyle\pm.20}$	$\mathbf{0.93}{\scriptstyle\pm.04}$	$\mathbf{0.33}{\scriptstyle\pm.10}$	$\mathbf{0.97}{\scriptstyle\pm.01}$	$\mathbf{0.22}{\scriptstyle\pm.05}$	$\mathbf{0.98}{\scriptstyle\pm.01}$	$\mathbf{0.16}{\scriptstyle\pm.03}$

Appendix

Table 11: Weight reconstruction of DP-MixMin across 14 NIH ChestX-ray14 tasks. Mean

\pm

std over

n{=}70

paired comparisons (14 tasks

\times

5 seeds). We compare the private weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Uniform references are CheXpert:

\mathrm{cos}_{\mathbf{u}}{=}0.75

L_{1,\mathbf{u}}{=}0.68

; MIMIC-CXR:

\mathrm{cos}_{\mathbf{u}}{=}0.80

L_{1,\mathbf{u}}{=}0.60

. The DP-MixMin selection budget

\rho=0.1

. Bold values improve over the corresponding uniform reference.

		$\rho{=}0.05$		$\rho{=}0.1$		$\rho{=}0.2$
		$n{\approx}4{,}485$ , #pos ${\approx}9$		$n{\approx}8{,}970$ , #pos ${\approx}18$		$n{\approx}17{,}939$ , #pos ${\approx}36$
Pretrain	$\varepsilon$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
CheXpert	$0.2$	$\mathbf{0.79}{\scriptstyle\pm 0.01}$	$\mathbf{0.64}{\scriptstyle\pm 0.02}$	$\mathbf{0.94}{\scriptstyle\pm 0.01}$	$\mathbf{0.32}{\scriptstyle\pm 0.01}$	$\mathbf{0.98}{\scriptstyle\pm 0.00}$	$\mathbf{0.17}{\scriptstyle\pm 0.01}$
	$0.5$	$\mathbf{0.96}{\scriptstyle\pm 0.01}$	$\mathbf{0.27}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.15}{\scriptstyle\pm 0.01}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.10}{\scriptstyle\pm 0.01}$
	$1.0$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.10}{\scriptstyle\pm 0.01}$	$\mathbf{1.00}{\scriptstyle\pm 0.00}$	$\mathbf{0.08}{\scriptstyle\pm 0.01}$
	$2.0$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.11}{\scriptstyle\pm 0.02}$	$\mathbf{1.00}{\scriptstyle\pm 0.00}$	$\mathbf{0.08}{\scriptstyle\pm 0.01}$	$\mathbf{1.00}{\scriptstyle\pm 0.00}$	$\mathbf{0.08}{\scriptstyle\pm 0.01}$

Appendix

Table 12: Weight reconstruction of DP-MixMin for NIH Hernia (0.20% prevalence in

D^{\mathrm{priv}}

, 227 total positives). Mean

\pm

std over 5 paired comparisons (5 seeds). We compare the private weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Uniform references: CheXpert

\mathrm{cos}_{\mathbf{u}}{=}0.64

L_{1,\mathbf{u}}{=}0.96

; MIMIC-CXR

\mathrm{cos}_{\mathbf{u}}{=}0.73

L_{1,\mathbf{u}}{=}0.82

. Bold values improve over the corresponding uniform reference.

		$\rho{=}0.05$		$\rho{=}0.1$		$\rho{=}0.2$
		$n{\approx}4{,}485$ , #pos ${\approx}57$		$n{\approx}8{,}970$ , #pos ${\approx}114$		$n{\approx}17{,}939$ , #pos ${\approx}229$
Pretrain	$\varepsilon$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
CheXpert	$0.2$	$\mathbf{0.78}{\scriptstyle\pm 0.02}$	$\mathbf{0.70}{\scriptstyle\pm 0.03}$	$\mathbf{0.94}{\scriptstyle\pm 0.01}$	$\mathbf{0.38}{\scriptstyle\pm 0.02}$	$\mathbf{0.98}{\scriptstyle\pm 0.00}$	$\mathbf{0.23}{\scriptstyle\pm 0.01}$
	$0.5$	$\mathbf{0.95}{\scriptstyle\pm 0.01}$	$\mathbf{0.33}{\scriptstyle\pm 0.04}$	$\mathbf{0.98}{\scriptstyle\pm 0.00}$	$\mathbf{0.20}{\scriptstyle\pm 0.02}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.01}$
	$1.0$	$\mathbf{0.98}{\scriptstyle\pm 0.01}$	$\mathbf{0.22}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.01}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.01}$
	$2.0$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.18}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.01}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.15}{\scriptstyle\pm 0.01}$

Appendix

Table 13: Weight reconstruction of DP-MixMin for NIH Fibrosis (1.50% prevalence in

D^{\mathrm{priv}}

, 1,686 total positives). Mean

\pm

std over 5 paired comparisons (5 seeds). We compare the private weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Uniform references: CheXpert

\mathrm{cos}_{\mathbf{u}}{=}0.73

L_{1,\mathbf{u}}{=}0.77

; MIMIC-CXR

\mathrm{cos}_{\mathbf{u}}{=}0.83

L_{1,\mathbf{u}}{=}0.60

. Bold values improve over the corresponding uniform reference.

	$\rho{=}0.05$		$\rho{=}0.1$		$\rho{=}0.15$		$\rho{=}0.2$		$\rho{=}0.3$		$\rho{=}0.5$
	(6 users)		(12 users)		(18 users)		(24 users)		(35 users)		(59 users)
$\varepsilon$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
$0.2$	—	—	$0.08{\scriptstyle\pm 0.11}$	$1.89{\scriptstyle\pm 0.13}$	$0.01{\scriptstyle\pm 0.00}$	$1.98{\scriptstyle\pm 0.01}$	$0.34{\scriptstyle\pm 0.44}$	$\mathbf{1.41}{\scriptstyle\pm 0.72}$	$0.19{\scriptstyle\pm 0.16}$	$1.65{\scriptstyle\pm 0.16}$	$\mathbf{0.44}{\scriptstyle\pm 0.24}$	$\mathbf{1.25}{\scriptstyle\pm 0.28}$
$0.5$	$0.12{\scriptstyle\pm 0.15}$	$1.85{\scriptstyle\pm 0.19}$	$0.15{\scriptstyle\pm 0.20}$	$1.75{\scriptstyle\pm 0.30}$	$0.07{\scriptstyle\pm 0.04}$	$1.75{\scriptstyle\pm 0.13}$	$\mathbf{0.70}{\scriptstyle\pm 0.30}$	$\mathbf{0.80}{\scriptstyle\pm 0.46}$	$\mathbf{0.86}{\scriptstyle\pm 0.13}$	$\mathbf{0.70}{\scriptstyle\pm 0.13}$	$\mathbf{0.95}{\scriptstyle\pm 0.01}$	$\mathbf{0.54}{\scriptstyle\pm 0.08}$
$1$	$0.12{\scriptstyle\pm 0.15}$	$1.83{\scriptstyle\pm 0.19}$	$\mathbf{0.61}{\scriptstyle\pm 0.26}$	$\mathbf{1.08}{\scriptstyle\pm 0.38}$	$\mathbf{0.61}{\scriptstyle\pm 0.26}$	$\mathbf{1.01}{\scriptstyle\pm 0.27}$	$\mathbf{0.97}{\scriptstyle\pm 0.03}$	$\mathbf{0.36}{\scriptstyle\pm 0.15}$	$\mathbf{0.97}{\scriptstyle\pm 0.03}$	$\mathbf{0.46}{\scriptstyle\pm 0.07}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.40}{\scriptstyle\pm 0.03}$
$2$	$0.16{\scriptstyle\pm 0.13}$	$1.68{\scriptstyle\pm 0.15}$	$\mathbf{0.92}{\scriptstyle\pm 0.05}$	$\mathbf{0.65}{\scriptstyle\pm 0.22}$	$\mathbf{0.93}{\scriptstyle\pm 0.05}$	$\mathbf{0.45}{\scriptstyle\pm 0.14}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.28}{\scriptstyle\pm 0.06}$	$\mathbf{0.99}{\scriptstyle\pm 0.01}$	$\mathbf{0.37}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.36}{\scriptstyle\pm 0.01}$

Appendix

Table 14: Enron weight recovery under user-level DP with

K{=}16

Common Pile proxies. Mean

\pm

std over 3 paired comparisons (3 seeds). We compare the private mixture weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Columns vary the DP-MixMin selection budget

\rho

, the fraction of the private mixture-learning split used to estimate the mixture weights; the second header row reports the corresponding number of Enron users. The uniform reference is

\mathrm{cos}_{\mathbf{u}}{=}0.40

and

L_{1,\mathbf{u}}{=}1.47

. Bold values improve over the uniform reference on that metric.

	$\rho{=}0.01$		$\rho{=}0.02$		$\rho{=}0.05$		$\rho{=}0.1$
	(275K tokens)		(552K tokens)		(1.4M tokens)		(2.8M tokens)
$\varepsilon$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
$0.2$	$0.17{\scriptstyle\pm 0.08}$	$1.66{\scriptstyle\pm 0.14}$	$0.34{\scriptstyle\pm 0.17}$	$1.38{\scriptstyle\pm 0.23}$	$\mathbf{0.78}{\scriptstyle\pm 0.10}$	$\mathbf{0.88}{\scriptstyle\pm 0.14}$	$\mathbf{0.92}{\scriptstyle\pm 0.02}$	$\mathbf{0.72}{\scriptstyle\pm 0.04}$
$0.4$	$0.32{\scriptstyle\pm 0.14}$	$1.39{\scriptstyle\pm 0.20}$	$\mathbf{0.68}{\scriptstyle\pm 0.13}$	$\mathbf{0.99}{\scriptstyle\pm 0.16}$	$\mathbf{0.92}{\scriptstyle\pm 0.02}$	$\mathbf{0.72}{\scriptstyle\pm 0.04}$	$\mathbf{0.94}{\scriptstyle\pm 0.00}$	$\mathbf{0.67}{\scriptstyle\pm 0.01}$
$0.6$	$\mathbf{0.49}{\scriptstyle\pm 0.15}$	$\mathbf{1.19}{\scriptstyle\pm 0.20}$	$\mathbf{0.82}{\scriptstyle\pm 0.07}$	$\mathbf{0.84}{\scriptstyle\pm 0.10}$	$\mathbf{0.94}{\scriptstyle\pm 0.01}$	$\mathbf{0.69}{\scriptstyle\pm 0.02}$	$\mathbf{0.95}{\scriptstyle\pm 0.00}$	$\mathbf{0.67}{\scriptstyle\pm 0.01}$
$0.8$	$\mathbf{0.64}{\scriptstyle\pm 0.12}$	$\mathbf{1.02}{\scriptstyle\pm 0.15}$	$\mathbf{0.88}{\scriptstyle\pm 0.04}$	$\mathbf{0.77}{\scriptstyle\pm 0.06}$	$\mathbf{0.94}{\scriptstyle\pm 0.01}$	$\mathbf{0.68}{\scriptstyle\pm 0.01}$	$\mathbf{0.95}{\scriptstyle\pm 0.00}$	$\mathbf{0.67}{\scriptstyle\pm 0.01}$

Appendix

Table 15: OpenWebMath weight recovery under example-level DP with

K{=}16

Common Pile proxies. Mean

\pm

std over 3 paired comparisons (3 seeds). We compare the private mixture weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Columns vary the DP-MixMin selection budget

\rho

, the fraction of the private mixture-learning split used to estimate the mixture weights; the second header row reports the corresponding number of OpenWebMath tokens. The uniform reference is

\mathrm{cos}_{\mathbf{u}}{=}0.40

and

L_{1,\mathbf{u}}{=}1.28

. Bold values improve over the uniform reference on that metric.

		Natural canary		Max-clipped-gradient canary
Class	% pos	Private ( $\varepsilon{=}1$ )	No-noise ctrl.	Private ( $\varepsilon{=}1$ )	No-noise ctrl.
Atelectasis	10.31	0.000	0.261	0.080	3.71
Consolidation	4.16	0.000	0.501	0.052	5.19
Infiltration	17.74	0.000	0.088	0.000	2.27
Pneumothorax	4.73	0.000	0.068	0.080	5.18
Edema	2.05	0.000	0.404	0.000	4.93

Appendix

Table 16: Empirical privacy audit of DP-MixMin’s mixture-learning step, CheXpert

\to

NIH ChestX-ray14, at

\bm{\varepsilon=1}

. We audit the released mixture weights

\bm{\lambda}=M_{\mathrm{mix}}(D_{\mathrm{mix}})

, reporting the Clopper–Pearson empirical lower bound

\varepsilon_{\mathrm{LB}}

\varepsilon=1

\delta=10^{-5}

\rho=0.1

, and

k=14

proxy sources, over 2000 runs on add/remove-neighboring datasets (attack fit on a SELECT half, bound computed on a disjoint EVAL half;

95\%

confidence). The natural canary is a real held-out example; the max-clip canary injects a worst-case gradient at the clipping norm. Private audits at

\varepsilon=1

; No-noise control removes the DP noise. Every private

\varepsilon_{\mathrm{LB}}\ll 1

while controls recover large leakage, confirming audit fidelity.

		Natural canary		Max-clipped-gradient canary
Class	% pos	Private ( $\varepsilon{=}1$ )	No-noise ctrl.	Private ( $\varepsilon{=}1$ )	No-noise ctrl.
Atelectasis	10.31	0.000	0.000	0.000	4.02
Consolidation	4.16	0.000	0.501	0.000	4.72
Infiltration	17.74	0.000	0.000	0.001	3.21
Pneumothorax	4.73	0.000	0.022	0.000	4.92
Edema	2.05	0.000	0.672	0.000	5.59

Appendix

Table 17: Empirical privacy audit of DP-MixMin’s mixture-learning step, MIMIC-CXR

\to

NIH ChestX-ray14, at

\bm{\varepsilon=1}

. Identical protocol to Table 16 : Clopper–Pearson

\varepsilon_{\mathrm{LB}}

\varepsilon=1

\delta=10^{-5}

\rho=0.1

, and

k=14

MIMIC-CXR proxy sources, over 2000 neighboring-dataset runs (SELECT/EVAL split,

95\%

confidence). As for CheXpert, every private

\varepsilon_{\mathrm{LB}}\ll 1

while controls recover substantial leakage.

Class	Non-private MixMin	DP-MixMin ( $\varepsilon{=}1$ )
Atelectasis	$0.745\pm 0.007$	$0.747\pm 0.006$
Consolidation	$0.753\pm 0.014$	$0.756\pm 0.012$
Infiltration	$0.669\pm 0.005$	$0.669\pm 0.006$
Pneumothorax	$0.796\pm 0.008$	$0.789\pm 0.018$
Edema	$0.819\pm 0.012$	$0.809\pm 0.019$
Emphysema	$0.715\pm 0.026$	$0.746\pm 0.025$

Appendix

Table 18: DP-MixMin vs. its non-private version, CheXpert

\to

NIH ChestX-ray14 at

\bm{\varepsilon=1}

. Per-class final AUC (mean

\pm

std over 3–5 seeds,

\delta=10^{-5}

). Classes are ordered by NIH index (Table 9 ). DP-MixMin matches the non-private upper bound to within

0.001

macro AUC.

Class	Non-private MixMin	DP-MixMin ( $\varepsilon{=}1$ )
Atelectasis	$0.758\pm 0.006$	$0.749\pm 0.010$
Consolidation	$0.764\pm 0.015$	$0.761\pm 0.013$
Infiltration	$0.678\pm 0.008$	$0.676\pm 0.003$
Pneumothorax	$0.779\pm 0.004$	$0.754\pm 0.014$
Edema	$0.838\pm 0.007$	$0.835\pm 0.010$
Emphysema	$0.745\pm 0.018$	$0.720\pm 0.028$

Appendix

Table 19: DP-MixMin vs. its non-private version, MIMIC-CXR

\to

NIH ChestX-ray14 at

\bm{\varepsilon=1}

. Per-class final AUC (mean

\pm

std over seeds,

\delta=10^{-5}

). Non-private MixMin finds the mixture with no privacy and fine-tunes on the full private set (both splits), so it is a strict upper bound; DP-MixMin reserves the mixing split (parallel composition) and privatizes both steps. DP-MixMin matches the non-private upper bound to within

0.006

macro AUC.

Explore similar work

Apr 22, 2026cs.LG

Differentially Private Model Merging

In machine learning, privacy requirements at inference or deployment time often evolve due to changing policies, regulations, or user preferences. In this work, we aim to construct a magnitude of models to satisfy any target differential privacy (DP) requirement without additional training, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. We propose two post-processing techniques, namely random selection and linear combination, to generate final private models satisfying any target privacy parameter. We provide privacy accounting of these approaches from the lens of R'enyi DP and privacy loss distributions on general problems, as well as on private mean estimation, where we precisely characterize the privacy/utility tradeoffs and compare the two mechanisms. Empirically, we demonstrate the effectiveness of our approaches and validate our analyses on several models and both synthetic and real-world datasets.

Qichuan Yin, Manzil Zaheer, Tian Li

The University of Chicago · Google DeepMind

Continual Model Merging Mixture Models

Ask PDF

Jun 25, 2026cs.LG

Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetwork

Differentially private (DP) training of neural networks is often hindered by the large amount of noise required by gradient-based methods such as DP-SGD, which repeatedly inject high-dimensional noise in parameter space throughout training. In this paper, we propose a new framework for DP learning that avoids iterative optimization in parameter space. Instead of updating the target model using privatized gradients, we employ a hypernetwork trained on public datasets to map a private dataset to the parameters of the target model. Specifically, each example is embedded into a low-dimensional representation, the embeddings are aggregated and perturbed to obtain a DP dataset embedding, and the hypernetwork generates the target model parameters from this noisy embedding. Because privacy noise is injected only once into a low-dimensional dataset representation, our approach can significantly reduce the adverse effect of noise. We theoretically show in a synthetic setting that, under a fixed privacy budget, models produced by our approach achieve higher utility than those trained with DP-SGD. Moreover, we apply our approach to LoRA fine-tuning of diffusion models and show that it achieves lower FID than LoRA models trained with DP-SGD and other public-data-guided methods.

Naoki Nishikawa, Shokichi Takakura, Satoshi Hasegawa

The University of Tokyo · LY Corporation

Neural Network Training Stochastic Gradient Descent Differentiable Optimization Hypernetwork Learn Iterative

Ask PDF

Jul 21, 2026cs.LG

End-to-End Differential Privacy in Training Deep Neural Network Classifiers

Differentially private machine learning enables model training on sensitive data while ensuring that individual data is unlikely to be recoverable from the parameters of the resulting model. However, existing work often privatizes both training inputs and their labels, and these protections may be conservative when labels are public or can be safely made public. Therefore, in this work we propose a novel private training framework that instead privatizes training inputs while keeping labels public. We consider neural networks with softmax output layers, and thus the mapping from training inputs to the output of the softmax layer is a mapping onto the unit simplex. We randomize softmax outputs during training by applying the Dirichlet mechanism to enforce differential privacy for the training inputs, hence the ``end-to-end'' label. Because training data is reused across multiple training epochs, we use the notion of \Renyi differential privacy to formulate tight bounds on the strength of privacy provided by the Dirichlet mechanism across repeated uses. We show empirically that we attain new state-of-the-art accuracy when training from scratch on CIFAR10, MNIST, MedMNIST, FashionMNIST, and SVHN across all privacy budgets evaluated. Notably, when implementing

(ε, δ)

-differential privacy with

δ=10^{-5}

, we improve the prior state-of-the-art accuracy from

78.37\%

88.17\%

ε=4

on CIFAR10, and our approach has

82.96\%

accuracy even for

ε=1

, which significantly outperforms prior work.

Huaiyuan Rao, Calvin Hawkins, Alexander Benvenuti +1

School of Electrical and Computer Engineering, Georgia Institute of Technology Atlanta, GA 30332, USA · School of Electrical and Computer Engineering, Georgia Institute of TechnologyJul Atlanta, GA 30332, USA

Δ)$-Differential Privacy Privacy Novel Modular Learning Framework

Ask PDF