Differentially Private Mixing of Public Datasets Improves Private Learning

Yufei Chen; Tejumade Afonja; Anvith Thudi; Nicolas Papernot

cs.LGOct 5, 2026

Differentially Private Mixing of Public Datasets Improves Private Learning

Authors: Yufei Chen, Tejumade Afonja, Anvith Thudi, Nicolas Papernot

Organizations: University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany

Abstract

Many machine learning applications involve sensitive data and therefore require training under differential privacy (DP). However, DP training often degrades model utility. In some cases, first pre-training the model on "public" data before finetuning with DP on the sensitive data can reduce the drop in utility. However, the success of this depends on how relevant the selected public dataset is to the sensitive data. We introduce the first pipeline that privately learns the mixture of several public datasets to pretrain on for a given sensitive downstream task. Our key insight is that we can privately find the best mixture of multiple public datasets by privately learning a low-dimensional linear model. We tested our method on the NIH dataset for X-ray classification and the ENRON email dataset for language modeling. Applying our method to find tailored mixtures of X-ray datasets to pretrain on for diseases in the NIH ChestX-ray14 dataset, we improved macro AUC by up to 0.037 across privacy budgets compared to the baselines, with gains as large as +22.8% relative AUC on Cardiomegaly at $ε=1$ . For DP training on the ENRON dataset, pre-training on our mixture of The Common Pile (a collection of public-domain text datasets) decreased test perplexity by 16% relative to the baseline mixtures.

Figures & tables

Figure 1: Overview of DP-MixMin. Proxy models trained on public data are combined with mixture weights

\lambda

learned with DP-SGD on a fraction

\rho

of the sensitive data (split A). We then pretrain on public data according to

\lambda

and DP fine-tune on the rest (split B).

Figure 2: DP-MixMin beats Uniform and Random Search at every $\varepsilon$ . Macro AUC on CheXpert/MIMIC-CXR $\rightarrow$ NIH ( $\uparrow$ ) and Enron test perplexity at $\rho=0.3$ ( $\downarrow$ ).

Figure 3: Per-task relative AUC gain over Random Init on NIH Chest X-ray at

\epsilon=1

. Bars show

\big(\mathrm{AUC(method)}-\mathrm{AUC(Random\ Init)}\big)/\mathrm{AUC(Random\ Init)}

as a percentage, with propagated standard deviation over 3–5 seeds. The macro AUC gain (unweighted mean across 14 tasks) is reported next to each subplot title; DP-MixMin roughly doubles, and in most cases more than doubles, the gain of both Uniform and Random Search on each public pretraining source. Full raw AUCs are reported in Tables 7 and 8 .

Method	$\rho$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$
DP-MixMin	$0.3$	$\mathbf{16.78\pm 0.55}$	$\mathbf{16.44\pm 0.95}$	$\mathbf{15.71\pm 0.77}$
DP-MixMin	$0.2$	$17.52\pm 1.55$	—	—
DP-MixMin	$0.1$	$17.97\pm 2.72$	—	—
Uniform	—	$19.88\pm 1.10$	$19.18\pm 1.01$	$18.24\pm 0.89$
Random Search	$0.3$	$20.46\pm 0.70$	$19.72\pm 0.66$	$18.72\pm 0.62$

Table 1: Enron test perplexity (mean

\pm

std) after DP finetuning at

\delta{=}10^{-5}

. DP-MixMin beats both Uniform and Random Search at every privacy budget. At

\epsilon{=}1

, the mixture quality (and stability) improves monotonically with the mixture weight finding fraction

\rho

Method	$\rho$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$
DP-MixMin	$0.1$	$\mathbf{12.22\pm 0.17}$	—	—
Uniform	—	$14.05\pm 0.17$	$14.04\pm 0.17$	$14.02\pm 0.17$

Table 2: OpenWebMath test perplexity (mean

\pm

std) after DP finetuning at

\delta{=}10^{-5}

. DP-MixMin lowers test perplexity by roughly

13\%

over uniform Common Pile pretraining at

\epsilon{=}1

. The uniform baseline is essentially flat in

\epsilon

on this domain, showing DP-MixMin at

\epsilon=1

is still better.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Proxy epochs	Atelectasis AUC
1	$0.744\pm 0.005$
2	$0.745\pm 0.005$
4 (main experiments)	$0.742\pm 0.006$
6	$0.749\pm 0.003$

Appendix

Table 3: DP-MixMin is insensitive to proxy training budget. NIH Atelectasis AUC (mean

\pm

std over 3 seeds) with CheXpert pretraining at

\epsilon=1

Figure 4: Excess error scales roughly as

\sqrt{k}

for large

k

. Average excess error of DP-MixMin over the 10 CIFAR-100 tasks as the number of public sources

k

increases (both axes log scale). Error bars and the shaded band show

\pm 1

standard deviation across tasks. The dashed red line is a

\sqrt{k}

reference; the fitted slope for

k\geq 32

0.49

, consistent with the predicted

\sqrt{k}

rate.

$k$	class 0	class 1	class 2	class 3	class 4	class 5	class 6	class 7	class 8	class 9	Avg
2	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.2}$	$0.1{\scriptstyle\pm 0.2}$	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.2}$	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.2}$	$0.1{\scriptstyle\pm 0.1}$	$0.1{\scriptstyle\pm 0.2}$	$\mathbf{0.1{\scriptstyle\pm 0.1}}$
4	$0.1{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.1{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.0{\scriptstyle\pm 0.0}$	$0.1{\scriptstyle\pm 0.0}$	$\mathbf{0.0{\scriptstyle\pm 0.0}}$
8	$1.1{\scriptstyle\pm 0.1}$	$1.4{\scriptstyle\pm 0.2}$	$1.3{\scriptstyle\pm 0.2}$	$3.7{\scriptstyle\pm 0.2}$	$3.3{\scriptstyle\pm 0.2}$	$2.1{\scriptstyle\pm 0.1}$	$2.3{\scriptstyle\pm 0.2}$	$2.8{\scriptstyle\pm 0.1}$	$4.1{\scriptstyle\pm 0.2}$	$2.1{\scriptstyle\pm 0.1}$	$\mathbf{2.4{\scriptstyle\pm 1.0}}$
16	$5.7{\scriptstyle\pm 0.4}$	$7.0{\scriptstyle\pm 0.2}$	$3.3{\scriptstyle\pm 0.4}$	$2.9{\scriptstyle\pm 0.3}$	$2.4{\scriptstyle\pm 0.4}$	$7.0{\scriptstyle\pm 0.6}$	$4.2{\scriptstyle\pm 0.3}$	$2.8{\scriptstyle\pm 0.4}$	$5.8{\scriptstyle\pm 0.5}$	$7.8{\scriptstyle\pm 0.5}$	$\mathbf{4.9{\scriptstyle\pm 2.0}}$
32	$2.8{\scriptstyle\pm 0.2}$	$6.7{\scriptstyle\pm 0.5}$	$3.4{\scriptstyle\pm 0.3}$	$2.0{\scriptstyle\pm 0.1}$	$1.9{\scriptstyle\pm 0.1}$	$3.4{\scriptstyle\pm 0.2}$	$3.7{\scriptstyle\pm 0.2}$	$2.7{\scriptstyle\pm 0.3}$	$5.8{\scriptstyle\pm 0.3}$	$5.3{\scriptstyle\pm 0.3}$	$\mathbf{3.8{\scriptstyle\pm 1.6}}$
64	$3.8{\scriptstyle\pm 0.2}$	$2.1{\scriptstyle\pm 0.3}$	$3.7{\scriptstyle\pm 0.2}$	$3.4{\scriptstyle\pm 0.3}$	$2.6{\scriptstyle\pm 0.3}$	$4.4{\scriptstyle\pm 0.3}$	$4.4{\scriptstyle\pm 0.2}$	$5.0{\scriptstyle\pm 0.5}$	$1.8{\scriptstyle\pm 0.1}$	$2.3{\scriptstyle\pm 0.3}$	$\mathbf{3.4{\scriptstyle\pm 1.1}}$

Appendix

Table 4: DP-MixMin excess error

\alpha=L(\bm{\lambda}_{\mathrm{DP}})-L(\bm{\lambda}^{\star})

across the number of public sources

k

(ImageNet-1k

\to

CIFAR-100,

\varepsilon{=}1

\delta{=}10^{-5}

), for all 10 CIFAR-100 one-vs-rest tasks. Mean

\pm

std over 5 seeds, in units of

10^{-3}

. The last column ( Avg ) averages over all 5 seeds and 10 tasks.

Class	E1	E2	E6
Atelectasis	$0.868\pm 0.015$	$0.986\pm 0.003$	$0.972\pm 0.014$
Consolidation	$0.886\pm 0.004$	$0.941\pm 0.035$	$0.944\pm 0.012$
Infiltration	$0.895\pm 0.018$	$0.873\pm 0.046$	$0.848\pm 0.023$
Pneumothorax	$0.952\pm 0.019$	$0.976\pm 0.003$	$0.975\pm 0.003$
Edema	$0.978\pm 0.003$	$0.965\pm 0.003$	$0.887\pm 0.029$
Emphysema	$0.986\pm 0.006$	$0.988\pm 0.002$	$0.970\pm 0.009$

Appendix

Table 5: Learned mixture is stable across proxy training budget. Cosine similarity of the DP-MixMin mixture (early-stop,

\varepsilon{=}1

) at proxy epochs

\{1,2,6\}

to the main

E{=}4

mixture, per NIH class (mean

\pm

std, 3 seeds).

E{=}4

is the self-reference (

1.000

Class	Sequential	Parallel	Non-private
Atelectasis	$0.740\pm 0.010$	$0.747\pm 0.005$	$0.746\pm 0.006$

Appendix

Table 6: Sequential and parallel composition perform similarly. NIH AUC (mean

\pm

std over 3 seeds), CheXpert pretraining, total

\epsilon=1

	DP-MixMin			Uniform			Random Search			No Pretraining
Class	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$
Atelectasis	0.740 $\pm$ 0.012	0.747 $\pm$ 0.005	0.747 $\pm$ 0.007	0.707 $\pm$ 0.013	0.724 $\pm$ 0.009	0.736 $\pm$ 0.008	0.724 $\pm$ 0.009	0.733 $\pm$ 0.011	0.739 $\pm$ 0.010	0.679 $\pm$ 0.006	0.696 $\pm$ 0.008	0.715 $\pm$ 0.006
Consolidation	0.742 $\pm$ 0.014	0.745 $\pm$ 0.017	0.748 $\pm$ 0.014	0.745 $\pm$ 0.015	0.749 $\pm$ 0.014	0.752 $\pm$ 0.013	0.734 $\pm$ 0.008	0.740 $\pm$ 0.011	0.742 $\pm$ 0.010	0.710 $\pm$ 0.006	0.716 $\pm$ 0.010	0.725 $\pm$ 0.010
Infiltration	0.671 $\pm$ 0.004	0.674 $\pm$ 0.005	0.678 $\pm$ 0.002	0.668 $\pm$ 0.004	0.670 $\pm$ 0.003	0.672 $\pm$ 0.003	0.664 $\pm$ 0.004	0.670 $\pm$ 0.003	0.675 $\pm$ 0.003	0.663 $\pm$ 0.004	0.672 $\pm$ 0.003	0.680 $\pm$ 0.002
Pneumothorax	0.780 $\pm$ 0.007	0.788 $\pm$ 0.002	0.798 $\pm$ 0.004	0.697 $\pm$ 0.008	0.728 $\pm$ 0.008	0.753 $\pm$ 0.009	0.674 $\pm$ 0.005	0.701 $\pm$ 0.009	0.726 $\pm$ 0.007	0.650 $\pm$ 0.009	0.684 $\pm$ 0.005	0.723 $\pm$ 0.005
Edema	0.829 $\pm$ 0.017	0.836 $\pm$ 0.014	0.845 $\pm$ 0.013	0.829 $\pm$ 0.006	0.835 $\pm$ 0.007	0.838 $\pm$ 0.008	0.844 $\pm$ 0.008	0.850 $\pm$ 0.008	0.857 $\pm$ 0.009	0.804 $\pm$ 0.012	0.816 $\pm$ 0.010	0.829 $\pm$ 0.010

Appendix

Table 7: Per-class final AUC on NIH ChestX-ray14 after transfer from CheXpert. Each entry is the mean

\pm

standard deviation of AUC over 3–5 seeds at

\delta=10^{-5}

. Bold indicates the best method per (class,

\epsilon

) cell. DP-MixMin achieves the best macro AUC across all three privacy budgets, with absolute gains of

+0.029

+0.037

over uniform pretraining.

	DP-MixMin			Uniform			Random Search			No Pretraining
Class	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$	$\epsilon{=}1$	$\epsilon{=}2$	$\epsilon{=}5$
Atelectasis	0.751 $\pm$ 0.010	0.749 $\pm$ 0.010	0.737 $\pm$ 0.000	0.728 $\pm$ 0.009	0.734 $\pm$ 0.008	0.740 $\pm$ 0.008	0.742 $\pm$ 0.006	0.749 $\pm$ 0.006	0.755 $\pm$ 0.005	0.679 $\pm$ 0.006	0.696 $\pm$ 0.008	0.715 $\pm$ 0.006
Consolidation	0.756 $\pm$ 0.012	0.758 $\pm$ 0.011	0.759 $\pm$ 0.015	0.748 $\pm$ 0.010	0.751 $\pm$ 0.011	0.753 $\pm$ 0.011	0.743 $\pm$ 0.012	0.746 $\pm$ 0.012	0.749 $\pm$ 0.012	0.710 $\pm$ 0.006	0.716 $\pm$ 0.010	0.725 $\pm$ 0.010
Infiltration	0.674 $\pm$ 0.008	0.679 $\pm$ 0.004	0.682 $\pm$ 0.005	0.667 $\pm$ 0.004	0.673 $\pm$ 0.005	0.677 $\pm$ 0.006	0.673 $\pm$ 0.006	0.676 $\pm$ 0.006	0.680 $\pm$ 0.003	0.663 $\pm$ 0.004	0.672 $\pm$ 0.003	0.680 $\pm$ 0.002
Pneumothorax	0.754 $\pm$ 0.013	0.770 $\pm$ 0.004	0.782 $\pm$ 0.005	0.719 $\pm$ 0.012	0.741 $\pm$ 0.015	0.754 $\pm$ 0.015	0.679 $\pm$ 0.015	0.707 $\pm$ 0.016	0.732 $\pm$ 0.016	0.650 $\pm$ 0.009	0.684 $\pm$ 0.005	0.723 $\pm$ 0.005
Edema	0.845 $\pm$ 0.003	0.849 $\pm$ 0.004	0.853 $\pm$ 0.005	0.839 $\pm$ 0.007	0.841 $\pm$ 0.008	0.845 $\pm$ 0.007	0.826 $\pm$ 0.011	0.834 $\pm$ 0.008	0.841 $\pm$ 0.011	0.804 $\pm$ 0.012	0.816 $\pm$ 0.010	0.829 $\pm$ 0.010

Appendix

Table 8: Per-class final AUC on NIH ChestX-ray14 after transfer from MIMIC-CXR. Each entry is the mean

\pm

standard deviation of AUC over 3–5 seeds at

\delta=10^{-5}

. Bold indicates the best method per (class,

\epsilon

) cell. DP-MixMin achieves the best macro AUC across all three privacy budgets, with absolute gains of

+0.024

+0.033

over uniform pretraining.

Figure 5: DP-MixMin beats both uniform-mixture and random-search Common Pile pretraining from scratch on Enron email completion across all three privacy budgets, with the largest gain at the tightest budget. Test-set perplexity after DP fine-tuning on Enron (

\delta{=}10^{-5}

) at

\epsilon\in\{1,2,5\}

. All three methods pretrain GPT-2 medium from random initialization on Common Pile and are then DP-finetuned on Enron under identical settings. DP-MixMin reduces test perplexity by roughly

16\%/14\%/14\%

over uniform and

18\%/17\%/16\%

over random-mixture pretraining at

\epsilon\in\{1,2,5\}

Figure 6: DP-MixMin consistently outperforms uniform Common Pile pretraining on Enron at

\epsilon{=}1

across all proxy-split fractions, with larger fractions yielding both lower perplexity and tighter cross-seed variance. Test-set perplexity after DP fine-tuning on Enron (

\delta{=}10^{-5}

) at

\epsilon{=}1

, varying the proxy fraction

\rho\in\{0.1,0.2,0.3\}

used by MixMin to estimate domain weights. The uniform baseline is independent of

\rho

and is shown for reference at all three values. DP-MixMin reduces test perplexity by

10\%/12\%/16\%

over uniform at

\rho\in\{0.1,0.2,0.3\}

, with

\rho{=}0.3

giving the most stable mixture (cross-seed

\sigma

drops from

2.72\to 0.45

NIH Idx	Class Name	# Pos.	% Pos.	Match	CheXpert/MIMIC-CXR Label
0	Atelectasis	11,559	10.31%	Yes	Atelectasis
1	Consolidation	4,667	4.16%	Yes	Consolidation
2	Infiltration	19,894	17.75%	Partial	Lung Opacity
3	Pneumothorax	5,302	4.73%	Yes	Pneumothorax
4	Edema	2,303	2.05%	Yes	Edema
5	Emphysema	2,516	2.24%	None	—

Appendix

Table 9: NIH ChestX-ray14 class distribution and correspondence to CheXpert/MIMIC-CXR labels. The Match column indicates whether each NIH class has a direct ( Yes ), Partial , or no ( None ) equivalent in CheXpert/MIMIC-CXR.

CheXpert $\to$ NIH ChestX-ray14
	$\epsilon=1$		$\epsilon=2$		$\epsilon=5$
	abs	rel	abs	rel	abs	rel
Group A ( $n=7$ )	$+0.040$	$+5.6\%$	$+0.030$	$+4.0\%$	$+0.029$	$+3.8\%$
Group B ( $n=7$ )	$+0.034$	$+6.0\%$	$+0.034$	$+5.7\%$	$+0.028$	$+4.7\%$
MIMIC-CXR $\to$ NIH ChestX-ray14
Group A ( $n=7$ )	$+0.029$	$+4.0\%$	$+0.020$	$+2.7\%$	$+0.015$	$+1.8\%$

Appendix

Table 10: Average AUC gains of DP-MixMin over Uniform pretraining, grouped by class type. Group A: well-represented classes with a direct CheXpert match (Atelectasis, Cardiomegaly, Effusion, Pneumonia, Pneumothorax, Consolidation, Edema). Group B: rare or unmatched classes (Infiltration, Mass, Nodule, Emphysema, Fibrosis, Pleural Thickening, Hernia); see Table 9 . Each cell is the within-group mean of per-class absolute (abs) and relative (rel) AUC gain of DP-MixMin over Uniform pretraining at

\delta=10^{-5}

CheXpert $\to$ NIH ChestX-ray14
		$\epsilon=0.2$		$\epsilon=0.5$		$\epsilon=1$		$\epsilon=2$		$\epsilon=5$
src	$n$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
CheXpert	$70$	$0.70{\scriptstyle\pm.08}$	$0.81{\scriptstyle\pm.14}$	$\mathbf{0.93}{\scriptstyle\pm.03}$	$\mathbf{0.37}{\scriptstyle\pm.09}$	$\mathbf{0.97}{\scriptstyle\pm.01}$	$\mathbf{0.24}{\scriptstyle\pm.06}$	$\mathbf{0.98}{\scriptstyle\pm.01}$	$\mathbf{0.19}{\scriptstyle\pm.05}$	$\mathbf{0.99}{\scriptstyle\pm.01}$	$\mathbf{0.16}{\scriptstyle\pm.04}$
MIMIC-CXR $\to$ NIH ChestX-ray14
MIMIC-CXR	$70$	$0.58{\scriptstyle\pm.17}$	$1.04{\scriptstyle\pm.30}$	$\mathbf{0.82}{\scriptstyle\pm.09}$	$\mathbf{0.57}{\scriptstyle\pm.20}$	$\mathbf{0.93}{\scriptstyle\pm.04}$	$\mathbf{0.33}{\scriptstyle\pm.10}$	$\mathbf{0.97}{\scriptstyle\pm.01}$	$\mathbf{0.22}{\scriptstyle\pm.05}$	$\mathbf{0.98}{\scriptstyle\pm.01}$	$\mathbf{0.16}{\scriptstyle\pm.03}$

Appendix

Table 11: Weight reconstruction of DP-MixMin across 14 NIH ChestX-ray14 tasks. Mean

\pm

std over

n{=}70

paired comparisons (14 tasks

\times

5 seeds). We compare the private weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Uniform references are CheXpert:

\mathrm{cos}_{\mathbf{u}}{=}0.75

L_{1,\mathbf{u}}{=}0.68

; MIMIC-CXR:

\mathrm{cos}_{\mathbf{u}}{=}0.80

L_{1,\mathbf{u}}{=}0.60

. The DP-MixMin selection budget

\rho=0.1

. Bold values improve over the corresponding uniform reference.

		$\rho{=}0.05$		$\rho{=}0.1$		$\rho{=}0.2$
		$n{\approx}4{,}485$ , #pos ${\approx}9$		$n{\approx}8{,}970$ , #pos ${\approx}18$		$n{\approx}17{,}939$ , #pos ${\approx}36$
Pretrain	$\varepsilon$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
CheXpert	$0.2$	$\mathbf{0.79}{\scriptstyle\pm 0.01}$	$\mathbf{0.64}{\scriptstyle\pm 0.02}$	$\mathbf{0.94}{\scriptstyle\pm 0.01}$	$\mathbf{0.32}{\scriptstyle\pm 0.01}$	$\mathbf{0.98}{\scriptstyle\pm 0.00}$	$\mathbf{0.17}{\scriptstyle\pm 0.01}$
	$0.5$	$\mathbf{0.96}{\scriptstyle\pm 0.01}$	$\mathbf{0.27}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.15}{\scriptstyle\pm 0.01}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.10}{\scriptstyle\pm 0.01}$
	$1.0$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.10}{\scriptstyle\pm 0.01}$	$\mathbf{1.00}{\scriptstyle\pm 0.00}$	$\mathbf{0.08}{\scriptstyle\pm 0.01}$
	$2.0$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.11}{\scriptstyle\pm 0.02}$	$\mathbf{1.00}{\scriptstyle\pm 0.00}$	$\mathbf{0.08}{\scriptstyle\pm 0.01}$	$\mathbf{1.00}{\scriptstyle\pm 0.00}$	$\mathbf{0.08}{\scriptstyle\pm 0.01}$

Appendix

Table 12: Weight reconstruction of DP-MixMin for NIH Hernia (0.20% prevalence in

D^{\mathrm{priv}}

, 227 total positives). Mean

\pm

std over 5 paired comparisons (5 seeds). We compare the private weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Uniform references: CheXpert

\mathrm{cos}_{\mathbf{u}}{=}0.64

L_{1,\mathbf{u}}{=}0.96

; MIMIC-CXR

\mathrm{cos}_{\mathbf{u}}{=}0.73

L_{1,\mathbf{u}}{=}0.82

. Bold values improve over the corresponding uniform reference.

		$\rho{=}0.05$		$\rho{=}0.1$		$\rho{=}0.2$
		$n{\approx}4{,}485$ , #pos ${\approx}57$		$n{\approx}8{,}970$ , #pos ${\approx}114$		$n{\approx}17{,}939$ , #pos ${\approx}229$
Pretrain	$\varepsilon$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
CheXpert	$0.2$	$\mathbf{0.78}{\scriptstyle\pm 0.02}$	$\mathbf{0.70}{\scriptstyle\pm 0.03}$	$\mathbf{0.94}{\scriptstyle\pm 0.01}$	$\mathbf{0.38}{\scriptstyle\pm 0.02}$	$\mathbf{0.98}{\scriptstyle\pm 0.00}$	$\mathbf{0.23}{\scriptstyle\pm 0.01}$
	$0.5$	$\mathbf{0.95}{\scriptstyle\pm 0.01}$	$\mathbf{0.33}{\scriptstyle\pm 0.04}$	$\mathbf{0.98}{\scriptstyle\pm 0.00}$	$\mathbf{0.20}{\scriptstyle\pm 0.02}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.01}$
	$1.0$	$\mathbf{0.98}{\scriptstyle\pm 0.01}$	$\mathbf{0.22}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.01}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.01}$
	$2.0$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.18}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.16}{\scriptstyle\pm 0.01}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.15}{\scriptstyle\pm 0.01}$

Appendix

Table 13: Weight reconstruction of DP-MixMin for NIH Fibrosis (1.50% prevalence in

D^{\mathrm{priv}}

, 1,686 total positives). Mean

\pm

std over 5 paired comparisons (5 seeds). We compare the private weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Uniform references: CheXpert

\mathrm{cos}_{\mathbf{u}}{=}0.73

L_{1,\mathbf{u}}{=}0.77

; MIMIC-CXR

\mathrm{cos}_{\mathbf{u}}{=}0.83

L_{1,\mathbf{u}}{=}0.60

. Bold values improve over the corresponding uniform reference.

	$\rho{=}0.05$		$\rho{=}0.1$		$\rho{=}0.15$		$\rho{=}0.2$		$\rho{=}0.3$		$\rho{=}0.5$
	(6 users)		(12 users)		(18 users)		(24 users)		(35 users)		(59 users)
$\varepsilon$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
$0.2$	—	—	$0.08{\scriptstyle\pm 0.11}$	$1.89{\scriptstyle\pm 0.13}$	$0.01{\scriptstyle\pm 0.00}$	$1.98{\scriptstyle\pm 0.01}$	$0.34{\scriptstyle\pm 0.44}$	$\mathbf{1.41}{\scriptstyle\pm 0.72}$	$0.19{\scriptstyle\pm 0.16}$	$1.65{\scriptstyle\pm 0.16}$	$\mathbf{0.44}{\scriptstyle\pm 0.24}$	$\mathbf{1.25}{\scriptstyle\pm 0.28}$
$0.5$	$0.12{\scriptstyle\pm 0.15}$	$1.85{\scriptstyle\pm 0.19}$	$0.15{\scriptstyle\pm 0.20}$	$1.75{\scriptstyle\pm 0.30}$	$0.07{\scriptstyle\pm 0.04}$	$1.75{\scriptstyle\pm 0.13}$	$\mathbf{0.70}{\scriptstyle\pm 0.30}$	$\mathbf{0.80}{\scriptstyle\pm 0.46}$	$\mathbf{0.86}{\scriptstyle\pm 0.13}$	$\mathbf{0.70}{\scriptstyle\pm 0.13}$	$\mathbf{0.95}{\scriptstyle\pm 0.01}$	$\mathbf{0.54}{\scriptstyle\pm 0.08}$
$1$	$0.12{\scriptstyle\pm 0.15}$	$1.83{\scriptstyle\pm 0.19}$	$\mathbf{0.61}{\scriptstyle\pm 0.26}$	$\mathbf{1.08}{\scriptstyle\pm 0.38}$	$\mathbf{0.61}{\scriptstyle\pm 0.26}$	$\mathbf{1.01}{\scriptstyle\pm 0.27}$	$\mathbf{0.97}{\scriptstyle\pm 0.03}$	$\mathbf{0.36}{\scriptstyle\pm 0.15}$	$\mathbf{0.97}{\scriptstyle\pm 0.03}$	$\mathbf{0.46}{\scriptstyle\pm 0.07}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.40}{\scriptstyle\pm 0.03}$
$2$	$0.16{\scriptstyle\pm 0.13}$	$1.68{\scriptstyle\pm 0.15}$	$\mathbf{0.92}{\scriptstyle\pm 0.05}$	$\mathbf{0.65}{\scriptstyle\pm 0.22}$	$\mathbf{0.93}{\scriptstyle\pm 0.05}$	$\mathbf{0.45}{\scriptstyle\pm 0.14}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.28}{\scriptstyle\pm 0.06}$	$\mathbf{0.99}{\scriptstyle\pm 0.01}$	$\mathbf{0.37}{\scriptstyle\pm 0.03}$	$\mathbf{0.99}{\scriptstyle\pm 0.00}$	$\mathbf{0.36}{\scriptstyle\pm 0.01}$

Appendix

Table 14: Enron weight recovery under user-level DP with

K{=}16

Common Pile proxies. Mean

\pm

std over 3 paired comparisons (3 seeds). We compare the private mixture weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Columns vary the DP-MixMin selection budget

\rho

, the fraction of the private mixture-learning split used to estimate the mixture weights; the second header row reports the corresponding number of Enron users. The uniform reference is

\mathrm{cos}_{\mathbf{u}}{=}0.40

and

L_{1,\mathbf{u}}{=}1.47

. Bold values improve over the uniform reference on that metric.

	$\rho{=}0.01$		$\rho{=}0.02$		$\rho{=}0.05$		$\rho{=}0.1$
	(275K tokens)		(552K tokens)		(1.4M tokens)		(2.8M tokens)
$\varepsilon$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$	$\mathrm{cos}_{\star}$	$L_{1}$
$0.2$	$0.17{\scriptstyle\pm 0.08}$	$1.66{\scriptstyle\pm 0.14}$	$0.34{\scriptstyle\pm 0.17}$	$1.38{\scriptstyle\pm 0.23}$	$\mathbf{0.78}{\scriptstyle\pm 0.10}$	$\mathbf{0.88}{\scriptstyle\pm 0.14}$	$\mathbf{0.92}{\scriptstyle\pm 0.02}$	$\mathbf{0.72}{\scriptstyle\pm 0.04}$
$0.4$	$0.32{\scriptstyle\pm 0.14}$	$1.39{\scriptstyle\pm 0.20}$	$\mathbf{0.68}{\scriptstyle\pm 0.13}$	$\mathbf{0.99}{\scriptstyle\pm 0.16}$	$\mathbf{0.92}{\scriptstyle\pm 0.02}$	$\mathbf{0.72}{\scriptstyle\pm 0.04}$	$\mathbf{0.94}{\scriptstyle\pm 0.00}$	$\mathbf{0.67}{\scriptstyle\pm 0.01}$
$0.6$	$\mathbf{0.49}{\scriptstyle\pm 0.15}$	$\mathbf{1.19}{\scriptstyle\pm 0.20}$	$\mathbf{0.82}{\scriptstyle\pm 0.07}$	$\mathbf{0.84}{\scriptstyle\pm 0.10}$	$\mathbf{0.94}{\scriptstyle\pm 0.01}$	$\mathbf{0.69}{\scriptstyle\pm 0.02}$	$\mathbf{0.95}{\scriptstyle\pm 0.00}$	$\mathbf{0.67}{\scriptstyle\pm 0.01}$
$0.8$	$\mathbf{0.64}{\scriptstyle\pm 0.12}$	$\mathbf{1.02}{\scriptstyle\pm 0.15}$	$\mathbf{0.88}{\scriptstyle\pm 0.04}$	$\mathbf{0.77}{\scriptstyle\pm 0.06}$	$\mathbf{0.94}{\scriptstyle\pm 0.01}$	$\mathbf{0.68}{\scriptstyle\pm 0.01}$	$\mathbf{0.95}{\scriptstyle\pm 0.00}$	$\mathbf{0.67}{\scriptstyle\pm 0.01}$

Appendix

Table 15: OpenWebMath weight recovery under example-level DP with

K{=}16

Common Pile proxies. Mean

\pm

std over 3 paired comparisons (3 seeds). We compare the private mixture weights

\bm{\lambda}_{\mathrm{DP}}

to the non-private MixMin solution

\bm{\lambda}^{\star}

using cosine similarity

\mathrm{cos}_{\star}

and

L_{1}

distance. Columns vary the DP-MixMin selection budget

\rho

, the fraction of the private mixture-learning split used to estimate the mixture weights; the second header row reports the corresponding number of OpenWebMath tokens. The uniform reference is

\mathrm{cos}_{\mathbf{u}}{=}0.40

and

L_{1,\mathbf{u}}{=}1.28

. Bold values improve over the uniform reference on that metric.

		Natural canary		Max-clipped-gradient canary
Class	% pos	Private ( $\varepsilon{=}1$ )	No-noise ctrl.	Private ( $\varepsilon{=}1$ )	No-noise ctrl.
Atelectasis	10.31	0.000	0.261	0.080	3.71
Consolidation	4.16	0.000	0.501	0.052	5.19
Infiltration	17.74	0.000	0.088	0.000	2.27
Pneumothorax	4.73	0.000	0.068	0.080	5.18
Edema	2.05	0.000	0.404	0.000	4.93

Appendix

Table 16: Empirical privacy audit of DP-MixMin’s mixture-learning step, CheXpert

\to

NIH ChestX-ray14, at

\bm{\varepsilon=1}

. We audit the released mixture weights

\bm{\lambda}=M_{\mathrm{mix}}(D_{\mathrm{mix}})

, reporting the Clopper–Pearson empirical lower bound

\varepsilon_{\mathrm{LB}}

\varepsilon=1

\delta=10^{-5}

\rho=0.1

, and

k=14

proxy sources, over 2000 runs on add/remove-neighboring datasets (attack fit on a SELECT half, bound computed on a disjoint EVAL half;

95\%

confidence). The natural canary is a real held-out example; the max-clip canary injects a worst-case gradient at the clipping norm. Private audits at

\varepsilon=1

; No-noise control removes the DP noise. Every private

\varepsilon_{\mathrm{LB}}\ll 1

while controls recover large leakage, confirming audit fidelity.

		Natural canary		Max-clipped-gradient canary
Class	% pos	Private ( $\varepsilon{=}1$ )	No-noise ctrl.	Private ( $\varepsilon{=}1$ )	No-noise ctrl.
Atelectasis	10.31	0.000	0.000	0.000	4.02
Consolidation	4.16	0.000	0.501	0.000	4.72
Infiltration	17.74	0.000	0.000	0.001	3.21
Pneumothorax	4.73	0.000	0.022	0.000	4.92
Edema	2.05	0.000	0.672	0.000	5.59

Appendix

Table 17: Empirical privacy audit of DP-MixMin’s mixture-learning step, MIMIC-CXR

\to

NIH ChestX-ray14, at

\bm{\varepsilon=1}

. Identical protocol to Table 16 : Clopper–Pearson

\varepsilon_{\mathrm{LB}}

\varepsilon=1

\delta=10^{-5}

\rho=0.1

, and

k=14

MIMIC-CXR proxy sources, over 2000 neighboring-dataset runs (SELECT/EVAL split,

95\%

confidence). As for CheXpert, every private

\varepsilon_{\mathrm{LB}}\ll 1

while controls recover substantial leakage.

Class	Non-private MixMin	DP-MixMin ( $\varepsilon{=}1$ )
Atelectasis	$0.745\pm 0.007$	$0.747\pm 0.006$
Consolidation	$0.753\pm 0.014$	$0.756\pm 0.012$
Infiltration	$0.669\pm 0.005$	$0.669\pm 0.006$
Pneumothorax	$0.796\pm 0.008$	$0.789\pm 0.018$
Edema	$0.819\pm 0.012$	$0.809\pm 0.019$
Emphysema	$0.715\pm 0.026$	$0.746\pm 0.025$

Appendix

Table 18: DP-MixMin vs. its non-private version, CheXpert

\to

NIH ChestX-ray14 at

\bm{\varepsilon=1}

. Per-class final AUC (mean

\pm

std over 3–5 seeds,

\delta=10^{-5}

). Classes are ordered by NIH index (Table 9 ). DP-MixMin matches the non-private upper bound to within

0.001

macro AUC.

Class	Non-private MixMin	DP-MixMin ( $\varepsilon{=}1$ )
Atelectasis	$0.758\pm 0.006$	$0.749\pm 0.010$
Consolidation	$0.764\pm 0.015$	$0.761\pm 0.013$
Infiltration	$0.678\pm 0.008$	$0.676\pm 0.003$
Pneumothorax	$0.779\pm 0.004$	$0.754\pm 0.014$
Edema	$0.838\pm 0.007$	$0.835\pm 0.010$
Emphysema	$0.745\pm 0.018$	$0.720\pm 0.028$

Appendix

Table 19: DP-MixMin vs. its non-private version, MIMIC-CXR

\to

NIH ChestX-ray14 at

\bm{\varepsilon=1}

. Per-class final AUC (mean

\pm

std over seeds,

\delta=10^{-5}

). Non-private MixMin finds the mixture with no privacy and fine-tunes on the full private set (both splits), so it is a strict upper bound; DP-MixMin reserves the mixing split (parallel composition) and privatizes both steps. DP-MixMin matches the non-private upper bound to within

0.006

macro AUC.

Explore similar work

Differentially Private Model Merging
Apr 22, 2026Qichuan Yin, Manzil Zaheer, Tian LiContinual Model Merging Mixture Models
The University of Chicago · Google DeepMind
Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetwork
Jun 25, 2026Naoki Nishikawa, Shokichi Takakura, Satoshi HasegawaNeural Network Training Stochastic Gradient Descent
The University of Tokyo · LY Corporation
End-to-End Differential Privacy in Training Deep Neural Network Classifiers
Jul 21, 2026Huaiyuan Rao, Calvin Hawkins, Alexander Benvenuti +1Δ)$-Differential Privacy Privacy
School of Electrical and Computer Engineering, Georgia Institute of Technology Atlanta, GA 30332, USA · School of Electrical and Computer Engineering, Georgia Institute of TechnologyJul Atlanta, GA 30332, USA

Apr 22, 2026, cs.LGJ/K move · Enter open · S save

Differentially Private Model Merging

Qichuan Yin, Manzil Zaheer, Tian Li

The University of Chicago · Google DeepMind

Continual Model Merging Mixture Models

Open paper AskPDF arXiv

In machine learning, privacy requirements at inference or deployment time often evolve due to changing policies, regulations, or user preferences. In this work, we aim to construct a magnitude of models to satisfy any target differential privacy (DP) requirement without additional training, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. We propose two post-processing techniques, namely random selection and linear combination, to generate final private models satisfying any target privacy parameter. We provide privacy accounting of these approaches from the lens of R'enyi DP and privacy loss distributions on general problems, as well as on private mean estimation, where we precisely characterize the privacy/utility tradeoffs and compare the two mechanisms. Empirically, we demonstrate the effectiveness of our approaches and validate our analyses on several models and both synthetic and real-world datasets.

Abstract

Explore similar work

Differentially Private Model Merging

Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetwork

End-to-End Differential Privacy in Training Deep Neural Network Classifiers