Turning Black Box into White Box: Dataset Distillation Leaks

Huajie Chen; Tianqing Zhu; Yuchen Zhong; Yang Zhang; Shang Wang; Feng He; Lefeng Zhang; Jialiang Shen

cs.CRMar 1, 2026

Turning Black Box into White Box: Dataset Distillation Leaks

Authors: Huajie Chen, Tianqing Zhu, Yuchen Zhong, Yang Zhang, Shang Wang, Feng He, Lefeng Zhang, Jialiang Shen,

Organizations: City University of Macau · CISPA Helmholtz Center for Information Security · University of Technology Sydney · University of Sydney

Abstract

Dataset distillation compresses a large real dataset into a small synthetic one, enabling models trained on the synthetic data to achieve performance comparable to those trained on the real data. Although synthetic datasets are assumed to be privacy-preserving, we show that existing distillation methods can cause severe privacy leakage because synthetic datasets implicitly encode the weight trajectories of the distilled model, they become over-informative and exploitable by adversaries. To expose this risk, we introduce the Information Revelation Attack (IRA) against state-of-the-art distillation techniques. Experiments show that IRA accurately predicts both the distillation algorithm and model architecture, and can successfully infer membership and recover sensitive samples from the real dataset.

Figures & tables

Figure 1. The overview of IRA. The victim generates a publicly available synthetic dataset using the distillation algorithm with the victim model and the real dataset. However, with the synthetic dataset, the adversary can reveal the sensitive information by launching IRA.

Symbols	Definitions
$V,A$	The victim/adversary
$f,h$	The victim/local model
$\theta,\tau$	The weights of the victim/local model
$D_{\text{real}}$ , $D_{\text{syn}}$	The real/synthetic dataset
$D_{\text{aux}},D_{\text{inv}}$	The auxiliary/inversed dataset
$b,c$	The membership/target class label

Table 1. Primary Notations.

Figure 2. Architecture Inference Stage. For the AIA, $A$ combines different distillation algorithms $\gamma_{i}$ with model architectures $f_{j}$ to synthesize $l$ synthetic datasets. Then, these synthetic datasets are respectively used to train $u\times v\times l$ local models to create the set of loss trajectories $\mathcal{T}$ . $\mathcal{T}$ is then used to train the architecture attack model $A_{A}$ to predict which algorithm and architecture are used for the dataset distillation.

Figure 3. Membership Inference Stage. In the MIA, $A$ trains the membership attack model $A_{M}$ with the outputs from each layer of the local model $h$ to determine the membership of an arbitrary given sample $x$ .

Figure 4. Model Inversion Stage. The framework employs two models $\phi$ and $\psi$ . $\phi$ is trained to predict the noise $\epsilon$ in the DDPM way, whereas $\psi$ is trained to output the clean image $x_{0}$ and the coefficient $r_{t}$ with the denoised image $x^{*}_{0}$ as the input. The final output of this dual network framework is the combination of their outputs weighted using $r_{t}$ . Notably, except for the first term, the rest of the terms in the loss funcion are used to regularize $\psi$ .

Method	Model Arch.	Test Top-1 Acc. %				MIA BA $\uparrow$ /AUC $\uparrow$ /T@LF % $\uparrow$				AIA Top-1 Acc. % $\uparrow$				MIV Atk. Acc. $\uparrow$ /KNN Dist $\downarrow$
		IPC				IPC				IPC				IPC
		100	250	500	1000	100	250	500	1000	100	250	500	1000	100	250	500	1000
MTT	ConvNet	58.4	60.4	62.8	63.9	0.77/0.85/12.5	0.81/0.89/21.5	0.81/0.89/20.1	0.82/0.88/19.5	79.1	80.8	80.4	82.1	0.81/682.4	0.83/634.2	0.87/591.2	0.89/574.7
	AlexNet	56.7	58.2	60.5	61.3	0.71/0.78/4.6	0.72/0.80/5.9	0.80/0.88/20.2	0.80/0.88/20.2					0.79/653.7	0.82/613.2	0.86/588.2	0.86/572.8
	ResNet18	54.6	59.3	63.1	64.3	0.70/0.77/3.2	0.80/0.88/17.2	0.80/0.87/18.2	0.82/0.89/19.9					0.77/758.7	0.81/603.6	0.87/582.3	0.89/552.6
	VGG11-BN	57.2	60.7	63.5	65.9	0.77/0.85/11.2	0.79/0.87/14.3	0.81/0.88/16.7	0.89/0.95/53.6					0.81/633.7	0.84/617.8	0.86/599.2	0.89/542.8

Table 2. Evaluation of the IRA on the SOTA Dataset Distillation Algorithms on the CIFAR10 Dataset.

Figure 5. MIV Qualitative Results on CIFAR-10. The upper row are the samples in each class from the real dataset. The bottom row are the synthetic samples generated by the diffusion model.

Figure 6. Log-scale ROC curves for the MIAs on the synthetic CIFAR-10 dataset.

Datasets	W/O Features as Inputs
Datasets	BA	AUC	T@LF %
CIFAR-10	0.88/0.67	0.94/0.71	53.1/3.3
CIFAR-100	0.74/0.61	0.82/0.63	9.7/0.4
TinyImagenet-200	0.75/0.59	0.83/0.61	9.1/0.1

Table 3. The impact of using the features as the input in the membership inference attack.

Loss Functions		Atk. Acc.	KNN Dist
$L_{\text{cls}}$	$L_{\text{traj}}$	Atk. Acc.	KNN Dist
$\times$	$\times$	0.47	1048.3
$\checkmark$	$\times$	0.91	593.2
$\times$	$\checkmark$	0.68	832.4
$\checkmark$	$\checkmark$	0.94	489.1

Table 4. The impact of the loss functions used in the model inversion attack on the CIFAR-10 synthetic dataset.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7. MIV Qualitative Results on CIFAR-100. Real samples are in the upper part, and their corresponding recovered samples are in the lower part.

Figure 8. The qualitative results of the MIV attack on TinyImagenet-200.

Method	Model Arch.	Test Top-1 Acc %				MIA BA $\uparrow$ /AUC $\uparrow$ /T@LF % $\uparrow$				AIA Top-1 Acc. % $\uparrow$				MIV Atk. Acc. $\uparrow$ /KNN Dist $\downarrow$
		IPC				IPC				IPC				IPC
		10	25	50	100	10	25	50	100	10	25	50	100	10	25	50	100
MTT	ConvNet	21.5	28.2	34.8	40.5	0.60/0.64/0.6	0.65/0.71/1.8	0.73/0.81/9.5	0.79/0.86/18.9	81.5	83.2	82.7	84.1	0.54/1048.2	0.56/1032.7	0.55/1082.5	0.59/1023.4
	AlexNet	20.7	26.3	35.6	38.7	0.63/0.68/0.9	0.68/0.74/1.9	0.77/0.85/10.4	0.77/0.85/16.0					0.55/1103.7	0.55/1079.2	0.56/1043.9	0.58/1037.5
	ResNet18	22.4	30.1	33.5	41.9	0.62/0.67/1.1	0.65/0.71/1.3	0.73/0.80/7.4	0.81/0.88/24.8					0.53/1072.4	0.54/1034.0	0.56/1052.1	0.56/1089.2
	VGG11-BN	22.5	29.2	36.2	40.7	0.63/0.69/1.4	0.66/0.72/1.9	0.76/0.80/14.1	0.79/0.87/20.9					0.52/1132.1	0.54/1058.6	0.55/1075.2	0.58/1035.2

Appendix

Table 5. Evaluation of the IRA on the SOTA Dataset Distillation Algorithms on the CIFAR-100 Dataset.

Method	Model Arch.	Test Top-1 Acc %			MIA BA $\uparrow$ /AUC $\uparrow$ /T@LF % $\uparrow$			AIA Top-1 Acc % $\uparrow$			MIV Atk. Acc. $\uparrow$ /KNN Dist. $\downarrow$
		IPC			IPC			IPC			IPC
		10	25	50	10	25	50	10	25	50	10	25	50
MTT	ConvNet	15.3	19.7	21.5	0.65/0.70/0.7	0.68/0.75/1.8	0.71/0.78/3.3	75.2	73.6	77.0	0.32/1359.2	0.34/1344.2	0.36/1306.8
	AlexNet	15.1	17.8	19.9	0.63/0.68/1.3	0.66/0.71/1.6	0.70/0.77/2.8				0.33/1382.1	0.35/1363.7	0.35/1327.8
	ResNet18	14.7	19.4	22.1	0.62/0.69/0.9	0.68/0.74/2.2	0.71/0.78/3.5				0.32/1402.8	0.34/1352.8	0.35/1323.5
	VGG11-BN	15.4	18.6	21.7	0.61/0.66/1.0	0.67/0.74/2.8	0.69/0.75/3.7				0.32/1376.9	0.34/1368.9	0.36/1324.9

Appendix

Table 6. Evaluation of the IRA on the SOTA Dataset Distillation Algorithms on the TinyImageNet-200 Dataset.

Figure 9. Log-scale ROC curves for the MIAs on the synthetic CIFAR-100 dataset. Please zoom in to see the details due to the page limit.

Figure 10. Log-scale ROC curves for the MIAs on the synthetic TinyImagenet-200 dataset. Please zoom in due to the page limit.

Training Data Reconstruction Attacks

See all 17 papers

Osmosis Distillation: Model Hijacking with the Fewest Samples
Mar 5, 2026Yuchen Shi, Huajie Chen, Heng Xu +6Data Poisoning Attacks Transfer Learning
Faculty of Data Science, City University of Macau. · College of Cyber Security, Jinan University. · The University of Sydney.
On Privacy Leakage in Tabular Diffusion Models: Influential Factors, Attacker Knowledge, and Metrics
May 7, 2026Masoumeh Shafieinejad, D. B. Emerson, Behnoosh Zamanlooy +5Data Leakage Membership Inference Attacks
Vector Institute · McMaster University · University of Toronto +1
Leaky Students: Membership Inference against On-Policy Distillation
Sep 27, 2026Zhexi Lu, Mingzhi Zhu, Stacy Patterson +1Membership Inference Attacks Privacy Leakage in Language Models
Rensselaer Polytechnic Institute

Mar 5, 2026J/K move · Enter open · S save

Osmosis Distillation: Model Hijacking with the Fewest Samples

Yuchen Shi, Huajie Chen, Heng Xu, Zhiquan Liu+5

Faculty of Data Science, City University of Macau. · College of Cyber Security, Jinan University. · The University of Sydney.

Data Poisoning Attacks Transfer Learning

Open paper AskPDF arXiv

Transfer learning is devised to leverage knowledge from pre-trained models to solve new tasks with limited data and computational resources. Meanwhile, dataset distillation has emerged to synthesize a compact dataset that preserves critical information from the original large dataset. Therefore, a combination of transfer learning and dataset distillation offers promising performance in evaluations. However, a non-negligible security threat remains undiscovered in transfer learning using synthetic datasets generated by dataset distillation methods, where an adversary can perform a model hijacking attack with only a few poisoned samples in the synthetic dataset. To reveal this threat, we propose Osmosis Distillation (OD) attack, a novel model hijacking strategy that targets deep learning models using the fewest samples. Comprehensive evaluations on various datasets demonstrate that the OD attack attains high attack success rates in hidden tasks while preserving high model utility in original tasks. Furthermore, the distilled osmosis set enables model hijacking across diverse model architectures, allowing model hijacking in transfer learning with considerable attack performance and model utility. We argue that awareness of using third-party synthetic datasets in transfer learning must be raised.

Abstract

Training Data Reconstruction Attacks

Explore similar work

Osmosis Distillation: Model Hijacking with the Fewest Samples

On Privacy Leakage in Tabular Diffusion Models: Influential Factors, Attacker Knowledge, and Metrics

Leaky Students: Membership Inference against On-Policy Distillation