Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller

Sagi Habani; Yehonathan Refael; Ofir Lindenbaum

cs.LGOct 13, 2026

Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller

Authors: Sagi Habani, Yehonathan Refael, Ofir Lindenbaum

Organizations: Ben-Gurion University · Tel Aviv University · Bar Ilan University

Abstract

Releasing a neural network checkpoint can leak information about its training data, and existing reconstruction results rely on gradient-descent training. We ask whether the choice of optimizer affects the ability to recover the training data. Extending work connecting Karush--Kuhn--Tucker conditions to training-data reconstruction, we show that the implicit biases of popular optimizers (SGD, Adam, Muon) leave very different fingerprints of the training data. We argue that the key distinction is the weight-specificity of these implicit biases, namely how much continuous information about the final weights the optimizer-dependent target of the stationarity conditions encodes. The limiting stationarity conditions for models trained with SGD and Muon retain enough weight-specific information to reconstruct training examples in practice. For Adam in the $\ell_\infty$ implicit-bias regime, the optimizer-dependent part of these conditions encodes only a discrete pattern of maximal coordinates and their signs, and at nondegenerate KKT points the conditions are too few to identify the training set to first order. Experiments on MNIST and CIFAR-10 support these claims: models trained with Muon can yield stronger reconstructions at much lower training cost than models trained with SGD, although this advantage depends on the training regime and architecture, and models trained with Adam appear consistently more resistant to the evaluated attacks.

Figures & tables

Panel of Figure 1 — Figure 1: Training-data reconstructions from networks trained with Muon, SGD and Adam, without an image prior ( $n=500$ ). The top row is binary MNIST and the bottom row binary CIFAR-10. In each panel, odd rows show reconstructions and the row directly below shows their nearest training images under the one-to-one matching of Section 5 . Muon reconstructions are visually closest to their matches, SGD recovers coarser structure, and Adam reconstructions are largely unrecognizable.

Panel of Figure 2 — Figure 2: Per-sample SSIM versus training margin $\gamma_{i}$ on CIFAR-10 ( $n=500$ , range prior), for binary models. Each point is a training sample, colored by class; the dashed line marks $\mathrm{SSIM}=0.4$ . Recognizable reconstructions occur near the minimal margin of their class. Muon produces most of them, and Adam essentially none.

	MNIST (binary)			CIFAR-10 (binary)			CIFAR-10 (10-class)
Prior	SGD	Adam	Muon	SGD	Adam	Muon	SGD	Adam	Muon
None	0.3463	0.2487	0.4310	0.2287	0.1379	0.5508	0.3827	0.2066	0.5339
Range	0.4057	0.2994	0.6371	0.2336	0.1367	0.6130	0.4515	0.1771	0.5315
TV	0.3768	0.2882	0.5808	0.2574	0.2115	0.6132	0.4332	0.3442	0.5481
$\ell_{1}$	0.3614	0.1752	0.5302	0.2375	0.2112	0.5598	0.4141	0.2739	0.5371
TV $+\ell_{1}$	0.3977	0.2311	0.6211	0.2440	0.2369	0.6286	0.4343	0.3050	0.5890

Table 1: SSIM@30 (

\uparrow

) under different image priors (

n=500

). Best per task and prior in bold , second best underlined . Muon is best in all

18

settings, and in all

144

entries of the full tables (Appendix E ).

Configuration	Optimizer	SSIM $\uparrow$	SSIM@30 $\uparrow$	PSNR $\uparrow$	PSNR@30 $\uparrow$	$\ell_{1}\downarrow$	$\ell_{1}$ @30 $\downarrow$	$\ell_{2}\downarrow$	$\ell_{2}$ @30 $\downarrow$
$n=50$	SGD	0.3024	0.4486	13.188	15.066	0.1586	0.1250	0.2191	0.1765
	Adam	0.1452	0.1809	11.733	12.663	0.1989	0.1808	0.2590	0.2327
	Muon	0.4070	0.5848	14.409	17.390	0.1353	0.0954	0.1904	0.1351
$n=100$	SGD	0.2949	0.6232	13.499	18.765	0.1554	0.0828	0.2114	0.1153
	Adam	0.0976	0.1488	11.214	12.570	0.2087	0.1775	0.2750	0.2352
	Muon	0.3501	0.6984	14.257	21.465	0.1404	0.0639	0.1937	0.0845

Table 2: Effect of sample size (top,

1000\!-\!1000

hidden units) and architecture (bottom,

n=500

) on binary CIFAR-10. All models are trained to a training loss below

10^{-8}

; the attack uses the range prior. Best per configuration in bold .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

optimizer	cone $C(\bm{\theta})$	codim at KKT stratum	specificity $S$	reachability
SGD	ray through $\bm{\theta}$	$p-1$	$p-1$	slow
Muon	$\prod_{k}\{U_{1,k}MV_{1,k}^{\top}:M\succeq 0\}$	$\geq\tfrac{1}{2}(p-\sum_{k}r_{k})$	$\sum_{k}S_{k}(\rho_{k})$	fast
Adam	simplicial cone on $A$ , signs of $\bm{\theta}\|_{A}$	$p-s\leq n_{I}$ (Thm. A.10 )	$0$ (combinatorial only)	fast

Appendix

Table 3: Summary. The cone

C(\bm{\theta})

is the optimizer-dependent part of the constraint and, for linear-in-parameter models, is the part through which the checkpoint affects stationarity feasibility (Lemma A.5 ). Codimensions are at a KKT direction on the stratum of

\bm{\theta}

; at a generic

\bm{\theta}

all three are

\Theta(p)

in the usual nontrivial matrix-block regime and separate nothing.

S_{k}(\rho)=\rho(m_{k}+n_{k}-\rho)-\rho(\rho+1)/2

r_{k}=\min(m_{k},n_{k})

Panel of Figure 3 — Figure 3: Reconstructions on binary MNIST ( $n=500$ ) without a prior (top) and with the range prior (bottom). In each panel, odd rows show reconstructions and the row below shows the matched training images. Muon recovers sharp digit structure even without a prior.

Task	Optimizer	Test acc. (%)	Test loss	Min. margin
MNIST (binary)	SGD	$90.0$	$0.790$	$9.21$
	Adam	$91.0$	$0.753$	$16.64$
	Muon	$92.0$	$0.853$	$16.65$
CIFAR-10 (binary)	SGD	$77.0$	$2.25$	$9.56$
	Adam	$79.0$	$3.41$	$16.64$
	Muon	$79.0$	$2.57$	$16.63$

Appendix

Table 4: Trained models (

n=500

). All models interpolate the training set. Because the network is homogeneous, an unnormalized margin depends on the parameter scale and is not comparable across optimizers.

Panel of Figure 4 — Figure 4: Per-sample SSIM versus training margin $\gamma_{i}$ on CIFAR-10 ( $n=500$ , range prior), for multiclass models. Each point is a training sample, colored by class; the dashed line marks $\mathrm{SSIM}=0.4$ . Recognizable reconstructions occur only near the minimal margin. Muon produces the most of them and Adam essentially none.

Panel of Figure 5 — Figure 5: Reconstructions on binary CIFAR-10 ( $n=500$ ), in the layout of Figure 3 . Muon reconstructions best preserve object shape and color.

Prior	Optimizer	SSIM $\uparrow$	SSIM@30 $\uparrow$	PSNR $\uparrow$	PSNR@30 $\uparrow$	$\ell_{1}\downarrow$	$\ell_{1}$ @30 $\downarrow$	$\ell_{2}\downarrow$	$\ell_{2}$ @30 $\downarrow$
None	SGD	0.1661	0.3463	10.857	14.867	0.2059	0.1260	0.2865	0.1806
	Adam	0.1527	0.2487	10.345	10.903	0.2077	0.1926	0.3039	0.2850
	Muon	0.2350	0.4310	11.495	16.997	0.1718	0.0935	0.2662	0.1413
Range	SGD	0.2007	0.4057	11.186	15.464	0.1700	0.1008	0.2759	0.1686
	Adam	0.1898	0.2994	10.640	11.933	0.1958	0.1415	0.2938	0.2531
	Muon	0.3242	0.6371	12.115	20.154	0.1339	0.0517	0.2479	0.0982

Appendix

Table 5: Reconstruction on binary MNIST (

n=500

). Best per prior in bold .

Prior	Optimizer	SSIM $\uparrow$	SSIM@30 $\uparrow$	PSNR $\uparrow$	PSNR@30 $\uparrow$	$\ell_{1}\downarrow$	$\ell_{1}$ @30 $\downarrow$	$\ell_{2}\downarrow$	$\ell_{2}$ @30 $\downarrow$
None	SGD	0.0986	0.2287	11.671	13.523	0.2027	0.1623	0.2609	0.2108
	Adam	0.1365	0.1379	12.422	12.902	0.1849	0.1743	0.2393	0.2264
	Muon	0.1771	0.5508	12.567	17.802	0.1807	0.0974	0.2353	0.1288
Range	SGD	0.0994	0.2336	11.665	13.435	0.2027	0.1634	0.2611	0.2129
	Adam	0.1299	0.1367	12.334	13.441	0.1863	0.1624	0.2417	0.2128
	Muon	0.1921	0.6130	12.808	19.190	0.1750	0.0825	0.2289	0.1098

Appendix

Table 6: Reconstruction on binary CIFAR-10 (

n=500

). Best per prior in bold .

Prior	Optimizer	SSIM $\uparrow$	SSIM@30 $\uparrow$	PSNR $\uparrow$	PSNR@30 $\uparrow$	$\ell_{1}\downarrow$	$\ell_{1}$ @30 $\downarrow$	$\ell_{2}\downarrow$	$\ell_{2}$ @30 $\downarrow$
None	SGD	0.2031	0.3827	11.264	13.435	0.1912	0.1657	0.2535	0.1906
	Adam	0.1530	0.2066	12.254	12.740	0.1856	0.1758	0.2440	0.2307
	Muon	0.2446	0.5339	13.284	18.826	0.1651	0.0884	0.2167	0.1145
Range	SGD	0.2157	0.4515	12.999	16.445	0.1722	0.1158	0.2239	0.1506
	Adam	0.1423	0.1771	12.250	13.238	0.1874	0.1653	0.2441	0.2178
	Muon	0.2445	0.5315	13.278	18.987	0.1651	0.0863	0.2168	0.1124

Appendix

Table 7: Reconstruction on 10-class CIFAR-10 (

n=500

). Best per prior in bold .

Panel of Figure 6 — Figure 6: Sorted image-level SSIM values (from highest to lowest) for reconstructed samples on CIFAR-10 and MNIST without prior constraints ( $n=500$ ). The dashed horizontal reference line at $\text{SSIM}=0.4$ indicates a threshold for acceptable visual reconstruction quality. Reconstructions under Muon consistently achieve higher SSIM scores, resulting in a substantially larger fraction of images reconstructed above $0.4$ compared to SGD and Adam ( $\text{Muon}>\text{SGD}>\text{Adam}$ ).

Training Data Reconstruction Attacks

See all 17 papers

Efficient Techniques for Data Reconstruction, with Finite-Width Recovery Guarantees
May 7, 2026Edward Tansley, Roy Makhlouf, Estelle Massart +1Training Data Reconstruction Attacks Random Feature Methods
Mathematical Institute, Woodstock Road, University of Oxford, Oxford, UK, OX2 6GG. · ICTEAM Institute, UCLouvain, Euler Building, Avenue Georges Lemaˆıtre, 4 - bte L4.05.01, Louvain-la-Neuve, B - 1348, Belgium.
Reducing information dependency does not cause training data privacy. Adversarially non-robust features do
Jul 14, 2026Rasmus Torp, Shailen K. Smith, Adam BreuerModel Inversion Attacks Neural Network Robustness
Department of Computer Science Dartmouth College · Department of Computer Science & Department of Government Dartmouth College
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Adversarial Training Orthogonal Optimization
IT College, Shanghai Ocean University, Shanghai, 201306 · School of Computer Science and Technology, Tongji University, Shanghai, 201804 · SEIE, Suzhou University of Science and Technology, Suzhou, 215009 +2

May 7, 2026J/K move · Enter open · S save

Efficient Techniques for Data Reconstruction, with Finite-Width Recovery Guarantees

Edward Tansley, Roy Makhlouf, Estelle Massart, Coralia Cartis

Mathematical Institute, Woodstock Road, University of Oxford, Oxford, UK, OX2 6GG. · ICTEAM Institute, UCLouvain, Euler Building, Avenue Georges Lemaˆıtre, 4 - bte L4.05.01, Louvain-la-Neuve, B - 1348, Belgium.

Training Data Reconstruction Attacks Random Feature Methods

Open paper AskPDF arXiv

Data reconstruction attacks on trained neural networks aim to recover the data on which the network has been trained and pose a significant threat to privacy, especially if the training dataset contains sensitive information. Here, we propose a unified optimization formulation of the data reconstruction problem based on initial and trained parameter values, incorporating state-of-the-art proposals. We show that in the random feature model, this formulation provably leads to training data reconstruction with high probability, provided the network width is sufficiently large; this unprecedented finite-width result uses PAC-style bounds. Furthermore, when the data lies in a low-dimensional subspace, we show that the network width requirement for successful reconstruction can be relaxed, with bounds depending on the subspace dimension rather than the ambient dimension. For general neural network models and unknown data orientations, we propose an efficient reconstruction algorithm that approximates the low-dimensional data subspace through the change in the first-layer weights during training and uses only the last-layer weights for reconstruction, thus reducing the search space dimension and the required network width for high-quality reconstructions. Our numerical experiments on synthetic datasets and CIFAR-10 confirm that our subspace-aware reconstruction approach outperforms standard full-space techniques.

Abstract

Training Data Reconstruction Attacks

Explore similar work

Efficient Techniques for Data Reconstruction, with Finite-Width Recovery Guarantees

Reducing information dependency does not cause training data privacy. Adversarially non-robust features do

When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study