Efficient Exact Unlearning for Federated Zeroth-Order Optimization

Weijun Yin; Yanqing Yao; Falko Dressler; Di Wang; Youming Tao

cs.LGOct 26, 2026

Efficient Exact Unlearning for Federated Zeroth-Order Optimization

Authors: Weijun Yin, Yanqing Yao, Falko Dressler, Di Wang, Youming Tao

Organizations: School of Cyber Science and Technology, Beihang University, Beijing 100191, China · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University, Beijing 100083, China · School of Electrical Engineering and Computer Science, TU Berlin, 10623 Berlin, Germany · Division of Computer, Electrical and Mathematical Sciences and Engineering, King Abdullah University of Science and Technology, Saudi Arabia

Abstract

Federated zeroth-order optimization (FedZOO) reduces the memory and communication costs of collaborative training for edge intelligence by avoiding backpropagation while communicating only random seeds and scalar coefficients. However, existing methods lack efficient mechanisms for exact unlearning to fully remove the influence of training data when users request deletion. A key insight is that total variation stability of the learning algorithm creates overlap between model distributions before and after data deletion, allowing exact unlearning to reuse computation through coupling. Based on this observation, we jointly design two stable zeroth-order learning algorithms and their corresponding exact-unlearning procedures: FedZOO-S obtains stability through client and mini-batch subsampling, enabling FedZOUL-S to perform efficient unlearning by preserving earlier updates and restarting from the first affected step. FedZOO-NS further combines subsampling with Gaussian noise to increase overlap between update distributions; FedZOUL-NS exploits this overlap through rejection--reflection coupling to reuse affected updates when the coupling is accepted. We prove exact unlearning for both algorithms: the model and saved state have the same joint distribution as retraining with the corresponding learner on retained data. We further establish stability and convergence guarantees for the learning algorithms and derive upper bounds on expected unlearning cost. Experiments demonstrate effective unlearning with competitive model utility and up to $56.1\%$ less cumulative unlearning time than re-optimization. Compared with first-order baselines, our methods reduce communication overhead by up to approximately four orders of magnitude.

Figures & tables

Dataset	Algorithm	$\alpha=0.1$			$\alpha=0.5$			$\alpha=1.0$
Dataset	Algorithm	Acc $\uparrow$	Time $\downarrow$	Comm $\downarrow$	Acc $\uparrow$	Time $\downarrow$	Comm $\downarrow$	Acc $\uparrow$	Time $\downarrow$	Comm $\downarrow$
	FedKSeed Pro	78.18	7.05	0.17	79.42	7.28	0.17	79.44	7.10	0.17
	DeComFL	66.59	7.93	0.02	80.03	7.91	0.02	79.63	7.98	0.02
	FedZOO-S	83.22	8.19	0.17	83.81	8.21	0.17	84.09	8.17	0.17
	FedZOO-NS ( $\sigma=0.1$ )	79.14	8.25	0.17	82.08	8.23	0.17	80.73	8.24	0.17
MNIST	FedZOO-NS ( $\sigma=0.2$ )	75.04	8.28	0.17	79.00	8.26	0.17	79.02	8.24	0.17

TABLE I: Comparison of accuracy (

\%

), runtime (s), and communication cost (MB) against zeroth-order baselines under varying levels of data heterogeneity on vision tasks. Our methods are shaded in gray, with the best accuracies in bold.

Model	Algorithm	Split99			Split95			Split90
Model	Algorithm	ROUGE-L $\uparrow$	Time $\downarrow$	Comm $\downarrow$	ROUGE-L $\uparrow$	Time $\downarrow$	Comm $\downarrow$	ROUGE-L $\uparrow$	Time $\downarrow$	Comm $\downarrow$
	FedKSeed Pro	40.66	63.41	0.17	38.87	65.77	0.17	38.42	68.38	0.17
	DeComFL	42.09	72.55	0.01	43.72	74.53	0.01	44.22	77.14	0.01
	FedZOO-S	44.07	74.07	0.17	44.47	76.47	0.17	45.01	78.61	0.17
	FedZOO-NS ( $\sigma=10$ )	42.58	74.17	0.17	43.57	76.57	0.17	42.65	79.23	0.17
Llama-3.2- 1B-Instruct	FedZOO-NS ( $\sigma=20$ )	38.61	74.08	0.17	41.03	76.59	0.17	40.68	79.47	0.17

TABLE II: Comparison of ROUGE-L scores (

\%

), runtime (s), and communication cost (MB) with zeroth-order baselines under different split settings on TOFU. Our methods are shaded in gray, with the best ROUGE-L scores in bold.

Panel of Figure 3 — Fig. 1: Accuracy comparison of FedZOUL-S/NS against FedZOO-S/NS (before unlearning) and re-optimization under stream unlearning requests on four vision datasets. Top-left:MNIST, top-right: FashionMNIST, bottom-left: SVHN, bottom-right: CIFAR-10.

Panel of Figure 4 — Fig. 2: ROUGE-L scores of FedZOUL-S and FedZOUL-NS compared with re-optimization and Oblivionis before and after stream unlearning requests on TOFU (Split95). Oblivionis-FT and Oblivionis-U denote fine-tuning and unlearning under the Oblivionis framework, respectively. Left: Llama-3.2-1B-Instruct. Right: Llama-3.2-3B-Instruct.

Fig. 3: Cumulative time cost of FedZOUL-S/NS against re-optimization under stream unlearning requests on four vision datasets. Left: CIFAR-10, right: TOFU (Llama-3.2-3B-Instruct).

Panel of Figure 6 — Fig. 4: Cumulative communication cost of FedZOUL-S/NS against re-optimization and first-order unlearning baselines under stream unlearning requests on four vision datasets. Top-left: MNIST, top-right: FashionMNIST, bottom-left: SVHN, bottom-right: CIFAR-10.

Fig. 5: Cumulative communication cost of FedZOUL-S/NS against re-optimization and Oblivionis under stream unlearning requests on TOFU (Split95). Left: Llama-3.2-1B-Instruct, right: Llama-3.2-3B-Instruct.

Panel of Figure 8 — Fig. 6: Impact of the client sampling ratio ( $m/M$ ) on accuracy, cumulative runtime, and cumulative communication cost on FashionMNIST. Top row: accuracy; bottom row: cumulative runtime and communication cost. Left column: FedZOO-S/NS; right column: FedZOUL-S/NS.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 7: Peak GPU memory of FedZOUL-S/NS against re-optimization and first-order unlearning baselines under stream unlearning requests. Left: four vision datasets. Right: TOFU (Split95).

Fig. 8: Impact of the number of per-sample perturbation directions (

q

) and the scalar gradient accumulator size (

K

) on the accuracy, per-round communication cost, and per-round time cost of FedZOO-S and FedZOO-NS on FashionMNIST. Left column: FedZOO-S, right column: FedZOO-NS. Top row: accuracy, middle row: communication cost, bottom row: time cost.

Federated Unlearning

Federated unlearning removes a client’s or record’s influence from a collaboratively trained model while preserving useful knowledge.

See all 29 papers

May 23, 2026

Rethinking Federated Unlearning via the Lens of Memorization

Federated learning (FL) increasingly needs machine unlearning to comply with privacy regulations. However, existing federated unlearning approaches may overlook the overlapping information between the unlearning and remaining data, leading to ineffective unlearning and unfairness between clients. In this work, we revisit federated unlearning through the lens of memorization. We argue that unlearning should mainly remove the unique memorized information attributable to the data to be forgotten, while preserving overlapping patterns that are also supported by the remaining data. Specifically, we propose Grouped Memorization Evaluation, an example-level metric that separates memorized knowledge from overlapping knowledge. Building on this metric, we introduce Federated Memorization Pruning (FedMemPrune), a pruning-based unlearning approach that resets redundant parameters responsible for memorization. Extensive experiments show that FedMemPrune closely matches retraining-based unlearning baselines while more effectively eliminating memorization than existing federated unlearning algorithms, yielding strong unlearning performance without sacrificing the utility of retained knowledge.

Jiaheng Wei, Yanjun Zhang, He Zhang +5

Royal Melbourne Institute of Technology · Griffith University · Swinburne University of Technology

Federated Unlearning Neural Network Memorization Machine Unlearning Federated Learning

Ask PDF

Apr 5, 2026

Jellyfish: Zero-Shot Federated Unlearning Scheme with Knowledge Disentanglement

With the increasing importance of data privacy and security, federated unlearning emerges as a new research field dedicated to ensuring that once specific data is deleted, federated learning models no longer retain or disclose related information. In this paper, we propose a zero-shot federated unlearning scheme, named Jellyfish. It distinguishes itself from conventional federated unlearning frameworks in four key aspects: synthetic data generation, knowledge disentanglement, loss function design, and model repair. To preserve the privacy of forgotten data, we design a zero-shot unlearning mechanism that generates error-minimization noise as proxy data for the data to be forgotten. To maintain model utility, we first propose a knowledge disentanglement mechanism that regularises the output of the final convolutional layer by restricting the number of activated channels for the data to be forgotten and encouraging activation sparsity. Next, we construct a comprehensive loss function that incorporates multiple components, including hard loss, confusion loss, distillation loss, model weight drift loss, gradient harmonization, and gradient masking, to effectively align the learning trajectories of the objectives of forgetting" and retaining". Finally, we propose a zero-shot repair mechanism that leverages proxy data to restore model accuracy within acceptable bounds without accessing users' local data. To evaluate the performance of the proposed zero-shot federated unlearning scheme, we conducted comprehensive experiments across diverse settings. The results validate the effectiveness and robustness of the scheme.

Houzhe Wang, Xiaojie Zhu, Chi Chen

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · King Abdullah University of Science and Technology, Thuwal, Kingdom of Saudi Arabia

Federated Unlearning

Ask PDF

Apr 14, 2026

PrivEraserVerify: Efficient, Private, and Verifiable Federated Unlearning

Federated learning (FL) enables collaborative model training without sharing raw data, offering a promising path toward privacy preserving artificial intelligence. However, FL models may still memorize sensitive information from participants, conflicting with the right to be forgotten (RTBF). To meet these requirements, federated unlearning has emerged as a mechanism to remove the contribution of departing clients. Existing solutions only partially address this challenge: FedEraser improves efficiency but lacks privacy protection, FedRecovery ensures differential privacy (DP) but degrades accuracy, and VeriFi enables verifiability but introduces overhead without efficiency or privacy guarantees. We present PrivEraserVerify (PEV), a unified framework that integrates efficiency, privacy, and verifiability into federated unlearning. PEV employs (i) adaptive checkpointing to retain critical historical updates for fast reconstruction, (ii) layer adaptive differentially private calibration to selectively remove client influence while minimizing accuracy loss, and (iii) fingerprint based verification, enabling participants to confirm unlearning in a decentralized and noninvasive manner. Experiments on image, handwritten character, and medical datasets show that PEV achieves up to 2 to 3 times faster unlearning than retraining, provides formal indistinguishability guarantees with reduced performance degradation, and supports scalable verification. To the best of our knowledge, PEV is the first framework to simultaneously deliver efficiency, privacy, and verifiability for federated unlearning, moving FL closer to practical and regulation compliant deployment.

Parthaw Goswami, Md Khairul Islam, Ashfak Yeafi

Department of Electronics and Communication Engineering, Khulna University of Engineering & Technology, Khulna-9203, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA · Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna-9203, Bangladesh

Federated Unlearning Privacy-Preserving Federated Learning

Ask PDF

Abstract

Federated Unlearning

Explore similar work

Rethinking Federated Unlearning via the Lens of Memorization

Jellyfish: Zero-Shot Federated Unlearning Scheme with Knowledge Disentanglement

PrivEraserVerify: Efficient, Private, and Verifiable Federated Unlearning