Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models

Yijun Quan; Wentai Wu; Giovanni Montana

cs.LGMar 13, 2026

Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models

Authors: Yijun Quan, Wentai Wu, Giovanni Montana

Organizations: WMG, University of Warwick, Coventry CV4 7AL, UK · Department of Computer Science, Jinan University, Guangzhou 510632, China

Abstract

Foundation models are commonly deployed as frozen feature extractors with a small trainable head to adapt to private, user-generated data in federated settings. The ``right to be forgotten'' requires removing the influence of specific samples or users from the trained model on demand. Existing federated unlearning methods target general deep models and rely on approximate reconstruction or selective retraining, making exactness costly or elusive. We study this problem in a practically relevant but under-explored regime: a frozen foundation model with a ridge-regression head. The exact optimum depends on the data only through two additive sufficient statistics, which we turn into a communication protocol supporting an arbitrary stream of add and delete requests via fixed-size messages. The server maintains a head that is, in exact arithmetic, pointwise identical to centralized retraining after every request. We provide deterministic retrain-equivalence guarantees, order and partition invariance, two server-side variants, and a Bayesian certificate of zero KL divergence. Experiments on four benchmarks confirm the guarantees: both variants match centralized ridge retraining to within $10^{-9}$ relative Frobenius error and complete each request at orders-of-magnitude lower cost than federated retraining baselines.

Figures & tables

	$\displaystyle T_{t}^{+}$	$\displaystyle=T_{t-1}-T_{t-1}{U_{t}^{+}}^{\top}(I_{r}+U_{t}^{+}T_{t-1}{U_{t}^{+}}^{\top})^{-1}U_{t}^{+}T_{t-1},$
	$\displaystyle W_{t}^{+}$	$\displaystyle=(I-T_{t}^{+}{U_{t}^{+}}^{\top}U_{t}^{+})W_{t-1}+T_{t}^{+}G_{t}^{+},$

Algorithm 2 Variant B at round

t

(SMW updates using client

R

factors)

Panel of Figure 1 — Figure 1: Protocol overview (Variant A). Clients compute frozen features locally and transmit only fixed-size sufficient-statistic messages $(S^{\pm}_{k,t},G^{\pm}_{k,t})$ . The server maintains a retained-statistics ledger ( $S_{t}\leftarrow S_{t-1}{+}S_{t}^{+}{-}S_{t}^{-}$ , $G_{t}\leftarrow G_{t-1}{+}G_{t}^{+}{-}G_{t}^{-}$ ) and recovers the exact ridge head by solving $(S_{t}{+}\gamma I)W_{t}=G_{t}$ .

	Communication per round	Server computation
Variant A	$\Theta(d^{2}+dc)$	$\mathcal{O}(d^{3}+d^{2}c)$
Variant B	$\Theta(dr+dc)$	$\mathcal{O}(rd^{2}+r^{3})$

Table 1: Per-round cost summary in the typical regime

d\gg r

Dataset	Task	Frozen backbone $\phi$	Feature dim. $d$
CIFAR-10	Image Classification	DINOv2	768
CIFAR-100	Image Classification	DINOv2	768
FeMNIST	Character Classification	DINOv2	768
Sentiment140	Sentiment Analysis	RoBERTa	768

Table 2: Experimental setup: datasets, tasks, frozen backbones, and feature dimensions used in all evaluations.

Dataset	Central ridge retrain	FedAvg retrain	FATS	Exact-Fun	Variant A	Variant B
CIFAR-10	0.9802	0.9740	0.9634	0.9664	0.9801	0.9802
CIFAR-100	0.8640	0.8570	0.8623	0.8471	0.8640	0.8640
FeMNIST	0.8084	0.7624	0.7217	0.7562	0.8080	0.8080
Sentiment 140	0.8291	0.8261	0.8264	0.8190	0.8291	0.8291

Table 3: Server model test accuracy on the evaluated datasets and tasks

	Variant A			Variant B
	$K{=}10$	$K{=}50$	$K{=}100$	$K{=}10$	$K{=}50$	$K{=}100$
fp32	5.55e-3	5.52e-3	5.05e-3	1.27e-4	1.26e-4	1.21e-4
fp64	1.47e-9	1.42e-9	1.42e-9	6.18e-10	1.37e-9	1.42e-9

Table 4: Relative Frobenius deviation

\|W_{\text{fed}}-W_{c}\|_{F}/\|W_{c}\|_{F}

on FeMNIST for different numbers of clients

K

and numerical precision used for server-side computation and aggregation of second-order statistics.

Panel of Figure 2 — Figure 2: Test Accuracy and Cumulative Time for CIFAR-10 Chunked Deletions. Unlearning performance on CIFAR-10 with repeated deletions of $20\%$ of the total data per step. Deleted samples are randomly selected from all clients. Left: test accuracy tracking utility across deletion rounds. Right: cumulative unlearning time demonstrating system efficiency.

Figure 3: Wall time measurement of 200 unlearning requests (single data points) performed by different methods

	Deletions only		Delete + add-back
Method	100 steps	200 steps	100 steps	200 steps
Variant A	2.72e-11	3.18e-11	3.14e-11	3.81e-11
Variant B	3.00e-11	3.66e-11	2.82e-11	5.10e-12

Table 5: Relative Frobenius deviation

\|W_{\text{fed}}-W_{c}\|_{F}/\|W_{c}\|_{F}

from the central ridge solution after 100 and 200 single-point deletions and the corresponding add-back experiment on CIFAR-10.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	Dataset	#Rounds	Clients per round	Local data per round
FedAvg retrain	CIFAR-10/100	120	20	full local data
Exact-Fun	CIFAR-10/100	120	20	full local data
FATS	CIFAR-10/100	60	5	64 samples
FedAvg retrain	FeMNIST	120	20	full local data
Exact-Fun	FeMNIST	120	20	full local data
FATS	FeMNIST	120	5	50 samples

Appendix

Table 6: The hyperparameter setting for baselines.

Federated Unlearning

Federated unlearning removes a client’s or record’s influence from a collaboratively trained model while preserving useful knowledge.

See all 27 papers

Rethinking Federated Unlearning via the Lens of Memorization
May 23, 2026Jiaheng Wei, Yanjun Zhang, He Zhang +5Federated Unlearning Neural Network Memorization
Royal Melbourne Institute of Technology · Griffith University · Swinburne University of Technology
FedUP: One-Shot Federated Unlearning via Centroid-Guided Plug-in Filters
Jun 23, 2026Feihong Nan, Zhengyi Zhong, Pan Wang +4Federated Unlearning Machine Unlearning
1National Key Laboratory of Big Data and Decision, National University of Defense Technology, China.
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
Jul 31, 2026Shengkun Zhu, Jinshan Zeng, Zhihua Allen-Zhao +5Federated Learning Aggregation Federated Fine-Tuning
The Hong Kong Polytechnic University · OceanBase, Ant Group · School of Management, Xi’an Jiaotong University +3

May 23, 2026J/K move · Enter open · S save

Rethinking Federated Unlearning via the Lens of Memorization

Jiaheng Wei, Yanjun Zhang, He Zhang, Leo Yu Zhang+4

Royal Melbourne Institute of Technology · Griffith University · Swinburne University of Technology

Federated Unlearning Neural Network Memorization Machine Unlearning Federated Learning

Open paper AskPDF arXiv

Federated learning (FL) increasingly needs machine unlearning to comply with privacy regulations. However, existing federated unlearning approaches may overlook the overlapping information between the unlearning and remaining data, leading to ineffective unlearning and unfairness between clients. In this work, we revisit federated unlearning through the lens of memorization. We argue that unlearning should mainly remove the unique memorized information attributable to the data to be forgotten, while preserving overlapping patterns that are also supported by the remaining data. Specifically, we propose Grouped Memorization Evaluation, an example-level metric that separates memorized knowledge from overlapping knowledge. Building on this metric, we introduce Federated Memorization Pruning (FedMemPrune), a pruning-based unlearning approach that resets redundant parameters responsible for memorization. Extensive experiments show that FedMemPrune closely matches retraining-based unlearning baselines while more effectively eliminating memorization than existing federated unlearning algorithms, yielding strong unlearning performance without sacrificing the utility of retained knowledge.

Abstract

Federated Unlearning

Explore similar work

Rethinking Federated Unlearning via the Lens of Memorization

FedUP: One-Shot Federated Unlearning via Centroid-Guided Plug-in Filters

Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients