FIRM: Fisher-Informed Retention Merging for Federated LLM Unlearning

Reeyansh Shah; Maram Hasan; Subhasis Chaudhuri; Biplab Banerjee

cs.LGOct 15, 2026

FIRM: Fisher-Informed Retention Merging for Federated LLM Unlearning

Authors: Reeyansh Shah, Maram Hasan, Subhasis Chaudhuri, Biplab Banerjee

Organizations: Indian Institute of Technology Bombay

Abstract

Federated LLM unlearning addresses targeted knowledge removal when forget and retain data remain distributed across private clients and are inaccessible to the server. The resulting locally trained adapters encode conflicting objectives, making their aggregation fundamentally different from conventional federated optimization. A key failure mode is magnitude explosion, whereby unnormalized cross-client aggregation increases the merged update magnitude and degrades retained utility under stronger unlearning. We propose FIRM (Fisher-Informed Retention Merging), a single-round framework for stable and retention-aware server-side aggregation. FIRM uses client-local Fisher statistics as a unified sensitivity signal to construct a Fisher-normalized base merge and impose explicit control over retain-sensitive parameters. Its second-order retention projection admits a unique minimum-distance solution, while the corresponding dual scales with the number of retain clients rather than the model dimension. Experiments across WMDP, TOFU, and MUSE-HP using Zephyr-7B and Llama-3-8B show utility gains at comparable forgetting on WMDP and MUSE-HP, with metric-dependent trade-offs on TOFU. Controlled comparisons support benefits of the combined base merge and additional retention correction at selected operating points.

Figures & tables

Figure 1: Overview of FIRM. Clients transmit task vectors and raw Fisher diagonals once. The server pools forget-domain summaries, normalizes and sparsifies each Fisher tensor, and applies separately normalized forget and retain merging with optional retention correction.

Method	$\bm{\gamma}$	Bio $\downarrow$	Cyber $\downarrow$	Util $\uparrow$
Pretrained	—	63.08	40.11	54.46
FULM	0.05	62.37	40.01	54.35
FULM	0.10	59.15	38.95	53.84
FULM	0.15	50.27	36.03	49.52
FULM	0.17	43.13	31.76	45.88
FULM	0.20	28.20	26.17	35.73

Table 1: WMDP sweep on Zephyr-7B. FIRM uses Fisher-weighted merging without QP (

\epsilon=\infty

); matched-forgetting QP comparisons are reported in Table 7 .

\downarrow/\uparrow

: lower/higher is better.

\dagger

: Utility collapse (

\leq 25\%

, near random). Bold : FIRM utility values for ease of comparison.

Method	Setting	Bio $\downarrow$	Cyber $\downarrow$	Util $\uparrow$
AVG	scale=1.00	24.59	25.87	24.18
SUM	scale=0.05	38.41	30.60	43.06
TIES	topk=0.50	24.74	25.11	22.80
FULM	$\gamma=0.17$	43.13	31.76	45.88
FIRM (ours)	$\gamma=1.00$	42.42	31.91	48.29

Table 2: WMDP results at the listed evaluated settings (Zephyr-7B). Scale-controlled averaging is compared separately in Table 6 . Bold marks the highest utility among the listed settings.

Panel of Figure 2 — Figure 2: Forgetting–utility trade-off on WMDP with Zephyr-7B. Each point denotes a scaling value $\gamma$ . The discussed comparable-forgetting pair favors FIRM; connecting segments visualize the sweep and do not establish uniform dominance.

Method	$\bm{\gamma}$	Fgt $\downarrow$	Ret $\uparrow$	RA $\uparrow$	RW $\uparrow$	Util $\uparrow$
Expert baseline	—	96.74	94.64	47.15	77.78	39.49
FULM	1.0	71.85	95.51	44.65	75.93	37.35
FULM	1.5	61.68	91.51	42.45	70.80	35.94
FULM	2.0	51.84	78.16	34.03	66.97	32.60
FULM	2.5	43.71	67.84	23.20	64.34	33.78
FIRM	1.0	90.15	96.75	49.15	77.64	38.40

Table 3: TOFU unlearning on Zephyr-7B (base merge without QP). Fgt

\downarrow

: forget ROUGE-L (lower = better). Ret

\uparrow

, RA

\uparrow

, RW

\uparrow

: retain, real authors, real world ROUGE-L. Util

\uparrow

: MMLU accuracy. Bold marks the highest retention and utility scores among the unlearned models.

Method	$\bm{\gamma}$	HP $\downarrow$	Util $\uparrow$
Pretrained	—	50.57	61.12
FULM	0.10	5.52	59.59
FULM	0.20	0.00	57.52
FULM	0.30	0.00	53.95
FULM	0.50	0.00	35.67
FULM	0.70	0.00	22.95 †

Table 4: MUSE-HP unlearning on Llama-3-8B (base merge without QP). HP

\downarrow

: ROUGE-L on forget QA (lower = better forgetting). Util

\uparrow

: MMLU accuracy.

\dagger

: utility collapse (

\leq 25\%

, near random). Bold marks the highest utility at zero HP ROUGE-L.

Configuration	Bio $\downarrow$	Cyber $\downarrow$	Util $\uparrow$
FIRM-base ( $\alpha=0.10$ )	42.42	31.91	48.29
A1: No sum-norm	63.00	40.01	54.37
A2: Uniform weights	24.82	25.87	24.54 †
A3: Top-20% ( $\alpha=0.20$ )	34.64	29.59	42.92
A3: Full Fisher ( $\alpha=1.0$ )	24.82	25.87	24.54 †
Pretrained	63.08	40.11	54.46

Table 5: Ablation study on WMDP (Zephyr-7B,

\gamma=1.0

\dagger

: catastrophic utility collapse (

\leq

25%).

Method	$\gamma$	Bio $\downarrow$	Cyber $\downarrow$	MMLU $\uparrow$
Uniform avg.	0.30	45.80	34.78	48.58
FIRM-base	1.00	45.56	34.88	51.33
FULM-MEAN	0.35	58.21	38.70	53.77
FIRM-base	0.60	58.05	38.70	54.38

Table 6: FIRM-base versus scale-tuned averaging on shared WMDP checkpoints, without QP. Each block compares closely matched forgetting; all accuracies are in %.

Pair ( $\gamma$ )	Bio gap	Cyber gap	$\Delta$ MMLU
Q/F (1.5/1.1)	0.000	0.201	+0.669
Q/F (2.2/1.5)	0.393	0.302	+1.723
T/Q (1.1/1.0)	0.079	0.101	+0.249
T/Q (1.5/1.4)	0.471	0.252	+0.349
T/Q (2.4/2.3)	0.393	0.302	+1.453

Table 7: Representative retention-control comparisons on WMDP (Zephyr-7B), using shared regenerated checkpoints. F: FIRM-base (

\epsilon=\infty

); Q: FIRM-QP with

\epsilon=4.1\times 10^{-5}

; T: FIRM-QP with

\epsilon=10^{-5}

. Parentheses give the first/second method’s

\gamma

. Accuracy gaps are absolute;

\Delta

MMLU is first minus second. All differences are in percentage points.

$m$	Active	$\max_{k}b_{k}$		Time	Bio $\downarrow$	Util $\uparrow$
	$\lambda_{k}$	Before	After	(s)	no-QP/QP	no-QP/QP
1	1/1	$5.7{\times}10^{-4}$	$1.0{\times}10^{-4}$	0.77	42.97/49.73	48.16/51.69
2	2/2	$3.6{\times}10^{-4}$	$5.7{\times}10^{-5}$	0.99	42.81/49.80	47.63/51.66
3	3/3	$4.2{\times}10^{-4}$	$6.3{\times}10^{-5}$	1.66	42.50/50.98	47.67/52.12

Table 8: Retention-constraint checks with heterogeneous retain clients on WMDP. Damage is measured per tensor; Bio and Utility are reported as no-QP/QP. For

m=1,2,3

, respectively,

\epsilon\approx(9.98,5.68,6.30)\times 10^{-5}

, calibrated separately as 0.3 times the mean no-QP damage across retain clients and tensors.

Panel of Figure 3 — Figure 3: Merged adapter $\ell_{2}$ norm versus $\gamma$ over three WMDP LoRA tensors. FIRM has smaller norms at equal $\gamma$ . This illustrates method-dependent update scaling; performance comparisons require tuning $\gamma$ and matching forgetting.

$n$	Adapter norm		Utility $\uparrow$
	FULM	FIRM	FULM	FIRM
1	0.73 ( $\times 1.0$ )	1.65 ( $\times 1.0$ )	54.22	52.68
2	1.44 ( $\times 2.0$ )	2.59 ( $\times 1.6$ )	43.31	47.44
3	1.86 ( $\times 2.6$ )	2.78 ( $\times 1.7$ )	24.03	39.10

Table 9: Effect of increasing the number of heterogeneous forget-domain centroids

n

on WMDP: Bio, then Cyber, then Chemistry, with one forget client per domain and one retain client. Scales remain fixed at

\gamma=0.15

(FULM) and

1.0

(FIRM), with no active QP. Norms cover layers 5–7.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\Delta\theta_{c}$	Concatenated $\mathrm{vec}(sBA)$ , with $s=\alpha_{\mathrm{LoRA}}/r$
$\bm{\theta}_{0},\bm{\theta}_{F}$	Initial model and pretrained Fisher reference
$\alpha,\alpha_{\mathrm{LoRA}}$	Fisher retained fraction and LoRA scaling coefficient
$F_{c}(j)$	Raw empirical Fisher diagonal for parameter $j$ of client $c$
$\tilde{F}_{s}(j)$	Per-tensor max-normalized, Top- $\alpha d$ Fisher sensitivity
$\mathcal{C}_{f}$	Forget clients sharing unlearning domain $g$

Appendix

Table 10: Main notation used in FIRM.

Method	$\bm{\gamma}$	HP $\downarrow$	MMLU $\uparrow$
Pretrain	–	44.07	54.46
FULM	0.005	43.43	54.29
FULM	0.010	41.85	54.19
FULM	0.020	41.01	53.88
FULM	0.030	30.25	53.51
FULM	0.050	10.23	52.21

Appendix

Table 11: Supplementary sweep results on Zephyr-7B / MUSE-HP.

Method	$\bm{\gamma}$	Bio $\downarrow$	Cyber $\downarrow$	MMLU $\uparrow$
Pretrain	–	70.93	43.53	61.12
FULM	0.05	71.41	43.63	61.10
FULM	0.10	70.78	42.68	60.82
FULM	0.15	70.31	42.07	60.30
FULM	0.20	69.21	41.67	59.36
FULM	0.30	61.82	38.45	56.62

Appendix

Table 12: Supplementary sweep results on Llama-3-8B / WMDP.

Component (per client unless stated)	Time / Size
Adapter training	0.63 min
bf16 model load for Fisher	0.11 min
Fisher estimation, 20 batches	15.96 min ∗
Peak GPU memory: training / Fisher	19.7 / 38.4 ∗ GB
Task-vector checkpoint	336 MiB
Dense Fisher on adapter support (derived)	336 MiB values

Appendix

Table 13: Client-side and server-side efficiency breakdown for FIRM. ∗ The measurement keeps the 4-bit training model resident while loading the bf16 Fisher copy. These measurements include both resident model copies; an isolated Fisher pass was not measured.

Pair ( $\gamma$ )	Bio gap	Cyber gap	$\Delta$ MMLU
F/U (0.4/0.10)	0.000	0.000	-0.007
F/U (0.6/0.20)	0.236	0.050	+0.413
F/U (1.0/0.30)	0.236	0.101	+2.756
F/U (1.3/0.35)	0.550	0.151	+2.471

Appendix

Table 14: Representative scale-tuned comparisons on WMDP. F: FIRM-base; U: dense uniform. Parentheses give first/second

\gamma

. Accuracy gaps are absolute;

\Delta

MMLU is F minus U. Differences are percentage points.

$\epsilon$	Bio $\downarrow$	Cyber $\downarrow$	Util $\uparrow$
$\infty$	42.97	31.96	48.16
$1{\times}10^{-4}$	49.57	36.39	51.68
$6{\times}10^{-5}$	50.75	37.49	52.41
$4.1{\times}10^{-5}$	52.00	37.85	52.85
$2{\times}10^{-5}$	54.05	38.35	53.29
$1{\times}10^{-5}$	55.07	38.85	53.45

Appendix

Table 15: Retention tolerance

\epsilon

effects on WMDP (Zephyr-7B,

\gamma=1.0

). Smaller

\epsilon

enforces a tighter per-tensor proxy budget.

Pair ( $\gamma$ )	Bio gap	Cyber gap	$\Delta$ MMLU
Q/F (0.3/0.3)	0.157	0.050	-0.028
Q/F (0.4/0.4)	0.393	0.252	-0.036
Q/F (0.5/0.5)	0.393	0.101	-0.114
Q/F (0.7/0.6)	0.157	0.151	-0.085
Q/F (1.5/1.1)	0.000	0.201	+0.669
Q/F (2.2/1.5)	0.393	0.302	+1.723

Appendix

Table 16: Jointly close projection comparisons on WMDP. F: base; Q:

\epsilon=4.1\times 10^{-5}

; T:

\epsilon=10^{-5}

. Parentheses give first/second

\gamma

. Gaps are absolute;

\Delta

MMLU is first minus second, in percentage points. Selection uses forgetting gaps, not utility.

Federated Unlearning

Federated unlearning removes a client’s or record’s influence from a collaboratively trained model while preserving useful knowledge.

See all 29 papers

Not All Retained Clients Are Equal: Client Selection for Federated Unlearning
Oct 22, 2026Thanh Linh Nguyen, Quoc-Viet PhamFederated Unlearning Client Selection in Federated Learning
School of Computer Science and Statistics, Trinity College Dublin, Ireland
Rethinking Federated Unlearning via the Lens of Memorization
May 23, 2026Jiaheng Wei, Yanjun Zhang, He Zhang +5Federated Unlearning Neural Network Memorization
Royal Melbourne Institute of Technology · Griffith University · Swinburne University of Technology
Split Federated Unlearning
Oct 27, 2026Jia Chen, Yecheng Lv, Xinrong Hu +2Federated Unlearning Split Federated Learning
School of Computer Science and Artificial Intelligence, Wuhan Textile University, Wuhan, China · Engineering Research Center of Hubei Province for Clothing Information, Wuhan Textile University, Wuhan, China · Australian Institute for Machine Learning (AIML), Adelaide University, Adelaide, Australia

Oct 22, 2026J/K move · Enter open · S save

Not All Retained Clients Are Equal: Client Selection for Federated Unlearning

Thanh Linh Nguyen, Quoc-Viet Pham

School of Computer Science and Statistics, Trinity College Dublin, Ireland

Federated Unlearning Client Selection in Federated Learning

Open paper AskPDF arXiv

Federated unlearning (FU) aims to remove a designated target client's influence from a trained model without the prohibitive cost of retraining from scratch. Existing methods largely treat unlearning as a post-hoc, stateless operation, overlooking that client influence is path-dependent: contributions made during training, particularly during critical learning periods (CLPs), can have disproportionate and persistent effects. Moreover, knowledge overlap and intermittent client availability make relying on all retained clients inefficient and impractical. We introduce PDF, a path-dependent client-selection framework based on the insight that retained clients are not equally valuable for unlearning. During training, PDF maintains lightweight, target-agnostic, CLP-weighted histories summarizing each client's accumulated update direction and global-model alignment. Upon an unlearning request, these histories identify and rank retained clients that are globally informative yet weakly coupled to the target, enabling retraining-based unlearning with only the top-

k

clients over a small number of rounds. Experiments on FMNIST, CIFAR-10, and CIFAR-100 under IID and non-IID settings show that PDF achieves effective forgetting and strong retained utility with substantially fewer retained clients. In particular, PDF achieves

0\%

forgetting accuracy with retraining-like MIA AUC on non-IID CIFAR-10, improves retained accuracy by up to

6.91

percentage points over the strongest competing FU baseline on CIFAR-100, and reduces communication by approximately

69\%

and computation by

57

61\%

relative to Retrain.

Abstract

Federated Unlearning

Explore similar work

Not All Retained Clients Are Equal: Client Selection for Federated Unlearning

Rethinking Federated Unlearning via the Lens of Memorization

Split Federated Unlearning