FedLNS: Leverage LayerNorm Signature Modeling to Mitigate Adversarial Manipulation in Federated LLMs

Kai Li; Jong-Ik Park; Carlee Joe-Wong; Wei Ni; Falko Dressler

cs.LGAug 19, 2026

FedLNS: Leverage LayerNorm Signature Modeling to Mitigate Adversarial Manipulation in Federated LLMs

Authors: Kai Li, Jong-Ik Park, Carlee Joe-Wong, Wei Ni, Falko Dressler

Organizations: Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pennsylvania, United States · School of Engineering, Edith Cowan University, Perth, WA 6027, Australia · School of Electrical Engineering and Computer Science, TU Berlin, Germany

Abstract

Federated training enables language models to learn from distributed private text, but the server cannot directly verify the local supervision or optimization process that produces each client update. A malicious client can therefore train on corrupted targets, introduce incorrect context-token associations, and degrade the global model through repeated aggregation. Such degradation can also increase the risk of unreliable or hallucinatory generation. We propose Federated Learning with Normalization Signatures (FedLNS), a server-side framework for lightweight malicious-update screening. FedLNS represents each client update through changes in trainable normalization-layer parameters and screens suspicious updates against a robust, history-aware cross-client reference. Because the signatures are extracted at the server from the returned local models, FedLNS requires no additional client-to-server parameter or metadata exchange compared to standard federated learning (FL) methods. After screening, the retained full-model updates can be aggregated using standard FL or another compatible aggregation rule. FedLNS requires no raw client data, trusted server dataset, labeled attack examples, or separately trained detector. Experiments on GPT-style, BERT-style, and LLaMA-style models trained from scratch with 200 clients show that, under 40% population-level target manipulation, FedLNS achieves lower test perplexity than the strongest of six baselines for all three architectures under both IID (independently and identically distributed) and non-IID data partitions.

Figures & tables

Fig. 1 : Threat model for malicious target manipulation in federated language-model training. A malicious client changes its local supervision and optimizes incorrect context–target associations. If the resulting update is accepted, repeated aggregation can degrade the global model and increase the downstream risk of semantically unreliable generation.

Panel of Figure 2 — Fig. 2 : Perplexity (lower is better) under target manipulation affecting $40\%$ of the client population. Bars show the mean over three random seeds, and error bars show one sample standard deviation. FedLNS consistently shows the lowest perplexity, across both IID and non-IID data partitions.

Panel of Figure 3 — Fig. 3 : Token-level semantic entropy (lower is better) under target manipulation affecting $40\%$ of the client population. Bars show the mean over three random seeds, and error bars show one sample standard deviation. FedLNS consistently shows the lowest semantic entropy, across both IID and non-IID data partitions.

Appendix figures & tables98 assets

Supplementary material from the paper’s appendix.

Appendix

Bank coverage $\rho$	$M_{\mathrm{act}}=\rho K$	Expected activation round
$25\%$	$50$	$3$
$50\%$	$100$	$7$
$75\%$	$150$	$14$

Appendix

TABLE I : Expected round at which the bank reaches each coverage threshold under random participation with

K=200

and participation rate

q=0.1

Item	Value
Number of clients	$K=200$
Participation rate	$0.1$
Clients per round	$20$
Communication rounds	$200$
Random seeds	$100,200,300$
Data partitions	IID and non-IID

Appendix

TABLE II : Federated simulation protocol used across model families.

Model family	Dataset	Tokenizer	Sequence construction	Evaluation split used for selection and reporting
GPT-style LM	WikiText-2 raw	gpt2	Causal LM blocks of length $128$	Official validation split
BERT-style MLM	Tiny Shakespeare	bert-base-uncased	MLM blocks of length $128$ with [CLS]/[SEP]	Final $10\%$ of the text
LLaMA-style LM	StackOverflow posts	TinyLlama/TinyLlama-1.1B-Chat-v1.0	Causal LM blocks of length $128$	Evaluation split defined during dataset construction

Appendix

TABLE III : Dataset and tokenization settings. All models are initialized from scratch; tokenizer names specify tokenization only.

Model family	Training units before partitioning	IID partition	Non-IID partition
GPT-style LM / WikiText-2	Causal LM token blocks of length $128$	Shuffle all block indices and assign them round-robin to $200$ clients	Split ordered block indices into $400$ contiguous shards; assign two shuffled shards per client
BERT-style MLM / Tiny Shakespeare	BERT MLM blocks of length $128$ from the first $90\%$ of the text	Shuffle all block indices and assign them round-robin to $200$ clients	Split ordered block indices into $400$ contiguous shards; assign two shuffled shards per client
LLaMA-style LM / StackOverflow	Causal LM token blocks of length $128$ from up to $50{,}000$ training posts	Shuffle all block indices and assign them round-robin to $200$ clients	Split ordered block indices into $400$ contiguous shards; assign two shuffled shards per client

Appendix

TABLE IV : Client data partition construction for IID and non-IID settings.

Hyperparameter	GPT-style LM	BERT-style MLM	LLaMA-style LM
Hidden / embedding dimension	$256$	$256$	$256$
Number of layers	$12$	$12$	$8$
Attention heads	$16$	$16$	$16$
Key-value heads	–	–	$8$
Intermediate dimension	–	$1024$	$1536$
Context / block size	$128$	$128$	$128$

Appendix

TABLE V : Scratch transformer architectures.

Setting	GPT-style LM	BERT-style MLM	LLaMA-style LM
Local epochs	$3$	$3$	$2$
Batch size	$8$	$8$	$8$
Learning rate	$3\times 10^{-4}$	$3\times 10^{-4}$	$10^{-5}$
Weight decay	$0.01$	$0.01$	$0.01$
Train-evaluation batches	$200$	$200$	$50$
Evaluation batches	$200$	$200$	$50$

Appendix

TABLE VI : Optimization and evaluation settings.

Item	Value
Attack type	Target corruption
Input tokens	Kept unchanged
Corrupted quantity	Training targets / labels
Corruption probability	$p_{\mathrm{corr}}=1.0$
Malicious fractions in sweep	$0,0.1,0.2,0.3,0.4$
Strongest main-text attack fraction	$0.4$

Appendix

TABLE VII : Target-corruption attack settings.

Method	Implementation summary
FedAvg	Sample-weighted average of client updates
Norm-Bounded FedAvg	Each update is clipped by an $\ell_{2}$ norm bound before FedAvg
Coordinate Median	Coordinate-wise median of flattened updates
Trimmed Mean	Coordinate-wise trimming followed by averaging
Multi-Krum	Distance-based update selection followed by averaging selected updates
FLAME	Two-cluster filtering, majority-cluster retention, norm clipping, and Gaussian noise

Appendix

TABLE VIII : Aggregation methods compared in the experiments.

Malicious fraction	Trim fraction	Multi-Krum $f$	Multi-Krum $m$
$0.0$	$0.0$	$0$	$20$
$0.1$	$0.1$	$2$	$18$
$0.2$	$0.2$	$4$	$16$
$0.3$	$0.3$	$6$	$14$
$0.4$	$0.4$	$8$	$12$

Appendix

TABLE IX : Robust-aggregation parameters used for each malicious-client fraction. The number of selected clients per round is

20

Item	Setting
Signature source	Trainable normalization-layer parameters whose names contain ln or layernorm and end with weight or bias
Bank entry	One latest per-layer signature dictionary per client
Bank update timing	Before screening in the current round
Bank update rule	Latest replacement; EMA coefficient $\beta=0$
Robust center	Coordinate-wise bank median
Robust scale	Coordinate-wise MAD

Appendix

TABLE X : FedLNS implementation settings used in the experiments.

Method	0%	10%	20%	30%	40%
FedAvg	7.13 $\pm$ 0.01	7.17 $\pm$ 0.01	7.23 $\pm$ 0.00	7.29 $\pm$ 0.01	7.37 $\pm$ 0.02
Norm-Bound	7.50 $\pm$ 0.00	7.51 $\pm$ 0.01	7.59 $\pm$ 0.01	7.69 $\pm$ 0.02	7.89 $\pm$ 0.01
Median	7.30 $\pm$ 0.01	7.27 $\pm$ 0.01	7.28 $\pm$ 0.01	7.30 $\pm$ 0.01	7.47 $\pm$ 0.03
Trimmed	7.12 $\pm$ 0.01	7.19 $\pm$ 0.01	7.25 $\pm$ 0.01	7.32 $\pm$ 0.03	7.43 $\pm$ 0.01
Multi-Krum	7.12 $\pm$ 0.01	7.15 $\pm$ 0.01	7.19 $\pm$ 0.00	7.21 $\pm$ 0.00	7.28 $\pm$ 0.02
FLAME	7.19 $\pm$ 0.00	7.19 $\pm$ 0.01	7.20 $\pm$ 0.00	7.21 $\pm$ 0.00	7.33 $\pm$ 0.03

Appendix

TABLE XI : Test Loss under different malicious-client fractions for GPT / WikiText with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	1242.91 $\pm$ 6.58	1303.02 $\pm$ 9.59	1373.61 $\pm$ 1.63	1472.41 $\pm$ 15.31	1590.56 $\pm$ 35.51
Norm-Bound	1809.27 $\pm$ 8.31	1828.13 $\pm$ 10.69	1985.85 $\pm$ 25.34	2196.84 $\pm$ 33.99	2683.53 $\pm$ 34.31
Median	1484.69 $\pm$ 11.20	1441.44 $\pm$ 9.33	1452.99 $\pm$ 11.40	1477.26 $\pm$ 17.65	1751.07 $\pm$ 52.62
Trimmed	1241.55 $\pm$ 10.08	1326.15 $\pm$ 17.45	1411.15 $\pm$ 15.57	1516.84 $\pm$ 43.71	1689.16 $\pm$ 18.00
Multi-Krum	1242.08 $\pm$ 8.40	1275.34 $\pm$ 7.09	1320.41 $\pm$ 5.63	1356.21 $\pm$ 6.19	1449.72 $\pm$ 22.36
FLAME	1323.55 $\pm$ 4.87	1328.88 $\pm$ 7.21	1345.05 $\pm$ 2.82	1355.34 $\pm$ 3.27	1524.02 $\pm$ 40.71

Appendix

TABLE XII : Test PPL under different malicious-client fractions for GPT / WikiText with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	6.80 $\pm$ 0.06	7.08 $\pm$ 0.08	7.37 $\pm$ 0.18	7.56 $\pm$ 0.16	7.96 $\pm$ 0.07
Norm-Bound	6.55 $\pm$ 0.02	8.04 $\pm$ 0.12	8.31 $\pm$ 0.10	8.92 $\pm$ 0.03	9.69 $\pm$ 0.06
Median	6.89 $\pm$ 0.04	7.30 $\pm$ 0.15	7.27 $\pm$ 0.14	7.61 $\pm$ 0.13	7.55 $\pm$ 0.19
Trimmed	6.77 $\pm$ 0.03	7.30 $\pm$ 0.18	7.29 $\pm$ 0.19	7.47 $\pm$ 0.13	7.55 $\pm$ 0.39
Multi-Krum	6.80 $\pm$ 0.07	7.14 $\pm$ 0.24	7.07 $\pm$ 0.09	7.03 $\pm$ 0.11	7.13 $\pm$ 0.37
FLAME	7.11 $\pm$ 0.05	7.13 $\pm$ 0.10	7.20 $\pm$ 0.05	7.11 $\pm$ 0.09	6.96 $\pm$ 0.10

Appendix

TABLE XIII : Semantic Entropy under different malicious-client fractions for GPT / WikiText with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	7.10 $\pm$ 0.01	7.16 $\pm$ 0.01	7.23 $\pm$ 0.04	7.29 $\pm$ 0.06	7.37 $\pm$ 0.08
Norm-Bound	7.56 $\pm$ 0.01	7.54 $\pm$ 0.00	7.63 $\pm$ 0.01	7.75 $\pm$ 0.02	7.94 $\pm$ 0.09
Median	7.21 $\pm$ 0.00	7.18 $\pm$ 0.00	7.26 $\pm$ 0.02	7.33 $\pm$ 0.02	7.51 $\pm$ 0.03
Trimmed	7.11 $\pm$ 0.01	7.17 $\pm$ 0.01	7.25 $\pm$ 0.01	7.38 $\pm$ 0.03	7.51 $\pm$ 0.05
Multi-Krum	7.11 $\pm$ 0.01	7.18 $\pm$ 0.00	7.25 $\pm$ 0.01	7.30 $\pm$ 0.00	7.36 $\pm$ 0.03
FLAME	7.21 $\pm$ 0.00	7.25 $\pm$ 0.01	7.28 $\pm$ 0.01	7.29 $\pm$ 0.01	7.40 $\pm$ 0.04

Appendix

TABLE XIV : Test Loss under different malicious-client fractions for GPT / WikiText with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	1211.23 $\pm$ 6.70	1282.08 $\pm$ 13.22	1378.40 $\pm$ 55.98	1471.79 $\pm$ 90.09	1595.68 $\pm$ 132.68
Norm-Bound	1916.47 $\pm$ 13.00	1886.00 $\pm$ 3.63	2060.01 $\pm$ 13.63	2314.33 $\pm$ 44.03	2811.07 $\pm$ 257.29
Median	1350.32 $\pm$ 1.34	1309.94 $\pm$ 6.40	1419.46 $\pm$ 32.27	1529.64 $\pm$ 31.03	1820.62 $\pm$ 48.31
Trimmed	1225.89 $\pm$ 8.21	1295.29 $\pm$ 16.41	1406.67 $\pm$ 11.55	1599.60 $\pm$ 52.85	1819.34 $\pm$ 98.93
Multi-Krum	1225.07 $\pm$ 8.06	1311.67 $\pm$ 5.60	1408.79 $\pm$ 20.45	1474.93 $\pm$ 3.80	1579.81 $\pm$ 43.94
FLAME	1359.58 $\pm$ 3.82	1415.19 $\pm$ 13.32	1449.95 $\pm$ 14.72	1467.85 $\pm$ 14.75	1637.88 $\pm$ 63.90

Appendix

TABLE XV : Test PPL under different malicious-client fractions for GPT / WikiText with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	6.79 $\pm$ 0.07	7.31 $\pm$ 0.17	7.12 $\pm$ 0.37	7.61 $\pm$ 0.30	7.66 $\pm$ 0.26
Norm-Bound	6.37 $\pm$ 0.02	7.85 $\pm$ 0.19	8.44 $\pm$ 0.20	9.02 $\pm$ 0.09	9.60 $\pm$ 0.36
Median	6.74 $\pm$ 0.15	7.31 $\pm$ 0.34	7.14 $\pm$ 0.32	7.52 $\pm$ 0.10	7.45 $\pm$ 0.25
Trimmed	6.75 $\pm$ 0.15	7.36 $\pm$ 0.07	7.27 $\pm$ 0.07	7.39 $\pm$ 0.11	7.72 $\pm$ 0.28
Multi-Krum	6.79 $\pm$ 0.17	7.08 $\pm$ 0.12	7.07 $\pm$ 0.45	7.25 $\pm$ 0.47	7.18 $\pm$ 0.46
FLAME	7.04 $\pm$ 0.03	7.08 $\pm$ 0.15	7.32 $\pm$ 0.20	6.94 $\pm$ 0.07	7.13 $\pm$ 0.58

Appendix

TABLE XVI : Semantic Entropy under different malicious-client fractions for GPT / WikiText with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	6.49 $\pm$ 0.03	6.54 $\pm$ 0.01	6.57 $\pm$ 0.01	6.62 $\pm$ 0.02	6.68 $\pm$ 0.02
Norm-Bound	6.80 $\pm$ 0.03	6.80 $\pm$ 0.01	6.92 $\pm$ 0.01	7.11 $\pm$ 0.01	7.30 $\pm$ 0.05
Median	6.80 $\pm$ 0.03	6.82 $\pm$ 0.01	6.82 $\pm$ 0.02	6.80 $\pm$ 0.03	6.97 $\pm$ 0.08
Trimmed	6.49 $\pm$ 0.03	6.66 $\pm$ 0.01	6.72 $\pm$ 0.02	6.80 $\pm$ 0.03	6.97 $\pm$ 0.07
Multi-Krum	6.49 $\pm$ 0.03	6.51 $\pm$ 0.01	6.52 $\pm$ 0.01	6.54 $\pm$ 0.02	6.59 $\pm$ 0.02
FLAME	6.49 $\pm$ 0.03	6.47 $\pm$ 0.02	6.49 $\pm$ 0.02	6.52 $\pm$ 0.00	6.70 $\pm$ 0.18

Appendix

TABLE XVII : Test Loss under different malicious-client fractions for BERT / Shakespeare with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	659.93 $\pm$ 21.92	693.01 $\pm$ 4.36	710.15 $\pm$ 6.19	750.09 $\pm$ 12.08	794.54 $\pm$ 15.01
Norm-Bound	901.04 $\pm$ 29.47	897.17 $\pm$ 10.13	1008.61 $\pm$ 7.00	1227.23 $\pm$ 15.36	1477.67 $\pm$ 70.84
Median	894.85 $\pm$ 27.58	912.29 $\pm$ 9.07	913.38 $\pm$ 19.06	898.46 $\pm$ 22.55	1066.13 $\pm$ 91.56
Trimmed	660.99 $\pm$ 20.10	779.94 $\pm$ 9.18	831.19 $\pm$ 20.60	900.65 $\pm$ 26.88	1062.80 $\pm$ 79.49
Multi-Krum	661.00 $\pm$ 20.10	672.89 $\pm$ 7.97	676.44 $\pm$ 4.57	695.35 $\pm$ 12.21	725.76 $\pm$ 17.84
FLAME	658.52 $\pm$ 18.95	645.26 $\pm$ 15.47	657.21 $\pm$ 10.53	679.98 $\pm$ 3.26	818.57 $\pm$ 157.53

Appendix

TABLE XVIII : Test PPL under different malicious-client fractions for BERT / Shakespeare with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	6.26 $\pm$ 0.18	6.79 $\pm$ 0.19	6.95 $\pm$ 0.07	7.22 $\pm$ 0.36	7.37 $\pm$ 0.33
Norm-Bound	6.31 $\pm$ 0.03	7.27 $\pm$ 0.21	7.65 $\pm$ 0.07	8.07 $\pm$ 0.15	8.54 $\pm$ 0.16
Median	6.60 $\pm$ 0.06	6.92 $\pm$ 0.22	7.05 $\pm$ 0.34	7.41 $\pm$ 0.22	7.47 $\pm$ 0.78
Trimmed	6.28 $\pm$ 0.16	6.63 $\pm$ 0.17	6.97 $\pm$ 0.18	7.35 $\pm$ 0.54	6.76 $\pm$ 0.47
Multi-Krum	6.28 $\pm$ 0.16	6.40 $\pm$ 0.07	6.69 $\pm$ 0.38	6.53 $\pm$ 0.36	6.61 $\pm$ 0.28
FLAME	6.26 $\pm$ 0.03	6.26 $\pm$ 0.11	6.11 $\pm$ 0.05	6.15 $\pm$ 0.10	6.46 $\pm$ 0.16

Appendix

TABLE XIX : Semantic Entropy under different malicious-client fractions for BERT / Shakespeare with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	6.36 $\pm$ 0.03	6.38 $\pm$ 0.04	6.51 $\pm$ 0.10	6.56 $\pm$ 0.12	6.59 $\pm$ 0.11
Norm-Bound	6.83 $\pm$ 0.01	6.85 $\pm$ 0.01	6.95 $\pm$ 0.01	7.14 $\pm$ 0.02	7.34 $\pm$ 0.02
Median	6.86 $\pm$ 0.01	6.89 $\pm$ 0.01	6.86 $\pm$ 0.03	6.86 $\pm$ 0.01	6.95 $\pm$ 0.05
Trimmed	6.54 $\pm$ 0.03	6.72 $\pm$ 0.01	6.80 $\pm$ 0.02	6.84 $\pm$ 0.02	6.94 $\pm$ 0.02
Multi-Krum	6.54 $\pm$ 0.03	6.59 $\pm$ 0.02	6.60 $\pm$ 0.01	6.61 $\pm$ 0.02	6.69 $\pm$ 0.06
FLAME	6.57 $\pm$ 0.02	6.56 $\pm$ 0.03	6.56 $\pm$ 0.01	6.57 $\pm$ 0.01	6.86 $\pm$ 0.17

Appendix

TABLE XX : Test Loss under different malicious-client fractions for BERT / Shakespeare with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	576.49 $\pm$ 19.76	588.64 $\pm$ 24.25	672.55 $\pm$ 66.84	708.50 $\pm$ 88.05	731.73 $\pm$ 83.15
Norm-Bound	922.98 $\pm$ 7.18	941.70 $\pm$ 9.95	1039.58 $\pm$ 9.18	1262.16 $\pm$ 26.29	1533.59 $\pm$ 36.83
Median	950.49 $\pm$ 10.79	985.58 $\pm$ 6.47	949.59 $\pm$ 26.06	953.32 $\pm$ 10.76	1047.36 $\pm$ 53.66
Trimmed	691.27 $\pm$ 19.12	829.90 $\pm$ 7.29	893.88 $\pm$ 19.37	938.21 $\pm$ 17.35	1033.35 $\pm$ 24.28
Multi-Krum	691.27 $\pm$ 19.11	727.43 $\pm$ 13.60	732.73 $\pm$ 10.31	739.83 $\pm$ 12.52	804.39 $\pm$ 53.06
FLAME	710.27 $\pm$ 16.45	705.47 $\pm$ 23.82	704.63 $\pm$ 4.08	712.01 $\pm$ 8.24	963.65 $\pm$ 157.76

Appendix

TABLE XXI : Test PPL under different malicious-client fractions for BERT / Shakespeare with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	5.87 $\pm$ 0.14	6.34 $\pm$ 0.41	6.96 $\pm$ 0.20	7.19 $\pm$ 0.38	6.94 $\pm$ 0.14
Norm-Bound	6.27 $\pm$ 0.01	6.99 $\pm$ 0.14	7.63 $\pm$ 0.15	7.98 $\pm$ 0.19	8.48 $\pm$ 0.02
Median	6.59 $\pm$ 0.06	7.28 $\pm$ 0.10	7.20 $\pm$ 0.36	7.41 $\pm$ 0.26	7.20 $\pm$ 0.59
Trimmed	6.24 $\pm$ 0.10	6.61 $\pm$ 0.15	7.00 $\pm$ 0.13	7.23 $\pm$ 0.17	6.75 $\pm$ 0.27
Multi-Krum	6.24 $\pm$ 0.10	6.67 $\pm$ 0.24	6.43 $\pm$ 0.06	6.69 $\pm$ 0.13	6.82 $\pm$ 0.31
FLAME	6.19 $\pm$ 0.06	6.27 $\pm$ 0.16	6.30 $\pm$ 0.12	6.41 $\pm$ 0.12	6.62 $\pm$ 0.14

Appendix

TABLE XXII : Semantic Entropy under different malicious-client fractions for BERT / Shakespeare with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	5.36 $\pm$ 0.03	5.40 $\pm$ 0.03	5.46 $\pm$ 0.04	5.53 $\pm$ 0.02	5.66 $\pm$ 0.04
Norm-Bound	5.49 $\pm$ 0.03	5.56 $\pm$ 0.03	5.63 $\pm$ 0.04	5.77 $\pm$ 0.02	5.86 $\pm$ 0.08
Median	5.39 $\pm$ 0.03	5.40 $\pm$ 0.03	5.43 $\pm$ 0.03	5.46 $\pm$ 0.03	5.54 $\pm$ 0.03
Trimmed	5.36 $\pm$ 0.03	5.39 $\pm$ 0.03	5.43 $\pm$ 0.03	5.47 $\pm$ 0.03	5.54 $\pm$ 0.04
Multi-Krum	5.36 $\pm$ 0.03	5.37 $\pm$ 0.03	5.38 $\pm$ 0.03	5.39 $\pm$ 0.03	5.43 $\pm$ 0.04
FLAME	5.48 $\pm$ 0.03	5.48 $\pm$ 0.03	5.48 $\pm$ 0.03	5.49 $\pm$ 0.03	5.58 $\pm$ 0.05

Appendix

TABLE XXIII : Test Loss under different malicious-client fractions for LLaMA / StackOverflow with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	213.15 $\pm$ 6.77	221.94 $\pm$ 6.63	234.03 $\pm$ 8.53	251.01 $\pm$ 5.88	286.73 $\pm$ 10.99
Norm-Bound	243.23 $\pm$ 6.54	258.63 $\pm$ 6.54	279.81 $\pm$ 11.33	319.10 $\pm$ 6.04	349.83 $\pm$ 27.13
Median	218.98 $\pm$ 7.04	222.39 $\pm$ 7.10	227.13 $\pm$ 7.36	235.44 $\pm$ 6.60	254.94 $\pm$ 8.06
Trimmed	213.15 $\pm$ 6.77	219.60 $\pm$ 7.16	227.49 $\pm$ 7.80	237.43 $\pm$ 7.51	254.52 $\pm$ 9.07
Multi-Krum	213.15 $\pm$ 6.77	215.53 $\pm$ 7.04	217.38 $\pm$ 6.99	219.76 $\pm$ 6.92	228.52 $\pm$ 8.72
FLAME	240.02 $\pm$ 6.40	239.85 $\pm$ 6.77	240.36 $\pm$ 6.81	242.54 $\pm$ 7.33	264.80 $\pm$ 13.70

Appendix

TABLE XXIV : Test PPL under different malicious-client fractions for LLaMA / StackOverflow with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	5.52 $\pm$ 0.02	5.80 $\pm$ 0.02	5.97 $\pm$ 0.12	6.12 $\pm$ 0.06	6.58 $\pm$ 0.34
Norm-Bound	5.70 $\pm$ 0.01	5.96 $\pm$ 0.01	6.43 $\pm$ 0.22	6.83 $\pm$ 0.07	6.95 $\pm$ 0.43
Median	5.55 $\pm$ 0.02	5.73 $\pm$ 0.03	5.81 $\pm$ 0.08	5.91 $\pm$ 0.02	5.97 $\pm$ 0.09
Trimmed	5.52 $\pm$ 0.02	5.73 $\pm$ 0.04	5.81 $\pm$ 0.09	5.89 $\pm$ 0.04	5.95 $\pm$ 0.08
Multi-Krum	5.52 $\pm$ 0.02	5.54 $\pm$ 0.04	5.55 $\pm$ 0.03	5.56 $\pm$ 0.03	5.61 $\pm$ 0.03
FLAME	5.64 $\pm$ 0.05	5.61 $\pm$ 0.04	5.64 $\pm$ 0.05	5.65 $\pm$ 0.04	5.74 $\pm$ 0.05

Appendix

TABLE XXV : Semantic Entropy under different malicious-client fractions for LLaMA / StackOverflow with IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	5.37 $\pm$ 0.03	5.41 $\pm$ 0.03	5.47 $\pm$ 0.04	5.54 $\pm$ 0.03	5.67 $\pm$ 0.03
Norm-Bound	5.51 $\pm$ 0.03	5.57 $\pm$ 0.03	5.65 $\pm$ 0.04	5.79 $\pm$ 0.02	5.83 $\pm$ 0.05
Median	5.40 $\pm$ 0.03	5.42 $\pm$ 0.03	5.44 $\pm$ 0.03	5.48 $\pm$ 0.03	5.56 $\pm$ 0.03
Trimmed	5.37 $\pm$ 0.03	5.40 $\pm$ 0.03	5.44 $\pm$ 0.04	5.49 $\pm$ 0.03	5.56 $\pm$ 0.04
Multi-Krum	5.37 $\pm$ 0.03	5.38 $\pm$ 0.03	5.39 $\pm$ 0.03	5.41 $\pm$ 0.03	5.45 $\pm$ 0.04
FLAME	5.49 $\pm$ 0.03	5.49 $\pm$ 0.03	5.49 $\pm$ 0.03	5.50 $\pm$ 0.03	5.59 $\pm$ 0.04

Appendix

TABLE XXVI : Test Loss under different malicious-client fractions for LLaMA / StackOverflow with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	215.38 $\pm$ 6.80	224.27 $\pm$ 6.68	237.35 $\pm$ 9.12	255.84 $\pm$ 7.17	288.74 $\pm$ 8.31
Norm-Bound	246.30 $\pm$ 6.76	261.73 $\pm$ 6.66	283.49 $\pm$ 10.99	326.42 $\pm$ 6.26	340.88 $\pm$ 18.09
Median	221.84 $\pm$ 7.19	225.51 $\pm$ 7.38	230.73 $\pm$ 7.84	239.51 $\pm$ 6.99	260.33 $\pm$ 7.99
Trimmed	215.55 $\pm$ 6.92	222.49 $\pm$ 7.43	230.96 $\pm$ 8.21	241.46 $\pm$ 8.09	259.56 $\pm$ 9.62
Multi-Krum	215.55 $\pm$ 6.92	218.10 $\pm$ 7.52	219.94 $\pm$ 7.17	222.74 $\pm$ 7.63	233.42 $\pm$ 10.15
FLAME	243.12 $\pm$ 6.82	242.99 $\pm$ 6.85	243.25 $\pm$ 6.41	245.58 $\pm$ 7.14	269.08 $\pm$ 11.90

Appendix

TABLE XXVII : Test PPL under different malicious-client fractions for LLaMA / StackOverflow with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Method	0%	10%	20%	30%	40%
FedAvg	5.53 $\pm$ 0.03	5.80 $\pm$ 0.01	5.94 $\pm$ 0.11	6.22 $\pm$ 0.11	6.61 $\pm$ 0.27
Norm-Bound	5.70 $\pm$ 0.03	5.98 $\pm$ 0.01	6.44 $\pm$ 0.21	6.90 $\pm$ 0.09	6.56 $\pm$ 0.42
Median	5.56 $\pm$ 0.03	5.76 $\pm$ 0.02	5.84 $\pm$ 0.09	5.91 $\pm$ 0.04	5.99 $\pm$ 0.12
Trimmed	5.53 $\pm$ 0.03	5.75 $\pm$ 0.03	5.84 $\pm$ 0.10	5.92 $\pm$ 0.06	5.96 $\pm$ 0.10
Multi-Krum	5.53 $\pm$ 0.03	5.55 $\pm$ 0.04	5.56 $\pm$ 0.03	5.56 $\pm$ 0.02	5.63 $\pm$ 0.06
FLAME	5.66 $\pm$ 0.06	5.63 $\pm$ 0.06	5.62 $\pm$ 0.01	5.66 $\pm$ 0.07	5.74 $\pm$ 0.06

Appendix

TABLE XXVIII : Semantic Entropy under different malicious-client fractions for LLaMA / StackOverflow with Non-IID partition. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	6.7 $\pm$ 0.6	6.7 $\pm$ 0.6	6.7 $\pm$ 0.6
75%	13.3 $\pm$ 0.6	13.3 $\pm$ 0.6	13.3 $\pm$ 0.6

Appendix

TABLE XXIX : Estimated activation rounds for the bank-size ablation under IID partition with

0\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0
75%	13.0 $\pm$ 0.0	13.0 $\pm$ 0.0	13.0 $\pm$ 0.0

Appendix

TABLE XXX : Estimated activation rounds for the bank-size ablation under IID partition with

10\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0
75%	13.3 $\pm$ 0.6	13.3 $\pm$ 0.6	13.3 $\pm$ 0.6

Appendix

TABLE XXXI : Estimated activation rounds for the bank-size ablation under IID partition with

20\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	7.3 $\pm$ 0.6	7.3 $\pm$ 0.6	7.3 $\pm$ 0.6
75%	14.0 $\pm$ 1.0	14.0 $\pm$ 1.0	14.0 $\pm$ 1.0

Appendix

TABLE XXXII : Estimated activation rounds for the bank-size ablation under IID partition with

30\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0
75%	14.0 $\pm$ 1.0	14.0 $\pm$ 1.0	14.0 $\pm$ 1.0

Appendix

TABLE XXXIII : Estimated activation rounds for the bank-size ablation under IID partition with

40\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	6.7 $\pm$ 0.6	6.7 $\pm$ 0.6	6.7 $\pm$ 0.6
75%	13.3 $\pm$ 0.6	13.3 $\pm$ 0.6	13.3 $\pm$ 0.6

Appendix

TABLE XXXIV : Estimated activation rounds for the bank-size ablation under Non-IID partition with

0\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0
75%	13.0 $\pm$ 0.0	13.0 $\pm$ 0.0	13.0 $\pm$ 0.0

Appendix

TABLE XXXV : Estimated activation rounds for the bank-size ablation under Non-IID partition with

10\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0
75%	13.3 $\pm$ 0.6	13.3 $\pm$ 0.6	13.3 $\pm$ 0.6

Appendix

TABLE XXXVI : Estimated activation rounds for the bank-size ablation under Non-IID partition with

20\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	7.3 $\pm$ 0.6	7.3 $\pm$ 0.6	7.3 $\pm$ 0.6
75%	14.0 $\pm$ 1.0	14.0 $\pm$ 1.0	14.0 $\pm$ 1.0

Appendix

TABLE XXXVII : Estimated activation rounds for the bank-size ablation under Non-IID partition with

30\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
25%	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0	3.0 $\pm$ 0.0
50%	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0	7.0 $\pm$ 0.0
75%	14.0 $\pm$ 1.0	14.0 $\pm$ 1.0	14.0 $\pm$ 1.0

Appendix

TABLE XXXVIII : Estimated activation rounds for the bank-size ablation under Non-IID partition with

40\%

malicious clients. The activation round is the first communication round where the bank reaches

M_{\mathrm{act}}

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.50 $\pm$ 0.00	7.33 $\pm$ 0.01	7.72 $\pm$ 0.04
25%	7.48 $\pm$ 0.01	7.20 $\pm$ 0.03	7.37 $\pm$ 0.04
50%	7.44 $\pm$ 0.01	7.04 $\pm$ 0.04	6.87 $\pm$ 0.09
75%	7.38 $\pm$ 0.01	6.94 $\pm$ 0.02	6.42 $\pm$ 0.03

Appendix

TABLE XXXIX : Bank-size threshold ablation under IID partition with

0\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1802.40 $\pm$ 2.97	1519.44 $\pm$ 14.38	2258.01 $\pm$ 88.68
25%	1764.16 $\pm$ 11.60	1341.88 $\pm$ 42.96	1584.71 $\pm$ 60.01
50%	1695.11 $\pm$ 18.37	1143.14 $\pm$ 48.58	969.72 $\pm$ 83.50
75%	1601.83 $\pm$ 17.25	1028.47 $\pm$ 24.40	616.50 $\pm$ 19.83

Appendix

TABLE XL : Bank-size threshold ablation under IID partition with

0\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.01 $\pm$ 0.03	7.46 $\pm$ 0.02	9.95 $\pm$ 0.02
25%	6.96 $\pm$ 0.09	7.26 $\pm$ 0.38	9.67 $\pm$ 0.03
50%	6.90 $\pm$ 0.00	6.39 $\pm$ 0.16	8.88 $\pm$ 0.18
75%	6.90 $\pm$ 0.06	6.37 $\pm$ 0.12	7.49 $\pm$ 0.04

Appendix

TABLE XLI : Bank-size threshold ablation under IID partition with

0\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.50 $\pm$ 0.00	7.32 $\pm$ 0.01	7.72 $\pm$ 0.04
25%	7.48 $\pm$ 0.01	7.18 $\pm$ 0.05	7.37 $\pm$ 0.04
50%	7.44 $\pm$ 0.01	7.06 $\pm$ 0.04	6.84 $\pm$ 0.03
75%	7.39 $\pm$ 0.00	6.92 $\pm$ 0.02	6.44 $\pm$ 0.03

Appendix

TABLE XLII : Bank-size threshold ablation under IID partition with

10\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1808.57 $\pm$ 3.95	1512.51 $\pm$ 12.15	2256.49 $\pm$ 90.48
25%	1772.48 $\pm$ 13.23	1310.31 $\pm$ 71.29	1585.07 $\pm$ 62.27
50%	1701.03 $\pm$ 11.07	1163.34 $\pm$ 42.36	933.03 $\pm$ 30.89
75%	1615.82 $\pm$ 5.52	1011.33 $\pm$ 20.24	625.74 $\pm$ 17.90

Appendix

TABLE XLIII : Bank-size threshold ablation under IID partition with

10\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.01 $\pm$ 0.10	7.51 $\pm$ 0.04	9.95 $\pm$ 0.02
25%	6.98 $\pm$ 0.06	6.88 $\pm$ 0.12	9.67 $\pm$ 0.03
50%	6.92 $\pm$ 0.05	6.45 $\pm$ 0.01	8.79 $\pm$ 0.06
75%	6.91 $\pm$ 0.06	6.33 $\pm$ 0.03	7.54 $\pm$ 0.04

Appendix

TABLE XLIV : Bank-size threshold ablation under IID partition with

10\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.50 $\pm$ 0.01	7.32 $\pm$ 0.01	7.72 $\pm$ 0.04
25%	7.48 $\pm$ 0.01	7.19 $\pm$ 0.05	7.37 $\pm$ 0.04
50%	7.43 $\pm$ 0.01	7.05 $\pm$ 0.01	6.84 $\pm$ 0.03
75%	7.39 $\pm$ 0.01	6.93 $\pm$ 0.04	6.43 $\pm$ 0.02

Appendix

TABLE XLV : Bank-size threshold ablation under IID partition with

20\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1810.07 $\pm$ 10.64	1505.92 $\pm$ 11.61	2256.98 $\pm$ 91.12
25%	1774.75 $\pm$ 21.70	1328.38 $\pm$ 66.28	1585.63 $\pm$ 62.41
50%	1693.93 $\pm$ 23.24	1157.90 $\pm$ 17.29	933.10 $\pm$ 31.26
75%	1612.54 $\pm$ 12.60	1027.72 $\pm$ 37.62	617.43 $\pm$ 15.49

Appendix

TABLE XLVI : Bank-size threshold ablation under IID partition with

20\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.02 $\pm$ 0.07	7.52 $\pm$ 0.06	9.95 $\pm$ 0.02
25%	6.92 $\pm$ 0.03	6.91 $\pm$ 0.13	9.67 $\pm$ 0.03
50%	6.95 $\pm$ 0.04	6.48 $\pm$ 0.11	8.79 $\pm$ 0.06
75%	6.88 $\pm$ 0.07	6.29 $\pm$ 0.05	7.50 $\pm$ 0.07

Appendix

TABLE XLVII : Bank-size threshold ablation under IID partition with

20\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.50 $\pm$ 0.01	7.32 $\pm$ 0.07	7.72 $\pm$ 0.04
25%	7.48 $\pm$ 0.01	7.18 $\pm$ 0.04	7.37 $\pm$ 0.04
50%	7.44 $\pm$ 0.01	7.06 $\pm$ 0.05	6.81 $\pm$ 0.05
75%	7.39 $\pm$ 0.01	6.92 $\pm$ 0.03	6.40 $\pm$ 0.03

Appendix

TABLE XLVIII : Bank-size threshold ablation under IID partition with

30\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1813.92 $\pm$ 12.17	1508.04 $\pm$ 110.34	2257.69 $\pm$ 91.15
25%	1778.81 $\pm$ 10.18	1312.65 $\pm$ 46.41	1585.24 $\pm$ 62.65
50%	1694.88 $\pm$ 16.38	1169.29 $\pm$ 61.52	903.36 $\pm$ 43.16
75%	1619.16 $\pm$ 11.42	1011.10 $\pm$ 35.22	600.06 $\pm$ 15.82

Appendix

TABLE XLIX : Bank-size threshold ablation under IID partition with

30\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.03 $\pm$ 0.08	7.51 $\pm$ 0.07	9.95 $\pm$ 0.02
25%	6.96 $\pm$ 0.02	6.94 $\pm$ 0.18	9.67 $\pm$ 0.03
50%	6.96 $\pm$ 0.04	6.43 $\pm$ 0.04	8.71 $\pm$ 0.12
75%	6.88 $\pm$ 0.07	6.40 $\pm$ 0.07	7.42 $\pm$ 0.08

Appendix

TABLE L : Bank-size threshold ablation under IID partition with

30\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.51 $\pm$ 0.00	7.33 $\pm$ 0.07	7.79 $\pm$ 0.07
25%	7.49 $\pm$ 0.01	7.24 $\pm$ 0.04	7.42 $\pm$ 0.05
50%	7.45 $\pm$ 0.00	7.07 $\pm$ 0.05	6.87 $\pm$ 0.02
75%	7.39 $\pm$ 0.01	6.95 $\pm$ 0.02	6.41 $\pm$ 0.04

Appendix

TABLE LI : Bank-size threshold ablation under IID partition with

40\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1820.21 $\pm$ 4.44	1532.76 $\pm$ 114.78	2408.15 $\pm$ 162.25
25%	1782.72 $\pm$ 10.82	1388.21 $\pm$ 62.55	1674.97 $\pm$ 86.91
50%	1718.82 $\pm$ 4.65	1171.92 $\pm$ 53.02	964.01 $\pm$ 22.51
75%	1626.87 $\pm$ 9.50	1043.96 $\pm$ 22.20	609.87 $\pm$ 24.44

Appendix

TABLE LII : Bank-size threshold ablation under IID partition with

40\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.11 $\pm$ 0.06	7.92 $\pm$ 0.48	9.99 $\pm$ 0.04
25%	7.02 $\pm$ 0.04	7.13 $\pm$ 0.22	9.73 $\pm$ 0.06
50%	6.97 $\pm$ 0.07	6.53 $\pm$ 0.06	8.88 $\pm$ 0.09
75%	6.93 $\pm$ 0.03	6.32 $\pm$ 0.10	7.46 $\pm$ 0.14

Appendix

TABLE LIII : Bank-size threshold ablation under IID partition with

40\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.58 $\pm$ 0.01	7.35 $\pm$ 0.03	7.73 $\pm$ 0.05
25%	7.55 $\pm$ 0.02	7.25 $\pm$ 0.01	7.37 $\pm$ 0.05
50%	7.51 $\pm$ 0.03	7.11 $\pm$ 0.06	6.88 $\pm$ 0.09
75%	7.44 $\pm$ 0.02	6.96 $\pm$ 0.04	6.43 $\pm$ 0.04

Appendix

TABLE LIV : Bank-size threshold ablation under Non-IID partition with

0\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1959.50 $\pm$ 22.90	1550.67 $\pm$ 47.69	2273.70 $\pm$ 102.86
25%	1896.90 $\pm$ 42.06	1403.02 $\pm$ 11.73	1596.15 $\pm$ 71.84
50%	1823.97 $\pm$ 46.01	1221.73 $\pm$ 76.35	976.65 $\pm$ 92.33
75%	1710.88 $\pm$ 32.52	1050.53 $\pm$ 37.04	622.54 $\pm$ 22.03

Appendix

TABLE LV : Bank-size threshold ablation under Non-IID partition with

0\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	6.86 $\pm$ 0.10	7.82 $\pm$ 0.38	9.95 $\pm$ 0.02
25%	6.82 $\pm$ 0.03	6.95 $\pm$ 0.18	9.68 $\pm$ 0.04
50%	6.72 $\pm$ 0.03	6.60 $\pm$ 0.15	8.88 $\pm$ 0.20
75%	6.71 $\pm$ 0.03	6.33 $\pm$ 0.07	7.48 $\pm$ 0.06

Appendix

TABLE LVI : Bank-size threshold ablation under Non-IID partition with

0\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.55 $\pm$ 0.02	7.33 $\pm$ 0.01	7.73 $\pm$ 0.05
25%	7.52 $\pm$ 0.02	7.25 $\pm$ 0.01	7.37 $\pm$ 0.05
50%	7.48 $\pm$ 0.03	7.10 $\pm$ 0.07	6.84 $\pm$ 0.04
75%	7.42 $\pm$ 0.02	6.95 $\pm$ 0.05	6.44 $\pm$ 0.03

Appendix

TABLE LVII : Bank-size threshold ablation under Non-IID partition with

10\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1899.89 $\pm$ 44.69	1531.88 $\pm$ 13.60	2265.83 $\pm$ 100.94
25%	1853.08 $\pm$ 43.58	1402.99 $\pm$ 7.86	1592.05 $\pm$ 71.15
50%	1768.44 $\pm$ 51.11	1213.39 $\pm$ 83.14	937.25 $\pm$ 35.56
75%	1672.14 $\pm$ 37.85	1047.93 $\pm$ 54.20	628.96 $\pm$ 18.82

Appendix

TABLE LVIII : Bank-size threshold ablation under Non-IID partition with

10\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	6.76 $\pm$ 0.01	7.63 $\pm$ 0.05	9.96 $\pm$ 0.02
25%	6.71 $\pm$ 0.01	6.97 $\pm$ 0.15	9.68 $\pm$ 0.04
50%	6.68 $\pm$ 0.05	6.61 $\pm$ 0.19	8.80 $\pm$ 0.08
75%	6.71 $\pm$ 0.02	6.42 $\pm$ 0.02	7.53 $\pm$ 0.05

Appendix

TABLE LIX : Bank-size threshold ablation under Non-IID partition with

10\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.55 $\pm$ 0.02	7.27 $\pm$ 0.12	7.73 $\pm$ 0.05
25%	7.52 $\pm$ 0.02	7.14 $\pm$ 0.17	7.38 $\pm$ 0.05
50%	7.47 $\pm$ 0.04	7.08 $\pm$ 0.16	6.85 $\pm$ 0.04
75%	7.42 $\pm$ 0.03	6.95 $\pm$ 0.11	6.43 $\pm$ 0.02

Appendix

TABLE LX : Bank-size threshold ablation under Non-IID partition with

20\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1892.83 $\pm$ 45.80	1439.58 $\pm$ 163.65	2284.63 $\pm$ 122.32
25%	1843.09 $\pm$ 43.59	1270.95 $\pm$ 202.01	1603.79 $\pm$ 84.10
50%	1759.53 $\pm$ 65.05	1199.79 $\pm$ 182.48	941.83 $\pm$ 41.03
75%	1673.23 $\pm$ 42.10	1048.68 $\pm$ 111.65	622.50 $\pm$ 15.12

Appendix

TABLE LXI : Bank-size threshold ablation under Non-IID partition with

20\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	6.78 $\pm$ 0.08	7.47 $\pm$ 0.31	9.96 $\pm$ 0.03
25%	6.84 $\pm$ 0.10	6.84 $\pm$ 0.45	9.69 $\pm$ 0.05
50%	6.75 $\pm$ 0.06	6.59 $\pm$ 0.16	8.81 $\pm$ 0.08
75%	6.74 $\pm$ 0.02	6.48 $\pm$ 0.10	7.50 $\pm$ 0.06

Appendix

TABLE LXII : Bank-size threshold ablation under Non-IID partition with

20\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.56 $\pm$ 0.02	7.30 $\pm$ 0.06	7.73 $\pm$ 0.05
25%	7.52 $\pm$ 0.04	7.20 $\pm$ 0.09	7.38 $\pm$ 0.05
50%	7.47 $\pm$ 0.03	7.09 $\pm$ 0.07	6.81 $\pm$ 0.06
75%	7.43 $\pm$ 0.02	6.97 $\pm$ 0.03	6.41 $\pm$ 0.03

Appendix

TABLE LXIII : Bank-size threshold ablation under Non-IID partition with

30\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1914.41 $\pm$ 35.81	1484.60 $\pm$ 89.71	2279.76 $\pm$ 116.94
25%	1849.76 $\pm$ 65.54	1339.75 $\pm$ 121.30	1600.19 $\pm$ 79.79
50%	1760.25 $\pm$ 53.04	1206.50 $\pm$ 80.07	911.93 $\pm$ 52.64
75%	1677.76 $\pm$ 37.72	1063.79 $\pm$ 26.84	605.22 $\pm$ 18.26

Appendix

TABLE LXIV : Bank-size threshold ablation under Non-IID partition with

30\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	6.74 $\pm$ 0.07	7.62 $\pm$ 0.11	9.96 $\pm$ 0.03
25%	6.79 $\pm$ 0.01	6.92 $\pm$ 0.14	9.68 $\pm$ 0.05
50%	6.75 $\pm$ 0.05	6.53 $\pm$ 0.10	8.72 $\pm$ 0.15
75%	6.78 $\pm$ 0.05	6.40 $\pm$ 0.04	7.43 $\pm$ 0.08

Appendix

TABLE LXV : Bank-size threshold ablation under Non-IID partition with

30\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	7.56 $\pm$ 0.03	7.32 $\pm$ 0.09	8.33 $\pm$ 0.82
25%	7.53 $\pm$ 0.03	7.21 $\pm$ 0.11	7.96 $\pm$ 0.82
50%	7.49 $\pm$ 0.04	7.13 $\pm$ 0.08	7.28 $\pm$ 0.61
75%	7.44 $\pm$ 0.03	6.98 $\pm$ 0.03	6.60 $\pm$ 0.31

Appendix

TABLE LXVI : Bank-size threshold ablation under Non-IID partition with

40\%

malicious clients. Metric: Test Loss. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	1920.35 $\pm$ 56.71	1514.80 $\pm$ 139.13	5266.58 $\pm$ 4615.45
25%	1860.94 $\pm$ 60.73	1355.84 $\pm$ 138.76	3651.86 $\pm$ 3205.90
50%	1783.26 $\pm$ 65.14	1248.37 $\pm$ 101.92	1649.52 $\pm$ 1092.41
75%	1698.50 $\pm$ 44.56	1078.90 $\pm$ 33.23	759.15 $\pm$ 249.94

Appendix

TABLE LXVII : Bank-size threshold ablation under Non-IID partition with

40\%

malicious clients. Metric: Test PPL. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

$M_{\mathrm{act}}/K$	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
0%	6.80 $\pm$ 0.12	7.85 $\pm$ 0.34	10.12 $\pm$ 0.17
25%	6.81 $\pm$ 0.10	6.90 $\pm$ 0.19	9.96 $\pm$ 0.31
50%	6.73 $\pm$ 0.04	6.62 $\pm$ 0.15	9.36 $\pm$ 0.66
75%	6.76 $\pm$ 0.06	6.29 $\pm$ 0.18	8.00 $\pm$ 0.97

Appendix

TABLE LXVIII : Bank-size threshold ablation under Non-IID partition with

40\%

malicious clients. Metric: Semantic Entropy. For each seed and threshold, we select the checkpoint with the lowest test loss within the first 10 communication rounds after the bank reaches

M_{\mathrm{act}}

. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.13 $\pm$ 0.00	6.50 $\pm$ 0.05	5.36 $\pm$ 0.03
Forced 2-GMM	7.13 $\pm$ 0.00	6.50 $\pm$ 0.05	5.36 $\pm$ 0.03

Appendix

TABLE LXIX : GMM-selection ablation under IID partition with

0\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1246.57 $\pm$ 2.52	663.68 $\pm$ 29.76	213.42 $\pm$ 6.57
Forced 2-GMM	1246.57 $\pm$ 2.52	663.68 $\pm$ 29.76	213.42 $\pm$ 6.57

Appendix

TABLE LXX : GMM-selection ablation under IID partition with

0\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.73 $\pm$ 0.03	6.28 $\pm$ 0.09	5.52 $\pm$ 0.02
Forced 2-GMM	6.73 $\pm$ 0.03	6.28 $\pm$ 0.09	5.52 $\pm$ 0.02

Appendix

TABLE LXXI : GMM-selection ablation under IID partition with

0\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.13 $\pm$ 0.00	6.49 $\pm$ 0.02	5.37 $\pm$ 0.03
Forced 2-GMM	7.13 $\pm$ 0.00	6.49 $\pm$ 0.02	5.37 $\pm$ 0.03

Appendix

TABLE LXXII : GMM-selection ablation under IID partition with

10\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1247.28 $\pm$ 5.97	655.97 $\pm$ 11.66	213.93 $\pm$ 6.56
Forced 2-GMM	1247.28 $\pm$ 5.97	655.97 $\pm$ 11.66	213.93 $\pm$ 6.56

Appendix

TABLE LXXIII : GMM-selection ablation under IID partition with

10\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.81 $\pm$ 0.01	6.38 $\pm$ 0.16	5.52 $\pm$ 0.04
Forced 2-GMM	6.81 $\pm$ 0.01	6.38 $\pm$ 0.16	5.52 $\pm$ 0.04

Appendix

TABLE LXXIV : GMM-selection ablation under IID partition with

10\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.13 $\pm$ 0.00	6.49 $\pm$ 0.02	5.37 $\pm$ 0.03
Forced 2-GMM	7.13 $\pm$ 0.00	6.49 $\pm$ 0.02	5.37 $\pm$ 0.03

Appendix

TABLE LXXV : GMM-selection ablation under IID partition with

20\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1249.11 $\pm$ 1.78	661.38 $\pm$ 11.94	214.57 $\pm$ 6.75
Forced 2-GMM	1249.11 $\pm$ 1.78	661.38 $\pm$ 11.94	214.57 $\pm$ 6.75

Appendix

TABLE LXXVI : GMM-selection ablation under IID partition with

20\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.75 $\pm$ 0.02	6.19 $\pm$ 0.09	5.54 $\pm$ 0.03
Forced 2-GMM	6.75 $\pm$ 0.02	6.19 $\pm$ 0.09	5.54 $\pm$ 0.03

Appendix

TABLE LXXVII : GMM-selection ablation under IID partition with

20\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.14 $\pm$ 0.00	6.51 $\pm$ 0.01	5.37 $\pm$ 0.03
Forced 2-GMM	7.14 $\pm$ 0.00	6.51 $\pm$ 0.01	5.37 $\pm$ 0.03

Appendix

TABLE LXXVIII : GMM-selection ablation under IID partition with

30\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1258.21 $\pm$ 4.70	669.37 $\pm$ 5.86	215.61 $\pm$ 6.70
Forced 2-GMM	1258.21 $\pm$ 4.70	669.37 $\pm$ 5.86	215.61 $\pm$ 6.70

Appendix

TABLE LXXIX : GMM-selection ablation under IID partition with

30\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.74 $\pm$ 0.04	6.24 $\pm$ 0.10	5.52 $\pm$ 0.04
Forced 2-GMM	6.74 $\pm$ 0.04	6.24 $\pm$ 0.10	5.52 $\pm$ 0.04

Appendix

TABLE LXXX : GMM-selection ablation under IID partition with

30\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.16 $\pm$ 0.00	6.55 $\pm$ 0.02	5.38 $\pm$ 0.03
Forced 2-GMM	7.16 $\pm$ 0.00	6.55 $\pm$ 0.02	5.38 $\pm$ 0.03

Appendix

TABLE LXXXI : GMM-selection ablation under IID partition with

40\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1281.63 $\pm$ 1.74	697.62 $\pm$ 13.62	217.14 $\pm$ 6.85
Forced 2-GMM	1281.62 $\pm$ 1.73	697.62 $\pm$ 13.62	217.14 $\pm$ 6.85

Appendix

TABLE LXXXII : GMM-selection ablation under IID partition with

40\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.80 $\pm$ 0.06	6.24 $\pm$ 0.06	5.52 $\pm$ 0.02
Forced 2-GMM	6.80 $\pm$ 0.06	6.24 $\pm$ 0.06	5.52 $\pm$ 0.02

Appendix

TABLE LXXXIII : GMM-selection ablation under IID partition with

40\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.11 $\pm$ 0.01	6.47 $\pm$ 0.05	5.37 $\pm$ 0.03
Forced 2-GMM	7.11 $\pm$ 0.01	6.47 $\pm$ 0.05	5.37 $\pm$ 0.03

Appendix

TABLE LXXXIV : GMM-selection ablation under Non-IID partition with

0\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1225.90 $\pm$ 12.06	643.56 $\pm$ 33.46	215.55 $\pm$ 6.85
Forced 2-GMM	1225.89 $\pm$ 12.06	643.56 $\pm$ 33.46	215.55 $\pm$ 6.85

Appendix

TABLE LXXXV : GMM-selection ablation under Non-IID partition with

0\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.80 $\pm$ 0.09	6.12 $\pm$ 0.05	5.54 $\pm$ 0.03
Forced 2-GMM	6.80 $\pm$ 0.09	6.12 $\pm$ 0.05	5.54 $\pm$ 0.03

Appendix

TABLE LXXXVI : GMM-selection ablation under Non-IID partition with

0\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.12 $\pm$ 0.00	6.36 $\pm$ 0.06	5.38 $\pm$ 0.03
Forced 2-GMM	7.12 $\pm$ 0.00	6.36 $\pm$ 0.06	5.38 $\pm$ 0.03

Appendix

TABLE LXXXVII : GMM-selection ablation under Non-IID partition with

10\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1238.52 $\pm$ 4.97	576.63 $\pm$ 36.58	216.13 $\pm$ 6.70
Forced 2-GMM	1238.52 $\pm$ 4.97	576.63 $\pm$ 36.58	216.13 $\pm$ 6.70

Appendix

TABLE LXXXVIII : GMM-selection ablation under Non-IID partition with

10\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.79 $\pm$ 0.05	6.06 $\pm$ 0.18	5.54 $\pm$ 0.03
Forced 2-GMM	6.79 $\pm$ 0.05	6.06 $\pm$ 0.18	5.54 $\pm$ 0.03

Appendix

TABLE LXXXIX : GMM-selection ablation under Non-IID partition with

10\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.13 $\pm$ 0.01	6.42 $\pm$ 0.14	5.38 $\pm$ 0.03
Forced 2-GMM	7.13 $\pm$ 0.01	6.42 $\pm$ 0.14	5.38 $\pm$ 0.03

Appendix

TABLE XC : GMM-selection ablation under Non-IID partition with

20\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1251.65 $\pm$ 11.37	618.30 $\pm$ 91.12	216.97 $\pm$ 6.91
Forced 2-GMM	1251.65 $\pm$ 11.37	618.30 $\pm$ 91.12	216.97 $\pm$ 6.91

Appendix

TABLE XCI : GMM-selection ablation under Non-IID partition with

20\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.04 $\pm$ 0.40	6.33 $\pm$ 0.09	5.53 $\pm$ 0.03
Forced 2-GMM	7.04 $\pm$ 0.40	6.33 $\pm$ 0.09	5.53 $\pm$ 0.03

Appendix

TABLE XCII : GMM-selection ablation under Non-IID partition with

20\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.15 $\pm$ 0.02	6.38 $\pm$ 0.18	5.38 $\pm$ 0.03
Forced 2-GMM	7.15 $\pm$ 0.02	6.38 $\pm$ 0.18	5.38 $\pm$ 0.03

Appendix

TABLE XCIII : GMM-selection ablation under Non-IID partition with

30\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1277.28 $\pm$ 23.52	594.61 $\pm$ 108.75	218.02 $\pm$ 7.07
Forced 2-GMM	1277.81 $\pm$ 22.93	594.63 $\pm$ 108.74	218.02 $\pm$ 7.07

Appendix

TABLE XCIV : GMM-selection ablation under Non-IID partition with

30\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.90 $\pm$ 0.02	6.00 $\pm$ 0.38	5.53 $\pm$ 0.02
Forced 2-GMM	6.87 $\pm$ 0.06	6.00 $\pm$ 0.38	5.53 $\pm$ 0.02

Appendix

TABLE XCV : GMM-selection ablation under Non-IID partition with

30\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	7.16 $\pm$ 0.01	6.48 $\pm$ 0.13	5.39 $\pm$ 0.03
Forced 2-GMM	7.16 $\pm$ 0.01	6.48 $\pm$ 0.13	5.39 $\pm$ 0.03

Appendix

TABLE XCVI : GMM-selection ablation under Non-IID partition with

40\%

malicious clients. Metric: Test Loss. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	1286.50 $\pm$ 15.34	658.21 $\pm$ 91.66	220.18 $\pm$ 6.45
Forced 2-GMM	1286.50 $\pm$ 15.34	658.21 $\pm$ 91.66	220.18 $\pm$ 6.45

Appendix

TABLE XCVII : GMM-selection ablation under Non-IID partition with

40\%

malicious clients. Metric: Test PPL. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

GMM rule	GPT / WikiText	BERT / Shakespeare	LLaMA / StackOverflow
BIC 1/2-GMM	6.92 $\pm$ 0.24	6.19 $\pm$ 0.13	5.52 $\pm$ 0.03
Forced 2-GMM	6.92 $\pm$ 0.24	6.19 $\pm$ 0.13	5.52 $\pm$ 0.03

Appendix

TABLE XCVIII : GMM-selection ablation under Non-IID partition with

40\%

malicious clients. Metric: Semantic Entropy. Values are mean

\pm

standard deviation over three seeds. For each seed, all metrics are taken from the checkpoint with the lowest test loss. The best case is highlighted in bold.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 951 papers

May 8, 2026

Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs

Federated fine-tuning (FFT) has emerged as a privacy-preserving paradigm for collaboratively adapting large language models (LLMs). Built upon federated learning, FFT enables distributed agents to jointly refine a shared pretrained LLM by aggregating local LLM updates without sharing local raw data. However, FFT-based LLMs remain vulnerable to model manipulation threats, in which adversarial participants upload manipulated LLM updates that corrupt the aggregation process and degrade the performance of the global LLM. In this paper, we propose an Augmented Model maniPulation (AugMP) strategy against FFT-based LLMs. Specifically, we design a novel graph representation learning framework that captures feature correlations among benign LLM updates to guide the generation of malicious updates. To enhance manipulation effectiveness and stealthiness, we develop an iterative manipulation algorithm based on an augmented Lagrangian dual formulation. Through this formulation, malicious updates are optimized to embed adversarial objectives while preserving benign-like parameter characteristics. Experimental results across multiple LLM backbones demonstrate that the AugMP strategy achieves the strongest manipulation performance among all competing baselines, reducing the global LLM accuracy by up to

26\%

and degrading the average accuracy of local LLM agents by up to

22\%

. Meanwhile, AugMP maintains high statistical and geometric consistency with benign updates, enabling it to evade conventional distance- and similarity-based defense methods.

Hanlin Cai, Kai Li, Houtianfu Wang +4

Centre for neXt Communications (CXC), Department of Engineering, University of Cambridge, CB3 0FA Cambridge, U.K. · Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg · Telecommunication Networks group (TKN) at the School of Electrical Engineering and Computer Science, TU Berlin, Germany +1

LLM Fine-Tuning LLM Security Federated Fine-Tuning Federated Learning

Ask PDF

May 7, 2026

FedAttr: Towards Privacy-preserving Client-Level Attribution in Federated LLM Fine-tuning

Watermark radioactivity testing type of methods can detect whether a model was trained on watermarked documents, and have become key tools for protecting data ownership in the fine-tuning of large language models (LLMs). Existing works have proved their effectiveness in centralized LLM fine-tuning. However, this type of method faces several challenges and remains underexplored in federated learning (FL), a widely-applied paradigm for fine-tuning LLMs collaboratively on private data across different users. FL mainly ensures privacy through secure aggregation (SA), which allows the server to aggregate updates while keeping clients' updates private. This mechanism preserves privacy but makes it difficult to identify which client trained on watermarked documents. In this work, we propose FedAttr, a new client-level attribution protocol for FL. FedAttr identifies which clients trained on watermarked data via a paired-subset-difference mechanism, while preserving the privacy guarantees of SA and FL performance. FedAttr proceeds in three steps: (i) estimate each client's update by differencing two SA queries, (ii) score the estimate with the watermark detector via differential scoring, and (iii) combine scores across rounds via Stouffer method. We theoretically show that FedAttr produces an unbiased estimator of each client's update with bounded mutual information leakage (i.e.,

O(d^*/N)

per-round update). Moreover, FedAttr empirically achieves 100% TPR and 0% FPR, outperforming all baselines by at least 44.4% in TPR or 19.1% in FPR, with only 6.3% overhead relative to FL training time. Ablation studies confirm that FedAttr is robust to protocol parameters and configurations.

Su Zhang, Junfeng Guo, Heng Huang

Department of Computer Science, University of Maryland, College Park

Privacy-Preserving Language Models LLM Fine-Tuning Federated Fine-Tuning Training Data Attribution Federated Learning

Ask PDF

May 4, 2026

Adversarial Update-Based Federated Unlearning for Poisoned Model Recovery

Federated learning (FL) is vulnerable to poisoning attacks, where malicious clients upload manipulated updates to degrade the performance of the global model. Although detection methods can identify and remove malicious clients, the model remains affected. Retraining from scratch is effective but costly, and existing unlearning methods remain unsatisfactory in both effectiveness and efficiency. We propose Federated Adversarial Unlearning (FAUN), a lightweight framework that retains only a short window of malicious clients' updates and employs adversarial optimization on a proxy dataset to derive updates that eliminate malicious directions. Applying these updates for a few unlearning rounds, followed by benign fine-tuning, enables fast removal of malicious effects and stable recovery. Experiments on three canonical datasets show that FAUN achieves recovery comparable to retraining while requiring far fewer rounds and reduces attack success rates to near zero, confirming FAUN successfully eliminates the contributions of unlearned clients.

Wenwei Zhao, Xiaowen Li, Yao Liu +1

Department of Electrical Engineering, University of South Florida,Tampa, FL, USA · Bellini College, University of South Florida, Tampa, FL, USA

Federated Unlearning Machine Unlearning Federated Learning

Ask PDF

Abstract

Federated Learning

Explore similar work

Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs

FedAttr: Towards Privacy-preserving Client-Level Attribution in Federated LLM Fine-tuning

Adversarial Update-Based Federated Unlearning for Poisoned Model Recovery