Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing

Kai Li; Hanlin Cai; Zihao Liu; Yichen Li; Jose Alcaraz Calero; Wei Ni; Falko Dressler; Özgür B. Akan

cs.LGOct 8, 2026

Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing

Authors: Kai Li, Hanlin Cai, Zihao Liu, Yichen Li, Jose Alcaraz Calero, Wei Ni, Falko Dressler, Özgür B. Akan

Organizations: School of Computer Science and Digital Technologies, Aston University, Birmingham, UK · Centre for neXt Communications (CXC), Department of Engineering, University of Cambridge, CB3 0FA Cambridge, U.K. · School of Engineering, Edith Cowan University, Perth, WA 6027 · School of Computer Science and Engineering, University of New South Wales, Sydney, NSW 2052, Australia ·

Abstract

Federated Large Language Models (FedLLMs) are promising to deliver privacy-preserving, collaborative intelligence. The decentralized training process is vulnerable to Byzantine attacks that corrupt semantic consistency and factual resilience through malicious model updates. This paper presents a new resilience strategy that prevents semantically corrupted knowledge from propagating during federated aggregation. A defense problem is formulated to jointly maximize global accuracy and suppress manipulated model updates, enabling trustworthy knowledge aggregation without accessing private local data. To strengthen the resilience of FedLLMs, we develop a novel Hypergraph Message Passing Graph Autoencoder (HMP-GAE) framework, where low-rank model updates are represented as nodes in a hypergraph and high-order semantic relationships among multiple agents are exploited through iterative message passing. By leveraging historical checkpoints and cross-agent relational reasoning, HMP-GAE identifies and suppresses semantically inconsistent updates while preserving knowledge alignment among agents throughout federated training. HMP-GAE is implemented using Qwen-2.5 and Llama-3.2 as pretrained FedLLM backbones and evaluated on AG News and Yahoo! Answers datasets. Compared to existing defenses, HMP-GAE improves global accuracy by $1.21$ %, reduces final-model classification semantic entropy by $44.08$ %, and lowers perplexity by $13.34$ %. HMP-GAE outperforms the GAE-based defense by improving the accuracy $1.39$ %, reducing the semantic entropy by $54.5$ %, and lowering perplexity by $35.5$ %.

Figures & tables

Fig. 1 : In FedLLMs, users train their local LLMs and produce local model updates.

Fig. 2 : The architecture of the proposed HMP-GAE framework, which is applied at the server. By modeling each agent’s update as a node in a hypergraph and capturing high-order semantic relationships via message passing, HMP is developed with GAE to identify and suppress semantically inconsistent or malicious contributions.

Dataset	Ref.	Method	Qwen-2.5 [ 42 ]				Llama-3.2 [ 43 ]
Dataset	Ref.	Method	Accuracy	CSE mean	CSE final	PPL	Accuracy	CSE mean	CSE final	PPL
AG News	[ 44 ]	Multi-Krum	85.68	0.2830	0.0870	83.15	89.16	0.1525	0.0539	69.16
	[ 45 ]	FoolsGold	88.00	0.1608	0.0397	78.81	89.63	0.1639	0.0290	60.04
	[ 46 ]	FLTrust	87.47	0.2859	0.1870	76.08	90.23	0.2379	0.1161	53.68
	Ours	HMP-GAE	88.95	0.1267	0.0395	69.49	91.44	0.1067	0.0258	46.52
Yahoo! Answers	[ 44 ]	Multi-Krum	54.13	1.0865	0.6247	1257.59	62.52	0.9452	0.4692	689.66

TABLE I: Global model performance of the proposed HMP-GAE, compared to Multi-Krum, FoolsGold, and FLTrust methods. Accuracy and CSE mean are averaged over 50 rounds, while CSE final and PPL are evaluated based on the final global model.

Fig. 3 : Global model CSE of four defenses under the Byzantine attack across 50 communication rounds.

Fig. 4 : Local CSE averaged over all benign agents under four defenses against the Byzantine attack across 50 communication rounds.

Dataset	Method	Qwen-2.5 [ 42 ]		Llama-3.2 [ 43 ]
Dataset	Method	Acc.	CSE	Acc.	CSE
AG News	Multi-Krum [ 44 ]	83.69	0.2052	87.10	0.1276
	FoolsGold [ 45 ]	84.76	0.1495	87.04	0.1256
	FLTrust [ 46 ]	83.74	0.2259	86.76	0.1704
	HMP-GAE (Ours)	85.00	0.1257	87.69	0.1085
Yahoo! Answers	Multi-Krum [ 44 ]	46.20	0.7477	52.36	0.6443

TABLE II : Local accuracy (Acc.) and local CSE averaged over all benign local agents under the Byzantine attack.

Fig. 5 : Per-user trust weight of four different defenses over 50 communication rounds under the Byzantine attack, with the Qwen-2.5 and Llama-3.2 backbones on AG News dataset .

Fig. 6 : Per-class PPL of the final global model under four different defenses against the Byzantine attack, with the Qwen-2.5 and Llama-3.2 backbones on AG News dataset and Yahoo! Answers dataset .

Model	Dataset	Method	Accuracy (%) $\uparrow$	CSE $\downarrow$	PPL $\downarrow$
Qwen-2.5	AG News	HMP-GAE	88.95	0.1267	69.49
		GAE-based defense (w/o HMP)	88.58	0.1746	76.21
		HMP-GAE w/o Hyperedge Reconstruction	88.71	0.1476	72.75
	Yahoo! Answers	HMP-GAE	61.12	0.6378	1061.74
		GAE-based defense (w/o HMP)	59.73	1.1034	1220.49
		HMP-GAE w/o Hyperedge Reconstruction	60.01	0.7449	1130.91

TABLE III : Ablation study of HMP-GAE under the Byzantine attack, comparing the full HMP-GAE framework with a GAE-based defense without HMP and with HMP-GAE without hyperedge reconstruction. Accuracy and CSE are averaged over 50 rounds, and PPL is evaluated on the final global model.

Byzantine-Robust Federated Learning

Byzantine-robust federated learning protects collaboratively trained models from clients that send corrupted, poisoned, or strategically manipulated updates.

See all 84 papers

FedGraph: Defending Federated Large Language Model Fine-Tuning Against Backdoor Attacks via Graph-Based Aggregation
Oct 13, 2026Xi Chen, Chunyi Zhou, Rui Zeng +3Backdoor Attacks in Federated Learning LLM Backdoor Attacks
Zhejiang University, Hangzhou, China
Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs
May 8, 2026Hanlin Cai, Kai Li, Houtianfu Wang +4LLM Fine-Tuning LLM Security
Centre for neXt Communications (CXC), Department of Engineering, University of Cambridge, CB3 0FA Cambridge, U.K. · Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg · Telecommunication Networks group (TKN) at the School of Electrical Engineering and Computer Science, TU Berlin, Germany +1
Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approach
Apr 23, 2026Guilin Deng, Silong Chen, Yuchuan Luo +6Membership Inference Attacks Privacy Leakage in Language Models
Colleague of Computer Science and Technology, National University of Defense Technology, Changsha, China · City University of Hong Kong, Hong Kong, China · Shenzhen University, Shenzhen, China

Oct 13, 2026J/K move · Enter open · S save

FedGraph: Defending Federated Large Language Model Fine-Tuning Against Backdoor Attacks via Graph-Based Aggregation

Xi Chen, Chunyi Zhou, Rui Zeng, Xiaogang Xu+2

Zhejiang University, Hangzhou, China

Backdoor Attacks in Federated Learning LLM Backdoor Attacks Federated Fine-Tuning Federated Learning Aggregation

Open paper AskPDF arXiv

Federated fine-tuning of large language models (LLMs) allows multiple clients to collaboratively adapt models without sharing raw data, offering strong privacy benefits. However, this setting is highly vulnerable to backdoor attacks, in which malicious clients manipulate their updates to induce targeted behaviors while maintaining normal performance on benign inputs. In this work, we make a key observation specific to federated LLM fine-tuning: malicious clients tend to exhibit strong consistency in their model updates, whereas benign clients produce more diverse and dispersed updates. This structural difference is not well captured by existing defense methods, which typically rely on direct anomaly detection in high-dimensional parameter space and therefore struggle in the presence of entangled LLM updates. Motivated by this observation, we propose FedGraph, a graph-based aggregation framework that explicitly models relationships among client updates. By representing clients as nodes in a dynamically constructed graph and extracting structural features that characterize their relative positions and interactions, FedGraph derives compact client representations that effectively separate malicious and benign behaviors. These representations are then used to identify and suppress malicious contributions during aggregation. Extensive experiments demonstrate that FedGraph significantly improves backdoor defense. It consistently reduces attack success rates to below 10% while achieving high detection accuracy (95.5% on average) and a low false positive rate (2.33%), substantially outperforming existing federated learning defenses.

Abstract

Byzantine-Robust Federated Learning

Explore similar work

FedGraph: Defending Federated Large Language Model Fine-Tuning Against Backdoor Attacks via Graph-Based Aggregation

Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs

Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approach