HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learning

Zihao Peng; Nan Zou; Jiandian Zeng; Guo Li; Ke Chen; Boyuan Li; Tian Wang

cs.CVMar 3, 2026

HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learning

Authors: Zihao Peng, Nan Zou, Jiandian Zeng, Guo Li, Ke Chen, Boyuan Li, Tian Wang

Organizations: Beijing Normal University · Zhengzhou University

Abstract

Vision Transformers (ViTs) have been widely adopted in vision tasks due to their strong transferability. In Federated Learning (FL), where full fine-tuning is communication heavy, Low-Rank Adaptation (LoRA) provides an efficient and communication-friendly way to adapt ViTs. However, existing LoRA-based federated tuning methods overlook latent client structures in real-world settings, limiting shared representation learning and hindering effective adaptation to unseen clients. To address this, we propose HiLoRA, a hierarchical LoRA framework that places adapters at three levels: root, cluster, and leaf, each designed to capture global, subgroup, and client-specific knowledge, respectively. Through cross-tier orthogonality and cascaded optimization, HiLoRA separates update subspaces and aligns each tier with its residual personalized objective. In particular, we develop a LoRA-Subspace Adaptive Clustering mechanism that infers latent client groups via subspace similarity analysis, thereby facilitating knowledge sharing across structurally aligned clients. Theoretically, we establish a tier-wise generalization analysis that supports HiLoRA's design. Experiments on ViT backbones with CIFAR-100 and DomainNet demonstrate consistent improvements in both personalization and generalization.

Figures & tables

Figure 1: Challenges of Dual-LoRA . Six clients own non-IID data that form three latent clusters (vehicles, insects, fruits). Global LoRA forces a “one-size-fits-all” adapter, causing (a) gradient drift and (b) loss of cluster-level cues. Fully personalized LoRA removes sharing but (c) overfits scarce local data. These limitations motivate a hierarchical LoRA design.

Figure 2: HiLoRA overview. Top-right: cascaded tier-wise optimization with progressive freezing. (A) Train the Root-LoRA as the global adapter; (B) Perform LoRA-Subspace Adaptive Clustering to identify client communities. (C) Update the Cluster-LoRA with an orthogonality constraint to the frozen root; (D) Adapt the Leaf-LoRA orthogonal to both root and cluster tiers. Each client $i$ with cluster index $j=j(i)$ updates its effective LoRA as $\Delta\mathbf{W}_{i}=\mathbf{B}_{r}\mathbf{A}_{r}+\mathbf{B}_{c,j}\mathbf{A}_{c,j}+\mathbf{B}_{\ell,i}\mathbf{A}_{\ell,i}$ .

Method	(i) Personalization Accuracy (↑)						(ii) Unseen-Client Adaptation (↑)
	GL–Dir(0.3)		SC–Dir(3)		Patho(10)		GL–Dir(0.3)	SC–Dir(0.3)	Patho(10)
	Mean Acc.	Worst Acc.	Mean Acc.	Worst Acc.	Mean Acc.	Worst Acc.	Test Acc.	Test Acc.	Test Acc.
Local-LoRA	$0.691\scriptstyle\pm 0.06$	$0.579\scriptstyle\pm 0.03$	$0.867\scriptstyle\pm 0.11$	$0.601\scriptstyle\pm 0.10$	$0.844\scriptstyle\pm 0.12$	$0.577\scriptstyle\pm 0.09$	$0.521\scriptstyle\pm 0.09$	$0.639\scriptstyle\pm 0.17$	$0.756\scriptstyle\pm 0.09$
FedIT	$0.772\scriptstyle\pm 0.05$	$0.711\scriptstyle\pm 0.04$	$0.578\scriptstyle\pm 0.16$	$0.315\scriptstyle\pm 0.03$	$0.869\scriptstyle\pm 0.07$	$0.766\scriptstyle\pm 0.07$	$0.753\scriptstyle\pm 0.04$	$0.687\scriptstyle\pm 0.16$	$0.848\scriptstyle\pm 0.17$
FlexLoRA	$\underline{0.818\scriptstyle\pm 0.06}$	$0.695\scriptstyle\pm 0.04$	$0.718\scriptstyle\pm 0.20$	$0.301\scriptstyle\pm 0.15$	$0.862\scriptstyle\pm 0.06$	$0.775\scriptstyle\pm 0.06$	$0.808\scriptstyle\pm 0.05$	$0.719\scriptstyle\pm 0.17$	$0.834\scriptstyle\pm 0.19$
FedSA-LoRA	$0.786\scriptstyle\pm 0.05$	$0.693\scriptstyle\pm 0.02$	$0.734\scriptstyle\pm 0.12$	$0.462\scriptstyle\pm 0.08$	$0.860\scriptstyle\pm 0.09$	$0.638\scriptstyle\pm 0.11$	$0.813\scriptstyle\pm 0.06$	$0.765\scriptstyle\pm 0.10$	$0.846\scriptstyle\pm 0.09$

Table 1: Performance analysis on CIFAR-100. We evaluate across statistical heterogeneity patterns: (i) Client-level Personalization : reporting the mean (Avg) and worst-case (10%) accuracies

\pm

std across all clients. (ii) Unseen-Client Adaptation : accuracy on unseen clients’ test distributions. Best results are in bold ; second-best are underlined .

Method	(i) Client-level (↑)		(ii) Domain-wise Accuracy (↑)						(iii) Unseen (↑)
Method	Mean Acc.	Worst Acc.	C	I	P	Q	R	S	Test Acc.
Local-LoRA	$0.755\scriptstyle\pm 0.17$	$0.374\scriptstyle\pm 0.11$	$0.716\scriptstyle\pm 0.07$	$0.428\scriptstyle\pm 0.12$	$0.850\scriptstyle\pm 0.06$	$0.748\scriptstyle\pm 0.06$	$0.954\scriptstyle\pm 0.02$	$0.794\scriptstyle\pm 0.05$	$0.665\scriptstyle\pm 0.21$
FedIT	$0.822\scriptstyle\pm 0.15$	$0.494\scriptstyle\pm 0.05$	$0.886\scriptstyle\pm 0.09$	$0.539\scriptstyle\pm 0.08$	$0.898\scriptstyle\pm 0.06$	$0.762\scriptstyle\pm 0.07$	$0.966\scriptstyle\pm 0.02$	$0.854\scriptstyle\pm 0.03$	$0.813\scriptstyle\pm 0.14$
FlexLoRA	$0.839\scriptstyle\pm 0.14$	$0.526\scriptstyle\pm 0.07$	$\underline{0.887\scriptstyle\pm 0.05}$	$0.567\scriptstyle\pm 0.09$	0.926±0.06	$0.779\scriptstyle\pm 0.05$	$0.975\scriptstyle\pm 0.02$	0.874±0.04	$\underline{0.840\scriptstyle\pm 0.11}$
FedSA-LoRA	$0.814\scriptstyle\pm 0.13$	$0.532\scriptstyle\pm 0.05$	$0.850\scriptstyle\pm 0.07$	$0.568\scriptstyle\pm 0.07$	$0.867\scriptstyle\pm 0.06$	$0.810\scriptstyle\pm 0.07$	$0.971\scriptstyle\pm 0.02$	$0.792\scriptstyle\pm 0.06$	$0.797\scriptstyle\pm 0.17$
FDLoRA	$0.822\scriptstyle\pm 0.13$	$0.535\scriptstyle\pm 0.05$	$0.860\scriptstyle\pm 0.07$	$0.562\scriptstyle\pm 0.07$	$0.882\scriptstyle\pm 0.05$	$0.822\scriptstyle\pm 0.06$	$0.974\scriptstyle\pm 0.02$	$0.800\scriptstyle\pm 0.05$	$0.804\scriptstyle\pm 0.17$

Table 2: Comparative analysis on DomainNet. We evaluate: (i) Client-level Personalization , reporting the mean (Avg) and worst-case (10%) accuracies

\pm

std across all clients; (ii) Domain-wise accuracy , averaging client accuracies within each domain (C/I/P/Q/R/S); (iii) Unseen-Client Adaptation , accuracy on test distributions of unseen clients. Best results are in bold ; second-best are underlined .

Panel of Figure 3 — Figure 3: t-SNE of clustering results on CIFAR-100. Clients are visualized by Jensen–Shannon distances of their label distributions: (a) Patho-10, $K^{\star}{=}10$ ; (b) SC–Dir( $\alpha{=}3$ ), $K^{\star}{=}20$ .

Panel of Figure 4 — Figure 4: Principal-angle distributions between tiers in HiLoRA. Computed from the column spaces of the LoRA- $\mathbf{B}$ matrices. Lower $\cos^{2}\theta$ indicates stronger subspace orthogonality.

Stage	CIFAR-100		DomainNet
	Mean $\pm\textbf{std}$	Gains (R/C) %	Mean $\pm\textbf{std}$	Gains (R/C) %
Root	0.663 $\pm$ 0.18	—-/—-	0.815 $\pm$ 0.15	—-/—-
+ Cluster	0.889 $\pm$ 0.10	+22.6% / —-	0.864 $\pm$ 0.13	+4.9% / —-
+ Leaf	0.934 $\pm$ 0.06	+27.1% / +4.5%	0.877 $\pm$ 0.11	+6.2% / +1.3%
Second-best	0.912 $\pm$ 0.06		0.860 $\pm$ 0.12

Table 3: HiLoRA multi-level gain comparison across Root, Cluster, and Leaf stages on CIFAR-100 and DomainNet, reporting Mean±Std and accuracy gains relative to Root and Cluster.

Panel of Figure 5 — Figure 5: Unseen-client accuracy ( mean ± std ) across adaptation epochs on (a) CIFAR-100 and (b) DomainNet . The starred point indicates the Root+Cluster initialization, while subsequent epochs correspond to leaf-level LoRA adaptation.

Hierarchical LoRA	LoRA subspace clustering	Orthogonality loss	CIFAR-100		DomainNet
Hierarchical LoRA	LoRA subspace clustering	Orthogonality loss	Per. $\uparrow$	Gen. $\uparrow$	Per. $\uparrow$	Gen. $\uparrow$
✓			89.7	87.5	85.4	84.2
✓	✓		92.8	92.2	86.0	85.1
✓	✓	✓	94.1	94.0	87.7	86.1

Table 4: Ablation study of HiLoRA components. We report personalized accuracy (Per.) and unseen-client adaptation (Gen.) on CIFAR-100 and DomainNet.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 943 papers

Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models
Jun 4, 2026Sunny Gupta, Shambhavi Shanker, Amit SethiFederated Learning Aggregation Non-IID Federated Learning
Indian Institute of Technology, Bombay
FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning
Sep 29, 2026Mengjun Yi, Huaian Gu, Yinghao Ai +2Low-Rank Adaptation Federated Fine-Tuning
State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science +1
Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers
Jul 23, 2026Hariharan Ramesh, Jyotikrishna DassFederated Learning Aggregation Communication-Efficient Distributed Training
Department of Electrical and Computer Engineering University of Arizona, Tucson, AZ 85719

Jun 4, 2026J/K move · Enter open · S save

Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models

Sunny Gupta, Shambhavi Shanker, Amit Sethi

Indian Institute of Technology, Bombay

Federated Learning Aggregation Non-IID Federated Learning Low-Rank Adaptation Hypernetworks Federated Fine-Tuning Federated Learning

Open paper AskPDF arXiv

Federated fine-tuning of foundation models using Low-Rank Adaptation (LoRA) offers a communication efficient solution for distributed learning. However, existing federated LoRA methods suffer from two fundamental limitations: (1) structural aggregation bias, where independently averaging low rank factors fails to approximate the true combined update, and (2) client side initialization lag, as clients repeatedly reinitialize LoRA parameters across communication rounds, slowing convergence. We propose HyperLoRA, a unified framework that addresses both issues through amortized federated adaptation through hypernetwork-driven LoRA generation and product space aggregation. Instead of iterative per-client optimization, HyperLoRA employs a learned generator that maps client distribution signatures to LoRA initializations, effectively amortizing per client adaptation. On the server side, we introduce a learned aggregation module that directly synthesizes updates in the low-rank product space, eliminating the inconsistencies of factor-wise averaging. A lightweight residual correction module further improves stability under heterogenous (non-IID) client distributions.By replacing iterative optimization and heuristic averaging with learned operators, HyperLoRA jointly enables efficient personalization, unbiased aggregation, and faster convergence. Experiments on federated vision and vision-language benchmarks show that HyperLoRA achieves improved convergence speed, greater robustness to distribution shift, and stronger personalization performance compared to prior federated LoRA methods.

Abstract

Federated Learning

Explore similar work

Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers