ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation

Chuiyang Meng; Ming Tang; Vincent W. S. Wong

cs.LGFeb 19, 2026

ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation

Authors: Chuiyang Meng, Ming Tang, Vincent W. S. Wong

Organizations: Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada · Department of Computer Science and Engineering, Southern University of Science and Technology, China

Abstract

Federated fine-tuning of large language models (LLMs) enables collaborative tuning across distributed clients. However, due to the large size of LLMs, local updates in federated learning (FL) may incur substantial video random-access memory (VRAM) usage. Moreover, frequent model exchange may lead to significant communication overhead. To tackle these challenges, in this paper we propose ZorBA, a zeroth-order optimization-based federated fine-tuning framework with heterogeneous block activation. ZorBA leverages zeroth-order optimization to eliminate the storage of gradients at the clients by forward passes. ZorBA includes a heterogeneous block activation mechanism in which the central server allocates different subsets of transformer blocks to clients in order to accelerate the convergence rate and reduce the VRAM usage. Furthermore, ZorBA utilizes shared random seeds and the finite differences of gradients in order to reduce the communication overhead. We conduct theoretical analysis to characterize the effect of block activation decisions on the convergence rate and VRAM usage. To jointly enhance the convergence rate and reduce the VRAM usage, we formulate an optimization problem to optimize the block activation decisions. We propose an $ε$ -constraint lexicographic algorithm to solve this problem. Experimental results show that ZorBA outperforms three federated fine-tuning baselines in VRAM usage by up to 62.41% and incurs a low communication overhead.

Figures & tables

Figure 1: VRAM usage during zeroth-order optimization with OPT-125M. The forward-pass activations per block include tensors for hidden tensors, projections for Q, K, and V in transformer blocks, and feed-forward networks (FFNs).

Figure 2: Illustration of our proposed ZorBA framework.

Figure 3: An example of different block activation decisions on three clients. We consider that each client’s model has three blocks. Each column denotes the blocks for each client. The blue square denotes the activated block. The pink square denotes the frozen block. The value of the block in the $m$ -th row and $n$ -th column corresponds to the aggregation weight, i.e., $\frac{a_{m,n}}{\sum_{n^{\prime}\in\mathcal{N}}a_{m,n^{\prime}}}$ .

Model	Datasets	ZorBA	FedIT	FedZO	DeComFL
OPT-125M	AG-News	138	49	155	181
	SST-2	167	70	161	205
	SNLI	231	112	255	246
OPT-1.3B	AG-News	61	23	79	68
	SST-2	99	48	104	104
	SNLI	157	82	186	173

Table I : Comparison of the number of fine-tuning rounds required to achieve the target average testing accuracy.

Model	Datasets	Metrics	ZorBA	FedIT	FedZO	DeComFL
OPT-125M	AG-News	VRAM	47.31	118.67	95.37	95.37
	AG-News	Commun.	$1.24\times 10^{5}$	$6.13\times 10^{11}$	$1.94\times 10^{12}$	$1.36\times 10^{5}$
	SST-2	VRAM	59.32	118.67	95.37	95.37
	SST-2	Commun.	$1.67\times 10^{5}$	$8.75\times 10^{11}$	$2.01\times 10^{12}$	$1.54\times 10^{5}$
	SNLI	VRAM	131.39	190.74	167.44	167.44
	SNLI	Commun.	$2.43\times 10^{5}$	$1.40\times 10^{12}$	$3.19\times 10^{12}$	$1.85\times 10^{5}$

Table II : Comparison of the total VRAM usage of all clients per round (in GB) and total communication overhead (i.e., Commun.) of all clients to achieve the target accuracy.

Panel of Figure 4 — Figure 4: (a) Pareto front of $\Lambda$ versus the total VRAM usage ratio. (b) Comparison between SC1, SC2, and SC3 for the number of rounds to achieve the target average testing accuracy and the total VRAM usage.

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 115 papers

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization
Oct 1, 2026Hang Zou, Chao Zhang, Yuzhi Yang +3Federated Learning Aggregation Low-Rank Adaptation
6G Research Center, Khalifa University, 127788 Abu Dhabi, UAE · Department of Computer Science, Central South University, 410083, Changsha, China · Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France
CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-Tuning Federated Learning
City University of Hong Kong · The University of Hong Kong
FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment
Feb 19, 2026Chuiyang Meng, Ming Tang, Vincent W. S. WongLLM Fine-Tuning Federated Fine-Tuning
The University of British Columbia · Southern University of Science and Technology

Oct 1, 2026J/K move · Enter open · S save

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization

Hang Zou, Chao Zhang, Yuzhi Yang, Yu Tian+2

6G Research Center, Khalifa University, 127788 Abu Dhabi, UAE · Department of Computer Science, Central South University, 410083, Changsha, China · Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France

Federated Learning Aggregation Low-Rank Adaptation Federated Fine-Tuning Federated Learning

Open paper AskPDF arXiv

Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental "aggregation dilemma" between the accurate Sum-of-Products (SoP) and the communication-efficient Product-of-Sums (PoS) implementations. To tackle these challenges, we propose FedFit. First, to significantly reduce communication overhead, we introduce a disjoint shared vector-bank parameterization that reconstructs high-dimensional adapter matrices from two compact and disjoint global vector banks. Second, to address the aggregation dilemma, we devise an alternating optimization schedule. By cycling between decoupled single-bank updates (which allow for accurate aggregation) and joint updates corrected by a Residual Spectral Aggregation mechanism, we resolve the conflict between SoP and PoS. Additionally, we integrate blockwise quantization with client-side error feedback to further compress the transmitted vectors. Furthermore, we establish theoretical convergence guarantees for the proposed algorithm. Extensive experiments on Qwen2.5 models demonstrate that FedFit achieves perplexity performance comparable to standard federated LoRA methods, while providing compression ratios up to 100x higher.

Abstract

Federated Fine-Tuning

Explore similar work

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment