cs.AISep 11, 2026

FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks

Authors: Xinlu Zhang, Na Yan, Yang Su, Yansha Deng, Toktam Mahmoodi

Organizations: Department of Engineering, King’s College London, Strand, London WC2R 2LS, U.K.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) have demonstrated strong capabilities across a wide range of natural language processing tasks. However, conventional fine-tuning typically relies on centralized data collection, bringing in privacy concerns. Federated learning (FL) enables collaborative LLM fine-tuning without sharing raw client data, but its deployment over bandwidth-constrained wireless networks is hindered by the communication overhead of model-parameter transmission. Although Low-Rank Adaptation (LoRA) reduces the number of trainable parameters, its communication cost still increases with model scale. Knowledge distillation avoids parameter sharing via output logits, but token-level logits in LLMs incur high communication cost due to sequence length and vocabulary size. Reducing logits lowers the cost but weakens supervision and degrades accuracy. To address these limitations, we propose FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of LLMs over wireless networks, which communicates intermediate LoRA activations as the distillation signal rather than logits or full parameters. Since transmitting all blocks over the entire public dataset remains costly, we further propose a transformer block importance scoring framework that selectively transmits the most informative blocks, and two dataset selection strategies that discard public samples deviating from the local data distribution and prioritise those most informative for distillation. Extensive experiments across multiple generative language datasets, including WikiText-103, PTB, and Dialog, demonstrate that our proposed framework reduces communication overhead by 50-65% while achieving rapid convergence to competitive perplexity compared to baselines.

Figures & tables

Fig. 1: The workflow of the proposed FLoKD framework.
ParameterValueParameterValue
DatasetsWikiText-103, PTB, Dialog
BackboneGPT-2 (12 Transformer blocks)
EvaluationPerplexity (PPL), 3 runs averaged
LoRA rank ( rr )8Client pool size30
LoRA α\alpha32Participating clients ( NN )10
LoRA dropout0.1Comm. rounds50/100
TABLE I: Experimental Settings
Fig. 2: Performance of FLoKD and Baseline frameworks under Limited Bandwidth on WikiText-103
Fig. 3: Convergence Behavior of FLoKD Against Three Baseline Methods
Fig. 4: Communication Cost for Target PPL (lower is better)
Fig. 5: Performance Comparison of the Proposed Block Selection and Other frameworks
Fig. 6: Comparison of Data Selection Methods under Different Selection Ratios (with Mixed PTB & DailyDialog Dataset)
Fig. 7: Pareto Frontier: Bandwidth vs Perplexity (59 tests)
More in this topic

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

Explore similar work

CardsList
Oct 1, 2026

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization

Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental "aggregation dilemma" between the accurate Sum-of-Products (SoP) and the communication-efficient Product-of-Sums (PoS) implementations. To tackle these challenges, we propose FedFit. First, to significantly reduce communication overhead, we introduce a disjoint shared vector-bank parameterization that reconstructs high-dimensional adapter matrices from two compact and disjoint global vector banks. Second, to address the aggregation dilemma, we devise an alternating optimization schedule. By cycling between decoupled single-bank updates (which allow for accurate aggregation) and joint updates corrected by a Residual Spectral Aggregation mechanism, we resolve the conflict between SoP and PoS. Additionally, we integrate blockwise quantization with client-side error feedback to further compress the transmitted vectors. Furthermore, we establish theoretical convergence guarantees for the proposed algorithm. Extensive experiments on Qwen2.5 models demonstrate that FedFit achieves perplexity performance comparable to standard federated LoRA methods, while providing compression ratios up to 100x higher.

Hang Zou, Chao Zhang, Yuzhi Yang +3

6G Research Center, Khalifa University, 127788 Abu Dhabi, UAE · Department of Computer Science, Central South University, 410083, Changsha, China · Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France

Federated Learning AggregationLow-Rank AdaptationFederated Fine-TuningFederated Learning
AskPDF
Feb 24, 2026

Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA

Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge integration.However, directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) catastrophic knowledge forgetting during fine-tuning process, arising from conflicting update directions caused by data heterogeneity; (ii) inefficient communication and convergence during model aggregation process, due to bandwidth-intensive redundant model transmissions; and (iii) multi-task knowledge interference during inference process, resulting from incompatible knowledge representations coexistence during inference. To address these issues in a fully decentralized scenario, we first propose a sparse-and-orthogonal LoRA that ensures orthogonality between model updates to eliminate direction conflicts during fine-tuning.Then, we analyze how device connection topology affects multi-task performance, prompting a cluster-based topology design during aggregation.Finally, we propose an implicit mixture of experts (MoE) mechanism to avoid the coexistence of incompatible knowledge during inference. Simulation results demonstrate that the proposed approach effectively reduces communication resource consumption by up to 73%73\% and enhances average performance by 5%5\% compared with the traditional LoRA method.

Nuocheng Yang, Sihua Wang, Ouwen Huan +3

Beijing Laboratory of Advanced Information Network, and the Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Department of Electrical and Computer Engineering and Institute for Data Science and Computing, University of Miami, Coral Gables, FL, 33146 USA · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, 487372, Singapore

Federated Fine-TuningFederated LearningMulti-Task LearningParameter-Efficient Fine-TuningClustered Federated Learning
AskPDF
Feb 19, 2026

FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment

Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt to downstream tasks efficiently. Federated learning (FL) further facilitates this process by enabling collaborative fine-tuning across distributed clients without sharing private data. However, the use of two separate low-rank matrices in LoRA for federated fine-tuning introduces two types of challenges. First, aggregation error can arise from separately aggregating the two low-rank matrices. Second, even if the server aggregates the product of two low-rank matrices, it needs to decompose the aggregated matrix back into low-rank matrices. Since the decomposition is not unique, it can lead to decomposition drift. To tackle the aforementioned challenges, we propose federated low-rank Gram-matrix aggregation (FLoRG), a federated fine-tuning framework which employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix (i.e., the matrix of inner products of its column vectors). FLoRG can eliminate the aggregation error and reduce the communication overhead. It also minimizes the decomposition drift by introducing a Procrustes alignment approach which aligns the decomposed matrix between consecutive fine-tuning rounds for consistent updates. We theoretically analyze the convergence of FLoRG and prove that adopting the Procrustes alignment results in a tighter convergence bound. Experimental results across multiple LLM fine-tuning benchmarks demonstrate that FLoRG outperforms five state-of-the-art baseline schemes by providing higher downstream task accuracy and can reduce the communication overhead by up to 2041×\times.

Chuiyang Meng, Ming Tang, Vincent W. S. Wong

The University of British Columbia · Southern University of Science and Technology

LLM Fine-TuningFederated Fine-TuningFederated LearningFederated Learning AggregationLow-Rank Adaptation
AskPDF