cs.LGMar 30, 2026

Task-Centric Personalized Federated Fine-Tuning of Language Models

Authors: Gabriel U. Talasso, Meghdad Kurmanji, Allan M. de Souza, Nicholas D. Lane, Leandro A. Villas

Organizations: Universidade Estadual de Campinas · University of Cambridge · Flower Labs

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Federated Learning (FL) has emerged as a promising technique for training language models on distributed and private datasets of diverse tasks. However, aggregating models trained on heterogeneous tasks often degrades the overall performance of individual clients. To address this issue, Personalized FL (pFL) aims to create models tailored for each client's data distribution. Although these approaches improve local performance, they usually lack robustness in two aspects: (i) generalization: when clients must make predictions on unseen tasks, or face changes in their data distributions, and (ii) intra-client tasks interference: when a single client's data contains multiple distributions that may interfere with each other during local training. To tackle these two challenges, we propose FedRouter, a clustering-based pFL that builds specialized models for each task rather than for each client. FedRouter uses adapters to personalize models by employing two clustering mechanisms to associate adapters with specific tasks. A local clustering that associate adapters with task data samples and a global one that associates similar adapters from different clients to construct task-centric personalized models. Additionally, we propose an evaluation router mechanism that routes test samples to the best adapter based on the created clusters. Experiments comparing our method with existing approaches across a multitask dataset, FedRouter demonstrate strong resilience in these challenging scenarios performing up to 6.1% relatively better under tasks interference and up to 136% relative improvement under generalization evaluation.

Figures & tables

Figure 1 : FedRouter Workflow Overview. Each client first computes embeddings from its local data and applies clustering to partition the dataset into task-specific subsets. The client then sends the resulting centroids and adapters to the server, which performs global clustering to associate similar tasks across clients and aggregate their corresponding adapters collaboratively. Finally, the server sends the updated adapters back to the clients, which then associate each received model with the appropriate local task-specific dataset for next round of training.
Figure 2 : FedRouter Evaluation Modes. During inference, each client computes the embedding of a new data sample and associates it with the nearest centroid based on the minimum Euclidean distance. The association can be performed using either the local centroids, to obtain a personalized evaluation, or the global centroids, to enable a generalized evaluation across the federation.
MethodSingleDualAllAverage
FedIT0.546 ±\pm 0.0120.550 ±\pm 0.0090.560 ±\pm 0.0100.552
Local0.553 ±\pm 0.0050.525 ±\pm 0.0140.534 ±\pm 0.0050.537
FedCluster0.561 ±\pm 0.0080.551 ±\pm 0.0120.553 ±\pm 0.0120.555
FedSA0.554 ±\pm 0.0080.530 ±\pm 0.0100.531 ±\pm 0.0100.538
FedDPA0.556 ±\pm 0.0090.551 ±\pm 0.0130.549 ±\pm 0.0100.552
FedRouter0.561 ±\pm 0.0040.558 ±\pm 0.0190.566 ±\pm 0.0340.562
Table 1 : Performance comparison (mean ±\pm std) across different data scenarios. The last column reports the average performance across scenarios. Best results per column are highlighted in bold.
Figure 4
Figure 5
Figure 7 : Silhouette Score method to choose the number of global clusters in different scenarios for FedRouter.
Figure 8 : Silhouette Score method to choose the number of local clusters in different scenarios for FedRouter.

Explore similar work

CardsList
Sep 30, 2026cs.LG

From Task Mixtures to Specialized Experts

In collaborative foundation model fine-tuning, client data is rarely homogeneous. Instead, clients typically possess unknown mixtures of distinct data distributions, or tasks. Conventional federated learning primarily addresses heterogeneity across clients without explicitly resolving latent task mixtures within each client. We study this setting as compound heterogeneity, where data is heterogeneous both across and within clients. We study adaptation over a common frozen representation and show that, when tasks share the same feature geometry, the optimal model for a client's task mixture under squared loss is a convex combination of the optimal models for its underlying tasks. Thus, a single locally trained model represents the client's overall task mixture, while individual inputs may be drawn from different underlying task distributions. This motivates routing inputs to specialized experts, and we show that, when the task optima form a simplex, task-aligned routing achieves lower risk than any single adapted model for genuinely mixed clients. With access to a small set of task-labeled public samples, we derive a convex program to recover task experts and match them to their corresponding tasks. Our routing analysis shows that effective specialization requires input-dependent expert selection aligned with each client's task mixture. Motivated by this analysis, we propose FedSEE. Across our experiments, FedSEE avoids the negative transfer observed in the evaluated baselines and improves performance by 2.9 points overall and 3.7 points for the worst-served quartile.

Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari +1

Florida International University, Miami, FL, USA · Security, Optimization, and Learning for InterDependent networks laboratory (solid lab), Miami, FL, USA · The Pennsylvania State University, University Park, PA, USA +1

Multi-Task LearningMixture-of-Experts ModelsFederated Fine-TuningHeterogeneous Federated Learning
AskPDF
Sep 29, 2026cs.AI

CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning

Federated LoRA fine-tuning enables parameter-efficient adaptation of pre-trained models without sharing private data, but suffers from two fundamental mismatches under heterogeneous client data: a structural aggregation mismatch caused by independently averaging LoRA factors, and a statistical collaboration mismatch caused by enforcing a single global adapter across divergent clients. To address these issues, we propose CF-LoRA, a clustered federated LoRA fine-tuning framework that combines decoupled factor aggregation with adaptation-aware client clustering. CF-LoRA first learns a globally shared AA factor while retaining personalized BiB_i factors, then identifies clients with similar adaptation patterns based on the cosine similarity of their learned BiB_i factors, and finally performs intra-cluster BB-factor aggregation with a frozen AA factor. By decoupling LoRA factor aggregation, CF-LoRA preserves the low-rank structure and mitigates the structural aggregation mismatch, while adaptation-aware clustering promotes collaboration among clients with similar adaptation patterns and reduces negative transfer caused by statistical heterogeneity. Experiments on four language tasks and four vision datasets with RoBERTa and ViT show that CF-LoRA achieves the highest average accuracy in both modalities while communicating only one LoRA factor per optimization round.

Mengjun Yi, Langxing Yang, Suhan Guo +2

State Key Laboratory for Novel Software Technology · School of Artificial Intelligence School of Electronic Science and Engineering Nanjing University

Federated Learning AggregationFederated Fine-TuningClustered Federated LearningFederated LearningParameter-Efficient Fine-TuningHeterogeneous Federated Learning
AskPDF
Sep 29, 2026cs.AI

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but statistical heterogeneity makes a single global adapter insufficient for personalized prediction. Existing personalized methods typically use the same low-rank structure for both shared and private adaptation, overlooking their distinct requirements for aggregation and personalization. We propose FedLAFP, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch. The shared branch provides an efficient interface for transferring common knowledge, whereas the private branch combines fixed random low-rank bases with learned scaling coefficients to provide expressive client-specific adaptation without additional communication. Client- and layer-specific mixing coefficients jointly fuse the two branches, and only the shared LoRA parameters are exchanged. A controlled linear study supports this role assignment: LoRA yields more aligned client updates and lower aggregation error, while RandLoRA more accurately recovers client-specific residuals. Experiments across four visual recognition benchmarks show that FedLAFP consistently outperforms local-only and federated LoRA baselines, achieving an average personalized accuracy of 86.93%86.93\% and exceeding the best baseline average by 1.301.30 percentage points.

Mengjun Yi, Huaian Gu, Yinghao Ai +2

State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science +1

Low-Rank AdaptationFederated Fine-TuningFederated LearningParameter-Efficient Fine-Tuning
AskPDF