SMELL: Sparsity-aware Memory Efficient Federated Fine-tuning for Long-context LLMs
Organizations: School of Computer Science and Technology, University of Science and Technology of China, Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China · Department of Data and Systems Engineering, The University of Hong Kong, Hong Kong, China
Abstract
Federated fine-tuning enables Large Language Models (LLMs) to collaboratively adapt to distributed private data. However, long-context fine-tuning on resource-constrained devices faces substantial challenges arising from excessive activation memory, heterogeneous token-importance patterns across clients, and communication overhead. To address these challenges, we propose SMELL, a Sparsity-aware and Memory-Efficient federated fine-tuning framework for Long-context LLMs. SMELL integrates three complementary mechanisms: Local Sparse-Forward Adaptation (LSFA), Consensus-Anchored Sparsity Coordination (CASC), and Communication-Efficient Progressive Update Recovery (CPUR). These mechanisms jointly enable memory-efficient local adaptation, heterogeneity-aware sparsity coordination, and lightweight model-update transmission. Experiments on OPT-350M and LLaMA2-7B demonstrate that SMELL reduces peak memory consumption by up to 88.35% compared with conventional fine-tuning and 13.25% compared with forward-only LoRA fine-tuning at a 16K context length. Furthermore, SMELL improves fine-tuning performance under certain non-IID settings and supports compact update transmission with progressive reconstruction, demonstrating its potential for resource-efficient federated long-context LLM adaptation.
Figures & tables
| Symbol | Definition |
|---|---|
| trainable parameters and transformer layers | |
| , | forward/backward activations per layer |
| context length and hidden dimension | |
| attention heads, batch size, peak activation memory | |
| block keep ratio, LoRA parameters, training loss | |
| finite-difference step, -th perturbation direction |
| LLaMA2-7B | OPT-350M | |||||||||||||||
| Context | Full FT | LoRA | Jenga | FwdLLM+LoRA | SMELL (Ours) | LoRA | FwdLLM+LoRA | SMELL (Ours) | ||||||||
| Scale | Mem | Scale | Mem | Scale | Mem | Scale | Mem | Scale | Mem | Scale | Mem | Scale | Mem | Scale | Mem | |
| 1K | 8.19 | 113 | 1.52 | 21 | 1.19 | 16.5 | 1.01 | 13.9 | 1 | 13.8 | 2.01 | 2.00 | 1.03 | 1.02 | 1 | 0.99 |
| 2K | 8.28 | 121 | 2.05 | 30 | 1.38 | 20.2 | 1.02 | 15.0 | 1 | 14.6 | 2.60 | 3.28 | 1.04 | 1.31 | 1 | 1.26 |
| 4K | 8.20 | 137 | 2.75 | 46 | 1.66 | 27.8 | 1.03 | 17.2 | 1 | 16.7 | 3.51 | 4.89 | 1.08 | 1.50 | 1 | 1.39 |
| 8K | 8.25 | 170 | 3.83 | 79 | — | 40 | 1.08 | 22.2 | 1 | 20.6 | 4.86 | 8.10 | 1.14 | 1.89 | 1 | 1.66 |
| Window | Pre-r0 | End | % | ||
|---|---|---|---|---|---|
| 16 | pre-r0–r7 | 33.6653 | 32.7144 | ||
| 22 | pre-r0–r7 | 33.6653 | 32.5335 | ||
| 32 | pre-r0–r7 | 33.6653 | 32.5071 |
| (a) OPT-350M/Discovery (full G-PPL) | |||||
| Protocol | r0(G-PPL) | r29(G-PPL) | % | ||
| 0.1 | CASC off | 33.525 | 31.533 | ||
| 0.1 | CASC on(train) | 33.525 ∗ | 31.378 | ||
| 0.1 | CASC on(train+eval) | 33.921 | 31.635 | ||
| 0.3 | CASC off | 33.530 | 31.457 | ||
| 0.3 | CASC on(train) | 33.530 ∗ | 31.416 | ||
| Run | NLL r0 | NLL r59 | Acc r59 | Uplink | |
|---|---|---|---|---|---|
| FwdLLM | 1.5923 | 1.3437 | 0.520 | 1.0 | |
| FwdLLM+Jenga | 1.6906 | 1.5110 | 0.474 | 1.0 | |
| + CPUR 0.5 | 1.6907 | 1.5139 | 0.474 | 0.50 | |
| Combo (train) | 1.6904 | 1.5231 | 0.470 | 0.50 | |
| Combo (train+eval) | 1.7425 | 1.4896 | 0.478 | 0.50 |
Federated Fine-Tuning
Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.
See all 128 papers