Adaptive Rank Allocation

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 20

All topics

Beyond Uniform Compression: Budgeted Transmission Allocation for Extreme Federated Learning
Sep 30, 2026Pengfei Li, Mohammad KhalilCommunication-Efficient Distributed Training Low-Rank Compression
Centre for the Science of Learning & Technology (SLATE), University of Bergen, Bergen, Norway
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization
Sep 24, 2026Zebang Xie, Chuanyang Zheng, Yik-Chung Wu +1Low-Rank Adaptation Adaptive Rank Allocation
National University of Singapore, Singapore · Huawei, Hong Kong · The University of Hong Kong, Hong Kong +1
Not All Task Vectors Need Equal Rank: Energy-Proportional Allocation for Model Merging
Sep 21, 2026Hyunjoong Cho, Jinhyeok JangMulti-Task Learning Task Vector Merging
LG CNS · ETRI · UST
MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training
Sep 14, 2026Keunyoung Kim, Nojun KwakLanguage Model Pretraining Low-Rank Approximation
Seoul National University
LoRA-GA $^2$ : Low Rank Adaptation with Multi-step Gradient Adaptive Alignment
Aug 20, 2026Haonan He, Xinyue FanLLM Fine-Tuning Low-Rank Adaptation
University of Science and Technology of China · Independent Researcher
Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs
Aug 9, 2026Mohanad Odema, Gabrielle De Micheli, Dayin Gou +3Model Compression LLM Compression
LG Electronics, North America
DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation
Jul 27, 2026Tan T. Nguyen, Quan V. DangKV Caching Low-Rank Approximation
∗Full Stack Data Science · †Department of Computer Science, University College London
IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
Jul 24, 2026Wei Zhang, Xinwu Liu, Yihang ChengLLM Fine-Tuning Low-Rank Adaptation
Faculty of Artificial Intelligence in Education Laboratory for Artificial Intelligence and New Forms of Education Central China Normal University Wuhan, Hubei, China
Statistical Inference for Rank Allocation in Low-Rank Adaptation
Jul 22, 2026Yihang Gao, Vincent Y. F. TanLLM Fine-Tuning Low-Rank Adaptation
Department of Mathematics, National University of Singapore · Department of Mathematics and Department of Electrical and Computer Engineering, National University of Singapore
RSLoRA: Training-free Rank Allocation for LoRA via Representational Sensitivity Probing
Jul 5, 2026Jiaqi Liu, Haidong Kang, Qihui Zhao +1Low-Rank Adaptation Adaptive Rank Allocation
School of Computer Science and Technology Dalian University of Technology · Hebei Key Laboratory of Marine Perception Network and Data Processing Northeastern University · School of Computer and Communication Engineering Northeastern University +1
TriageRA-CCF: Source-Side Clinical Confidence and Coverage Signals for Adaptive Rank Budgeting in Medical LLMs
Jun 28, 2026Shucan Ji, Yining Huang, Hongliang GuoLow-Rank Adaptation Medical QA
College of Computer Science, Sichuan University, Chengdu, China · Meta Emergence Laboratory
BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
Jun 28, 2026Zhibin Duan, Yuhong Wang, Jiahong Fu +3Bayesian Neural Networks Low-Rank Adaptation
National Key Lab of Radar Signal Processing, Xidian University, Xi’an, Shaanxi 710071, China · School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, 710049, China
UniRank: Unified Rank Allocation for Low-Rank LLM Compression
Jun 20, 2026Chao Han, Yongjie Du, Junjie Tan +1LLM Compression Low-Rank Matrix Decomposition
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo
Energy-Structured Low-Rank Adaptation for Continual Learning
May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual Learning Low-Rank Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China
FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
May 16, 2026Ramakrishnan SathyavageeswaranAdapter Tuning Low-Rank Adaptation
Intuit
AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
May 11, 2026Barbara Su, Fangshuo Liao, Anastasios KyrillidisLLM Fine-Tuning Adapter Tuning
Department of Computer Science Rice University
Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study
May 8, 2026Yash Ganpat SawantGroup Relative Policy Optimization Low-Rank Adaptation
1Independent Researcher.
Post-Optimization Adaptive Rank Allocation for LoRA
Apr 30, 2026Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. SrijithLow-Rank Adaptation Low-Rank Compression
1Indian Institute of Technology Hyderabad · 2Deakin University
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
Jan 8, 2026Guanzhi Deng, Bo Li, Ronghao Chen +5Mixture-of-Experts Language Models Fine-Tuning
City University of Hong Kong · Tsinghua University · Peking University +2

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Beyond Uniform Compression: Budgeted Transmission Allocation for Extreme Federated Learning

Pengfei Li, Mohammad Khalil

Centre for the Science of Learning & Technology (SLATE), University of Bergen, Bergen, Norway

Communication-Efficient Distributed Training Low-Rank Compression Adaptive Rank Allocation Federated Learning

Open paper AskPDF arXiv

Federated learning faces severe communication bottlenecks when clients upload high-dimensional model updates. Existing methods often compress these updates uniformly across all layers. This uniform approach ignores the heterogeneous value of different parameter blocks and wastes limited bandwidth on insensitive layers. To address this issue, we propose Layer-wise Budgeted Adaptive Transmission (LBAT). LBAT reframes federated communication under extreme uplink budgets as a resource allocation problem. Our framework dynamically estimates the transmission value of different layers utilising local training signals. It then employs an exact byte dynamic programming allocator to determine optimal rank and bit configurations under strict budgets. We validate LBAT on highly heterogeneous federated tabular prediction and data generation tasks. Extensive experiments demonstrate that LBAT consistently outperforms uniform rank, uniform quantisation, and fixed compression baselines across various extreme budget regimes. Furthermore, it achieves significantly better communication and utility tradeoffs while preserving essential distributional fidelity.

Momentum

Related topics

Latest papers 20