Adaptive Rank Allocation

Ask about this topic

Overview

Adaptive rank allocation studies how to divide a limited low-rank budget so models preserve useful capacity without spending equally everywhere. In LoRA and other parameter-efficient fine-tuning, methods use gradient or Fisher scores, activation sensitivity, uncertainty, or information-flow structure; spectral methods use singular values to allocate capacity for compression and model merging. These approaches are evaluated on settings such as language understanding, math and code reasoning, image generation, and long-context cache compression, often at matched parameter or memory budgets. A central tension is that importance scores do not always predict capacity needs: signals that help in supervised fine-tuning may fail in reinforcement-learning alignment, and dynamic allocation must justify its added estimation or training cost.

Latest papers

CardsList

32 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

  1. SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection

    Oct 27, 2026Igor Shalygin, Grigorii Davydenko, Andrey Veprikov +4Low-Rank AdaptationFine-Tuning Language Models

    Moscow Independent Research Institute of Artificial Intelligence (MIRAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab +2

  2. LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

    Oct 26, 2026Shenghui Li, Thiemo voigtLow-Rank CompressionAdaptive Rank Allocation

    Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden

  3. Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons

    Oct 19, 2026Motti Goldberger, Nils RudiHuman-in-the-Loop EvaluationPreference Elicitation

    Yale School of Management

  4. Residual-Guided Adaptive Tucker Rank Allocation for Parameter-Efficient MCI Conversion Prediction

    Oct 19, 2026Guanghua He, Hancan Zhu, Gaohang Yu +1Adaptive Rank AllocationParameter-Efficient Fine-Tuning

    Department of Mathematics, Hangzhou Dianzi University, Hangzhou, 310018, China · School of Mathematics, Physics and Information, Shaoxing University, Shaoxing, 312000, China · Department of Mathematics, Zhejiang University of Science and Technology, Hangzhou, 310023, China

  5. Data-Driven Prediction of Optimal MPI Rank Placement and Node Count for Memory-Bound CFD on AMD EPYC Clusters

    Oct 16, 2026Marcin Lawenda, Łukasz Szustak, Aleksandra KrasickaHigh-Performance ComputingAdaptive Rank Allocation

    Poznan Supercomputing and Networking Center, Jana Pawła II 10, 61-139 Poznań, Poland · Czestochowa University of Technology, Dąbrowskiego 69, 42-201 Częstochowa, Poland

  6. How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts

    Oct 13, 2026Mei Okonkwo, Owen Castellanos, Ingrid Solberg +1Low-Rank AdaptationAdaptive Rank Allocation

    University of Edinburgh · Delft University of Technology

  7. AnyRank: Reliable Anytime Policy Ranking via Sequential Testing

    Oct 10, 2026Apurva Badithela, David Snyder, Tenny Yin +6Robotic Policy EvaluationAdaptive Rank Allocation

    Princeton University · University of Pennsylvania · Toyota Research Institute

  8. Beyond Uniform Compression: Budgeted Transmission Allocation for Extreme Federated Learning

    Sep 30, 2026Pengfei Li, Mohammad KhalilCommunication-Efficient Distributed TrainingLow-Rank Compression

    Centre for the Science of Learning & Technology (SLATE), University of Bergen, Bergen, Norway

  9. Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

    Sep 24, 2026Zebang Xie, Chuanyang Zheng, Yik-Chung Wu +1Low-Rank AdaptationAdaptive Rank Allocation

    National University of Singapore, Singapore · Huawei, Hong Kong · The University of Hong Kong, Hong Kong +1

  10. Not All Task Vectors Need Equal Rank: Energy-Proportional Allocation for Model Merging

    Sep 21, 2026Hyunjoong Cho, Jinhyeok JangMulti-Task LearningTask Vector Merging

    LG CNS · ETRI · UST

  11. MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training

    Sep 14, 2026Keunyoung Kim, Nojun KwakLanguage Model PretrainingLow-Rank Approximation

    Seoul National University

  12. Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks

    Sep 3, 2026Avinash Amballa, Yashas Malur Saidutta, Wenbo Li +2Model MergingLow-Rank Adaptation

    Samsung Research America

  13. LoRA-GA2^2: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment

    Aug 20, 2026Haonan He, Xinyue FanLLM Fine-TuningLow-Rank Adaptation

    University of Science and Technology of China · Independent Researcher

  14. Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs

    Aug 9, 2026Mohanad Odema, Gabrielle De Micheli, Dayin Gou +3Model CompressionLLM Compression

    LG Electronics, North America

  15. DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation

    Jul 27, 2026Tan T. Nguyen, Quan V. DangKV CachingLow-Rank Approximation

    ∗Full Stack Data Science · †Department of Computer Science, University College London

  16. IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning

    Jul 24, 2026Wei Zhang, Xinwu Liu, Yihang ChengLLM Fine-TuningLow-Rank Adaptation

    Faculty of Artificial Intelligence in Education Laboratory for Artificial Intelligence and New Forms of Education Central China Normal University Wuhan, Hubei, China

  17. Statistical Inference for Rank Allocation in Low-Rank Adaptation

    Jul 22, 2026Yihang Gao, Vincent Y. F. TanLLM Fine-TuningLow-Rank Adaptation

    Department of Mathematics, National University of Singapore · Department of Mathematics and Department of Electrical and Computer Engineering, National University of Singapore

  18. RSLoRA: Training-free Rank Allocation for LoRA via Representational Sensitivity Probing

    Jul 5, 2026Jiaqi Liu, Haidong Kang, Qihui Zhao +1Low-Rank AdaptationAdaptive Rank Allocation

    School of Computer Science and Technology Dalian University of Technology · Hebei Key Laboratory of Marine Perception Network and Data Processing Northeastern University · School of Computer and Communication Engineering Northeastern University +1

  19. TriageRA-CCF: Source-Side Clinical Confidence and Coverage Signals for Adaptive Rank Budgeting in Medical LLMs

    Jun 28, 2026Shucan Ji, Yining Huang, Hongliang GuoLow-Rank AdaptationMedical QA

    College of Computer Science, Sichuan University, Chengdu, China · Meta Emergence Laboratory

  20. BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    Jun 28, 2026Zhibin Duan, Yuhong Wang, Jiahong Fu +3Bayesian Neural NetworksLow-Rank Adaptation

    National Key Lab of Radar Signal Processing, Xidian University, Xi’an, Shaanxi 710071, China · School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, 710049, China

  21. UniRank: Unified Rank Allocation for Low-Rank LLM Compression

    Jun 20, 2026Chao Han, Yongjie Du, Junjie Tan +1LLM CompressionLow-Rank Matrix Decomposition

    Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo

  22. Energy-Structured Low-Rank Adaptation for Continual Learning

    May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual LearningLow-Rank Adaptation

    School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

  23. FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation

    May 16, 2026Ramakrishnan SathyavageeswaranAdapter TuningLow-Rank Adaptation

    Intuit

  24. AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery

    May 11, 2026Barbara Su, Fangshuo Liao, Anastasios KyrillidisLLM Fine-TuningAdapter Tuning

    Department of Computer Science Rice University

  25. Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study

    May 8, 2026Yash Ganpat SawantGroup Relative Policy OptimizationLow-Rank Adaptation

    1Independent Researcher.

  26. Post-Optimization Adaptive Rank Allocation for LoRA

    Apr 30, 2026Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. SrijithLow-Rank AdaptationLow-Rank Compression

    1Indian Institute of Technology Hyderabad · 2Deakin University

  27. TLoRA: Task-aware Low Rank Adaptation of Large Language Models

    Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-TuningLow-Rank Adaptation

    College of Computer Science and Software Engineering, Shenzhen University, China

  28. Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation

    Mar 23, 2026Donald Shenaj, Federico Errica, Antonio CartaLow-Rank AdaptationAdaptive Rank Allocation

    University of Pisa · NEC Laboratories Europe

  29. CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention

    Mar 18, 2026Zhongzhu Zhou, Fengxiang Bie, Ziyan Chen +6Multi-Head Latent AttentionLow-Rank Approximation

    University of Sydney · Together AI · University of Texas at Austin +1

  30. IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring

    Mar 14, 2026Xuan Cui, Huiyue Li, Run Zeng +5LLM Fine-TuningLow-Rank Adaptation

    Chongqing Technology and Business University · Shanghai Jiao Tong University