Low-Rank Adaptation

Also known as LoRA

Low-rank adaptation studies how to specialize pretrained models by representing weight changes as products of small matrices while keeping most backbone weights frozen. Methods choose where adapters are placed and how their rank is allocated, using approaches such as spectral or gradient-based initialization, adaptive rank selection, and quantized fine-tuning. For multiple tasks or users, adapter routing, expert mixtures, merging, and federated aggregation aim to share useful updates without interference. The central tension is how to gain enough capacity for new tasks while controlling memory and compute, preserving existing capabilities, and generalizing across data shifts; the best rank, placement, and composition strategy depend on the model and task.

Latest papers

538 papers92 papers in the last four weeks, up 59% on the four weeks before. 0.5% of all new papers.

LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights
Feb 19, 2026Kasun Dewage, Marianna Pensky, Shankadeep Mondal +1Tensor Decomposition Low-Rank Adaptation
University of Central Florida Orlando, FL 32816, USA
FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment
Feb 19, 2026Chuiyang Meng, Ming Tang, Vincent W. S. WongLLM Fine-Tuning Federated Fine-Tuning
The University of British Columbia · Southern University of Science and Technology
Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC
Feb 18, 2026Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov +3Proximal Optimization Kronecker-Factored Approximation
MBZUAI, Abu Dhabi, UAE · Yandex School of Data Analysis, Moscow, Russia · SB AI Lab, Moscow, Russia +2
Spanning the Visual Analogy Space with a Weight Basis of LoRAs
Feb 17, 2026Hila Manor, Rinon Gal, Haggai Maron +2Vision Foundation Model Adaptation Low-Rank Adaptation
2NVIDIA, Israel · 1Technion, Israel · 3Bar-Ilan University, Israel
Weight space Detection of Backdoors in LoRA Adapters
Feb 16, 2026David Puertolas Merenciano, Ekaterina Vasyagina, Kevin Zhu +2Backdoor Detection Backdoor Defense in LLMs
Algoverse AI Research · Independent
D2-LoRA: A Synergistic Approach to Differential and Directional Low-Rank Adaptation
Feb 16, 2026Nozomu Fujisawa, Masaaki KondoLLM Fine-Tuning Low-Rank Adaptation
Keio University
Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts
Feb 16, 2026Buze Zhang, Jinkai Tao, Zilang Zeng +4Parameter-Efficient Fine-Tuning Representation Geometry in Language Models
School of Software Engineering, Xi’an Jiaotong University, China · School of Information, Central University of Finance and Economics, Beijing, China · Department of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China +2
Parameter-Efficient Temporal Domain Generalization
Feb 12, 2026Zekun Cai, Xinyuan Song, Yiheng Yao +3Domain Generalization Parameter-Efficient Fine-Tuning
The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan · Emory University, Atlanta, GA, USA +1
Rank-Constrained Adaptation for Reliable Real-World Performance
Feb 6, 2026Abinitha Gourabathina, Hyewon Jeong, Teya Bergamaschi +2Subgroup Robustness Distribution Shift Robustness
Department of Electrical Engineering & Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA · Harvard School of Medicine, Harvard, Boston, MA, USA.
SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
Feb 6, 2026Yewei Liu, Xiyuan Wang, Yansheng Mao +3Knowledge Augmentation for Language Models Low-Rank Adaptation
Institute for Artificial Intelligence, Peking University · School of Electronics Engineering and Computer Science, Peking University · University of Oxford +1
Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions
Feb 5, 2026Yihao Ouyang, Shiwei Li, Haozhao Wang +5Nonlinear LoRA Low-Rank Adaptation
Huazhong University of Science and Technology, Wuhan, China
Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs
Feb 3, 2026Alessio Quercia, Arya Bangun, Ira Assent +1Low-Rank Adaptation Parameter-Efficient Fine-Tuning
IAS-8, Forschungszentrum Juelich, Juelich, Germany · Department of Computer Science, RWTH Aachen University, Aachen, Germany · Aarhus University, Aarhus, Denmark
Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
Jan 31, 2026Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3VLM Adaptation Continual Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China
Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
Jan 28, 2026Moule Lin, Shuhao Guan, Andrea Patane +2Bayesian Neural Networks Confidence Estimation in Language Models
School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
Jan 8, 2026Guanzhi Deng, Bo Li, Ronghao Chen +5Mixture-of-Experts Language Models Fine-Tuning
City University of Hong Kong · Tsinghua University · Peking University +2
Weight Space Representation Learning via Neural Field Adaptation
Dec 1, 2025Zhuoqian Yang, Mathieu Salzmann, Sabine SüsstrunkImplicit Neural Representations Low-Rank Adaptation
EPFL
K-Merge: Online Continual Merging of Adapters for On-device Large Language Models
Oct 15, 2025Donald Shenaj, Ondrej Bohdal, Taha Ceritli +3Continual Learning for LLMs On-Device Language Model Inference
Samsung R&D Institute UK · University of Pisa · University of Padova
Hierarchical Alignment: Block-Level LoRA-DPO for Preference Tuning in Large Language Models
Oct 14, 2025Yishen Chen, Qi Dong, Yukun ZhangLLM Alignment Preference Alignment
Fudan University Shanghai, China · The Chinese University of Hong Kong Hong Kong, China
On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
Sep 22, 2025Rongguang Ye, Ming Tang, Edith C. H. NgaiMixed-Precision Quantization LLM Quantization
Southern University of Science and Technology · The University of Hong Kong
LoRAtorio: An intrinsic approach to LoRA Skill Composition
Aug 15, 2025Niki Foteinopoulou, Ignas Budvytis, Stephan LiwickiT2I Diffusion Models Low-Rank Adaptation
Cambridge Research Laboratory, Toshiba Europe · Independent Researcher
From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning
Aug 7, 2025Jinda Liu, Yi Chang, Yuan WuMulti-Task Learning Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · 2Southern University of Science and Technology · 3International Center of Future Science, Jilin University +1
PLoRA: Efficient Concurrent LoRA Training for Large Language Models
Aug 4, 2025Minghao Yan, Zhuang Wang, Zhen Jia +2LLM Fine-Tuning Efficient Language Model Training
University of Wisconsin-Madison · Amazon Web Services
WaRA: Wavelet Low-Rank Adaptation for Medical Image Classification
Jun 25, 2025Moein Heidari, Yijin Huang, Yasamin Medghalchi +3Vision Foundation Model Adaptation Low-Rank Adaptation
School of Biomedical Engineering, The University of British Columbia, Vancouver, BC, Canada · Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China · Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, Canada +2
TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree
Jun 12, 2025Yu-Yang Qian, Yuan-Ze Xu, Zhen-Yu Zhang +2Continual Learning for LLMs Low-Rank Adaptation
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · RIKEN AIP, Tokyo, Japan
COLORA: Efficient Fine-Tuning for Convolutional Models with a Study Case on Optical Coherence Tomography Image Classification
May 23, 2025Mariano Rivera, Angello HoyosLow-Rank Adaptation Convolutional Neural Networks
Centro de Investigación en Matemáticas, A.C. (CIMAT) Guanajuato, Guanajuato 36023, México
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Feb 25, 2025Xin Zhang, Liang Bai, Xian YangContinual LoRA Low-Rank Adaptation
Institute of Intelligent Information Processing, Shanxi University, Taiyuan, 030006, China · Alliance Manchester Business School, The University of Manchester, Manchester, M13 9PL, UK
A LoRA is Worth a Thousand Pictures
Dec 16, 2024Chenxi Liu, Towaki Takikawa, Alec JacobsonWeight-Space Analysis Low-Rank Adaptation
University of Toronto · Outerport · Adobe Research
BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models
Aug 8, 2024Yupeng Chang, Yi Chang, Yuan WuSocial Bias in Language Models Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University · International Center of Future Science, Jilin University

Feb 19, 2026J/K move · Enter open · S save

LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

Kasun Dewage, Marianna Pensky, Shankadeep Mondal, Suranadi De Silva

University of Central Florida Orlando, FL 32816, USA

Tensor Decomposition Low-Rank Adaptation Transformer Attention Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

We introduce LoRA-CRAFT (Cross-layer Rank Adaptation via Frozen Tucker), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors. Existing tensor-based PEFT methods decompose gradient updates: LoTR applies Tucker decomposition with shared factor matrices, while SuperLoRA groups and reshapes

ΔW

across layers before applying Tucker decomposition. Separately, methods such as PiSSA apply SVD to pre-trained weights but operate independently per layer. CRAFT bridges these two lines of work: it performs full Tucker decomposition via Higher-Order SVD (HOSVD) directly on pre-trained weights organized as cross-layer 3D tensors, freezes all resulting factors, and adapts the model through lightweight trainable transformations applied to each factor matrix. Experiments on the GLUE benchmark using RoBERTa-base and RoBERTa-large, as well as commonsense reasoning benchmarks using LLaMA2-7B and LLaMA3-8B, demonstrate that CRAFT achieves competitive performance with existing methods while requiring only \rev{extremely low Tucker adaptation parameters}. \fixw{On LLaMA3-8B, CRAFT} \rev{exceeds the average accuracy of LoRA} using hundreds of times fewer parameters\fixw{; on LLaMA2-7B the same holds at a

0.252

M budget}. Our results suggest that CRAFT's efficiency advantage grows with model scale, as the frozen Tucker factors better capture the richer cross-layer structure of larger pre-trained models.