Low-Rank Adaptation

Also known as LoRA

Low-rank adaptation studies how to specialize pretrained models by representing weight changes as products of small matrices while keeping most backbone weights frozen. Methods choose where adapters are placed and how their rank is allocated, using approaches such as spectral or gradient-based initialization, adaptive rank selection, and quantized fine-tuning. For multiple tasks or users, adapter routing, expert mixtures, merging, and federated aggregation aim to share useful updates without interference. The central tension is how to gain enough capacity for new tasks while controlling memory and compute, preserving existing capabilities, and generalizing across data shifts; the best rank, placement, and composition strategy depend on the model and task.

Latest papers

542 papers92 papers in the last four weeks, up 59% on the four weeks before. 0.5% of all new papers.

LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection
May 12, 2026Lanxin Zhao, Bamdev Mishra, Pratik Jawanpuria +4Fine-Tuning Low-Rank Adaptation
The University of Sydney, Australia. · Microsoft India. · Indian Institute of Technology Bombay, India. +1
Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundary
May 11, 2026Ramchand KumaresanAdaptive Model Routing Evolutionary Optimization
Murai Labs
AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
May 11, 2026Barbara Su, Fangshuo Liao, Anastasios KyrillidisLLM Fine-Tuning Adapter Tuning
Department of Computer Science Rice University
Filtering Memorization from Parameter-Space in Diffusion Models
May 11, 2026Yu Zhe, Yang Jiayan, Wei Junhao +2Low-Rank Adaptation Memorization in Generative Models
RIKEN AIP · Institute of Science Tokyo · University of California, Berkeley +1
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
May 11, 2026Jia Wei, Zhonghao Zhang, Ping Chen +5Mixture-of-Experts Language Models Fine-Tuning
Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
SSDA: Bridging Spectral and Structural Gaps via Dual Adaptation for Vision-Based Time Series Forecasting
May 10, 2026Mingrui Zhang, Hanchen Yang, Wengen Li +4Domain Adaptation Time Series Forecasting
AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
May 9, 2026Ziyun Liu, Fengmiao Bian, Jian-Feng CaiLow-Rank Adaptation Adaptive Gradient Methods
Department of Mathematics The Hong Kong University of Science and Technology
Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms
May 8, 2026Omatharv Bharat Vaidya, Connor T. Jerzak, Nhat Ho +1Memory-Augmented Neural Networks Adapter Tuning
Department of Computer Science University of Texas at Austin · Department of Government University of Texas at Austin · University of Texas at Austin
Convergent Stochastic Training of Attention and Understanding LoRA
May 8, 2026Zhengkai Sun, Dibyakanti Kumar, Alejandro F Frangi +2Stochastic Optimization Low-Rank Adaptation
MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
May 8, 2026Ionut-Vlad Modoranu, Mher Safaryan, Dan AlistarhAdapter Tuning LLM Fine-Tuning
Institute of Science and Technology Austria (ISTA). · Lancaster University, UK · ISTA
Bayesian Fine-tuning in Projected Subspaces
May 8, 2026Viktar Dubovik, Patryk Marszałek, Jacek Tabor +1Bayesian Neural Networks Fine-Tuning
Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland
One Adapter, Every Resolution: Gated Low-Rank Adaptation for Remote Sensing VLMs
May 8, 2026Song Zhang, Yanlong Chen, Yining Chen +2Remote Sensing Image Understanding Remote Sensing VQA
Nanjing University · ETH Zurich · Nanjing University of Posts and Telecommunications
SR $^2$ -LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning
May 8, 2026Fengqiang Wan, Yipeng Lin, Kan Lv +1Representation Drift Low-Rank Adaptation
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study
May 8, 2026Yash Ganpat SawantGroup Relative Policy Optimization Low-Rank Adaptation
1Independent Researcher.
StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs
May 8, 2026Chang Che, Ziqi Wang, Hui Ma +2VLM Adaptation Low-Rank Adaptation
Hefei University of Technology · Tsinghua University
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
May 8, 2026Haozhan Tang, Xiuqi Zhu, Xinyin Zhang +3LLM Fine-Tuning Low-Rank Adaptation
Carnegie Mellon University · Tsinghua University · Infinigence AI
Rethinking Adapter Placement: A Dominant Adaptation Module Perspective
May 7, 2026Suoxin Zhang, Run He, Di Fang +3LLM Fine-Tuning Adapter Tuning
South China University of Technology, China · Zhejiang University, China
Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
May 7, 2026Jinqian Chen, Chang Liu, Jihua ZhuFederated Learning Aggregation Low-Rank Adaptation
School of Software Engineering Xi’an Jiaotong University Xi’an, China 710000
Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration
May 7, 2026Haisen He, Xiangyu Zou, SongLin Dong +3Low-Rank Adaptation All-in-One Image Restoration
Southern University of Science and Technology · Shenzhen University · Shenzhen University of Advanced Technology +1
Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning
May 7, 2026Myoungjun Kim, Sangwoo Park, Yoseob Han +1Differentially Private Federated Learning Federated Learning
Myongji University · King’s College London · Soongsil University
CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning
May 7, 2026Md Anwar Hossen, Fatema Siddika, Juan Pablo Munoz +2Continual Learning for LLMs Low-Rank Adaptation
Iowa State University · Maro Systems, USA · University of California, Berkeley +1
Low-Rank Adaptation of Geospatial Foundation Models for Wildfire Mapping Using Sentinel-2 Data
May 6, 2026Ali Shibli, Andrea Nascetti, Yifang BanGeospatial Foundation Models Spatial Generalization
KTH Royal Institute of Technology Stockholm, Sweden
Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference
May 5, 2026Mohammed Sabry, Anya BelzLLM Compression Low-Rank Adaptation
ADAPT Centre, Dublin City University, Ireland
Rethinking the Rank Threshold for LoRA Fine-Tuning
May 5, 2026Juneyoung ParkLow-Rank Adaptation Binary Classification
OptAI Inc.
Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
May 5, 2026Yihang Peng, Peng Jin, Jie Gong +4Commonsense Reasoning LLM Fine-Tuning
School of Computer Science and Software Engineering, Southwest Petroleum University · School of Electronic Information and Artificial Intelligence, Leshan Normal University
CERSA: Cumulative Energy-Retaining Subspace Adaptation for Memory-Efficient Fine-Tuning
May 5, 2026Jingze Ge, Xue Geng, Yun Liu +6Fine-Tuning Memory-Efficient Fine-Tuning
National University of Singapore · Institute for Infocomm Research (I2R), A*STAR · Nankai University +1
Ortho-Hydra: Orthogonalized Experts for DiT LoRA
May 5, 2026Seunghyun JiFine-Tuning Diffusion Transformer
Compress Then Adapt? No, Do It Together via Task-aware Union of Subspaces
May 4, 2026Jingze Ge, Yun Liu, Xue Geng +4Model Compression Adapter Tuning
National University of Singapore · Nankai University · Institute for Infocomm Research (I2R), A*STAR
Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks
May 3, 2026Zongqian Li, Yixuan Su, Han Zhou +2Low-Rank Adaptation Bayesian LoRA
University of Cambridge
SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability
May 2, 2026Shuaipeng Zhou, Yu ZhangUncertainty Quantification Adapter Tuning
School of Computer Science and Technology, Shenyang University of Chemical Technology · 2Liaoning Key Laboratory of Intelligent Technology for Chemical Process Industry

May 12, 2026J/K move · Enter open · S save

LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection

Lanxin Zhao, Bamdev Mishra, Pratik Jawanpuria, Lequan Lin+3

The University of Sydney, Australia. · Microsoft India. · Indian Institute of Technology Bombay, India.+1

Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Orthogonal parameter-efficient fine-tuning (PEFT) adapts pretrained weights through structure-preserving multiplicative transformations, but existing methods often conflate two distinct design choices: the subspace in which adaptation occurs and the transformation applied within that subspace. This paper introduces LOFT, a low-rank orthogonal fine-tuning framework that explicitly separates these two components. By viewing orthogonal adaptation as a multiplicative subspace rotation, LOFT provides a unified formulation that recovers representative orthogonal PEFT methods, including coordinate-, butterfly-, Householder-, and principal-subspace-based variants. More importantly, this perspective exposes support selection as a central design axis rather than a byproduct of a particular parameterization. We develop a first-order analysis showing that useful adaptation supports should be informed by the downstream training signal, motivating practical task-aware support selection strategies. Across language understanding, visual transfer, mathematical reasoning, and multilingual out-of-distribution adaptation, LOFT recovers principal-subspace orthogonal adaptation while gradient-informed supports improve the efficiency-performance trade-off under matched parameter, memory, and compute budgets. These results suggest that principled support selection is an important direction for improving orthogonal PEFT.