Low-Rank Adaptation
Also known as LoRA
Overview
Low-rank adaptation studies how to specialize pretrained models by representing weight changes as products of small matrices while keeping most backbone weights frozen. Methods choose where adapters are placed and how their rank is allocated, using approaches such as spectral or gradient-based initialization, adaptive rank selection, and quantized fine-tuning. For multiple tasks or users, adapter routing, expert mixtures, merging, and federated aggregation aim to share useful updates without interference. The central tension is how to gain enough capacity for new tasks while controlling memory and compute, preserving existing capabilities, and generalizing across data shifts; the best rank, placement, and composition strategy depend on the model and task.
Latest papers
581 papers58 papers in the last four weeks, down 36% on the four weeks before. 0.4% of all new papers.
SkillFormer: Skill-Decomposed Adaptation for Audio Language Models
RACER: Residual-Adaptive Closed-Loop Estimation for Sampling-Based Planning in Wheeled-Quadruped Racing
Conditional Rank Allocation for Taxonomy-Aware Medical Language Model Adaptation
Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA
A Fine-Grained Analysis of the LoRA Fine-Tuning Landscape with Implications for Data Selection
Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating
JIVEAdapter: A Multi-Task Additive Low-Rank Adapter via Joint and Individual Variation Explained (JIVE)
Robust Parameter-Efficient LLM Adaptation on Analog Hardware
LoRA's Second Descent Extends Beyond Parameter Parity
Continual Humanoid Motion Learning
PaLoRA: Paced Low-Rank Adaptation for Continual Learning
Predicting Steering Vectors and Adapter Weights for Few-Shot Author-Style Transfer
Query-aware routing for Cross-lingual performance gains in Encoders
Does Every User Need a Private LoRA? Decoupling Personalization from Per-User Adaptation
Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA
Learning to Predict Distributions over Weight Updates for Test-Time Adaptation
Task-Oriented Rank Adaptation for Continual Learning in Text Classification
MLoRA:Continual Low-Rank Adaptation via Relative-Response Subspace Selection and Momentum Orthogonalization
MLoRA, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.