Low-Rank Adaptation
Also known as LoRA
Overview
Low-rank adaptation studies how to specialize pretrained models by representing weight changes as products of small matrices while keeping most backbone weights frozen. Methods choose where adapters are placed and how their rank is allocated, using approaches such as spectral or gradient-based initialization, adaptive rank selection, and quantized fine-tuning. For multiple tasks or users, adapter routing, expert mixtures, merging, and federated aggregation aim to share useful updates without interference. The central tension is how to gain enough capacity for new tasks while controlling memory and compute, preserving existing capabilities, and generalizing across data shifts; the best rank, placement, and composition strategy depend on the model and task.
Latest papers
581 papers58 papers in the last four weeks, down 36% on the four weeks before. 0.4% of all new papers.
Doc2LoRA Provides Decodable Representations of Scientific Ideas
FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning
LoRA Direction Extraction for Controllable Light Toggling in FLUX.1 Kontext
Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention
Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
G-LoRA: Organizing Reward-Weighted Video Data with Gradient-Guided Grouped LoRA
Style-Driven Data Synthesis and Degradation-Aware Enhancement for Ultrasound Image Restoration
Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning
Rotated Manifold Optimization for Low-Rank Adaptation
Loop Dropout: Regularizing Shared Updates in Looped Language Models
Resource-Aware Parameter-Efficient Model Adaptation for Onboard High-Dimensional Data
CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
CFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates CFLoRA achieves convergence rate of the original LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that CFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.Stabilizing the Dynamic Low-Rank Training
What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignment
When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation
Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancing
What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation
New LoRA Skills Should Read but Never Write
JevSoup: System-One Routing for Training-Free LoRA Composition
Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models
LoRA Adaptation Strength in Aurora-WRF: Trade-offs in Regional Heavy-Rainfall Forecasting
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization
LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models
Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRA
LoRA-generating hypernetworks for efficient on-device LLM generative personalization
LLMs'), e.g. running on mobile phones, are ripe for improvement via personalization. The limited compute resources of mobile devices impose limits on model scale and thus model quality, making any realizable quality gains highly impactful. At the same time, their personal nature (i.e., the close coupling to a particular user) means that a given on-device LLM tends to be used in similar, predictable patterns over the course of time. This paper presents a novel method for personalizing on-device LLMs. It trains a hypernetwork to map a user's context tokens to a low-rank adaptation (LoRA') well-suited to that user. Once the trained common artifacts are deployed to users' devices, each user uses the hypernetwork to synthesize (entirely on device) a personalized LoRA. This approach blends the benefits while avoiding the drawbacks of two existing approaches to LLM customization: in-context learning (ICL') and parameter-efficient fine-tuning (PEFT'). Like ICL (and unlike PEFT), the on-device phase of our approach is computationally feasible, requiring only forward passes through neural networks. Like PEFT (and unlike ICL), our approach modifies the `target' base LLM via weights (the LoRA), avoiding negative consequences (e.g. increased latency) associated with extending the input sequence. Our approach is particularly well-suited to the mobile device regime. Apart from the on-device compute and latency benefits mentioned, it also requires minimal additional storage, as internally its architecture partly leverages the same LLM weights as belong to the target LLM to be personalized. We demonstrate the benefits of LoRA-generating hypernetworks on several representative personalization datasets, comparing against baselines like ICL and PEFT. Of note, our personalization experiments focus on more challenging and less studied long-form text generation tasks.