Low-Rank Adaptation

Also known as LoRA

Low-rank adaptation studies how to specialize pretrained models by representing weight changes as products of small matrices while keeping most backbone weights frozen. Methods choose where adapters are placed and how their rank is allocated, using approaches such as spectral or gradient-based initialization, adaptive rank selection, and quantized fine-tuning. For multiple tasks or users, adapter routing, expert mixtures, merging, and federated aggregation aim to share useful updates without interference. The central tension is how to gain enough capacity for new tasks while controlling memory and compute, preserving existing capabilities, and generalizing across data shifts; the best rank, placement, and composition strategy depend on the model and task.

Latest papers

542 papers92 papers in the last four weeks, up 59% on the four weeks before. 0.5% of all new papers.

A LoRA is Worth a Thousand Pictures
Dec 16, 2024Chenxi Liu, Towaki Takikawa, Alec JacobsonWeight-Space Analysis Low-Rank Adaptation
University of Toronto · Outerport · Adobe Research
BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models
Aug 8, 2024Yupeng Chang, Yi Chang, Yuan WuSocial Bias in Language Models Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University · International Center of Future Science, Jilin University

Dec 16, 2024J/K move · Enter open · S save

A LoRA is Worth a Thousand Pictures

Chenxi Liu, Towaki Takikawa, Alec Jacobson

University of Toronto · Outerport · Adobe Research

Weight-Space Analysis Low-Rank Adaptation Diffusion Model Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning of image diffusion models, with over 200K models trained to replicate artistic styles and shared in online communities. One way to analyze styles of LoRA models is to generate image samples and apply image-based features like CLIP. However, this generation requires additional computation and prompt selection. In this work, we propose analyzing styles directly in the LoRA weight space, bypassing these limitations while achieving better performance even on unseen LoRAs. Since in-the-wild LoRAs lack ground truth model details, we design an experimental setup and show that LoRA weights alone serve as effective feature representations of adapted styles, robust to variations in fine-tuning seeds and image selection. We extensively evaluate using SD1.5 (the most popular base model within online communities) across data domains (fine-art paintings, architectural photos) and diverse prompt settings, with a small-scale SDXL experiment further verifying generalization. These evaluations confirm that our approach, under the selected aspects of variability, preserves fine-grained styles and outperforms image-based methods in clustering and retrieval of unseen LoRAs.

Latest papers

A LoRA is Worth a Thousand Pictures

BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models