cs.LGJun 4, 2026

High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model

Authors: O. DuranthonF. BoncoraglioL. Zdeborová

Organizations: Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne

Abstract

We develop a high-dimensional statistical theory of low-rank adaptation (LoRA) in attention models, capturing the interplay between pre-training and fine-tuning. We introduce a solvable framework in which a single-head attention layer is first pre-trained on a data-abundant task and subsequently adapted via a rank-one LoRA update on limited data. In the high-dimensional limit, both stages admit a sharp asymptotic characterization in terms of a finite set of order parameters, yielding explicit predictions for test errors and representation alignment. Our analysis shows that the impact of pre-training on LoRA is summarized by an effective noise term, from which we derive prescriptions for the optimal pre-training procedure. We also demonstrate a regime with a mismatch between the value of the test error and representation quality, and propose an application of our theory to active fine-tuning.

Explore similar work

CardsList
  1. LoRA vs. Full Fine-Tuning: A Theoretical Perspective

    May 18, 2026Ali Zindari, Rotem Mulayoff, Sebastian U. StichLow-Rank AdaptationFinetuning

  2. Low-Rank Adaptation Redux for Large Models

    Apr 23, 2026Bingcong Li, Yilang Zhang, Georgios B. GiannakisLow-Rank AdaptationLarge Models

  3. Beyond LoRA: Is Sparsity-Induced Adaptation Better?

    Jun 11, 2026Elijah Cadenhead, Cristian McGee, Xin Li +2Low-Rank Adaptation