A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges

Xiaoyu Li; Zhizhou Sha; Chiwun Yang; Dai Shi

cs.LGOct 8, 2026

A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges

Authors: Xiaoyu Li, Zhizhou Sha, Chiwun Yang, Dai Shi

Organizations: University of New South Wales · University of Texas at Austin · City University of Hong Kong · University of Cambridge

Abstract

Parameter-efficient fine-tuning has hundreds of methods, usually compared by parameter count and benchmark score. We compare them as maps. A pretrained checkpoint is a base point in weight space, and a weight-space method is a smooth map from a small parameter space that starts at that point. These maps form a slice category in which the frozen model is initial and full fine-tuning terminal. Three invariants of the map do most of the work. Its image near the base point governs expressivity, through the first-order image, the dimension, a min-cut bound on rank and whether that point is smooth or an apex. Its fibres, with the optimizer's metric, govern training dynamics. Base change moves the base point and covers splitting initialisations, quantisation, restarts and merging. We prove, for example, that zero-initialised LoRA, and Householder reflection adaptation at its default start under stated rank and parity conditions, begin at the apex of a cone; that no exact initialisation lets LoRA reach an update of rank above twice its own; that exactly orthogonal methods never change a weight's singular values in exact arithmetic, however often they are merged; that under Adam with no epsilon, weight decay or clipping and one shared schedule, LoRA+ from the standard zero initialisation follows exactly the trajectory of plain LoRA with its scale multiplied by the learning-rate ratio; that Adam with shared hyperparameters respects only the signed permutations inside LoRA's continuous symmetry group of full-rank factors; and that the activation function decides when a scaling adapter folds into preceding weights. We classify 382 methods from 138 papers by seven coordinates read off their formulas, with 403 simulation arrows and 485 obstructions. Every numbered result is proved in full, and results due to others are credited where we re-derive them.

Figures & tables

Figure 1: Fine-tuning moves a point. Weight space

\Theta

is drawn as a chart around the pretrained point

\theta_{0}

; the grid is the lattice of a round-to-nearest quantiser

\kappa

through the zero weight

0

(marked

+

). Solid curves are image germs of methods pointed at

\theta_{0}

( Definition I.2 ), drawn darkest near

\theta_{0}

because only the germ there matters. LoRA ( Hu et al., 2021 ) is the cone

\theta_{0}+\mathcal{M}_{\leq r}

with its apex at

\theta_{0}

( Theorem II.2 ), shown as the rank-

\leq 1

double cone

x^{2}=y^{2}+z^{2}

of the symmetric

2\times 2

slice in perspective. OFT ( Qiu et al., 2023 ) is the orbit of

\theta_{0}

under rotations, an arc of the circle about

0

through

\theta_{0}

, the rest of which is dotted. (IA) 3 ( Liu et al., 2022a ) is a torus orbit, whose closure is the line through

0

and

\theta_{0}

( Theorem II.5 ). BitFit ( Ben-Zaken et al., 2021 ) moves along a coordinate line, parallel to the grid. The dashed curves are not pointed at

\theta_{0}

. Prefix tuning ( Li and Liang, 2021 ) extends the network and is unpointed ( Definition VI.1 ); drawn in weight space it is a curve that misses

\theta_{0}

, at the distance marked by the dotted segment. QLoRA ( Dettmers et al., 2023 ) is LoRA pointed at the nearest grid node

\kappa\theta_{0}

, so its cone is LoRA’s cone translated by the defect

e=\kappa\theta_{0}-\theta_{0}

(red arrow), and

\theta_{0}

is off its image unless

\rk e\leq r

( Proposition V.2 ).

Label	What it means here
Theorem, Proposition, Lemma, Corollary	Proved in this paper, with the complete proof in Appendix C ; the statement is set in italic. A heading that says known and names authors marks a result due to them, which we re-derive in our language, adding only its reading for fine-tuning.
Observation	True and formally easy; kept because it organises.
Definition	A definition.
Remark, Example	Commentary and instances, in upright type.
Conjecture, [Conj]	A conjecture.
Prediction, [Pred]	A falsifiable prediction, stated with its optimizer scope.

Table 1: How to read the labels. The coloured rule is the one that marks the environment in the text.

Symbol	Meaning
$W$	A linear box $y=Wx$ , $W\in\mathbb{R}^{m\times n}$ , with $m=d_{\mathrm{out}}$ and $n=d_{\mathrm{in}}$ . Row $i$ is output neuron $i$ .
$\Theta$	Weight space, a real vector space of adapted and frozen boxes.
$\theta_{0}$ , $W_{0}$	The pretrained point; $W_{0}$ for one box.
$\rho$ , $(Q,q_{0})$	The method, a pointed smooth map from trainable parameters $Q$ started at $q_{0}$ , with $\rho(q_{0})=\theta_{0}$ .
$\mathbf{Meth}(\Theta,\theta_{0})$	The pointed slice $\mathbf{Diff}_{*}/(\Theta,\theta_{0})$ , whose objects are methods $M=(Q,q_{0},\rho)$ . A morphism $h:M\to N$ is a pointed smooth map with $\rho_{N}\circ h=\rho_{M}$ , read “ $N$ simulates $M$ ”.
$\lvert M\rvert$	The budget, $\dim Q$ .

Table 2: Notation.

Panel of Figure 2 — Figure 2: The pointed slice $\mathbf{Meth}(\Theta,\theta_{0})$ . (a) The interval of weight-space methods. For a method $M$ , the first arrow picks the initialisation $q_{0}$ ; it lies over $\Theta$ because $M$ starts at $\theta_{0}$ . The second arrow is $\rho_{M}$ itself, the merge into the base weights, and the composite is the inclusion of $\theta_{0}$ . The frozen model is initial and full fine-tuning terminal ( Observation I.4 ). A pointing defect, $\rho_{M}(q_{0})\neq\theta_{0}$ , leaves the triangle commuting only up to that defect (Exposé V ). Adapters and prefixes fall outside the interval because they extend the network ( Definition VI.1 ). (b) A simulation is a pointed smooth map $h$ that makes the triangle over $(\Theta,\theta_{0})$ commute; a weak simulation drops $h(q_{0})=q_{0}^{\prime}$ . (c) An instance: LoRA pointed at $(0,A_{0})$ simulates LoRA-FA through $h(B)=(B,A_{0})$ .

In PEFT	In the pointed slice	Statement
pretrained checkpoint	base point $\theta_{0}$ of a pointed 1-cell $(\Theta,\theta_{0},f)$	Definition I.1
a weight-space PEFT method	an object $\rho:(Q,q_{0})\to(\Theta,\theta_{0})$ of $\mathbf{Meth}(\Theta,\theta_{0})$	Definition I.2
“ $N$ can do whatever $M$ can”	a morphism $h:M\to N$ , $\rho_{N}\circ h=\rho_{M}$ ; at the level of sets, image inclusion	Definition I.2 , Observation I.4 (c)
frozen model, full fine-tuning	initial and terminal objects	Observation I.4 (a)
merging into full-precision weights	the canonical morphism $\rho_{M}$ to the terminal object	Observation I.4 (a)
effective number of parameters	expressive dimension $d(M)$ ; the rest is gauge waste	Definition I.5

Table 3: The dictionary for Exposé I .

Figure 3: The apex and the smooth point. Panels (a) and (b) draw the symmetric slice

S=\left(\begin{smallmatrix}x+y&z\\ z&x-y\end{smallmatrix}\right)

2\times 2

matrices, in which rank

\leq 1

is the grey double cone

x^{2}=y^{2}+z^{2}

, and the model

\rho(c,\phi)=W_{\rm res}+c\,v_{\phi}v_{\phi}^{\top}

v_{\phi}=(\cos\phi,\sin\phi)

, which plays LoRA (

c

for

B

v_{\phi}

for

A

); its image is the whole cone, so

d=2

. (a) Zero initialisation,

c_{0}=0

and

W_{\rm res}=\theta_{0}

: the base point

\theta_{0}

is the vertex of the cone, and the first-order image

S_{1}

(teal) is a single line, one ruling, so the deficit is

1

and

\theta_{0}

is an apex ( Definition I.5 , Theorem II.2 (c)). (b) A split (PiSSA-type) initialisation,

c_{0}\neq 0

and

W_{\rm res}=\theta_{0}-P

with

P=c_{0}v_{\phi_{0}}v_{\phi_{0}}^{\top}

: the vertex moves to

\theta_{0}-P

(open diamond),

\theta_{0}

lies on a smooth sheet of the cone, and

S_{1}

is the tangent plane there, so the deficit is

0

. Both values of

\dim S_{1}

are Jacobian ranks of the model. (c) For

m=n=4096

and

r=16

, each pair of bars compares the expressive dimension

d(M)

(grey) with

\dim S_{1}

(teal); the dashed red box is the gap between them, the first-order deficit. Zero-init LoRA, stratum

\mathrm{T}_{A}

, has

\dim S_{1}=mr=65{,}536

against

d=r(m+n-r)=130{,}816

; a split initialisation has

\dim S_{1}=d

( Theorem II.2 (b)); HRA’s paired initialisation

u_{2i-1}=u_{2i}

k=r/2

, has

\dim S_{1}=kn-\tfrac{k(k+1)}{2}=32{,}732

against

d=rn-\tfrac{r(r+1)}{2}=65{,}400

, so it too starts at an apex ( Theorem II.7 (b)). The script checks each closed form against a Jacobian rank at

(m,n,r)=(11,9,4)

scheme	stratum	point
vanilla LoRA ( Hu et al., 2021 ) (Linear, Conv)	$\mathrm{T}_{A}$	a random point of $\Gr(r,n)$ : Haar ( Background D.14 ) under Gaussian $A_{0}$ , only approximately Haar under Kaiming-uniform
vanilla LoRA on nn.Embedding (loralib and HF default)	$\mathrm{T}_{B}$	$A_{0}=0$ , $B_{0}\sim\mathcal{N}(0,1)$ in the convention $y=Wx$ with $m$ the embedding dimension; deficit $r(m-r)$
EVA ( Paischer et al., 2024 )	$\mathrm{T}_{A}$	the top- $r$ principal subspace of the input activations, at its per-layer rank
LoRA-FA ( Zhang et al., 2023d )	sub-object of $\mathrm{T}_{A}$	the same point with $A$ frozen (an arrow $\mathrm{FA}(A_{0})\to\mathrm{LoRA}_{(0,A_{0})}$ )
“Init[B]” ( Hayou et al., 2024a )	$\mathrm{T}_{B}$	a random point of $\Gr(r,m)$
MiCA ( Rüdiger and Raschka, 2026 )	sub-object of $\mathrm{T}_{B}$	$B_{0}$ the minor left-singular frame of $W_{0}$ , frozen: $\mathrm{FB}(B_{0})\to\mathrm{LoRA}_{(B_{0},0)}$ , a linear image at a regular point (HF also allows $r=\min(m,n)$ , outside the theorem)

Table 4: LoRA initialisation schemes as points of the strata of Theorem II.2 , with the sub-objects and the exceptions. Conventions:

y=Wx

W\in\mathbb{R}^{m\times n}

; HF is Hugging Face PEFT.

pipeline	examples	defect $\rho(q_{0})-W$
quantise, then zero-init	QLoRA ( Dettmers et al., 2023 )	$e$
split, then quantise the residual	QPiSSA ( Meng et al., 2024a ) ; HF’s recommended PiSSA/CorDA workflow; LoftQ’s later quantisation steps	$\kappa(W_{\rm res})-W_{\rm res}$
quantise, then fit the error	LoftQ ( Li et al., 2023b ) with one iteration (HF default loftq_iter=1); HF replace_lora_weights_loftq	$e-\mathrm{SVD}_{r}(e)$ , the part of $e$ left after its best rank- $r$ approximation (unit LoRA scale)
quantise, split, re-quantise	HF olora_init on bitsandbytes weights	$e+\kappa(r^{\prime})-r^{\prime}$ , with $r^{\prime}=\kappa W-\delta_{0}$

Table 5: Pointing defects of quantised-base LoRA pipelines ( Proposition V.2 (b)).

action	group	complete invariant	methods that live here
$W\mapsto WR$ , input side	$\mathrm{O}(n)$	$K_{\mathrm{out}}=WW^{\top}$	exact-Cayley OFT and block OFT, HRA, and GOFT and ETHER as published
$W\mapsto PW$ , output side	$\mathrm{O}(m)$	$K_{\mathrm{in}}=W^{\top}W$	LyCORIS diag-OFT; HF PEFT OFT up to version 0.13
$W\mapsto PWR$	$\mathrm{O}(m)\times\mathrm{O}(n)$	singular values with multiplicity	any composite of the two rows above
$W\mapsto DW$	$T_{m}^{+}$ , $T_{m}$	row rays; zero rows and row lines	(IA) 3 on $k,v$ , SSF’s scale, DoRA’s magnitude (closure only)
$W\mapsto E\odot W$	$(\mathbb{R}^{\times})^{m\times n}$	zero pattern	HiRA (one way only: $Z(W_{0})\subseteq Z(W)$ )

Table 6: The five basic actions, their complete invariants ( Theorem II.5 ) and the methods whose images lie in one orbit or orbit closure.

implementation	map	exactly orthogonal?	what it keeps
BOFT in HF PEFT; HF PEFT OFT 0.14–0.15	$\diag(s)\,W_{0}R^{\top}$ , $R$ exactly orthogonal, $s$ free, initialised at $1$	only $R$	$K_{\mathrm{out}}\mapsto\diag(s)K_{\mathrm{out}}\diag(s)$ , and block Grams likewise. $\lvert\cos_{ij}\rvert$ survives when $s_{i}s_{j}\neq 0$ , $\cos_{ij}$ when $s_{i}s_{j}>0$ . At HF’s default single butterfly factor, BOFT is block OFT times $s$ .
GOFT, official code default	$W_{0}\,\diag(g)\,G$ , with a trainable input scale $g$	no	Nothing on this list, since $K_{\mathrm{out}}=W_{0}\diag(g)^{2}W_{0}^{\top}$ .
output-side OFT (LyCORIS diag-OFT at its defaults; HF PEFT OFT up to 0.13)	image in $\mathrm{O}(m)\cdot W_{0}$	yes, on the other side	$K_{\mathrm{in}}$ , the partial input Grams and the spectrum. Not $K_{\mathrm{out}}$ .
HF PEFT default OFT since 0.18 (use_cayley_neumann=True)	$R=I+2Q+2Q^{2}+2Q^{3}+Q^{4}=C(Q)(I-Q^{4})$ , with $C$ an exact Cayley map	no, $R^{\top}R=(I-Q^{4})^{2}$	Nothing exactly. The adapted weight already has $K_{\mathrm{out}}=W_{0}(I-Q^{4})^{2}W_{0}^{\top}$ , and for small $Q$ each merge shrinks $K_{\mathrm{out}}$ and the spectrum further. (In 0.16–0.17, $R^{\top}R=I+4Q^{6}+4Q^{8}$ .)
qGOFT	free $2\times 2$ blocks with a soft orthogonality penalty	no	Neither $K_{\mathrm{out}}$ nor the cosines. Once $n>2$ , one non-orthogonal block, conformal or not, generically moves a cosine.
ETHER+	two-sided, $I-uu^{\top}+vv^{\top}$ , singular when $u\perp v$	no	Neither $K_{\mathrm{out}}$ nor the cosines.

Table 7: Implementations that look orthogonal, read against Corollary II.6 . Each row is a separate computation (see the remarks after the corollary).

Figure 4: Rank is not volume ( Theorem II.10 , Proposition II.12 ). One

4096\times 4096

layer at the budget of LoRA 8 ( Hu et al., 2021 ) ,

\lvert M\rvert=8(m+n)=65{,}536

. Each method is placed by the largest rank of

\Delta W

it can reach (horizontal, logarithmic scale), which is bounded by the narrowest cut of its diagram ( Theorem II.10 ), and by its expressive dimension as a fraction of the layer,

d(M)/(mn)

(vertical, in units of

10^{-3}

, with the axis broken between

3.41

and

3.90

), from the gauge census of Proposition II.12 . The dashed line is

d=\lvert M\rvert

, a method without gauge, and the dotted line is

d=\lvert M\rvert-8^{2}=65{,}472

, the dimension of LoRA 8 with its

\mathrm{GL}_{8}

gauge. Along the dotted line the maximum rank runs from

8

for LoRA 8 , through GraLoRA ( Jung et al., 2025 ) with

k=2

4

and

8

blocks (the last two light), to full rank for HiRA 8 ( Huang et al., 2025b ) . MoRA ( Jiang et al., 2024 ) and C3A ( Chen et al., 2024a ) have no gauge and reach full rank; their points coincide and are drawn on either side of the full-rank line. FourierFT ( Gao et al., 2024 ) loses one dimension for each sampled conjugate pair, and its maximum rank is known only through a bound (open marker). KronA ( Edalati et al., 2022 ) is shown at the factor shapes closest to the budget. LoHa 4,4 ( Hyeon-Woo et al., 2021 ) , below the break, sits at the proved upper bound of its dimension: it reaches rank

16

with at least

8{,}159

fewer dimensions than LoRA 8 ( Proposition II.12 (c)). The parameters behind each point are in § E.6 .

Panel of Figure 5 — Figure 5: LoRA as a plug. (a) Cutting $W_{0}$ out of a linear layer $y=W_{0}x$ leaves a hole $V\to U$ , with $V=\mathbb{R}^{n}$ and $U=\mathbb{R}^{m}$ . (b) LoRA fills the hole with the frozen box $W_{0}$ (grey) plus the trainable chain $sBA$ (teal), a formal sum, over which rank is subadditive. The dashed cut puts $A$ on the input side and $B$ on the output side and crosses one wire of dimension $r$ , so $\rk(sBA)\leq r$ ( Theorem II.10 ). The dotted cuts put both boxes on one side and cross an open leg, with weights $n$ and $m$ ; the minimum over cuts is $\min(n,r,m)$ .

method	presentation used	bound on $\rk\Delta W$
LoRA family, $BA$ ( Hu et al., 2021 )	one bond of dimension $r$	$r$
split initialisations (PiSSA type), relative to $W_{0}$	$\Delta W=BA-B_{0}A_{0}$ , two bonds	$2r$
HRA ( Yuan et al., 2024 )	telescoping sum $R-I=\sum_{k}(H_{k}-I)H_{k+1}\cdots H_{r}$	$r$
Givens circuit with $g$ gates ( Ma et al., 2024 )	the same telescoping sum	$2g$
ReLoRA after $K$ phases ( Lialin et al., 2023b )	formal sum of $K$ LoRAs	$Kr$
GraLoRA ( Jung et al., 2025 )	sum of $k^{2}$ block LoRAs	$kr$

Table 8: Cut bounds of Theorem II.10 on

\rk\Delta W

for one

m\times n

box, with the presentation each bound uses.

Figure 6: Cut the tensor ( Theorem II.11 ). A

16\times 16

matrix

\Delta

is read as a four-leg tensor with output legs

U_{1},U_{2}

and input legs

V_{1},V_{2}

, through

\mathbb{R}^{16}=\mathbb{R}^{4}\otimes\mathbb{R}^{4}

on both sides. The legs can be split into two pairs in three ways (columns): the ordinary cut

\{U_{1}U_{2}\}\mid\{V_{1}V_{2}\}

, across which LoRA factors as

\Delta=BA

; the Kronecker cut

\{U_{1}V_{1}\}\mid\{U_{2}V_{2}\}

, across which a sum of Kronecker products

\sum_{i}C_{i}\otimes D_{i}

factors after the rearrangement of Van Loan and Pitsianis ( 1993 ) ; and the third cut

\{U_{1}V_{2}\}\mid\{U_{2}V_{1}\}

, the Kronecker cut with the two input legs swapped. The glyph above each column draws that factorisation as two boxes joined by a bond across the cut. Each panel shows the operator-Schmidt spectrum across one cut, the singular values

\sigma_{i}

of the corresponding

16\times 16

reshape of

\Delta

divided by

\lVert\Delta\rVert_{F}

, on a logarithmic scale, for (a) a random rank-one update

uv^{\top}

, (b) the identity

I_{16}=I_{4}\otimes I_{4}

, and (c)

C\otimes D

with random

4\times 4

factors. Filled markers lie above the rank threshold (dashed line), and open markers are numerically zero. The ranks are

(1,16,16)

for

uv^{\top}

and

(16,1,16)

for

I_{16}

and for

C\otimes D

. So a generic rank-one update has the maximal Kronecker rank

\kappa_{1}=16

, and a Kronecker product of full-rank factors has the maximal rank

\nu=16

, which makes zero-init LoRA r and

k

-term Kronecker sums incomparable for

r,k<16

( Theorem II.11 (b)–(d)). LoRA 1 fits

uv^{\top}

with

32

parameters where a Kronecker sum needs

16

terms and

512

parameters, and for

I_{16}

and

C\otimes D

the two costs trade places.

method	budget $\lvert M\rvert$	exhibited Lie gauge (orbit dim)	$d(M)$	max rank
LoRA r	$r(m+n)$	$\mathrm{GL}_{r}$ ( $r^{2}$ )	$r(m+n-r)$	$r$
GraLoRA ( $k\times k$ blocks, rank $r/k$ each; HF default hybrid_r=0)	$r(m+n)$	$\prod\mathrm{GL}_{r/k}$ ( $r^{2}$ )	$r(m+n-r)$	$\min(kr,m,n)$
LoHa ${}_{r_{1},r_{2}}$	$(r_{1}+r_{2})(m+n)$	$\mathrm{GL}_{r_{1}}\times\mathrm{GL}_{r_{2}}\times{}$ diagonal torus ( $r_{1}^{2}+r_{2}^{2}+m+n-1$ )	$\min(mn,F)$ [Num]; $F$ is a proved upper bound	$\min(r_{1}r_{2},m,n)$
HRA r ( $\rk W_{0}=n$ , apply_GS=False; even $r$ is needed only for HF’s identity init)	$rn$	non-linear; fibre dimension $r(r+1)/2$ ; exhibited Lie gauge $(\mathbb{R}^{\times})^{r}$	$rn-r(r+1)/2$ (for $r\in\{n-1,n\}$ this is $\dim\mathrm{SO}(n)$ [Num])	$r$
DoRA r	$r(m+n)+m$	$\mathrm{GL}_{r}$ for generic $W_{0}$ with $(m-r)(n-r)\geq m$ ; larger otherwise (dimension $6>r^{2}$ at $(m,n,r)=(3,3,2)$ , and $r^{2}+r$ for rank-one $W_{0}$ )	$\min\bigl(mn,\,r(m+n-r)+m\bigr)$ for generic $W_{0}$ ( Proposition II.8 )	up to full
VeRA	$m+r$	$\mathbb{R}^{\times}$	$m+r-1$ [Num]	$\min(r,m,n)$

Table 9: The gauge census of Proposition II.12 (b) for one

m\times n

box: budget, exhibited Lie gauge with the dimension of its generic orbit, expressive dimension and maximum rank. [Num] marks values checked as float64 Jacobian ranks (Appendix E );

F

is defined in Proposition II.12 (b).

method	$d(M)$	fibre $\lvert M\rvert-d(M)$	max rank	block structure
LoRA 16	$130{,}816$	$256$	$16$	none
GraLoRA, $k=2$	$130{,}816$	$256$	$32$	each of $4$ blocks has rank $\leq 8$
LoHa 8,8	$\leq 122{,}753$	$\geq 8{,}319$	$64$	none

Table 10: One

4096\times 4096

box and one budget,

\lvert M\rvert=131{,}072

, spent three ways ( Proposition II.12 ).

gauge	satisfy the hypothesis (reachable functions unchanged)	fail the sufficient condition
single global residual rotation (SliceGPT style)	LoRA, full FT, BitFit, PiSSA; (IA) 3 with HF defaults, which scales the output side of $k$ (or $q$ , for Qwen, Gemma and Phi) and $v$ and the input side of down, while the rotation acts on the other side, where (IA) 3 is $\mathrm{GL}$ -covariant; DoRA on $q$ , $k$ , $v$ , gate and up; OFT on $o$ and down (no scale); LoHa 1 , whose image is $W_{0}+\mathcal{M}_{\leq 1}$	block OFT and BOFT on residual-input projections (the block normaliser contains no Hadamard or PCA rotation); BOFT on $o$ and down (its output scale boft_s); DoRA on $o$ and down; LoHa with $r\geq 2$ ; HiRA; coordinate masks; FourierFT
query–key and value–output head gauges	LoRA	(IA) 3 on $q$ or $k$ and on $v$ ; OFT on $o$

Table 11: Methods against the sufficient condition of Observation II.15 . For the residual rotation the comparison is between the gain-fused model and the fused-and-rotated model.

Method (gradient flow or SGD)	Cometric $K(G)$
LoRA ( Hu et al., 2021 )	$s^{2}(\eta_{B}GA^{\top}A+\eta_{A}BB^{\top}G)$
LoRA-FA, original / corrected ( Zhang et al., 2023d )	$s^{2}\eta_{B}GA_{0}^{\top}A_{0}$ / $\eta GP_{V}$
scaled GD, undamped Riemannian preconditioning ( Tong et al., 2021 ; Zhang and Pilanci, 2024a )	$\eta s^{2}(P_{U}G+GP_{V})$
LoRA-Pro, SGD variant ( Wang et al., 2024i )	$\eta\,\Pi_{T\mathcal{M}_{r}}G$
GaLore, one period, scale $\alpha$ ( Zhao et al., 2024a ) ; Proposition IV.2	$\eta\alpha PP^{\top}G$ , or $\eta\alpha\,GQQ^{\top}$ when $m>n$ (default $\alpha=0.25$ )
GaLore with a random projector $A_{t}$ , one period (the compressed SGD of Hao et al., 2024 )	$\eta GA_{t}^{\top}A_{t}$

Table 12: Induced weight-space cometrics under gradient flow or SGD ( Observation III.2 , Proposition III.4 ). The last three rows are full-parameter methods, listed for comparison;

P

and

Q

are GaLore’s left and right projectors,

A_{t}

a random projector,

z

the random direction of MeZO.

Figure 7: Noether hyperbolas and the crossover ( Theorem III.5 , Corollary III.6 , Theorem III.9 ). (a) Scalar LoRA,

w=w_{0}+s\,ba

with

s=1

, loss

\tfrac{1}{2}(w-w^{\star})^{2}

w^{\star}-w_{0}=1

and rates

\eta_{A}=1

\eta_{B}=2

. Top: six gradient-flow trajectories (teal) run from the filled dots along their own charge hyperbolas

b^{2}/\eta_{B}-a^{2}/\eta_{A}=\Phi_{0}

(dashed grey; the slice

\Phi=0

dashed green) to the fibre

s\,ab=w^{\star}-w_{0}

(ochre). From the zero-

B

start

(1.2,0)

, Adam (red) leaves its hyperbola and lands at another point of the fibre. Bottom: the drift

\max_{t\leq 10}\lvert\Phi(t)-\Phi_{0}\rvert

from the same start against the step size

\eta

(learning rates

\eta\eta_{A}

\eta\eta_{B}

). A GD step changes the charge at order

\eta^{2}

, so over a fixed time the GD drift is proportional to

\eta

and vanishes in the flow limit. An Adam step changes the charge at first order in

\eta

, and its drift stays of order one at every step size: Adam lies outside Theorem III.5 ( Corollary III.6 ). (b) Matrix LoRA with

m=n=6

r=2

, loss

\tfrac{1}{2}\lVert W-W^{\star}\rVert_{F}^{2}

B_{0}=0

and

A_{0}A_{0}^{\top}=I

exactly, so that

\Phi=-\kappa I

with

\kappa=1

and the crossover lies at

\sigma^{*}=\mu\lvert\kappa\rvert/2=0.5

(dashed ochre). Top: the closed-form gains

g_{\kappa}(\sigma^{2}/\mu^{2})

(teal, the

B

-update) and

f_{\kappa}(\sigma^{2}/\mu^{2})

(purple, the

A

-update) of Theorem III.9 , with the plateau

\lvert\kappa\rvert

and the balanced asymptote

\sigma/\mu

dotted. The hollow circles, read off the factors along the integrated flow for both singular directions, lie on the curves. Shading marks the one-sided regime, the crossover, and the balanced regime (grey). Bottom: the singular values

\sigma_{1}(t)

and

\sigma_{2}(t)

\Delta W

, with time running downward on a log scale.

\sigma_{1}

crosses

\sigma^{*}

and settles in the balanced regime;

\sigma_{2}

stays below

\sigma^{*}

, where the

A

-update supplies only

7\%

of its growth rate, as in LoRA-FA. Integrators and tolerances are in § E.6 .

Figure 8: LoRA+ is

\alpha

in disguise ( Theorem III.10 , Corollary III.11 ). A seeded toy LoRA regression with

B_{0}=0

is trained by Adam in float64, once as LoRA+ with ratio

\lambda=16

(scale

s

\eta_{B}=16\,\eta_{A}

) and once as plain LoRA with

\alpha

multiplied by

16

(scale

16s

\eta_{B}=\eta_{A}

), at the same

\eta_{A}

and

A_{0}

. (a) At

\varepsilon=0

, without weight decay or clipping, the two loss curves coincide; the control, plain LoRA at the uncompensated scale

s

, ends far above them. (b) The weight gap

\lVert W_{t}-W^{\prime}_{t}\rVert_{F}

between the two runs at every step, on a log scale above an exact-zero floor. At

\varepsilon=0

the runs are bit-identical at every step: the torus element relating them,

c=(1,1/16)

, is a power of two and rescales floating-point numbers exactly. Each red curve switches on one ingredient with a single value shared by both parameter groups: Adam’s

\varepsilon

, decoupled weight decay, or global-norm clipping. Each breaks the hyperparameter torus; the control is dotted. Rescaling the same ingredients per block, as Theorem III.10 prescribes, restores the exact zero (green floor). The values used are in § E.6 .

Figure 9: Update rules that see the gauge ( Theorem III.12 , Proposition III.4 , Proposition V.4 ). (a) At a fixed seeded LoRA state

(B,A)

with loss gradient

G

, each cell is the relative gauge defect

\lVert\delta W(g)-\delta W(I)\rVert_{F}/\lVert\delta W(I)\rVert_{F}

of one update rule: the change in its first-order weight update

\delta W=\delta B\,A+B\,\delta A

when the factors are replaced by the gauge-equivalent pair

(Bg^{-1},gA)

. The columns are a rotation, a stretch, a signed permutation and a generic

g\in\mathrm{GL}_{2}

; under each name is the smallest of the groups

B_{2}=\mathrm{O}(2)\cap\mathrm{Mon}_{2}

\mathrm{O}(2)

\mathrm{Mon}_{2}

(monomial matrices) and

\mathrm{GL}_{2}

that contains it. Green cells vanish up to float64 round-off, and they occur exactly where

g

lies in the rule’s gauge group (right column):

\mathrm{O}(2)

for SGD and for damped scaled GD,

B_{2}

for Adam,

\mathrm{Mon}_{2}

for Adam whose per-rank-channel rates are rescaled jointly with a stretch, and all of

\mathrm{GL}_{2}

for undamped scaled GD and for LoRA-Pro with the Sylvester-optimal free matrix, whose weight updates depend only on

W

. Red cells are genuine defects. (b) Three LoRA modules

(B_{i},A_{i})

with weights

w_{i}

are merged after module 2 alone is re-gauged to

(cB_{2},A_{2}/c)

. Task arithmetic ( Ilharco et al., 2022 ) ,

\sum_{i}w_{i}B_{i}A_{i}

, does not move. The LoraHub merge ( Huang et al., 2023 ) ,

(\sum_{i}w_{i}B_{i})(\sum_{j}w_{j}A_{j})

, does, and the open circles, the closed form of Proposition V.4 , match the recomputed change. The settings are in § E.6 .

Update rule	Equivariance group	What it reads off $A_{0}$ at $B_{0}=0$
$\mathrm{GL}_{r}$ -equivariant along the whole trajectory (the example rule; LoRA-RITE’s Algorithm 1, checked numerically)	$\mathrm{GL}_{r}$	$\operatorname{row}A_{0}\in\Gr(r,n)$
SGD, GD	$\mathrm{O}(r)$	$A_{0}^{\top}A_{0}$
damped scaled GD	$\mathrm{O}(r)$	$A_{0}^{\top}A_{0}$ (through a $\delta$ -dependent function)
Adam, AdamW; scaled AdamW (Zhang and Pilanci; HF’s Riemannian optimizer with AdamW); HF’s LoRA-FA optimizer	$B_{r}$	$A_{0}$ up to signed permutation of its rows

Table 13: What an update rule reads off

A_{0}

from a zero-

B

start ( Corollary III.13 ).

Method	Compression $c_{t}$	Anchor $a_{t}$
GaLore ( Zhao et al., 2024a )	$P_{t}^{\top}$ , with $P_{t}$ from a gradient SVD	$P_{t}$
GaLore with a random projector ( Torroba-Hennigen et al., 2025 ; Hao et al., 2024 )	as GaLore, with $P_{t}$ random	$P_{t}$
LISA ( Pan et al., 2024 )	layer-block coordinate projection	the same projection
MeZO ( Malladi et al., 2023 )	$g\mapsto\langle z_{t},g\rangle$ , estimated by finite differences without a backward pass	$\lambda\mapsto\lambda z_{t}$
Flora as published ( Hao et al., 2024 )	identity: only the momentum or accumulation buffer is compressed, by a random $A_{t}$ , inside $\mathrm{opt}$	identity (full-space Adafactor step)

Table 14: Backward methods as compression–anchor pairs ( Definition IV.1 ). The last row is the only reading of Flora as published in the form of Definition IV.1 , the trivial lens with all compression inside

\mathrm{opt}

Projector / schedule	Frozen	Re-based periodically	Resampled every step
random	LoRA-FA ( Zhang et al., 2023d ) , with $K\propto GA_{0}^{\top}A_{0}$ under SGD a	GaLore with a random projector ( Torroba-Hennigen et al., 2025 ; Hao et al., 2024 ) ; idealised Flora b	MeZO ( Malladi et al., 2023 ) (rank $1$ , global), up to its finite-difference estimator
gradient or data SVD	EVA ( Paischer et al., 2024 ) , LoRA-GA ( Wang et al., 2024e ) (first step only)	GaLore ( Zhao et al., 2024a )	GaLore with $T=1$
coordinate blocks	BitFit ( Ben-Zaken et al., 2021 ) , FISH ( Sung et al., 2021 ) , LN-tuning ( Zhao et al., 2023 )	LISA ( Pan et al., 2024 )	block coordinate descent
two-sided, state-dependent (contrast)	LoRA ( Hu et al., 2021 )	ReLoRA ( Lialin et al., 2023b )	none in use

Table 15: The cotangent family by projector (rows) and schedule (columns). Proposition IV.2 covers the first three rows under its optimizer hypotheses, except EVA and LoRA-GA, which initialise two-sided LoRA and sit in the frozen column only for their first step. The last row is a contrast row outside the equivalence.

Move	Examples	What to compute	Where
a reparametrisation that keeps the function	permuting hidden units; SliceGPT-style rotations of the residual stream ( Ashkboos et al., 2024a )	nothing new; methods are carried along by composition	Definition V.1 (a), Observation II.15
a quantiser	QLoRA, LoftQ, QPiSSA, QA-LoRA	the pointing defect, and the invariants that move with the base	Proposition V.2
merge and restart	ReLoRA, re-merged OFT and BOFT, re-HRA; GaLore period by period	the monoid generated by one cycle	Theorem V.3
combining adapters	task arithmetic, LoraHub, TIES, DARE, KnOTS	whether the rule is a function of the adapters, and which symmetries it commutes with	Proposition V.4

Table 16: Four ways the base point moves, and what each one asks us to compute.

Figure 10: Rebasing reach ( Theorem V.3 ). (a) The rank staircase for

m\times n=24\times 16

: the rank of

\Delta W_{K}=\sum_{k\leq K}B_{k}A_{k}

after

K

merge-and-restart cycles of ReLoRA, each adding a fresh random rank-

r

product, for

r=3

(circles) and

r=5

(squares). The measured ranks (teal) lie on the bound

\min(Kr,m,n)

(thick grey; dotted, the line

Kr

) at every

K

, so ReLoRA reaches full rank exactly at

K^{*}=\lceil 16/r\rceil

cycles, and not before (corollary 1, Theorem V.3 (a)). LoRA-XS with its frames frozen at the injective base

W_{0}

(red,

r=5

) stays at rank

5

: an idempotent method gains nothing from rebasing (corollary 2). (b) Seven families of block-orthogonal factors on

n=16

coordinates: block OFT on a fixed partition with blocks of size

b=4

and

b=8

, BOFT with

b=2

and

m^{\prime}=2,3,4

butterfly factors, and Givens rotations in

7

planes and in

15

planes forming a spanning tree. Arcs are two-element blocks and brackets larger blocks; dots are coloured by the connected component of the hypergraph the blocks generate, and hollow dots are isolated coordinates. The light bars give the dimension

\sum_{C}\binom{\lvert C\rvert}{2}

of the generated Lie algebra ( Theorem V.3 (c)), and the dark bars the dimension reached in one cycle, against

\dim\mathrm{SO}(16)=120

(dashed), with green bars for the families that reach it. Only a connected hypergraph reaches

120

; for BOFT that takes the full depth

m^{\prime}=4

(corollary 4). The bar values are in § E.6 .

Method under rebasing	One cycle	After $K$ cycles, or $R_{\infty}$	Gain from restarting
ReLoRA ( Lialin et al., 2023b )	$W_{0}+\mathcal{M}_{\leq r}$	$W_{0}+\mathcal{M}_{\leq\min(Kr,m,n)}$	yes; full FT at $K\geq\lceil\min(m,n)/r\rceil$
LoRA-FA (fixed $A$ ), LoRA-XS (frozen frames), BitFit, (IA) 3 , SSF, fixed masks	a subspace or submonoid	the same set	none (idempotent)
VeRA ( Kopiczko et al., 2023 ) ( $B$ without zero entries)	$W_{0}+\Lambda_{b}B\Lambda_{d}A$	$\text{LoRA-FA}(A)$	yes, up to LoRA-FA
block OFT ( Qiu et al., 2023 ) , exact Cayley, fixed partition	$W_{0}\cdot\prod\mathrm{Cay}(b)$	$W_{0}\cdot\prod\mathrm{SO}(b)=\overline{R_{1}}$	measure zero
COFT, exact Cayley	an $\varepsilon$ -ball in each block	up to $W_{0}\cdot\prod\mathrm{SO}(b)$	within the blocks
block OFT, HF Cayley–Neumann default	not orthogonal	a contraction monoid for small $Q$	drifts; $K_{\rm out}$ shrinks

Table 17: What merge-and-restart reaches ( Theorem V.3 and its corollaries).

Panel of Figure 11 — Figure 11: The merge rules of Theorem VI.3 on a pre-LN transformer block, drawn as a string diagram with the residual stream along the bottom: (a) the attention sublayer under RoPE, (b) a SwiGLU feed-forward sublayer. Tide boxes are trainable boxes of the base network, $\bullet$ is a copy node, $\odot$ the product node of the gated block, and ochre diamonds are edits inserted by a method. Moss arrows are rewrites licensed for every parameter value, ochre arrows rewrites licensed under a side condition, and seal marks what the rules do not merge. A query scaling $\ell$ fuses into $W_{Q}$ by R1; R5 moves it onto $W_{K}$ only if $\ell$ is constant on each rotary pair and across a grouped-query sharing group. A value scaling $\ell_{v}$ fuses into $W_{V}$ . A norm gain is copied to every reader (R7) and absorbed (R6). SSF’s scale and shift $(s,t)$ fuse into $W_{O}$ only if $W_{O}$ has a bias slot. The (IA) 3 scale fuses into $W_{\mathrm{down}}$ (R1), and through the product node into $W_{\mathrm{up}}$ for every $\ell$ (R4, then R1); it does not slide through the SiLU gate, and in an ungated block R3 carries it through a ReLU only for $\ell\geq 0$ . A prefix enters the softmax with an input-dependent gate ( Theorem VI.5 (f)), a nonlinear adapter is a non-affine edit ( Theorem VI.3 (c)), and LoReFT acts at selected positions of the residual stream ( Proposition VI.4 (c)); R1–R7 merge none of the three. The companion site’s merge game runs these rewrites interactively, with a float64 check of each licensed merge on seeded weights.

Figure 12: A prefix is a gate ( Theorem VI.5 ). One query

q

attends to six content tokens

C

and four prefix tokens

P

, all seeded synthetic data. (a) Prefix tuning ( Li and Liang, 2021 ) , one softmax over

P\uplus C

. The prefix (ochre) takes the mass

\lambda_{q}=Z_{q}(P)/(Z_{q}(P)+Z_{q}(C))

, and each content weight (teal) is its value without the prefix (dashed outline) times

1-\lambda_{q}

. The strip below plots the ratio after/before for every content token; it is the same for all six, so the content weights keep their ratios and their order (part (c), due to Petrov et al. (2023) ). (b) LLaMA-Adapter ( Zhang et al., 2023h ) normalises

P

and

C

separately and adds the prefix branch with a gate

g

: the prefix weights sum to

g

, and the content weights equal the no-prefix ones exactly (part (d)). (c) The value vectors projected on their two leading principal axes, with the hulls of the content values (dashed) and of the prefix values (ochre). The prefix-tuning output (black dot) lies on the segment from

\mathrm{Attn}_{q}(C)

\mathrm{Attn}_{q}(P)\in\mathrm{conv}\,V_{P}

, the fraction

\lambda_{q}

of the way (parts (b) and (c); the identity (b) is due to He et al. (2021) ), and a larger

\lambda_{q}

moves it further towards the prefix hull. The LLaMA-Adapter output

\mathrm{Attn}_{q}(C)+g\,\mathrm{Attn}_{q}(P)

(open grey) is an additive step that leaves the segment. The rigidity holds within one layer, relative to the same layer without the prefix: prefixes at lower layers change the queries, keys and values of higher ones. The values used are in § E.6 .

Method	What it adds	Neutral point	Merge	Reason
LoRA, DoRA, OFT, HRA	nothing; weight-space reparametrisations	their pointing in $\mathbf{Meth}$	$\mathsf{M1}$	nonlinearity on the parameter side; $\mu=\rho$
(IA) 3 on $k$ , $v$	output scaling of $W_{K}$ , $W_{V}$	$\ell=\mathbf{1}$	$\mathsf{M1}$	R1
(IA) 3 on the FFN	input scaling of $W_{\mathrm{down}}$	$\ell=\mathbf{1}$	$\mathsf{M1}$ , $\mathsf{M1}{\rightarrow}$	R1 into $W_{\mathrm{down}}$ ; R4 into $W_{\mathrm{up}}$ under SwiGLU, R3 under ReLU with $\ell\geq 0$
SSF	scale and shift after a box	$\gamma=\mathbf{1}$ , $\beta=0$	$\mathsf{M1}{\rightarrow}$	R1 into the preceding box if it has a bias slot
LN-tuning	new values of existing $\gamma,\beta$	the pretrained values	$\mathsf{M1}$	in place; folding forward needs pre-LN, and $\beta$ a bias slot
linear adapter parallel to one box	$sUDx$ beside $W_{0}x$	$U=0$	$\mathsf{M1}$	Proposition VI.4 (a): it is LoRA

Table 18: Extensions and their merge types. The verdicts are box-local ( Definition VI.1 ), and the attention results are per layer.

Axis	Question	LoRA	(IA) 3	Prefix tuning
1 Kind	Where does it act?	R	R	E ∘
2 Shape	What is the image germ?	Cone, $d=r(m+n-r)$	Lin, a torus-orbit closure	Fun
3 Base point	Is $\rho(q_{0})=\theta_{0}$ ? Is $\dim S_{1}=d(M)$ ?	apex, deficit $r(n-r)$	regular	unpointed
4 Gauge	Which reparametrisations of $Q$ leave $\rho$ unchanged?	$\mathrm{GL}_{r}$	none	non-linear (Li–Liang’s MLP)
5 Covariance	For which $(g,h)$ is $\Imop M(gW_{0}h^{-1})=g\,\Imop M(W_{0})\,h^{-1}$ ?	$\mathrm{GL}\times\mathrm{GL}$	$\mathrm{Mon}\times\mathrm{GL}$ ; $\mathrm{GL}\times\mathrm{Mon}$ on $W_{\rm down}$	$\mathrm{GL}\times\mathrm{GL}$ , function level
6 Merge	Does the update fuse into its box, or by R1–R7 into a neighbour?	M1	M1 $\to$	M $\infty$

Table 20: The seven questions, answered for three methods. Every answer is read off the formula with no training; the values of each axis are defined in Table 21 .

Value	Definition
1 Kind (where the method acts). Does the method reparametrise the existing weights, extend the architecture (with or without a neutral point), act only on the backward pass or the schedule, operate post hoc on several trained points, or form a family over an auxiliary base $T$ ?
R reparametrisation	A pointed smooth map $\rho:(Q,q_{0})\to(\Theta,\theta_{0})$ into the existing weight space, an object of $\mathbf{Meth}(\Theta,\theta_{0})=\mathbf{Diff}_{*}/(\Theta,\theta_{0})$ . Test: the trained model is $f(\rho(q),-)$ with the original architecture $f$ .
E pointed extension	A function-preserving embedding $\iota(\theta)=(\theta,\theta_{0}^{\prime})$ into a larger architecture $\tilde{f}$ with a neutral point $\theta_{0}^{\prime}$ , so that $\tilde{f}(\theta,\theta_{0}^{\prime},-)=f(\theta,-)$ , followed by a reparametrisation of $\Theta\times\Theta^{\prime}$ . Test: new boxes are inserted, and some parameter value makes them the identity.
E ∘ unpointed extension	An extension with no neutral point: no parameter value reproduces the frozen function (for prompts and prefixes, Theorem VI.5 (f) proves this on a head with $W_{o}W_{v}\neq 0$ ). Test: try to solve $\tilde{f}(\theta_{0},p,-)=f(\theta_{0},-)$ for $p$ .
B backward lens or schedule	The forward map is $\mathrm{id}_{\Theta}$ and the update is $\theta\leftarrow\theta-\eta\,a_{t}\big(\mathrm{opt}(c_{t}(\nabla L))\big)$ with a compression $c_{t}$ and an anchor $a_{t}$ ; or the method is a sequence of merge-and-restart stages (a rebasing scheme); or it is an optimizer that compresses only its own state (Flora). Test: the method changes how gradients or optimizer state are compressed, or when the base is re-pointed, and leaves the parametrisation alone.
P post-hoc operation	An operation $\Theta^{N}\to\Theta$ (or on tuples of factors) applied to several trained points. Test: the inputs are already-trained adapters or checkpoints.

Table 21: The seven axes. For each axis, the question it answers and the definition of every value, with the test that decides it where one is needed. “–” (a null coordinate) means that the axis does not apply.

Method	Kind	Shape ( $d$ ; $\rho_{\max}$ )	Base pt.	Gauge	Covariance	Merge	Rebasing
Frozen / Full FT	R (initial / terminal)	$\{\theta_{0}\}$ / $\Theta$	– / regular	–	all / $\mathrm{GL}\times\mathrm{GL}$	–	id
LoRA	R	Cone $r(m{+}n{-}r)$ ; $r$	apex	$\mathrm{GL}_{r}$	$\mathrm{GL}\times\mathrm{GL}$	M1	$\to\Theta$ at $K=\lceil\min(m,n)/r\rceil$
rsLoRA, LoRA+	R 1	$=$ LoRA	apex	$\mathrm{GL}_{r}$	$\mathrm{GL}\times\mathrm{GL}$	M1	$\to\Theta$
LoRA-Pro, Riem. precond., LoRA-RITE	R + metric 2	$=$ LoRA	apex	$\mathrm{GL}_{r}$ 2	$\mathrm{GL}\times\mathrm{GL}$	M1	$\to\Theta$
DeLoRA	R	Cone $\subseteq\mathcal{M}_{\leq r}$ , $\lVert\Delta W\rVert_{F}\leq\lambda\lVert W_{0}\rVert$	apex (HF) / split (paper)	non-linear, fibre $r^{2}{+}1$ 3	$\mathrm{GL}\times\mathrm{GL}$	M1	$\to\Theta$
GraLoRA	R	$\prod$ block cones, $r(m{+}n{-}r)$ ; $\min(kr,m,n)$	apex	$\prod\mathrm{GL}_{r/k}$	$\Pi$ (blocks)	M1	$\to\Theta$

Table 22: The periodic table by method: kind, shape (with

d

and

\rho_{\max}

), base point, gauge, covariance, merge type and rebasing closure. The notation is explained before the table, and the marks †,

\approx

and (fn.) and the numbered notes after it.

Axis	Value: all / core
Kind	R $208/85$ $\cdot$ E $33/10$ $\cdot$ E ∘ $27/13$ $\cdot$ B $29/6$ $\cdot$ P $31/11$ $\cdot$ F T $54/17$
Shape	Lin $106/45$ $\cdot$ Cone $134/43$ $\cdot$ Cone ∗ $20/9$ $\cdot$ Orb $13/7$ $\cdot$ Sat $5/3$ $\cdot$ Meet $2/2$ $\cdot$ Fun $93/32$ $\cdot$ – $9/1$
Base point	regular $113/56$ $\cdot$ apex $111/40$ $\cdot$ neutral $40/13$ $\cdot$ defect $59/13$ $\cdot$ unpointed $57/18$ $\cdot$ – $2/2$
Gauge	none $139/62$ $\cdot$ scalar $9/6$ $\cdot$ torus $40/9$ $\cdot$ $\mathrm{GL}$ $152/51$ $\cdot$ $\mathrm{GL}\times{}$ torus $3/1$ $\cdot$ non-linear $23/10$ $\cdot$ – $16/3$
Covariance	$\mathrm{GL}\times\mathrm{GL}$ $166/53$ $\cdot$ $\mathrm{CO}\times\mathrm{CO}$ $47/20$ $\cdot$ $\mathrm{GL}\times\mathrm{CO}$ $12/5$ $\cdot$ $\mathrm{Mon}\times\mathrm{GL}$ $21/10$ $\cdot$ $\mathrm{Mon}\times\mathrm{Mon}$ $52/18$ $\cdot$ $\Pi$ $49/19$ $\cdot$ frame $19/10$ $\cdot$ – $16/7$
Merge	M1 $242/98$ $\cdot$ M1 $\to$ $11/5$ $\cdot$ Mq $11/2$ $\cdot$ M $\infty$ $97/32$ $\cdot$ – $21/5$

Table 23: The classification of all 382 catalogued methods: the number of methods with each coordinate value, over all methods / over the 142 core methods. “–” counts null coordinates, including the legacy merge key na. Per-method values are in Table 36 .

Figure 13: The catalogue in time. Each of the 382 catalogued methods is one dot, placed at the month of its paper’s first arXiv submission, read from the YYMM prefix of its identifier; within a month the dots are spread evenly in the order of their arXiv numbers, and the six entries without an identifier sit at mid-year. The dots form a centred beeswarm, coloured by modification kind (legend, with the number of entries of each kind) and ringed when the method ships in Hugging Face PEFT (82 entries). The one entry before 2017, linear probing on DeCAF features ( Donahue et al., 2013 ) , sits left of the axis break. The number under each year counts its entries, from

10

in 2020 to

113

in 2024; the busiest month is February 2024, with

20

. The 2026 count ends at the catalogue’s build date, 2 October 2026 (dotted line). Labels mark the bottleneck adapter ( Houlsby et al., 2019 ) , prefix tuning ( Li and Liang, 2021 ) , LoRA ( Hu et al., 2021 ) , (IA) 3 ( Liu et al., 2022a ) , QLoRA ( Dettmers et al., 2023 ) , DoRA ( Liu et al., 2024b ) , GaLore ( Zhao et al., 2024a ) and the newest entry, FrameFT ( Adepu et al., 2026 ) . Additive weight edits are the largest kind (

120

entries), followed by composition and routing (

65

) and optimizer-side methods (

51

Rule	Statement	The map $h$ , or the test	Examples
A1 interval	Frozen ${}\to M\to{}$ Full FT for every weight-space $M$ .	$h=\mathrm{const}_{q_{0}}$ ; $h=\rho_{M}$	Merging is the arrow to Full FT.
A2 freezing	Freezing coordinates at their base values gives a sub-object.	$h(q^{\prime})=(q^{\prime},q_{0}^{\prime\prime})$	$\mathrm{FA}(A_{0})\hookrightarrow{}$ LoRA pointed at $(0,A_{0})$ ; MiCA ${}\hookrightarrow{}$ LoRA pointed at $(B_{0},0)$ .
A3 factorisation	$\rho_{M}=\rho_{N}\circ h$ for an explicit pointed map $h$ , usually polynomial.	VeRA ${}\to\mathrm{FA}(A)$ : $h(b,d)=\Lambda_{b}B\Lambda_{d}$ . LoHa ${}_{r_{1},r_{2}}\to{}$ LoRA ${}_{r_{1}r_{2}}$ : face-splitting products. (IA) ${}^{3}\to{}$ HiRA 1 pointed at $(0,\mathbf{1}^{\top})$ : $h(\ell)=(\ell-\mathbf{1},\mathbf{1}^{\top})$ .	Isomorphisms: rsLoRA ${}\cong{}$ LoRA by $h=(\sqrt{r}\,B,A)$ ; MiSS (balance mode) ${}\cong\mathrm{FA}(\mathbf{1}^{\top}\otimes I_{r})$ ; a linear adapter parallel to one layer ${}\cong{}$ LoRA.
A4 products	A product comes with its two projections.	$R\mapsto B_{0}R$ , $R\mapsto RA_{0}$	LoRA-XS ${}\to\mathrm{FA}(A_{0})$ , $\mathrm{FB}(B_{0})$ . HRA’s image lies in those of OFT SO and LoRA r , an image-level inclusion only.
A5 sums	$M\to M\oplus N$ , and $K$ stages embed in $K+1$ .	$q\mapsto(q,q_{0}^{N})$	LoRA ${}_{r}\oplus{}$ LoRA ${}_{s}\cong{}$ LoRA r+s (unpointed); ReLoRA ${}^{K}\to{}$ ReLoRA K+1 .
T transport	$T_{*}$ for $T\in\mathrm{GL}(\Theta)$ is an automorphism of the slice, giving one object in two frames.	no arrow; record the frame $T$	Sums of Kronecker products are $\mathcal{R}^{-1}_{}{}$ LoRA; HiRA is $(D_{W_{0}})_{}{}$ LoRA at $\theta_{0}$ when $W_{0}$ has no zero entry.

Table 24: The arrow rules A1–A5, transport T and the obstruction tests O1–O3. An arrow

M\to N

is a pointed smooth map

h:Q_{M}\to Q_{N}

with

\rho_{N}\circ h=\rho_{M}

Test	Count	What fails
O1	288	an image invariant (Gram matrix, spectrum, row lines, zero pattern) or image inclusion
O2	129	dimension or cut-rank
O3	68	first-order image or base-point stratum

Table 25: The 485 recorded obstructions

M\not\to N

, by the test that rules the arrow out. The full list, with the failing invariant of each, is the ancillary file obstructions.csv.

Figure 14: The category of core methods. A layered Hasse diagram ( Background A.16 ), from Frozen (initial, bottom) to Full FT (terminal, top); the dashed outer frame stands for the A1 arrows Frozen

{}\to M\to{}

Full FT of every weight-space method

M

( Observation I.4 (a)). Chips are the 142 core methods, coloured by modification kind and dashed for the

57

that are not objects of the weight-space slice. An arrow

M\to N

points up and means that

N

simulates

M

through a pointed smooth map

h

with

\rho_{N}\circ h=\rho_{M}

( Definition I.2 ). All

161

recorded arrows between two core methods are drawn, coloured by rule (Table 24 ):

29

by freezing (A2, hooked tail),

122

by factorisation (A3),

2

by products (A4) and

8

by sums or stages (A5). The

7

loops land on another member of a family, as

\mathrm{LoRA}_{r}\to\mathrm{LoRA}_{r+1}

does, and the

5

classes of mutually simulating families share a rung in a grey box. Arrows into a method with four or more incoming arrows from outside its class end in one trunk (

57

into LoRA), and the

28

methods whose arrows all land in the LoRA class form the comb beneath it. The

26

lighter arrows are composites, also reached by a longer path of arrows.

Strength	What the entry asserts	Example
Definition	The phrase gets a precise, computable meaning.	“Effective number of parameters” is $d(M)$ , the generic rank of $d\rho$ ( Definition I.5 ).
Theorem	The phrase hides a claim, proved under stated hypotheses.	“No initialisation buys rank above $2r$ ” means that on one $m\times n$ box the exact pointings of $\mathrm{LoRA}_{r}$ together reach $\theta_{0}+\mathcal{M}_{\leq\min(2r,m,n)}$ ( Theorem II.1 ).
Analogy	A borrowed word names a pattern and adds nothing to the theorem behind it.	“Noether charge”, “gauge”, “Erlangen”, “moduli space”.

Table 26: The three strengths of a dictionary entry.

Order	Used by	Pointing defect
split, then quantise the residual	QPiSSA; HF’s PiSSA and CorDA workflow	$\kappa(W_{\mathrm{res}})-W_{\mathrm{res}}$
quantise, then fit the error	LoftQ with one iteration (HF default loftq_iter=1)	$e-s\,\mathrm{SVD}_{r}(e)$ , as HF does not divide the fit by $s$ ; at $s=1$ , $e$ minus its best rank- $r$ approximation
quantise, split, re-quantise	HF olora_init on bitsandbytes weights	$e+\kappa(r^{\prime})-r^{\prime}$ , with $r^{\prime}=\kappa W-\delta_{0}$

Table 27: Three orders of quantising and splitting, and the pointing defect

\rho(q_{0})-W

each leaves ( Proposition V.2 (b)).

#	PEFT	Categorical / geometric	Gloss
1	pretrained checkpoint	base point $\theta_{0}:1\to\Theta$ ; a pointed 1-cell $(\Theta,\theta_{0},f)$ of $\mathbf{Para}(\mathbf{Diff})$	Fine-tuning moves a point; everything is measured relative to it ( Def. I.1 ).
2	PEFT method	object of the pointed slice $\mathbf{Meth}(\Theta,\theta_{0})=\mathbf{Diff}_{*}/(\Theta,\theta_{0})$ , equivalently the opposite of the pointed coslice of reparametrisation 2-cells under $F$	A method is a pointed map $\rho:(Q,q_{0})\to(\Theta,\theta_{0})$ . The slice depends only on $(\Theta,\theta_{0})$ ; $\mathbf{Para}$ is bookkeeping for weight-space methods ( Rem. I.3 ).
3	“method $N$ can do whatever $M$ can”	morphism (simulation) $h:M\to N$ with $\rho_{N}\circ h=\rho_{M}$	Smooth simulations also carry parameter paths; at the level of sets they reduce to image inclusion ( Obs. I.4 ).
4	frozen model / full fine-tuning	initial / terminal object	Every weight-space method lies in the interval [frozen, full FT]; adapters, prefixes and ReFT are extensions, not objects of $\mathbf{Meth}$ ( Obs. I.4 ).
5	merging an adapter into the weights	the canonical morphism to the terminal object (it is $\rho$ itself); for extensions, a lift along the realisation $\Phi_{f}$	For weight-space methods merging is definitional; for extensions zero inference overhead is a lifting problem, certified box-locally by rewriting (sufficient conditions and box-local obstructions) under side conditions such as position-uniformity, no weight sharing and bias slots ( Thm VI.3 ).
6	expressivity / effective number of parameters	image germ of $\rho$ at $\theta_{0}$ ; its dimension $d(M)$ , the generic rank of $d\rho$	Rank is a cut invariant; dimension is a volume invariant ( Thm II.10 , Prop. II.12 ).

Table 28: Rosetta stone: the full dictionary from PEFT phrases to their categorical or geometric counterparts. The gloss names the results behind each pairing; their hypotheses are part of the entry.

No.	Prediction	Optimizer scope	Rests on	Kind
X.1	LoRA+ against $\alpha$ re-tuned by $\lambda$	Adam, AdamW	Thm III.10 , Cor. III.11	theorem
X.2	DoRA against LoRA + output gain	AdamW, shared by all arms	Prop. II.8 , Conj. D	conjecture
X.3	Hidden-unit permutations	coordinatewise (SGD, Adam)	Lemma VI.2 , Obs. II.15	theorem + data
X.4	Residual-stream rotations	reach: any; outcomes: depend on optimizer and init	Obs. II.15 , Conj. A	conjecture
X.5	Rank-one targets	any (image claim)	Thm II.11	theorem
X.6	Re-based OFT	any (reach); exact Cayley map	Thm V.3	theorem

Table 29: The eleven predictions, the optimizer scope each inherits, the results it rests on, and its kind.

Word	Meaning here	Proved	Not claimed
Grothendieck	The relative point of view: a method is a map $\rho:(Q,q_{0})\to(\Theta,\theta_{0})$ , methods form the pointed slice $\mathbf{Meth}(\Theta,\theta_{0})$ , and changing $\theta_{0}$ is base change.	Frozen model initial and full fine-tuning terminal; products are fibre products in $\mathbf{Diff}$ and can fail to exist ( Obs. I.4 ); a symmetric monoidal sum ( Obs. I.7 ); automorphisms by transport ( Prop. II.13 ); pushforward along function-preserving maps ( Def. V.1 (a)).	No schemes, sheaves, descent theory or stacks. Quantisation and rebasing borrow only the word “base change”, and idle constructions such as $\int\mathbf{Meth}\to\mathbf{PNet}$ are left out. “Exposé” and the rising sea are homage.
Erlangen	After Klein (1872) , a method’s geometry is its covariance group $\mathcal{G}_{M}$ ( Def. II.14 , Background D.17 ); what a group-type method can never change is a complete invariant of the action.	Complete invariants of the orthogonal, torus and Hadamard actions ( Thm II.5 ); exactly orthogonal input-side methods keep $K_{\mathrm{out}}$ and the spectrum through any number of merges ( Cor. II.6 , Thm V.3 ); covariance groups of named methods ( Prop. II.9 (e), Prop. II.16 ).	For cones such as LoRA’s image the word only names the shape and its covariance. Keeping an invariant concerns reach; that it causes knowledge retention is not shown.
Noether	The pattern that turns a continuous symmetry of $\rho$ into a conserved quantity.	Thm III.5 , for gradient flow driven by any signal $\Lambda D\rho^{\top}\gamma$ , a mild extension of Zhao et al. (2022b) ; for LoRA the charges are the entries of $\Phi$ .	No Lagrangian. Discrete steps drift by $O(\eta^{2})$ relative to the charge, and Adam and momentum break the law.
Gauge	The symmetry group of a parametrisation ( Def. I.5 , Background D.21 ); for LoRA, $\mathrm{GL}_{r}$ acting by $(Bg^{-1},gA)$ .	Generic gauges and fibre dimensions ( Prop. II.12 ); the part each optimizer respects, $B_{r}\subset\mathrm{O}(r)\subset\mathrm{GL}_{r}$ ( Thm III.12 ); the gauge dependence of factor-wise merging ( Prop. V.4 ).	No connection or curvature. Fibres are gauge orbits only on the full-rank stratum; away from it the fibres jump.
Moduli	A set of isomorphism classes ( Background C.20 ).	Three strata of full-rank LoRA pointings are classified, as a set, by $\Gr(r,n)\sqcup\Gr(r,m)\sqcup\mathcal{M}_{r}$ ( Thm II.2 ).	No topology: the natural quotient topology is not a disjoint union ( $\mathrm{T}_{A}$ lies in the closure of $\mathrm{S}$ ). No stack is built, and the strata miss some pointings (HF’s orthogonal init, defect pointings).
Fibre	The preimage $\rho^{-1}(\theta)$ , the parameters that give the same weights, as in the title.	Fibre dimensions ( Prop. II.12 (a)); which points of the apex fibre $\{BA=0\}$ an optimizer can tell apart ( Cor. III.13 ).	$\rho$ is no fibre bundle; the fibre dimension jumps between strata.

Table 30: Analogy versus theorem: seven borrowed words, their meaning here, what is proved under each, and what is not claimed.

Here	Due to	Added here
Def. I.1 , Exposé IV	Fong et al. (2017) ; Cruttwell et al. (2021)	The pointed slice of methods; tangent and cotangent sides.
Thm II.1	The PiSSA-to-LoRA conversion of Meng et al. (2024a) , in HF PEFT ( Mangrulkar et al., 2022 )	The rank- $2r$ bound for every exact pointing, and from any start.
Thm II.5	Classical: the first fundamental theorem for $\mathrm{O}(n)$ ; Witt’s extension theorem ( Background D.16 )	Which invariant each group-type method keeps.
Thm II.10	The standard cut-rank bound for tensor networks ( Background B.8 )	A rank census of PEFT plugs.
Thm II.11	Van Loan and Pitsianis (1993) ; the operator-Schmidt decomposition ( Background B.12 )	Kronecker sums as LoRA across another cut; the LoKr inclusion criterion.
Prop. II.12 , TT/Tucker row	Koch and Lubich (2010) ; Uschmajew and Vandereycken (2013) ; Holtz et al. (2012)	The same count, at minimal ranks, beside other PEFT families.

Table 31: Known results we re-derive, whom they are due to, and what is added here.

Result	Status	What the referees changed
Obs. I.4 , universal properties	repaired	Products need a fibre product in $\mathbf{Diff}$ and can fail to exist ( $\mathrm{LoRA}_{1}\times\text{Frozen}$ on $1\times 1$ weights).
Obs. I.6 , $S_{1}$ is functorial	verified	Monotonicity stated only along pointed morphisms.
Obs. I.7 , additive monoidal structure	repaired twice	$r\mapsto\mathrm{LoRA}_{r}$ is strong monoidal only unpointed or at degenerate pointings ( $\dim S_{1}$ is $mr$ against $2mr$ ); HF’s cat merge described exactly.
Prop. I.8 , a product and a meet	verified	LoRA-XS needs $\rk W_{0}\geq r$ ; the set-level meet in (b) is no new method.
Thm II.1 , pointings of an additive method	verified	Stated for exact pointings, with a robust form; credited to Meng et al. (2024a) .
Thm II.2 , three strata of LoRA initialisations	repaired	Full-rank pointings only, and only as a set; HF’s orthogonal init and QLoRA/LoftQ defects lie outside.

Table 32: The referee record: the status of each numbered result after both rounds of review, and what the referees changed.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

No.	Entry	Area	First places used
A.1	category	Categories	Table 2 , Def. I.2
A.2	commutative diagram	Categories	Figure 2 , Def. I.2
A.3	isomorphism	Categories	Obs. I.6 , Obs. I.7
A.4	functor; functorial invariant	Categories	Obs. I.4 (c), Obs. I.6
A.5	natural transformation; naturality	Categories	Thm III.3 , Prop. V.4
A.6	slice category	Categories	Table 2 , Def. I.2

Appendix

Table 33: Index of the background entries, with the area of each and the first places in the paper that use it.

Panel of Figure 15 — Figure 15: The interval of $\mathbf{Meth}(\Theta,\theta_{0})$ . The left arrow is the only arrow out of the initial object, the right arrow $\rho$ is the only arrow into the terminal object, and the triangle commutes because $\rho(q_{0})=\theta_{0}$ . Figure 2 (a) draws the same triangle in the main text, where a pointing defect $\rho_{M}(q_{0})\neq\theta_{0}$ leaves it commuting only up to that defect.

Panel of Figure 16 — Figure 16: A pullback square. It commutes, $\rho_{M}\circ p_{M}=\rho_{N}\circ p_{N}$ , and any test object $T$ whose maps $u,v$ agree over $\Theta$ factors through the corner by exactly one $t$ . The small corner mark is the usual sign of a pullback.

Panel of Figure 17 — Figure 17: LoRA as a string diagram, read left to right, with the scale $s$ absorbed into $B$ . The fork sends the input to both branches and the $+$ node adds their outputs. These are the copy and sum maps of the biproduct $\oplus$ ( Background B.2 ), so the diagram says $W=W_{0}+BA$ . The frozen $W_{0}$ is grey, and the trainable $A\in\mathbb{R}^{r\times n}$ and $B\in\mathbb{R}^{m\times r}$ are tinted. The dashed cut separates $A$ from $B$ and crosses one wire of dimension $r$ , so $\rk(W-W_{0})=\rk BA\leq r$ . For $W$ itself, subadditivity over the two branches gives only $\rk W\leq\rk W_{0}+r$ .

Family	What it covers	Methods	Papers	Corr.
LoRA core	LoRA and its variants	20	12	21
Initialisation and rank	Base-point splits; rank allocation	25	6	46
Shape	Kronecker, Hadamard, tensor and sparse shapes	29	9	49
Multiplicative	Group and monoid actions on $W_{0}$	32	9	54
Sharing	Shared or random bases	28	4	46
Selective	Subsets of existing weights	26	7	37

Appendix

Table 34: How the catalogue was assembled. Each family was written by one reader and attacked by another; the last column (Corr.) counts the logged corrections. The families overlap, so the records were merged into the catalogue on a canonical identifier.

Rule	Count	How $h$ arises
A2	61	freezing a block of coordinates at its base value, $h(q^{\prime})=(q^{\prime},q_{0}^{\prime\prime})$
A3	329	an explicit smooth pointed factorisation $\rho_{M}=\rho_{N}\circ h$
A4	2	a projection out of a fibre product
A5	11	a sum or an added stage, $h(q)=(q,q_{0}^{N})$

Appendix

Table 35: The 403 recorded arrows by rule (top) and the 485 recorded obstructions by test (bottom). The rules and tests are those of § VIII.C ;

161

of the arrows join two core methods. Every arrow is listed in Table 37 , and every obstruction, with its failing invariant, in the ancillary file obstructions.csv.

Method	Reference	Year	Kind	Shape	Base pt.	Gauge	Covar.	Merge	Rebase	HF
R, additive
ABBA	Singhal et al.,2025	2025	R	Cone	apex	GL $\times$ T	Mon $\times$ Mon	M1	span
$\bullet$ AdaLoRA	Zhang et al.,2023g	2023	R	Cone	apex	nonlin.	GL $\times$ GL	M1	span	✓
$\bullet$ AdaMSS	Zheng et al.,2025	2025	R	Cone	apex	GL	O $\times$ O	M1	span	✓
AdaPaD	Su et al.,2026	2026	R	Cone	defect	GL	GL $\times$ GL	M1	span
ALoRA	Liu et al.,2024d	2024	R	Cone	apex	GL	GL $\times$ GL	M1	span

Appendix

Table 36: The complete catalogue: every method with its seven coordinates (§ VIII ). Core methods are marked

\bullet

; HF marks methods available in Hugging Face PEFT at the pinned commit. “–” means the axis does not apply.

Source $M$	Target $N$	Rule	Pointed map $h$ with $\rho_{N}\circ h=\rho_{M}$
(IA) 3	DoRA	A2	$\ell\mapsto\big(B=0,\ A=A_{0},\ \mu=\ell\odot(\\|w_{0,i}\\|)_{i}\big)$
(IA) 3	HiRA	A3	$\ell\mapsto(\ell-\mathbf{1},\ \mathbf{1}^{\top})\ \ (k,v);\hskip 16.38895pt\ell\mapsto(\mathbf{1},\ (\ell-\mathbf{1})^{\top})\ \ (W_{\rm down})$
ABBA	LoHa	A3	$h(B_{1},A_{1},B_{2},A_{2})=(B_{1},A_{1},cB_{2},A_{2}),\ \ c=\frac{\alpha/\sqrt{r_{1}r_{2}}}{\gamma_{\rm LoHa}}$
ABBA	LoRA	A3	$h=\big(\tfrac{\alpha}{\sqrt{r_{1}r_{2}}}\,B_{1}\bullet B_{2},\;(A_{1}^{\top}\bullet A_{2}^{\top})^{\top}\big)$
AdaLomo	LOMO	A3	$h=\mathrm{id}_{\Theta}$
AdaLoRA	LoRA	A3	$h(P,\Lambda,Q)=(P\Lambda,\ Q)$

Appendix

Table 37: Every recorded simulation arrow

M\to N

(rules A2–A5 of § VIII ), with its explicit map. Notes on the instances (rank, pointing) are in the ancillary file arrows.csv.

Check	Supports	What the script computes	What it prints
Sections 1–11: the numerical claims of the framework
1	Thm II.7 (b)	HRA r without Gram–Schmidt, $u\mapsto W_{0}H(u_{1})\cdots H(u_{r})$ , with an injective $W_{0}\in\mathbb{R}^{(n+2)\times n}$ . Jacobian rank at a random point, and at HF’s paired initialisation ( $u_{2i-1}=u_{2i}$ , so $k=r/2$ distinct vectors).	At $(n,r)=(9,2)$ , $(9,4)$ , $(10,2)$ , $(10,4)$ , $(12,6)$ : generic $15$ , $26$ , $17$ , $30$ , $51$ , equal to $rn-r(r+1)/2$ ; paired $8$ , $15$ , $9$ , $17$ , $30$ , equal to $kn-k(k+1)/2$ .
2	Thm III.12 (b)	The first Adam step from zero state with $\varepsilon=0$ is sign descent. Its first-order weight update $dB\,A+B\,dA$ at $(B,A)$ and at $(Bg,g^{-1}A)$ , $(m,n,r)=(6,5,3)$ .	Defect $1.4\times 10^{-15}$ for a signed permutation $g$ . Controls: $11.3$ for $g=\diag(2,0.5,3)$ and $6.6$ for a random rotation. With per-channel rates $(\eta_{B}d_{j},\eta_{A}/d_{j})$ the diagonal $g=\diag(d)$ gives $5.0\times 10^{-16}$ .
3	Cor III.11 , Thm III.10	$200$ steps of bias-corrected Adam with $\varepsilon=0$ on a least-squares toy ( $W_{0}\in\mathbb{R}^{7\times 6}$ , $r=2$ , $B_{0}=0$ ): LoRA+ with ratio $\lambda=8$ and scale $s$ , against LoRA with scale $\lambda s$ and equal rates.	Largest weight difference over the whole trajectory $0.0$ , exactly, since $\lambda$ is a power of two. Control without the rescaling: $0.93$ .
4	Prop II.12	GraLoRA with $k\times k$ blocks of rank $r/k$ : Jacobian rank, and the rank of $\Delta W$ at a random point.	At $(m,n,r,k)=(8,8,4,2)$ , $(12,8,4,2)$ , $(12,12,6,3)$ : dimension $48$ , $64$ , $108$ , equal to $r(m+n-r)$ ; rank of $\Delta W$ $8$ , $8$ , $12$ , against LoRA’s cap $4$ , $4$ , $6$ .
5	Prop II.12	LoHa r,r , $(B_{1}A_{1})\odot(B_{2}A_{2})$ : Jacobian rank.	At $(m,n,r)=(8,7,2)$ , $(12,10,2)$ , $(12,10,3)$ : $38$ , $59$ , $93$ , equal to the bound $2r(m+n-r)-(m+n-1)$ .

Appendix

Table 38: The checks of framework_checks.py, in the script’s order, with the result each one supports, what it computes, and what it prints. Controls are cases in which the theory predicts a failure.

Component	Version	Used for
Python	3.9.6	the check script, the figure scripts, the generators of the tables and ancillary files
PyTorch ( Paszke et al., 2019 )	2.8.0	float64 automatic differentiation, linear algebra, seeded random draws
NumPy ( Harris et al., 2020 )	2.0.2	discrete Fourier transforms and ranks in the checks; the figure scripts
SciPy ( Virtanen et al., 2020 )	1.13.1	Sylvester equations (LoRA-Pro, check 12l) and matrix logarithms (check 12t)
Matplotlib ( Hunter, 2007 )	3.9.4	the figures of this paper
Hugging Face PEFT ( Mangrulkar et al., 2022 )	commit 85be3f3, 30 September 2026	the implementation that every library claim was checked against; not needed to run the checks

Appendix

Table 39: Software and hardware for the checks, the figures and the companion site.

Phase	Workflows	Agent runs	Fresh tokens	Generated tokens
Literature catalogue (first sweep, additions, 2026 sweep)	4	75	24.6 M	0.43 M
Theory and adversarial review (rounds 1–2)	2	145	16.3 M	2.03 M
Classification, site sections and figures	6	103	32.1 M	2.74 M
Site review, appendix and typography	4	49	22.1 M	1.68 M
Paper (writing, referees, sync)	4	50	22.0 M	1.75 M
All workflows	20	422	117.1 M	8.63 M

Appendix

Table 40: Agent runs and tokens by phase. Fresh tokens are prompts written to the cache plus generated text; re-reads from the prompt cache are not counted.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 610 papers

May 14, 2026

GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning

Low-rank adaptation (LoRA) has become a dominant paradigm for parameter-efficient fine-tuning (PEFT) of large-scale deep learning models. However, its bilinear parameterization induces a parameter-dependent geometry: the mapping from trainable parameters to weight updates is not generally distance-preserving. Related methods that project a low-dimensional vector into LoRA's parameter space, such as Uni-LoRA, improve parameter efficiency, but the subsequent bilinear map breaks end-to-end isometry. We propose GPart (Global Partition fine-tuning), a highly parameter-efficient fine-tuning method that maps a

d

-dimensional trainable vector directly into the full weight space through a sparse, isometric partition matrix. GPart retains a fixed global parameter-sharing prior while removing the additional low-rank reconstruction used by LoRA-based methods. This yields a simple parameterization with a single main hyperparameter (

d

), exact end-to-end isometry, and a minimal checkpoint representation consisting of the trainable vector and a random seed. GPart builds on the premise of effective fine-tuning within random low-dimensional subspaces of the full weight space without requiring a low-rank matrix factorization. Across natural language understanding, computer vision, and mathematical reasoning benchmarks, GPart matches or improves over existing PEFT methods at ultra-low parameter budgets. Beyond offering mathematical tractability and memory efficiency, the direct linear parameterization of GPart streamlines model selection and paves the way for compact adapter composition. Overall, GPart provides an elegant and competitive alternative for fine-tuning under small parameter budgets, with a fixed and predictable geometry between trainable coordinates and weight-space updates.

Paolo Mandica, Michał Brzozowski, Zuzanna Dubanowska +1

Samsung AI Center, Warsaw, Poland · University of Warsaw, Poland

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Sep 29, 2026

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.

Yihao Ouyang, Shiwei Li, Haozhao Wang +5

Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

May 29, 2026

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Valérie Castin, Kimia Nadjahi, Pierre Ablin +1

1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence