What Should We Freeze? Guarded Freezing: Connectivity Shapes the Fine-Tuning of Pretrained Models

Leonel Aguilar

cs.LGSep 26, 2026

What Should We Freeze? Guarded Freezing: Connectivity Shapes the Fine-Tuning of Pretrained Models

Authors: Leonel Aguilar

Organizations: Chair of Cognitive Science, ETH Zürich

Abstract

When adapting pre-trained models through fine-tuning, freezing weights alone might not preserve performance, as updates elsewhere can change the inputs to the frozen core, ultimately affecting overall performance. We first analyse the case where a selected frozen core can be isolated and propose removal-value, a capacity-based score that approximates HOPE's removal cost averaged over removal orders. We show that in VGG-8, cutting paths from trainable neurons into a frozen core makes selection using this score useful: $70\%$ frozen preserves $5.22\pm0.51$ percentage points more old-task accuracy than DEFT at similar new-task accuracy. In transformers, shared residual streams leave paths into frozen neurons open. For this case, we derive drift-value, a forward-only proxy for the output disturbance from updating each weight entry under a local update model. In language models, at 40 epochs, this policy exceeds adapted Wanda and RIA freezing scores in settings with substantial retention loss, while its differences from Fisher remain unresolved. After 160 epochs on Qwen2.5-1.5B, it retains $0.0433\pm0.0102$ more than static Fisher. In DINOv3 vision-transformer adaptation to point clouds, drift-value retains $0.440$ image accuracy versus $0.187$ for a random mask of the same count. These results motivate Guarded Freezing: select by removal-value when incoming paths are cut, and by drift-value when they remain.

Figures & tables

Figure 1: Connectivity determines which changes can reach a frozen neuron. Colors mark fixed or trainable neurons and attention, and arrows show activation flow. (a) Cut trainable-to-frozen connections to block changes for fixed external inputs and unchanged normalization. Keep frozen-to-trainable connections so new computation can reuse fixed features. Our removal-value scores units. (b) Freezing weights leaves the shared residual stream connected. The bold path carries changes into a frozen neuron even with attention frozen. Drift-value scores weight entries by this disturbance. Plus signs denote addition. Selected layers are shown without normalization.

Figure 2: Cutting incoming trainable paths protects selected cores, but does not similarly protect random cores. Twenty VGG-8 adaptations, with

40\%

70\%

of units frozen. Left: old-task accuracy change from pretrained (dashed line). Right: new-task accuracy. Filled/open markers show incoming paths cut/kept. Core arms train at rate

0.01

. Gentle fine-tuning (dotted line) uses

0.0005

. Error bars and the gentle fine-tuning band show one SE. The pale row highlights the

70\%

removal-value core. Appendix Table 8 gives H-score.

Policy / selection rule	Retention (worst-task accuracy)	Acquisition (fact recall)	$\Delta$ retention vs drift-value
Pretrained model	$0.577\pm 0.005$	–	–
Full fine-tuning	$0.263\pm 0.010$	$0.988\pm 0.004$	$-0.239\pm 0.006$
Attention frozen	$0.356\pm 0.011$	$0.990\pm 0.003$	$-0.146\pm 0.009$
Random mask (control)	$0.426\pm 0.008$	$0.994\pm 0.003$	$-0.076\pm 0.006$
RIA freezing score	$0.429\pm 0.009$	$0.990\pm 0.003$	$-0.073\pm 0.009$
Wanda freezing score	$0.458\pm 0.011$	$0.987\pm 0.003$	$-0.044\pm 0.005$

Table 1: Drift-value leads Wanda and RIA. Its differences from SSU and Fisher remain unresolved. Qwen3-1.7B, nine seeds, bf16 SR, learning rate

10^{-4}

, 40 epochs. Below the divider attention is frozen. Each selection rule additionally freezes

35.2\%

of every feed-forward matrix, except the attention-only reference. SSU approximately matches this fraction with whole input columns. Means

\pm

SE, differences paired by seed. Rows within each block are ordered by retention. Random freezes the same count. Pale columns mark the main readout in every table. Bold numeric differences exceed the stated decision threshold.

Figure 3: Drift-value and Fisher do not separate at 40 epochs. Selection gains grow where attention freezing leaves more retention loss. Paired retention difference between drift-value and each comparison across six models, nine seeds. Positive values favor drift-value. Bars show one paired SE. Parentheses give the retention loss after attention freezing. Rows group models by family, then by increasing size. The gray band marks

\pm 0.031

, the minimum decision threshold. A resolved difference must also exceed twice its paired SE. All policies freeze attention and

35.2\%

of each feed-forward matrix (approximately for SSU, which freezes whole input columns). Qwen3-0.6B uses float32. The others use bf16 SR. Appendix Table 14 lists every policy.

Figure 4: The annex retains the most image accuracy by adding units. Drift-value exceeds Wanda and RIA at a matched entry budget. DINOv3 point-cloud adaptation under the stress protocol: 30 epochs, backbone learning rate

10^{-3}

. Left: image accuracy change from pretrained (

0.728

, dotted line). Right: point-cloud acquisition. Attention is frozen in every entry-mask policy. At

35.2\%

of each feed-forward matrix, drift-value retains

0.253\pm 0.008

more image accuracy than a matched random mask across nine paired seeds, with similar acquisition. The random and drift-value masks and attention freezing use nine seeds. Other rows use four. Error bars show one SE. Policies below the divider add parameters. Axis ranges differ.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Required data	Interpretation
Rounding-induced freeze	Weights, learning rate and format	Estimates rounding-limited updates
Removal-value	One forward calibration pass	Ranks units by contribution size
Drift-value	One forward calibration pass	Estimates local update disturbance
Vision drift-value	One pass over the 256 removal-value calibration images	Estimates local update disturbance
Remaining retention loss	Pretrained evaluation and an attention-frozen reference run	Input to the fitted headroom rule
Drift projection shares	Fine-tuned checkpoints	Measures the contribution of each writer family

Appendix

Appendix Table 2: Removal-value and drift-value need only forward calibration. Fisher also needs backward passes. The headroom rule needs a fitted slope and an attention-frozen run. The shaded column lists required data.

Level	Frozen	Spread (CV)	Model estimate	Measured gain	Gain vs. threshold
Attention heads	0.500	0.30	0.009 [0.009, 0.017]	$-0.003\pm 0.012$ (2/9)	unresolved
Units, removal-value, bf16	0.400	2.05	0.029	$+0.022\pm 0.020$ (6/9)	unresolved
Units, removal-value, bf16 SR	0.400	2.05	0.052	$+0.013\pm 0.020$ (5/9)	unresolved
Entries, drift-value	0.352	6.74	0.078 [0.066, 0.089]	$\bm{+0.077\pm 0.011}$ (9/9)	exceeds threshold

Appendix

Appendix Table 3: Only entry selection gives a resolved gain in these comparisons. Screening estimates for Qwen3-1.7B use calibration scores and measured response scales. The measured column gives each tested policy’s gain over its random control, paired by seed. Estimates are model predictions, not certified accuracy bounds.

Model	Precision	Rate	Frozen	Remaining retention loss	Random-mask gain	Predicted selection gain	Selection gain
Qwen3-4B ‡	bf16 SR	$10^{-4}$	0.352	0.353 $\pm$ 0.015	+0.100 $\pm$ 0.024 (9/9)	+0.115	$\bm{+0.152\pm 0.021}$ (9/9)
SmolLM2-1.7B ‡	bf16 SR	$2\times 10^{-4}$	0.352	0.346 $\pm$ 0.014	+0.042 $\pm$ 0.005 (9/9)	+0.112	$\bm{+0.134\pm 0.011}$ (9/9)
Qwen3-8B ‡	bf16 SR	$10^{-4}$	0.352	0.313 $\pm$ 0.030	+0.132 $\pm$ 0.032 (8/9)	+0.102	$\bm{+0.099\pm 0.026}$ (9/9)
Qwen2.5-1.5B ‡	bf16 SR	$10^{-4}$	0.352	0.304 $\pm$ 0.020	+0.108 $\pm$ 0.011 (9/9)	+0.099	$\bm{+0.078\pm 0.011}$ (9/9)
Qwen3-1.7B †	Kahan bf16	$10^{-4}$	0.352	0.229 $\pm$ 0.010	+0.084 $\pm$ 0.008 (9/9)	+0.075	$\bm{+0.074\pm 0.006}$ (9/9)
Qwen3-1.7B †	bf16 SR	$10^{-4}$	0.352	0.221 $\pm$ 0.012	+0.067 $\pm$ 0.012 (8/9)	+0.072	$\bm{+0.077\pm 0.011}$ (9/9)

Appendix

Appendix Table 4: Remaining retention loss predicts the scope for entry selection. Fifteen configurations, nine paired seeds per policy. Five configurations fit the slope and ten test it. Random-mask gain is relative to attention freezing alone. Selection gain is drift-value minus random. Bold gains exceed

\max(0.031,2\,\mathrm{SE})

Appendix Figure 5: Remaining retention loss predicts selection benefit within the tested task. The line is fit to the five filled settings. Open markers show ten additional settings. Circles freeze

35.2\%

and triangles

20\%

. Diamonds mark a model or rate absent from the fit at either fraction. Error bars show one SE. The dotted line is the minimum decision threshold, not a confidence bound. The pale region falls below it. Appendix Table 4 identifies every setting.

Appendix Figure 6: On Qwen2.5-1.5B drift-value pulls ahead of Fisher as adaptation runs longer, and its selected set changes less than Fisher’s. (a) Drift-value minus Fisher on retention against epochs, nine seeds, means

\pm

SE, the

\pm 0.031

minimum decision band shaded. The hollow markers at 160 epochs are SmolLM2-1.7B at

2\times 10^{-4}

and Qwen3-0.6B in bf16 with stochastic rounding. (b) Jaccard overlap of each score’s recomputed set with the set it replaces at epochs 40, 80 and 120. The dotted line is the approximate expected overlap of two independent random sets of the same size,

f/(2-f)=0.21

f=0.352

Model	Seeds	Pretrained retention	Attention frozen	Random mask	Drift-value retention	Selection gain	Drift-value acquisition
Qwen2.5-1.5B	9	0.642	0.338	0.446	0.524	$+0.078\pm 0.011$	0.989
Qwen3-0.6B	9	0.449	0.297	0.328	0.398	$+0.070\pm 0.011$	0.990
Qwen3-1.7B	9	0.577	0.356	0.426	0.502	$+0.076\pm 0.006$	0.990
Qwen3-4B	9	0.659	0.307	0.407	0.558	$+0.152\pm 0.021$	0.991
Qwen3-8B	9	0.716	0.402	0.534	0.633	$+0.099\pm 0.026$	0.988
SmolLM2-1.7B	9	0.664	0.545	0.607	0.636	$+0.029\pm 0.005$	0.982

Appendix

Appendix Table 5: Selection gains vary by model and follow the loss the attention freeze leaves. Rows group models by family, then by increasing size, as in Figure 3 . With

35.2\%

of every feed-forward matrix frozen, stochastic rounding, and learning rate

10^{-4}

, nine seeds a model. Retention is worst-task accuracy, and acquisition is fact recall. The gain is drift-value minus random with its paired SE. The Qwen3-1.7B row holds the fixed

35.2\%

of every matrix of Table 1 , not the count-matched mask of Appendix Tables 3 , 4 and 10 .

Policy	Seeds	Image retention	Point-cloud acquisition
Full fine-tuning	4	$0.025\pm 0.001$	$0.871\pm 0.008$
Removal-value core	4	$0.036\pm 0.002$	$0.874\pm 0.007$
DEFT core	4	$0.044\pm 0.006$	$0.872\pm 0.003$
DEFT core, exposed	4	$0.030\pm 0.005$	$0.872\pm 0.004$
Attention frozen	9	$0.097\pm 0.003$	$0.887\pm 0.003$
Attention + random, 20.0%	4	$0.127\pm 0.015$	$0.893\pm 0.002$

Appendix

Appendix Table 6: The annex retains most, and drift-value beats a random mask of the same count at all three budgets. All tested policies on the vision stress protocol of Figure 4 , means

\pm

SE over the seeds in the second column. Percentages beside entry policies are the share of each feed-forward matrix frozen. Unit cores freeze approximately

40\%

of feed-forward units.

	Target accuracy		Source retention		H-score
Method	HOPE	Rep.	HOPE	Rep.	HOPE	Rep.
DEFT	89.79	90.85	52.14	52.09	65.82	65.98
Head-only	36.11	34.84	63.13	62.78	45.79	44.69
Full fine-tuning	94.09	93.66	7.52	6.04	13.88	11.31
EWC	93.94	93.69	6.74	6.09	12.54	11.41
BN-only tuning	81.91	83.05	5.44	6.11	10.18	11.36

Appendix

Appendix Table 7: The VGG-8 reproduction matches HOPE’s reported DEFT result. The removal-value core improves its H-score on the same local scenarios. Target accuracy, source retention and H-score are percentages. Published means come from HOPE Table 2 ( Mobahi & Bartlett, 2026 ) . Rep. denotes the local reproduction over 20 paired CIFAR-100-to-SVHN transfers. Our DEFT port omits merging. The removal-value (Ours) row changes only DEFT’s selection score. Its paired H-score gain over local DEFT is

3.94\pm 0.39

(SE).

Frozen	Policy / connectivity	Retention (old-task acc., %)	Acquisition (new-task acc., %)	H-score
40%	Random core, exposed	$7.36\pm 0.19$	$82.16\pm 0.32$	$13.49\pm 0.32$
40%	Random core, isolated	$5.03\pm 0.03$	$81.50\pm 0.37$	$9.48\pm 0.05$
40%	DEFT core, exposed	$6.21\pm 0.17$	$81.01\pm 0.43$	$11.53\pm 0.30$
40%	DEFT core, isolated	$13.88\pm 0.90$	$81.31\pm 0.40$	$23.49\pm 1.31$
40%	Removal-value core, exposed	$6.67\pm 0.23$	$81.95\pm 0.31$	$12.32\pm 0.40$
40%	Removal-value core, isolated	$18.54\pm 1.42$	$81.20\pm 0.35$	$29.66\pm 1.93$

Appendix

Appendix Table 8: Isolation mainly changes source retention, while all policies reach the target. Complete comparison behind Figure 2 : means

\pm

SE over the same 20 transfers, in %. H-score is the harmonic mean of retention and acquisition.

Model	Policy	Retention	Acquisition
Qwen3-0.6B	Full fine-tuning	$0.356\pm 0.012$	$0.988\pm 0.004$
	Removal-value core	$0.384\pm 0.010$	$0.991\pm 0.003$
	DEFT core	$0.377\pm 0.012$	$0.990\pm 0.004$
	Random core	$0.383\pm 0.013$	$0.987\pm 0.003$
	Removal-value core, 70%	$0.393\pm 0.013$	$0.958\pm 0.028$
	Random core, 70%	$0.389\pm 0.010$	$0.992\pm 0.002$

Appendix

Appendix Table 9: At the lower learning rates, removal-value, DEFT and random cores do not separate. Two Qwen3 models, nine seeds, means

\pm

SE. Cores freeze

40\%

of feed-forward units, or

70\%

where marked. Qwen3-0.6B uses float32 at learning rate

2\times 10^{-5}

, and Qwen3-1.7B bf16 SR at

5\times 10^{-5}

Frozen feed-	Random mask		Drift-value mask		Paired
forward entries	Retention (worst-task acc.)	Acquisition (fact recall)	Retention (worst-task acc.)	Acquisition (fact recall)	gain in retention
20%	$0.388\pm 0.015$	$0.985\pm 0.004$	$0.474\pm 0.011$	$0.990\pm 0.003$	$+0.086\pm 0.013$
35.2%	$0.423\pm 0.008$	$0.990\pm 0.002$	$0.500\pm 0.008$	$0.991\pm 0.002$	$+0.077\pm 0.011$
50%	$0.471\pm 0.012$	$0.992\pm 0.002$	$0.518\pm 0.006$	$0.990\pm 0.003$	$+0.047\pm 0.012$

Appendix

Appendix Table 10: Drift-value exceeds random selection at each tested budget. Qwen3-1.7B, nine seeds and bf16 with stochastic rounding. Both masks freeze attention. Retention and acquisition are means

\pm

SE, and the final column gives drift-value minus random at each matched budget. Bold gains exceed

\max(0.031,2\,\mathrm{SE})

. Published-method comparisons appear in Table 1 and Appendix Table 13 .

Model	Removal-value minus random	Removal-value minus uniform allocation
Mistral-7B	$\bm{+0.138\pm 0.024}$	$-0.029\pm 0.003$
OLMo-2-7B	$\bm{+0.109\pm 0.020}$	$+0.002\pm 0.003$
Qwen3-0.6B	$\bm{+0.017\pm 0.006}$	$+0.009\pm 0.003$
Qwen3-1.7B	$+0.016\pm 0.013$	$+0.021\pm 0.004$
Qwen3-4B	$\bm{+0.043\pm 0.002}$	$+0.016\pm 0.001$
Qwen3-8B	$\bm{+0.030\pm 0.003}$	$-0.016\pm 0.009$

Appendix

Appendix Table 11: Removal-value selection improves on random selection, and changing allocation gives smaller gains. Eight-model control study. The first difference holds uniform allocation fixed, and the second holds the removal-value ranking fixed. Values are worst-task accuracy differences averaged over seven densities, with SE across three seeds. Bold selection differences exceed twice their paired SE.

Policy / selection rule	Retention (worst-task accuracy)	Acquisition (fact recall)	$\Delta$ retention vs drift-value
Pretrained model	$0.577\pm 0.005$	–	–
Output row gain alone	$0.466\pm 0.009$	$0.986\pm 0.005$	$-0.034\pm 0.007$
Input moment alone	$0.492\pm 0.010$	$0.990\pm 0.002$	$-0.008\pm 0.007$
Drift-value	$0.500\pm 0.008$	$0.991\pm 0.002$	reference
Drift-value, random tokens	$0.503\pm 0.006$	$0.986\pm 0.004$	$+0.003\pm 0.007$

Appendix

Appendix Table 12: Input-only and random-token variants have unresolved differences from drift-value. Row gain alone trails it. Score ablations on Qwen3-1.7B at the count the rounding rule of Appendix E names. Retention and acquisition are means

\pm

SE across nine seeds under stochastic rounding. The final column is paired retention minus drift-value.

Policy / selection rule	Retention (worst-task accuracy)	Acquisition (fact recall)	$\Delta$ retention vs drift-value
Pretrained model	$0.577\pm 0.005$	–	–
Random mask (control)	$0.471\pm 0.012$	$0.992\pm 0.002$	$-0.047\pm 0.012$
Half fine-tuning (HFT)	$0.474\pm 0.015$	$0.988\pm 0.003$	$-0.043\pm 0.013$
SSU columns	$0.504\pm 0.008$	$0.990\pm 0.004$	$-0.014\pm 0.005$
Drift-value	$0.518\pm 0.006$	$0.990\pm 0.003$	reference

Appendix

Appendix Table 13: Drift-value leads HFT and random entries at a

50\%

budget. Its difference from SSU is unresolved. Random entries, HFT, SSU, and drift-value at a

50\%

feed-forward freezing budget, with attention frozen. All use Qwen3-1.7B, bf16 with stochastic rounding, and nine seeds. SSU freezes whole input columns. The HFT row freezes two of the three feed-forward matrices in a random half of the blocks and one in each other block, so its budget is matched in aggregate rather than in each matrix. HFT itself also trains two of the four attention matrices in every block, which every row here holds frozen. Means

\pm

SE. The final column is paired retention minus drift-value.

Policy	Precision	LR	Retention (worst-task accuracy)	Change from pretrained accuracy	Acquisition (fact recall)	$\Delta$ retention vs attention + drift-value
Qwen2.5-1.5B
Full fine-tuning	bf16 SR	1e-4	$0.306{\pm}0.009$	$-0.336{\pm}0.013$	$0.986{\pm}0.003$	$-0.218{\pm}0.008$
DEFT core	bf16 SR	1e-4	$0.336{\pm}0.010$	$-0.307{\pm}0.018$	$0.991{\pm}0.003$	$-0.189{\pm}0.011$
SSU columns	bf16 SR	1e-4	$0.503{\pm}0.007$	$-0.139{\pm}0.011$	$0.990{\pm}0.004$	$-0.021{\pm}0.004$
Empirical Fisher	bf16 SR	1e-4	$0.508{\pm}0.010$	$-0.134{\pm}0.016$	$0.988{\pm}0.003$	$-0.017{\pm}0.009$
Wanda freezing score	bf16 SR	1e-4	$0.469{\pm}0.010$	$-0.173{\pm}0.016$	$0.986{\pm}0.004$	$-0.055{\pm}0.007$

Appendix

Appendix Table 14: With backbone precision matched, drift-value retains more than LoRA on Qwen3-0.6B and Qwen2.5-1.5B. Adaptation policies across six models. The final column is paired retention minus attention plus drift-value. Bold differences exceed

\max(0.031,2\,\mathrm{SE})

in magnitude. Qwen3-0.6B uses float32. The other initial blocks compare bf16 LoRA with stochastic-rounding freezing, so precision and policy both differ. The final three blocks use ordinary bf16 backbones for both methods. LoRA adapters remain float32. Of these three, only Qwen2.5-1.5B has matched acquisition. Rows with unmatched or unstable acquisition omit retention differences, as defined in the final note.

Policy	Precision	LR	Retention (worst-task accuracy)	Change from pretrained accuracy	Acquisition (fact recall)	$\Delta$ retention vs attention + drift-value
Qwen3-8B
Full fine-tuning	bf16 SR	1e-4	$0.271{\pm}0.016$	$-0.445{\pm}0.016$	$0.858{\pm}0.089$	variable acq.
DEFT core	bf16 SR	1e-4	$0.348{\pm}0.031$	$-0.367{\pm}0.033$	$0.989{\pm}0.002$	$-0.285{\pm}0.033$
SSU columns	bf16 SR	1e-4	$0.613{\pm}0.007$	$-0.103{\pm}0.007$	$0.990{\pm}0.004$	$-0.021{\pm}0.010$
Empirical Fisher	bf16 SR	1e-4	$0.632{\pm}0.009$	$-0.083{\pm}0.009$	$0.993{\pm}0.003$	$-0.001{\pm}0.009$
Wanda freezing score	bf16 SR	1e-4	$0.551{\pm}0.036$	$-0.164{\pm}0.038$	$0.989{\pm}0.004$	$-0.082{\pm}0.042$

Appendix

Table 21

Appendix Figure 7: Freezing the backbone preserves image accuracy while the annex learns point clouds. Single seed, stress protocol. Figure 4 gives the comparison across multiple seeds. (a) Point-cloud acquisition and (b) image retention, with accuracy before adaptation dotted. (c) Annex units’ share of HOPE’s unit capacity (Equation 1 ), measured separately on point-cloud and image inputs after training. Each share divides the annex units’ total capacity by that of all units in its feed-forward layer. The pale row marks the frozen-backbone annex.

Explore similar work

Sep 1, 2026cs.LG

Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through frozen-core adaptation: a calibration pass fixes left and right bases for each weight matrix, and fine-tuning optimizes only an

r\times r

core. This removes the ability of trainable factors to repair a poor initial span and makes subspace quality directly observable. We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates. Under a matched

r^2

budget, we compare eight basis constructors on 11 tasks, four model settings, and three seeds. On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks. It ranks first at all three Qwen scales and finishes within 0.13 points of the best method on Llama-3.2-1B. Controlled ablations show gains of 2.7--17.2 points from whitening and identify QR as necessary for stable core optimization in the tested regime. Finally, FCCA comes within 0.32 and 0.23 average points of LoRA and DoRA while optimizing 36.9K rather than roughly 7.4M parameters. These results show that a carefully selected fixed span can recover most of the benefit of movable low-rank factors at a much smaller trainable and optimizer-state cost.

Wentao Ye, Zhanming Shen, Zhiqing Xiao +3

Zhejiang University · Hunyuan, Tencent

Memory-Efficient Fine-Tuning LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Jul 25, 2026cs.LG

LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment

We present LARA (Lightweight Additive Residual Adaptation), a method for efficient adaptation that operates in the residual stream of a frozen model rather than in its weights. Where LoRA adds an update of low rank to weight matrices, LARA reads the hidden state at a small set of layers and adds a correction of low rank back to the residual stream, leaving all base weights untouched. On a code fine-tuning task and on preference optimization (DPO), LARA matches LoRA at equal parameter counts. Because adaptation is a frozen base plus a residual, LARA exposes a scale γ, applied at inference, that interpolates smoothly between base and adapted behavior, a form of graded control that adaptation in weight space does not offer. Finally, because each behavior is a small residual module over a shared frozen base, many behaviors can be held resident at once and routed automatically per token. We place seven behaviors, six fine-tuned and one optimized for preference, on one frozen 1.5B model for roughly 33 MB of overhead, against one full model for each behavior. Because the base is untouched, behaviors are trained separately and selected per token rather than loaded on demand, which suits hosting many behaviors, and adding new ones, on a single model on a device.

Pascal Ekin, Hyosun Choi, Wei Jie

Royal Holloway, University of London, United Kingdom · University of West London, United Kingdom

LLM Alignment Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Jul 18, 2026cs.AI

TopoTuner: Topological Finetuning of Large Language Models

Full fine-tuning remains a strong way to adapt pretrained LLMs, but it updates all weights and can be expensive. LoRA reduces the number of trainable parameters, but it does not directly answer which pretrained components should be trained and which can be frozen during adaptation. We introduce TopoTuner, a topology-guided fine-tuning framework for selective freezing of attention projection matrices. \method treats each projection matrix as a row cloud and uses Wasserstein distances between persistence diagrams to measure how its topology changes during fine-tuning. TopoTuner learns a reusable freezing profile from a source dataset and transfers it to efficiently fine-tune models on out-of-domain datasets, evaluating whether task-specific topological drift generalizes across question answering and sentiment analysis tasks. Across LLaMA-3.1-8B, Mistral-7B-v0.3, and Qwen3-8B-Base, TopoTuner is competitive with full fine-tuning while training only 1-2% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57% of the projection parameters. Along with minimized updates, TopoTuner reduces training time by 20.4% relative to full fine-tuning and 5.5% relative to LoRA on average. TopoTuner opens a new direction for reusable freezing profiles, where fine-tuning behavior learned on one dataset can be shared across multiple tasks.

Abdulkadir Erol, Yash Mahajan, Vepaul Hariprashad +4

Department of Computer Science University of Central Florida Orlando, USA · Department of Computer Science University of Manitoba Canada · Department of Finance University of Central Florida Orlando, USA

Fine-Tuning LLM Fine-Tuning Selective Fine-Tuning Parameter-Efficient Fine-Tuning

Ask PDF