Reducing the Adaptation Gap Through Reachable Fisher Geometry

Wasif Jalal; Sachin Deb; Asif Salekin

cs.LGSep 28, 2026

Reducing the Adaptation Gap Through Reachable Fisher Geometry

Authors: Wasif Jalal, Sachin Deb, Asif Salekin

Organizations: Arizona State University Tempe, Arizona, USA

Abstract

Parameter-efficient fine-tuning (PEFT) determines not only how many parameters are trained, but also which local directions a model can move in, so similar adapters can affect subgroup losses differently. Since curvature matrices are infeasible to form at adapter scale, scalar summaries such as the Fisher trace are often used instead. We study what the trace reveals and what it loses through the reachable Fisher: each subgroup's full-model Fisher pulled back through the adapter Jacobian. Under likelihood losses, it represents the Gauss-Newton curvature accessible to the adapter, and its trace can be computed from score-gradient norms without forming the full matrix. Under matched subgroup gradients, a positive-definite reachable-Fisher difference, with a margin exceeding the Hessian-Fisher defect, implies that every sufficiently small nonzero model-changing update increases the signed gap. In contrast, the restricted operator norm determines worst-case quadratic change, while a matrix-free Frobenius discrepancy bounds its reachable-Fisher component. Trace alone cannot certify definiteness or control matrix mismatch. Equal traces rule out a positive-definite difference but can still hide large operator discrepancies. Across 306 single-seed models, higher trace accompanies greater subgroup difficulty in 75.7 percent of 1,218 eligible evaluations, while trace matching reduces the best-worst subgroup gap in all 30 dataset-encoder-adapter combinations. However, held-out audits show that operator discrepancy decreases in 23 of 30 combinations, while the unbiased squared-Frobenius statistic decreases in only 16 of 30. Fisher trace is therefore a scalable diagnostic and training heuristic, but not a certificate of local gap behavior or matrix alignment.

Figures & tables

Question	Exact governing quantity	Reachable-Fisher certificate
Curvature term positive in every direction?	$\lambda^{\mathcal{H}}_{\min}(C_{ab})>0$	$\lambda^{\mathcal{H}}_{\min}(F^{\mathrm{R}}_{a}-F^{\mathrm{R}}_{b})>\eta^{\mathcal{H}}_{ab}$
Worst-case curvature change?	$\tfrac{1}{2}\\|C_{ab}\\|^{\mathcal{H}}_{2}\varepsilon^{2}$	at most $\tfrac{1}{2}\big(\\|F^{\mathrm{R}}_{a}-F^{\mathrm{R}}_{b}\\|^{\mathcal{H}}_{2}+\eta^{\mathcal{H}}_{ab}\big)\varepsilon^{2}$

Table 1: What controls direction and magnitude after slope matching. All quantities are restricted to

\mathcal{H}_{\phi}

; without slope matching, the linear term generally dominates small updates.

Dataset	Task	Subgroup axes (groups)	Models
Fitzpatrick17k ( Groh et al., 2021 )	3-class diagnosis	skin tone (7), binary tone (3)	17 encoders
UTKFace ( Zhang et al., 2017 )	age regression	race (5)	17 encoders
FHIBE face ( Xiang et al., 2025 )	age regression	ancestry (6)	17 encoders
Diabetes, 130 hospitals ( Strack et al., 2014 )	30-day readmission	race (6)	FT-Transformer ( Gorishniy et al., 2021 )

Table 2: Diagnostic datasets and model grid. A subgroup axis is the attribute defining the groups; group counts in parentheses. Full details in Table 5 .

	task-metric difficulty			surrogate-loss difficulty
Setting	cells	positive sign	encoder-cluster mean $r$ [95% CI]	cells	positive sign	mean $r$
Diabetes FT-Transformer	18	61.1%	$0.117$ [not estimable]	6	66.7%	$+0.284$
Fitzpatrick17k	600	59.3%	$0.320\ [0.184,\,0.424]$	200	81.5%	$+0.611$
UTKFace	300	98.3%	$0.930\ [0.901,\,0.950]$	100	98.0%	$+0.886$
FHIBE face	300	86.7%	$0.601\ [0.511,\,0.679]$	100	82.0%	$+0.411$
All cells (pooled)	1,218	75.7%	$0.637$	406	85.5%	$+0.668$

Table 3: Subgroup Fisher–difficulty alignment. Image-task means weight encoders equally, with encoder-bootstrap 95% intervals; sign rates are descriptive. Surrogate-loss results use the test split only, which is why their cell counts are smaller. The pooled row is a cell-level Fisher-

z

mean and is not comparable with the encoder-cluster means.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Training pipeline. After training, the pairwise Gram discrepancy of Equation 16 audits both arms (Section C.4 ). The task loss uses every row in the batch. The curvature penalty uses observed-label gradients with respect to the trainable adapter parameters and task head, aggregates their squared norms by subgroup, and combines the resulting penalty gradient with the task gradient.

	ConvNeXt–DoRA ( $p=1.58$ M)			ViT–LoRA ( $p=1.34$ M)
Step	s/step	$\times$ trace	peak MB	s/step	$\times$ trace	peak MB
no penalty	0.478	—	14,656	0.289	—	6,539
trace penalty	177.63	1.00	14,668	6.562	1.00	6,549
Frobenius penalty	176.98	1.00	14,668	6.093	0.93	6,549
trace statistic only	2.953	0.02	2,568	1.556	0.24	2,195
Frobenius statistic only	2.692	0.02	5,832	1.436	0.22	5,248

Appendix

Table 4: Measured cost of one training step, median of 20 timed steps after 3 warm-up steps on UTKFace at the training batch composition (

25

examples per subgroup

\times\,5

groups). The optimizer update is excluded; it is identical in every row. NVIDIA RTX 5090, PyTorch 2.14.0+cu130, CUDA 13.0.

Dataset	task	output	train/val/test	axes (groups)	encoders
Fitzpatrick17k	image classification	3 classes	11,592/2,484/2,484	tone (7), binary tone (3)	17
UTKFace	image regression	age	13,796/2,956/2,957	race (5)	17
FHIBE face	image regression	age	5,556/1,192/1,210	ancestry (6)	17
Diabetes 130H	tabular classification	binary	28,495/20,353/10,177	race (6)	FT-T

Appendix

Table 5: Evaluation settings. Split sizes are the stored fine-tuning train/validation/test partitions; diabetes additionally uses 42,741 earlier encounters for pretraining before applying FFT or PEFT. “Axes” lists the criteria of subgroups used in curvature-difficulty correlations.

Figure 2: Encoder-level alignment by task. Blue points are encoder-level Fisher-

z

means, red diamonds are equal-encoder task means, and error bars bootstrap encoders; percentages are descriptive positive-sign rates over cells. Summary values are in Table 3 and method-conditioned means in Table 8 .

MMethod	params (%)	$\Delta$ Fitz BAcc	$\Delta$ UTK MAE	$\Delta$ FHIBE MAE	Fisher-CV ratio	gap ratio
LoRA	1.57	$+2.81$	$-0.34$	$-1.33$	1.077	0.823
DoRA	1.66	$+3.21$	$-0.31$	$-1.38$	1.095	0.902
rsLoRA	1.57	$+1.46$	$-0.28$	$-0.79$	1.123	0.901
PiSSA	1.55	$+1.01$	$+0.00$	$-0.88$	0.956	0.946
VeRA	0.10	$-0.19$	$-0.04$	$-1.23$	0.603	0.745

Appendix

Table 6: Balanced 16-encoder comparison to full fine-tuning. Fitzpatrick17k entries are balanced-accuracy changes in percentage points (higher is better); UTKFace and FHIBE face are MAE changes in years (lower is better). Ratios are geometric means across the three image tasks; values below one denote a reduction. Both ratios compare different parameterizations and are descriptive.

Quantity	task	LoRA	DoRA	rsLoRA	PiSSA	VeRA
Fisher CV	Fitzpatrick17k	0.953	0.845	1.009	0.907	0.913
	UTKFace	0.933	0.927	0.915	0.950	0.537
	FHIBE face	1.405	1.676	1.536	1.016	0.446
Performance gap	Fitzpatrick17k	0.943	1.195	1.031	0.977	1.013
	UTKFace	0.856	0.938	0.915	1.019	0.929
	FHIBE face	0.690	0.656	0.777	0.851	0.440

Appendix

Table 7: Per-task ratios relative to full fine-tuning on the balanced 16-encoder roster.

Setting	full FT	LoRA	DoRA	rsLoRA	PiSSA	VeRA
Fitzpatrick17k	0.337	0.217	0.422	0.321	0.381	0.330
UTKFace	0.897	0.929	0.942	0.932	0.934	0.945
FHIBE face	0.400	0.700	0.626	0.587	0.552	0.655
Diabetes FT-T	0.149	0.130	$-0.039$	0.163	0.470	$-0.206$

Appendix

Table 8: Fisher-

z

mean aligned Pearson

r

by setting and method.

Task	PEFT	gap wins	plain gap	trace gap	median $R_{\mathrm{gap}}$ (%)	gap and utility	$\Delta$ utility
Fitzpatrick17k	LoRA	3/3	0.247	0.146	$+36$	1/3	$+0.1$
Fitzpatrick17k	DoRA	3/3	0.238	0.201	$+15$	2/3	$-0.0$
Fitzpatrick17k	PiSSA	3/3	0.219	0.153	$+32$	2/3	$+0.5$
UTKFace (MAE, yr)	LoRA	3/3	1.57	1.39	$+14$	3/3	$-0.26$
UTKFace (MAE, yr)	DoRA	3/3	1.52	1.28	$+15$	2/3	$-0.22$
UTKFace (MAE, yr)	PiSSA	3/3	1.87	1.59	$+12$	3/3	$-0.23$

Appendix

Table 9: Primary analysis by task and adapter, each cell at its selected batch size. Image rows pool the three encoders; each diabetes row is its single FT-Transformer cell, so its counts are out of one. Gap wins count cells with

R_{\mathrm{gap}}>0

. Plain and trace gaps are means over the cells in the row. “Gap and utility” counts cells that narrow the gap without a loss of pooled utility.

\Delta

utility is the mean change in balanced accuracy (percentage points, higher is better) or MAE (years, lower is better).

Figure 3: Share of the 30 cells, each at its selected batch size, in which trace matching lowers the gap, EOD or AOD relative to the plain arm, by dataset, adapter and encoder. EOD and AOD are undefined for the regression tasks (n/a); 0% marks a measured value that never favoured trace matching. EOD and AOD count the 12 classification cells only. The dotted line marks 50%.

Figure 4: Best–worst gap, plain $\to$ trace matching, for all 30 cells, each at its selected batch size. Colour is $R_{\mathrm{gap}}$ ; blue means trace matching narrows the gap. The image encoders do not apply to diabetes and the FT-Transformer does not apply to the image tasks (n/a).

Figure 5: Primary cells, from the experiment pipeline. The axes use a change convention,

\Delta G=100(G_{\mathrm{H}}-G_{\mathrm{P}})/G_{\mathrm{P}}=-R_{\mathrm{gap}}

and likewise for the CV, so negative values are reductions. Left:

\Delta

CV against

\Delta G

over all 30 cells (Spearman

\rho=+0.23

; no

p

-value is reported because the cells share tasks, encoders and adapters). Right: relative change in pooled utility (right is better) against

\Delta G

. The per-cell CV values are in Table 14 .

Dataset	Encoder	PEFT	BS	Worst P $\to$ H	Pooled P $\to$ H	Gap P $\to$ H [drop]	EOD P $\to$ H [drop]
Fitzpatrick17k	ConvNeXt	DoRA	256	$0.603\!\to\!0.594$	$0.708\!\to\!0.716$	$0.182\!\to\!0.154$ [ $15.5\%$ ]	$0.182\!\to\!0.096$ [ $47.4\%$ ]
Fitzpatrick17k	ConvNeXt	LoRA	256	$0.598\!\to\!0.617$	$0.733\!\to\!0.725$	$0.194\!\to\!0.145$ [ $25.7\%$ ]	$0.153\!\to\!0.116$ [ $24.1\%$ ]
Fitzpatrick17k	ConvNeXt	PiSSA	256	$0.557\!\to\!0.613$	$0.716\!\to\!0.721$	$0.177\!\to\!0.170$ [ $3.7\%$ ]	$0.064\!\to\!0.145$ [ $-127.7\%$ ]
Fitzpatrick17k	ViT	DoRA	128	$0.566\!\to\!0.570$	$0.747\!\to\!0.737$	$0.244\!\to\!0.217$ [ $10.9\%$ ]	$0.108\!\to\!0.072$ [ $34.0\%$ ]
Fitzpatrick17k	ViT	LoRA	128	$0.496\!\to\!0.644$	$0.738\!\to\!0.727$	$0.273\!\to\!0.118$ [ $56.9\%$ ]	$0.063\!\to\!0.072$ [ $-13.8\%$ ]
Fitzpatrick17k	ViT	PiSSA	128	$0.534\!\to\!0.598$	$0.740\!\to\!0.715$	$0.260\!\to\!0.139$ [ $46.5\%$ ]	$0.145\!\to\!0.059$ [ $59.6\%$ ]

Appendix

Table 10: Primary Eval 1 results for all 30 cells. P and H denote plain and Huber-curvature training. BS is the selected batch size shared by the two arms of each pair. Bracketed values are percentage reductions, so positive values favor Huber. Balanced accuracy is reported for classification and MAE for regression; regression EOD is undefined (—).

Figure 6: Fitzpatrick17k primary cells: matched plain versus Huber-curvature results at the selected batch size. Rows are the three encoders, columns the three PEFT methods. Higher is better for balanced accuracy; lower is better for gap and EOD.

Figure 7: UTKFace primary cells: matched plain versus Huber-curvature results at the selected batch size. Rows are the three encoders, columns the three PEFT methods. Higher is better for balanced accuracy; lower is better for MAE and gap. UTKFace is a regression task, so EOD is undefined.

Figure 8: FHIBE Face primary cells: matched plain versus Huber-curvature results at the selected batch size. Rows are the three encoders, columns the three PEFT methods. Higher is better for balanced accuracy; lower is better for MAE and gap. FHIBE Face is a regression task, so EOD is undefined.

Figure 9: Diabetes FT-Transformer primary cells: matched plain versus Huber-curvature results at the selected batch size, one panel per PEFT method. Higher is better for balanced accuracy; lower is better for gap and EOD.

			train time (s)			peak mem (GB)		GPU util (%)
Encoder	Dataset	PEFT	adapter	full	$\Delta t$	adapter	full	adapter / full
ViT	FHIBE face	LoRA	$63.30\pm 0.21$	$101.30\pm 0.34$	$+60\%$	6.42	13.55	$92.2$ / $95.1$
		DoRA	$87.41\pm 0.32$	$123.74\pm 0.58$	$+42\%$	12.69	20.55	$90.7$ / $94.8$
		PiSSA	$62.55\pm 0.44$	$100.67\pm 0.32$	$+61\%$	6.39	13.53	$92.2$ / $95.0$
	UTKFace	LoRA	$155.35\pm 0.49$	$248.97\pm 0.37$	$+60\%$	6.42	13.55	$95.6$ / $97.3$
		DoRA	$212.89\pm 0.52$	$304.80\pm 0.57$	$+43\%$	12.69	20.55	$92.8$ / $96.2$

Appendix

Table 11: Measured cost of adapter versus full penalty scope on a primary evaluation device NVIDIA RTX 5090. Each entry is the mean

\pm

standard deviation over repeated timed runs. Train time is seconds per run;

\Delta t

is the increase from adapter to full scope. Peak memory is the maximum allocated during the run, and is independent of the dataset because the batch composition is fixed, so the two datasets repeat the same value. GPU utilization is the mean over the run. This table reports cost only; it makes no quality comparison between the two scopes.

			train time (s)			peak mem (GB)		GPU util (%)
Encoder	Dataset	PEFT	adapter	full	$\Delta t$	adapter	full	adapter / full
ViT	FHIBE face	LoRA	$377.83\pm 0.30$	$594.81\pm 0.68$	$+57\%$	6.42	13.55	$93.9$ / $94.3$
		DoRA	$542.34\pm 0.87$	$757.80\pm 0.76$	$+40\%$	12.69	20.55	$94.1$ / $94.3$
		PiSSA	$344.20\pm 0.72$	$590.89\pm 0.75$	$+72\%$	6.39	13.53	$93.6$ / $94.5$
	UTKFace	LoRA	$940.16\pm 0.64$	$1477.05\pm 0.49$	$+57\%$	6.42	13.55	$94.6$ / $94.9$
		DoRA	$1350.09\pm 0.69$	$1883.82\pm 1.25$	$+40\%$	12.69	20.55	$94.5$ / $94.6$

Appendix

Table 12: Measured cost of adapter versus full penalty scope, on a second evaluation device (NVIDIA GB10). Each entry is the mean

\pm

standard deviation over the three training epochs. Train time is seconds per epoch;

\Delta t

Task	Encoder	Adapter	trace gap	Frobenius	operator norm	loss gap
Fitzpatrick17k	ConvNeXt	DoRA	-25	+98	+1	-49
Fitzpatrick17k	ConvNeXt	LoRA	-31	-84	-4	-40
Fitzpatrick17k	ConvNeXt	PiSSA	+20	-41	+10	+19
Fitzpatrick17k	SigLIP 2	DoRA	+4	-18	-3	-33
Fitzpatrick17k	SigLIP 2	LoRA	+20	+66	-4	-16
Fitzpatrick17k	SigLIP 2	PiSSA	-17	-25	+19	-32

Appendix

Table 13: Post-training audit, per cell. Each entry is the change from the plain arm to trace matching, in percent, so negative favours trace matching; bold marks a reduction. Trace gap, Frobenius discrepancy and operator norm are normalised by

s_{ab}

and averaged over subgroup pairs and held-out batches; the loss gap is the best–worst subgroup gap in the training loss.

Task	Encoder	Adapter	CV plain	CV matched	$\Delta$ CV
Fitzpatrick17k	ConvNeXt	DoRA	0.095	0.154	+62
Fitzpatrick17k	ConvNeXt	LoRA	0.206	0.142	-31
Fitzpatrick17k	ConvNeXt	PiSSA	0.161	0.327	+104
Fitzpatrick17k	SigLIP 2	DoRA	0.164	0.123	-25
Fitzpatrick17k	SigLIP 2	LoRA	0.165	0.095	-43
Fitzpatrick17k	SigLIP 2	PiSSA	0.139	0.088	-37

Appendix

Table 14: Expected-Fisher trace coefficient of variation across the constrained subgroups, per cell, recomputed from the stored checkpoints on the test split by mitigation/mechanism_cv.py .

\Delta

CV is the change from the plain arm to trace matching in percent, so negative means trace matching made the subgroup traces more even; bold marks a reduction. These are the per-cell values behind Figure 5 .

Check	Result
Gauge (Prop. 4 )	$\dim\ker J=4=r^{2}$ ; $\\|F^{\mathrm{R}}_{c}v\\|\leq 10^{-15}$ for $v\in\ker J$ ; the unrestricted $\lambda_{\min}(F^{\mathrm{R}}_{a}-F^{\mathrm{R}}_{b})=-4.45<0$ .
Expansion (Prop. 1 )	over 8 random horizontal directions, the remainder after the slope and curvature terms scales as $\varepsilon^{3.05}$ (median log–log slope).
Worst case (Thm. 2 )	along the dominant restricted eigenvector (eigenvalue $-3.13$ , so $\\|C_{ab}\\|^{\mathcal{H}}_{2}=3.13$ ), the second-order change matches $\mathcal{Q}_{ab}(\varepsilon)=\tfrac{1}{2}\\|C_{ab}\\|^{\mathcal{H}}_{2}\varepsilon^{2}$ in magnitude to within 34.6%, 5.8%, 2.2%, 1.2% and 0.6% at $\varepsilon=0.2,0.1,0.05,0.025,0.0125$ .
Sign on $S$	$C_{ab}$ restricted to $S$ is indefinite (eigenvalues in $[-2.01,1.27]$ ); the sign of the actual gap change matched $\operatorname{sign}(\delta^{\top}C_{ab}\delta)$ in 216 of 216 random directions at $\varepsilon=0.01$ .
Forced increase on $S_{+}$ (Thm. 1 )	$\dim S_{+}=6$ , $\rho_{\min}=0.0105$ ; all 1,200 sampled steps increased the gap, by at least $2.79\times\tfrac{1}{4}\rho_{\min}\\|\delta\\|^{2}$ .
Certificate (Cor. 1 )	on $S_{+}$ , $\lambda_{\min}(F^{\mathrm{R}}_{a}-F^{\mathrm{R}}_{b})=-0.100$ against a defect of $0.171$ : the reachable-Fisher certificate fails although $C_{ab}$ is positive definite there; on $\mathcal{H}_{\phi}$ , $\eta^{\mathcal{H}}_{ab}/\\|F^{\mathrm{R}}_{a}-F^{\mathrm{R}}_{b}\\|^{\mathcal{H}}_{2}=0.66$ .

Appendix

Table 15: Exact checks on the small network ( validate_exact.py ).

S

is the part of

\mathcal{H}_{\phi}

orthogonal to

g_{a}-g_{b}

, on which the slope term vanishes for every update (the one-step constraint), and

S_{+}

the span of the positive eigenvectors of

C_{ab}

restricted to

S

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 605 papers

Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning
Jun 8, 2026Ghodsiyeh Rostami, Po-Han Chen, Mahdi S. HosseiniFine-Tuning Vision Foundation Model Adaptation
Concordia University · Concordia University Mila - Québec AI Institute
Loss-Parameterized Fisher Width Along Learning Trajectories
Aug 21, 2026Vu Khac KyFisher Information Neural Network Training Dynamics
Department of Mathematics, FPT University, Vietnam
Learning Only What Valid Adapters Can Express: Subspace-Constrained Adaptation Against Fine-Tuning Poisoning
Jul 6, 2026Fabien PollyLLM Backdoor Attacks Low-Rank Adaptation
Independent researcher

Jun 8, 2026J/K move · Enter open · S save

Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning

Ghodsiyeh Rostami, Po-Han Chen, Mahdi S. Hosseini

Concordia University · Concordia University Mila - Québec AI Institute

Fine-Tuning Vision Foundation Model Adaptation Selective Fine-Tuning Fisher Information Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parameter-efficient fine-tuning often selects trainable parameters before adaptation using architectural heuristics, without accounting for their varying importance during training. We introduce FisherAdapTune, which progressively selects parameter groups based on temporal drift in their Fisher information. Under a local Gaussian approximation, we bound the divergence between the fine-tuned posterior and pretrained prior by accumulated Fisher-weighted update costs, motivating curvature-aware selection. FisherAdapTune measures Jensen-Shannon distance between successive Fisher-value distributions and uses an adaptive threshold to freeze stabilized groups. Across VTAB-1k classification tasks, it achieves the highest macro Top-1 accuracy among the compared methods with a smaller average trainable set than full fine-tuning. Across four segmentation backbones, it maintains competitive in-distribution performance and improves zero-shot transfer in several settings. Its selections reveal architecture-dependent patterns where input, output, and normalization parameters can remain trainable as attention and MLP groups freeze at different rates. The results support Fisher structural drift as a task-dependent signal for allocating updates during adaptation. We release our code publicly to enable further application of our proposed approach.

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning

Loss-Parameterized Fisher Width Along Learning Trajectories

Learning Only What Valid Adapters Can Express: Subspace-Constrained Adaptation Against Fine-Tuning Poisoning