Beyond Endpoint Gains: A Weight-Delta Audit of Medical Specialization

Praphul Singh; Shanu Kumar; Akshat Agarwal

cs.AIAug 21, 2026

Beyond Endpoint Gains: A Weight-Delta Audit of Medical Specialization

Authors: Praphul Singh, Shanu Kumar, Akshat Agarwal

Organizations: Oracle Health AI · IIT Kanpur · MBZUAI

Abstract

Specialist language models are usually understood through endpoint gains: the generalist scores lower, the specialist scores higher, and the difference is treated as evidence of specialization. This leaves the released update itself largely unexamined. We propose a paired weight-delta path audit and apply it to two public, aligned generalist-to-medical-specialist checkpoint pairs: Gemma-3-4B-IT to MedGemma-4B-IT and Qwen2.5-7B-Instruct to HuatuoGPT-o1-7B. In both pairs, the full decoder-side update strongly reconstructs measured medical benchmark movement (0.974 and 1.183 endpoint-normalized retention), making each decoder delta an appropriate substrate for the audit. Yet the movement is not cleanly localized. MLP is the strongest broad component family in both pairs, but mixed off-domain movements, 10-seed matched controls, and endpoint-anchored rollbacks prevent a unique coarse-family explanation. The audit therefore separates update-level reconstruction from component-level explanation. Its claims concern text-only multiple-choice benchmark movement, not clinical validation, repair, or circuit-level mechanism.

Figures & tables

Figure 1: Paired checkpoint audit protocol, illustrated with the primary Gemma $\rightarrow$ MedGemma pair. For each aligned release, we first test whether the full decoder-side path reconstructs measured benchmark movement, then whether component-family interpretations survive matched controls and endpoint rollback.

View	Sources	Count
Medical composite	MedQA; MMLU Clinical Knowledge; MMLU Professional Medicine	1,810
Full non-clinical	GPQA Diamond; MMLU-Pro; CommonsenseQA; TruthfulQA MC1; Kaleidoscope text-only	7,325
Selected regressions	Endpoint drops with at least 80 examples and gap at least 0.02	2,469
Selected gains	Endpoint gains with at least 80 examples and gap at least 0.02	2,606

Table 1: Evaluation views. The medical composite is the target-domain readout; the full non-clinical suite checks aggregate endpoint behavior; selected regression and gain slices are conditional diagnostics used for path analysis.

Endpoint comparison	$\Delta$	95% CI
Medical composite	+0.085	[ +0.062, +0.108 ]
Selected-regression slices	-0.035	[ -0.058, -0.015 ]
Selected-gain slices	+0.063	[ +0.032, +0.084 ]

Table 2: Endpoint movements used to normalize the path readouts. Deltas are MedGemma minus Gemma.

Panel of Figure 2 — Figure 2: Decoder-side $\Delta W$ screen. The released update is structured across layers and projection families rather than concentrated in a single obvious component. This screen motivates the tested buckets; behavioral claims come from path sweeps and controls.

Figure 3: Full decoder-side paths for both checkpoint pairs. Each observed update reconstructs the medical endpoint movement while carrying mixed selected off-domain movements. The Qwen curves connect only evaluated path points. Selected slices are endpoint-conditioned diagnostics, not estimates of broad non-clinical forgetting.

Figure 4: Scoped component-family paths show partial sufficiency, not localization. The MLP family is strongest among scoped buckets, but no tested bucket matches the full decoder path and the strongest bucket is also large.

Bucket	Best scale	Med. ret.	Reg. rec.
Attention	0.70	0.438	0.767
MLP	0.80	0.797	0.628
Attention+down	0.70	0.601	0.744
Full-down	0.70	0.667	0.465

Table 3: Best observed points for scoped decoder-side component-family paths. Medical retention is normalized by the endpoint medical gain; selected-regression recovery is endpoint-conditioned.

Diagnostic	Point	Med. ret.	Reg. rec.
MLP scoped best	$t=0.80$	0.797	0.628
MLP param. ctrl (10)	$t=1.00$	0.814 $\pm$ 0.067	0.236 $\pm$ 0.175
MLP $\Delta$ E ctrl (10)	$t=1.00$	0.563 $\pm$ 0.100	0.262 $\pm$ 0.242
Attn+down scoped best	$t=0.70$	0.601	0.744
Attn+down matrix ctrl (10)	$t=1.00$	0.703 $\pm$ 0.112	0.388 $\pm$ 0.159
Attn+down param. ctrl (10)	$t=1.00$	0.417 $\pm$ 0.086	0.585 $\pm$ 0.166

Table 4: Controls and anchored rollbacks test whether component sufficiency becomes explanation. The table distinguishes scoped-best path points from controls at

t=1

and full rollbacks at

\alpha=0

. Control entries are mean

\pm

standard deviation over 10 seeds.

Diagnostic	Gemma–MedGemma	Qwen–Huatuo
Medical endpoint gain	+0.0845	+0.0453
Full decoder retention	0.974	1.183
MLP retention at $t=1$	0.699	1.024
MLP parameter control	0.814	1.115
MLP $\Delta W$ -energy control	0.563	1.098
Attention rollback: med./reg.	0.699 / 0.279	0.756 / 0.387

Table 5: Cross-pair audit summary. Control entries are mean medical retention at

t=1

; rollback entries report medical retention / selected-regression recovery after fully removing the named family.

Figure 5: Endpoint-anchored rollback sweeps show that no family is a clean repair knob. MLP rollback recovers selected regressions more than attention rollback, but at a much larger medical-retention cost.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

View	Sources / rule	Count
Medical composite	MedQA; MMLU Clinical Knowledge; MMLU Professional Medicine	1,810
Full non-clinical	GPQA Diamond; MMLU-Pro; CommonsenseQA; TruthfulQA MC1; Kaleidoscope text-only	7,325
Selected regressions	Endpoint drops with at least 80 examples and gap at least 0.02	2,469
Selected gains	Endpoint gains with at least 80 examples and gap at least 0.02	2,606

Appendix

Table 6: Evaluation views used in the audit. Selected regression and gain views are endpoint-conditioned diagnostics, not population estimates of all non-clinical behavior.

View	$\Delta$	95% CI
Medical	+0.085	[+0.062, +0.108]
Selected regressions	-0.035	[-0.058, -0.015]
Selected gains	+0.063	[+0.032, +0.084]

Appendix

Table 7: Endpoint movements used to normalize the path readouts. Deltas are MedGemma minus Gemma.

Figure 6: Endpoint movement summary with bootstrap intervals. The full non-clinical aggregate is used descriptively in the main text and is not the denominator for the selected-slice readouts.

Source	Count	Gemma	MedGemma	$\Delta$	95% CI
MedQA (USMLE 4-option)	1,273	0.469	0.551	+0.082	[+0.053, +0.109]
MMLU Clinical Knowledge	265	0.604	0.645	+0.042	[-0.019, +0.098]
MMLU Professional Medicine	272	0.529	0.669	+0.140	[+0.081, +0.199]
Composite	1,810	0.498	0.582	+0.085	[+0.062, +0.108]

Appendix

Table 8: Medical endpoint performance by source. This table makes explicit that the composite gain is positive across the three medical sources, while the composite is example-weighted and therefore MedQA-dominated.

View	Count	Gemma	MedGemma	$\Delta$	95% CI
Full non-clinical	7,325	0.410	0.418	+0.009	[-0.002, +0.020]

Appendix

Table 9: Full non-clinical endpoint aggregate. The small positive aggregate movement motivates treating selected regressions as diagnostics rather than broad forgetting.

Table 10: Exact full non-clinical source composition and endpoint movements.

Table 11: Endpoint-selected diagnostic slices. Selection uses source-level endpoint gaps of at least 0.02 accuracy and at least 80 examples.

$t$	Med. acc.	Med. ret.	Reg. rec.	Gain acq.
0.10	0.5193	0.255	0.953	0.220
0.20	0.5293	0.373	0.965	0.354
0.25	0.5365	0.458	1.047	0.445
0.30	0.5448	0.556	0.895	0.506
0.40	0.5619	0.758	0.849	0.610
0.50	0.5707	0.863	0.744	0.665

Appendix

Table 12: Full decoder-side path grid. Gain acquisition is the acquired fraction of selected endpoint gains; selected-regression recovery is computed on endpoint-regression slices.

Figure 7: Bootstrap confidence bands for the Gemma–MedGemma full decoder path. Medical reconstruction is the most stable readout; selected-slice diagnostics have wider intervals and are interpreted directionally.

Panel of Figure 8 — Figure 8: Per-source selected-slice diagnostics along the full decoder-side path. These views are conditional on endpoint slice selection and are used as diagnostics rather than broad non-clinical estimates.

Bucket	Matrices	Param. %	$\Delta W$ energy %	Role
Full decoder	238	100.0	100.0	Reference
Attention family	136	16.7	35.0	Component
MLP family	102	83.3	65.0	Component
Attn+ down	170	44.4	44.2	Mixed
Full– down	204	72.2	90.8	Leave-one-out

Appendix

Table 13: Bucket definitions for decoder-side component-family sweeps. Shares are relative to the full decoder-side bucket.

Bucket	Scale	Med. ret. [CI]	Reg. rec. [CI]
Attention	0.70	0.438 [0.205, 0.673]	0.767 [-0.122, 1.276]
MLP	0.80	0.797 [0.608, 1.029]	0.628 [-0.154, 1.193]
Attention+down	0.70	0.601 [0.374, 0.849]	0.744 [0.020, 1.485]
Full-down	0.70	0.667 [0.454, 0.892]	0.465 [-0.286, 0.922]

Appendix

Table 14: Best observed scoped component-family points with bootstrap intervals.

Target	Control	$n$	Med. retention	Reg. recovery	Structured
Attention+down	delta-energy	10	0.416 $\pm$ 0.151	0.431 $\pm$ 0.136	0.510/0.570
Attention+down	matrix-count	10	0.703 $\pm$ 0.112	0.388 $\pm$ 0.159	0.510/0.570
Attention+down	parameter-count	10	0.417 $\pm$ 0.086	0.585 $\pm$ 0.166	0.510/0.570
MLP	delta-energy	10	0.563 $\pm$ 0.100	0.262 $\pm$ 0.242	0.699/0.279
MLP	matrix-count	10	0.437 $\pm$ 0.077	0.513 $\pm$ 0.224	0.699/0.279
MLP	parameter-count	10	0.814 $\pm$ 0.067	0.236 $\pm$ 0.175	0.699/0.279

Appendix

Table 15: Matched random controls over 10 seeded draws. Entries report mean

\pm

standard deviation; the final column gives the corresponding structured-bucket medical retention / selected-regression recovery, with both controls and structured comparators evaluated at

t=1

Figure 9: Matched random buckets approach the structured component buckets. Stars denote the structured MLP and attention+ down_proj buckets; other markers are 10-seed random-control means, with one-standard-deviation bars, matched by parameter count, matrix count, or $\Delta W$ energy. Because several controls approach the stars, the plot supports partial sufficiency rather than unique component-family localization.

Rolled-back family	$\alpha$	Med. ret.	Reg. rec.
Attention	0.00	0.699	0.279
Attention	0.25	0.784	0.419
Attention	0.50	0.902	0.314
Attention	0.75	0.993	0.256
Attention	1.00	0.974	0.070
MLP	0.00	0.359	0.593

Appendix

Table 16: Anchored rollback paths. Lower

\alpha

rolls back the named family while leaving the rest of the decoder update fixed at the specialist endpoint.

Endpoint view	Estimate	95% CI
Medical composite	+0.0453	+0.0254	+0.0641
Selected regressions	-0.0617	-0.0813	-0.0417
Selected gains	+0.0554	+0.0387	+0.0721

Appendix

Table 17: Qwen–HuatuoGPT endpoint movements and evaluated full-path grid.

Group	Matrices	Param. share	$\Delta W$ energy	Mean rank
Attention	112	12.6%	12.2%	477.7
MLP	84	87.4%	87.8%	1522.3
gate_proj	28	29.1%	30.6%	1471
up_proj	28	29.1%	29.0%	1354
down_proj	28	29.1%	28.2%	1742

Appendix

Table 18: Qwen decoder projection coverage and update geometry.

Figure 10: Qwen structured-family medical-retention trajectories at the evaluated path points. MLP is strongest, but its size motivates matched controls.

Target	Match	Med. mean	Range	Reg. mean	Gain mean
Attn+down	$\Delta W$ energy	0.660	0.573–0.780	0.744	0.674
Attn+down	Matrices	1.117	1.061–1.256	0.685	0.941
Attn+down	Parameters	0.667	0.476–0.817	0.799	0.707
MLP	$\Delta W$ energy	1.098	1.037–1.159	0.548	1.102
MLP	Matrices	0.680	0.427–0.902	0.837	0.713
MLP	Parameters	1.115	1.024–1.195	0.527	1.082

Appendix

Table 19: Qwen matched controls at

t=1

over 10 seeded draws. Ranges show the observed minimum and maximum, not standard deviations or confidence intervals.

Figure 11: Qwen control means and observed ranges over 10 seeded draws. Stars mark the corresponding structured bucket at

t=1

Module-only path	Med. retention
down_proj	0.512
up_proj	0.439
gate_proj	0.366
o_proj	0.183
v_proj	0.061
q_proj	0.012

Appendix

Table 20: Qwen module-only and leave-one-module-out paths at

t=1

Full rollback	Med. retention	Reg. recovery	Gain acquisition
Attention	0.756 [0.494, 1.000]	0.387 [0.212, 0.600]	0.882 [0.723, 1.084]
MLP	0.293 [0.011, 0.541]	0.986 [0.800, 1.234]	0.471 [0.297, 0.677]
Attn+down	0.549 [0.295, 0.841]	0.570 [0.342, 0.789]	1.000 [0.790, 1.296]
Full-minus-down	0.317 [0.085, 0.557]	0.866 [0.671, 1.083]	0.559 [0.394, 0.774]

Appendix

Table 21: Qwen full rollback results with 95% bootstrap intervals.

Figure 12: Qwen rollback tradeoff. No tested family simultaneously preserves the medical movement and fully recovers selected regressions.

Ranking comparison	Top-100	Top-1k	Top-5k
Medical imp. vs medical drift	0.212	0.255	0.275
Medical imp. vs regression drift	0.250	0.277	0.283
Medical imp. vs regression imp.	0.786	0.619	0.603
Medical drift vs regression drift	0.515	0.514	0.520
Regression imp. vs regression drift	0.266	0.278	0.317
Medical imp. vs gain imp.	0.802	0.630	0.618

Appendix

Table 22: Top-

k

Jaccard overlaps for the MLP neuron diagnostic. “Imp.” denotes the down-projection-weighted activation-magnitude proxy. The diagnostic uses 64 examples per group and is reported only as descriptive support for the coarse localization analysis.

Module	Family	All $n$	All rel.	All top-32	All eff. rank	Dec. rel.	Dec. top-32
q_proj	attention	61	2.486	0.178	786.8	0.368	0.137
v_proj	attention	61	2.121	0.136	668.4	0.338	0.104
k_proj	attention	61	1.941	0.174	611.8	0.373	0.142
down_proj	MLP	34	0.370	0.039	2171.0	0.370	0.039
o_proj	attention	34	0.337	0.122	1086.6	0.337	0.122
gate_proj	MLP	34	0.261	0.033	2175.5	0.261	0.033

Appendix

Table 23: Module-level

\Delta W

summaries. “All” aggregates all selected aligned matrices, including the vision tower where applicable. “Dec.” recomputes the same relative-change and mean top-32 energy summaries over decoder-side matrices only. Relative change is computed as the aggregate Frobenius norm of

\Delta W

divided by the corresponding aggregate base-model norm.

Layer	Rel. change	RMS	Attn rel.	MLP rel.	Attn/MLP	Eff. rank	Max module
19	1.286	0.0222	2.649	0.248	5.81	1208.2	v_proj
26	1.283	0.0215	2.463	0.247	5.97	1173.2	v_proj
2	1.254	0.0219	2.375	0.255	5.67	1049.2	q_proj
20	1.241	0.0216	2.193	0.262	5.62	1196.8	v_proj
18	1.239	0.0219	2.510	0.248	5.62	1212.1	v_proj
17	1.239	0.0214	2.438	0.256	5.50	1160.1	v_proj

Appendix

Table 24: Top decoder-side layers by layer-level relative change. The attention/MLP ratio compares Frobenius norms of the attention and feed-forward parts of each layer’s

\Delta W

Layer	Module	Rel. change	RMS	Top-32 energy	Eff. rank	$k_{80}$
27	k_proj	0.854	0.0173	0.165	665.4	476
7	k_proj	0.843	0.0176	0.152	673.8	455
27	v_proj	0.837	0.0170	0.084	822.9	548
11	down_proj	0.830	0.0044	0.053	2070.3	1441
19	o_proj	0.825	0.0083	0.114	1073.8	734
7	v_proj	0.814	0.0170	0.099	771.4	508

Appendix

Table 25: Top decoder-side matrices by matrix-level relative change.

k_{80}

is the number of singular directions needed to explain 80% of the matrix

\Delta W

energy.

Panel of Figure 13 — Figure 13: Additional decoder-side layer diagnostics. These views show that the decoder-side update is depth-structured, while the intervention analysis in the main text focuses on component-level bucket sufficiency.

Panel of Figure 14 — Figure 14: Decoder-side module diagnostics complementing Table 23 .

Panel of Figure 15 — Figure 15: Additional spectral diagnostics for decoder-side $\Delta W$ matrices.

Panel of Figure 16 — Figure 16: Vision-side descriptive $\Delta W$ context. These plots explain why the global raw $\Delta W$ screen contains large vision-tower changes, even though the main intervention analysis targets the decoder backbone for text-only medical evaluation.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists
Aug 21, 2026Ye Chen, Weining ZhangLLM Fine-Tuning LLM Evaluation
Alibaba Group · Cheung Kong Graduate School of Business
Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models
Sep 11, 2026Darin Keng, Zhewei SunDomain Adaptation for LLMs Medical LLMs
Department of Statistics, University of Chicago, Chicago, Illinois · Toyota Technological Institute at Chicago, Chicago, Illinois
MSA-CITE: A Co-Adapted LoRA Specialist Ecology for Fixed-Budget Small-Model Inference
Aug 11, 2026Ruitong Li, Binjie Guo, Aisheng Mo +3Language Model Ensembles Efficient Language Model Inference
The University of Hong Kong · Zhejiang University · Independent Researcher

Aug 21, 2026J/K move · Enter open · S save

UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists

Ye Chen, Weining Zhang

Alibaba Group · Cheung Kong Graduate School of Business

LLM Fine-Tuning LLM Evaluation

Open paper AskPDF arXiv

Organizations maintain task-specific adapters for open-weight language models, and each new base-model release forces a migration decision: retain existing specialists, port adapters, refresh from preserved behavior, or retrain. Prior transfer work evaluates isolated model pairs, without studying these choices across real model release sequences. We present UpgradeBench, a decision-driven longitudinal benchmark covering four consecutive Qwen releases, one continuation checkpoint, six tasks, and two model sizes, augmented by OLMo checkpoints with known training lineage. The benchmark disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether specialization assets transfer across versions, and what recovery resources are usable. We observe upgrade gains differ across task-scale-release episodes: some retrained baselines improve while others stay within training noise, with durability ranging from under one release interval for text-to-SQL to over fourteen months for intent classification. Direct adapter copying depends neither on architecture nor model family: on OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens; annealing and model souping introduce no extra harm, with portability decaying with continued-pretraining distance. Given preserved input data, teacher relabeling recovers target-base specialists without fresh gold annotations, though compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross-version adapter portability (Spearman 0.74 across eight model pairs). We release per-example predictions, cost logs, split manifests, and evaluation code.

Abstract

LLM Auditing

Explore similar work

UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists

Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models

MSA-CITE: A Co-Adapted LoRA Specialist Ecology for Fixed-Budget Small-Model Inference