Reversion, Retention, and Semantic Intervention: Measuring Functional Content in CLIP Fine-Tuning Deltas

Chih-Ting Liao; Kezhen Bao; Xin Cao

cs.CVOct 25, 2026

Reversion, Retention, and Semantic Intervention: Measuring Functional Content in CLIP Fine-Tuning Deltas

Authors: Chih-Ting Liao, Kezhen Bao, Xin Cao

Organizations: University of New South Wales

Abstract

Fine-tuning adapts pretrained models to new tasks, but identifying which weight changes support the resulting behavior remains difficult. Weight-space interpretability assigns semantic descriptions to these changes, called fine-tuning deltas. Such descriptions leave open how much task performance depends on the identified updates, particularly when other updates can support the same predictions. We address this gap by pairing reversion, which restores selected parameters to their pretrained values, with retention, which keeps only selected fine-tuning updates. We evaluate Contrastive Language-Image Pretraining (CLIP) models on synthetic financial charts, Flowers102, EuroSAT, and FGVCAircraft with ViT-L/14, and on Flowers102 with ViT-B/16, using a shared recipe that tunes the final six visual blocks. Our contributions are threefold. First, attention value-output (VO) updates alone recover 43-87% of the accuracy improvement, although reverting them leaves chart accuracy unchanged. Cross-entropy loss preserves this contrast on Charts and EuroSAT, while correct-class score margins reveal additional contributions. Second, VO weights recover more loss improvement than all ten equally sized, layer-matched random subsets in each setting. VO and multilayer perceptron (MLP) retentions jointly correct 25-81% of pretrained errors, demonstrating overlapping recovered behavior, which we call operational redundancy. Third, removing semantically selected directions reduces VO-retained recovery more than matched-random removal in all five settings. Together, these results distinguish semantic alignment, removal cost, and retained recovery as complementary evidence about the functional role of fine-tuning updates within the evaluated CLIP recipe.

Figures & tables

Figure 1: Semantic reading, reversion, and retention answer different questions. The semantic readout acts on per-head effective VO deltas. Reversion measures the effect of removing selected updates from the tuned network; component retention measures recovery when the complementary updates are reverted. Both networks retain the pretrained computation. The schematic uses “copy” as shorthand for retained recovery. Direction removal is a targeted intervention on the readout, whereas the retention evidence is at component level.

Figure 2: A small accuracy-reversion effect coexists with substantial retention recovery. Left: VO-revert accuracy costs on three chart tuning seeds, as reported in the original audit. Right: base, VO-retained, and full-model accuracies on the primary 780-example chart evaluation set. VO retention uses pretrained LayerNorms and recovers 0.808 of the gain. This is component-level recovery relative to the pretrained network.

Figure 3: From effective VO deltas to semantic readouts. The instrument forms the per-head effective delta, extracts its leading left singular vectors, maps them through the frozen gain and projection, and applies dictionary-selection and random-direction criteria. The top-eight pre/post subspace overlap of 0.9993 on Charts explains why delta reading reveals structure that changes little in the dominant absolute-weight subspace.

Setting	$n$	Base	Full	$\mathrm{Rec}_{A}(\mathrm{VO})$	$\mathrm{Rec}_{A}(\mathrm{MLP})$
Charts	780	0.1359	0.9846	0.808	0.989
Flowers102	2000	0.7870	0.9900	0.547	0.862
EuroSAT	2000	0.4695	0.9770	0.869	0.960
Aircraft	2000	0.3105	0.7430	0.428	0.682
Flowers102 (B/16)	2000	0.6970	0.9735	0.438	0.808

Table 1: Accuracy and recovery relative to the shared pretrained network. The first four settings use ViT-L/14; the final setting uses ViT-B/16. Both retained models use pretrained LayerNorm parameters. Bias conventions are specified in Appendix A .

Setting	VO revert	QK revert	MLP revert
Charts	0.000 [-0.008, 0.008]	0.012 [0.004, 0.021]	0.014 [0.001, 0.027]
Flowers102	0.015 [0.009, 0.022]	0.003 [-0.001, 0.006]	0.050 [0.040, 0.060]
EuroSAT	0.010 [0.005, 0.016]	0.004 [0.001, 0.008]	0.024 [0.016, 0.032]
Aircraft	0.061 [0.046, 0.076]	0.036 [0.024, 0.049]	0.127 [0.108, 0.148]
Flowers102 (B/16)	0.030 [0.022, 0.039]	0.005 [0.001, 0.009]	0.124 [0.109, 0.139]

Table 2: Accuracy necessity costs with paired-bootstrap 95% intervals. Values are accuracy fractions: 0.05 is five accuracy points. Reversion operators follow the bias convention in Appendix A ; VO reversion leaves the output-projection bias tuned.

Setting	$N_{L}(\mathrm{VO})$ [95% CI]	$R_{L}(\mathrm{VO})$	$R_{L}(\mathrm{MLP})$
Charts	-0.0081 [-0.0245, 0.0055]	0.746	0.988
Flowers102	0.0395 [0.0254, 0.0531]	0.631	0.880
EuroSAT	0.0035 [-0.0154, 0.0211]	0.822	0.980
Aircraft	0.2024 [0.1710, 0.2337]	0.505	0.698
Flowers102 (B/16)	0.1146 [0.0954, 0.1357]	0.486	0.846

Table 3: Cross-entropy necessity and retention recovery. The VO loss-cost interval spans zero on Charts and EuroSAT, while VO retention recovers 74.6% and 82.2% of the loss improvement. Table 7 reports all three components and margins.

Setting	Weights-only VO	Random mean	Random range
Charts	0.745	0.555	[0.554, 0.556]
Flowers102	0.629	0.344	[0.344, 0.345]
EuroSAT	0.820	0.609	[0.608, 0.611]
Aircraft	0.504	0.280	[0.279, 0.281]
Flowers102 (B/16)	0.484	0.260	[0.259, 0.261]

Table 4: Loss recovery under per-layer parameter-count matching. Both weights-only VO and random masks use pretrained biases and LayerNorms. Ranges describe ten mask seeds, not confidence intervals.

Setting	VO/full	MLP/full	VO/MLP
Charts	0.8256	0.9859	0.8321
Flowers102	0.9015	0.9670	0.9290
EuroSAT	0.9190	0.9700	0.9255
Aircraft	0.5500	0.7090	0.6820
Flowers102 (B/16)	0.8315	0.9375	0.8725

Table 5: Prediction agreement on all evaluation examples. Shared corrections of pretrained errors are reported separately to distinguish overlap in recovered behavior from shared mistakes.

Setting	Collapse difference [95% CI]	Random range	Majority criterion
Charts	0.366 [0.325, 0.409]	0.004	Fail
Flowers102	0.081 [0.039, 0.124]	0.005	Fail
EuroSAT	0.736 [0.699, 0.775]	0.002	Pass
Aircraft	0.102 [0.068, 0.136]	0.009	Fail
Flowers102 (B/16)	0.177 [0.138, 0.216]	0.007	Fail

Table 6: Readout-guided minus random collapse in normalized accuracy recovery. The intervention applies sequential removal projections within VO-retained models. Ranges describe the three random seeds.

Figure 4: Semantic alignment and accuracy necessity measure different properties. Left: coherent-head fraction and the fraction of tuning accuracy gain lost after VO reversion. Right: Charts produces 41 coherent heads under financial anchors and zero under generic anchors; points show 40, 41, and 43 heads across the original three tuning seeds. The null-tune comparison holds images and hyperparameters fixed.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Component retention and VO necessity in the original audit. Top: VO and MLP accuracy recoveries; the reference lines show the original VO insufficiency and MLP positive-control criteria. Bottom: VO-revert accuracy costs. The complete VO, QK, and MLP necessity measurements, including the larger MLP costs on Aircraft and ViT-B/16, appear in Table 2 .

Figure 6: Pretrained semantic readouts depend on dictionary and corpus. Left: selected, null-passing, and jointly passing vector counts under the original and competitor-enriched chart audits. Right: technically coherent pretrained heads across corpora and architectures. The sole OpenAI ViT-L/14 hit remains polysemy-confounded; LAION hits lack a full selection trace.

Figure 7: The original chart accuracy intervention battery. Direction surgery and matched-random removal are shown alongside coherent-head VO, all-head VO, QK, and joint VO+QK reverts. The original 15-point line is a decision gate for this battery. Joint VO+QK reversion is distinct from MLP reversion; the subsequently measured MLP-revert cost of 1.4 accuracy points appears in Table 2 .

Setting	Group	$N_{L}$	$N_{M}$	$R_{L}$	$R_{M}$
Charts	VO	-0.008 [-0.025, 0.006]	1.519 [1.450, 1.582]	0.746 [0.736, 0.757]	0.283 [0.275, 0.291]
	QK	0.024 [0.013, 0.037]	0.946 [0.901, 0.992]	0.466 [0.453, 0.480]	0.147 [0.143, 0.152]
	MLP	0.187 [0.160, 0.214]	5.082 [4.941, 5.221]	0.988 [0.982, 0.994]	0.687 [0.680, 0.693]
Flowers102	VO	0.040 [0.025, 0.053]	1.458 [1.399, 1.516]	0.631 [0.611, 0.655]	0.259 [0.256, 0.263]
	QK	0.009 [0.004, 0.014]	1.022 [0.989, 1.053]	0.241 [0.222, 0.263]	0.157 [0.151, 0.162]
	MLP	0.160 [0.130, 0.188]	3.226 [3.139, 3.316]	0.880 [0.860, 0.902]	0.545 [0.540, 0.550]

Appendix

Table 7: Complete continuous-metric results with paired-bootstrap 95% intervals.

N_{L}

and

N_{M}

are raw loss and margin costs;

R_{L}

and

R_{M}

are normalized recovery. Every condition uses the same fixed logit scale within a checkpoint.

Setting	VO	QK	MLP
Charts	0.808 [0.775, 0.839]	0.337 [0.302, 0.373]	0.989 [0.979, 0.998]
Flowers102	0.547 [0.497, 0.598]	0.202 [0.159, 0.246]	0.862 [0.824, 0.899]
EuroSAT	0.869 [0.846, 0.892]	0.354 [0.323, 0.384]	0.960 [0.946, 0.973]
Aircraft	0.428 [0.390, 0.465]	0.187 [0.157, 0.219]	0.682 [0.643, 0.720]
Flowers102 (B/16)	0.438 [0.397, 0.481]	0.132 [0.101, 0.165]	0.808 [0.773, 0.843]

Appendix

Table 8: Accuracy recovery with paired-bootstrap 95% intervals, including QK-only retention.

Setting	Pair	$n$	Agreement [95% CI]	Both correct	Same wrong
Charts	VO/full	674	0.7982 [0.7685, 0.8279]	532	6
	MLP/full	674	0.9837 [0.9733, 0.9926]	653	10
	VO/MLP	674	0.8056 [0.7745, 0.8353]	532	11
Flowers102	VO/full	426	0.5493 [0.5000, 0.5962]	223	11
	MLP/full	426	0.8521 [0.8192, 0.8850]	352	11
	VO/MLP	426	0.6714 [0.6244, 0.7160]	222	64

Appendix

Table 9: Agreement on the fixed pretrained-error subset. Both-correct counts joint corrections; same-wrong counts shared incorrect predictions. Agreement includes both categories.

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 268 papers

[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning
Oct 10, 2026Anton Nuzhdin, Andrei Gruzitski, Balázs Egyed +1VLM Interpretability Saliency Map Evaluation
University of Amsterdam
Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
May 15, 2026Fabian Morelli, Arnas Uselis, Ankit Sonthalia +1Distribution Shift Robustness VLM Robustness
University of Tübingen ∗ · University of Tübingen
Why Constrained Updates Still Forget: A Functional-Path Analysis of Continual SFT
Oct 21, 2026Qingsen Ma, Dianyun Wang, Jiahong Liu +1Supervised Fine-Tuning Continual Learning for LLMs
The Chinese University of Hong Kong · Beijing University of Posts and Telecommunications

Oct 10, 2026J/K move · Enter open · S save

[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning

Anton Nuzhdin, Andrei Gruzitski, Balázs Egyed, Aleksandr Raudvee

University of Amsterdam

VLM Interpretability Saliency Map Evaluation Adversarial Training ML Reproducibility VLM Robustness

Open paper AskPDF arXiv

This paper presents a reproducibility study of ``Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning'', published at ICLR 2025, which proposes an unsupervised adversarial fine-tuning (AFT) method with norm regularisation to enhance the visual interpretability of CLIP's image encoder. We attempt to reproduce the key claims regarding improved saliency map quality, increased concept alignment, transferability to out-of-distribution datasets, and the trade-off with zero-shot accuracy. Beyond reproduction, we propose a saliency-guided regularisation extension that introduces an Energy Pointing Game loss, directly supervising the spatial alignment of Simple Gradient saliency maps with target objects. We evaluate our extension across a range of saliency-loss weights and show that explicit saliency supervision \rev{substantially} improves localisation metrics at a modest cost: the reduction in zero-shot accuracy and adversarial robustness is small at the lowest saliency-loss weight (

w_s=0.2

) and moderate at the strongest setting (

w_s=0.6

). We restructure the codebase for configurable, systematic experimentation.

Abstract

VLM Interpretability

Explore similar work

[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning

Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models

Why Constrained Updates Still Forget: A Functional-Path Analysis of Continual SFT