[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning

Anton Nuzhdin; Andrei Gruzitski; Balázs Egyed; Aleksandr Raudvee

cs.CVOct 10, 2026

[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning

Authors: Anton Nuzhdin, Andrei Gruzitski, Balázs Egyed, Aleksandr Raudvee

Organizations: University of Amsterdam

Abstract

This paper presents a reproducibility study of ``Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning'', published at ICLR 2025, which proposes an unsupervised adversarial fine-tuning (AFT) method with norm regularisation to enhance the visual interpretability of CLIP's image encoder. We attempt to reproduce the key claims regarding improved saliency map quality, increased concept alignment, transferability to out-of-distribution datasets, and the trade-off with zero-shot accuracy. Beyond reproduction, we propose a saliency-guided regularisation extension that introduces an Energy Pointing Game loss, directly supervising the spatial alignment of Simple Gradient saliency maps with target objects. We evaluate our extension across a range of saliency-loss weights and show that explicit saliency supervision \rev{substantially} improves localisation metrics at a modest cost: the reduction in zero-shot accuracy and adversarial robustness is small at the lowest saliency-loss weight ( $w_s=0.2$ ) and moderate at the strongest setting ( $w_s=0.6$ ). We restructure the codebase for configurable, systematic experimentation.

Figures & tables

Figure 1: Saliency map comparison between CLIP (middle row) and CLIP with AFT (bottom row). (a) Simple Gradient maps for different text prompts. (b) Grad-ECLIP visualisations for the same images. (c) Out-of-distribution evaluation on medical and wildlife images. (d) Linear probing results on COCO classification.

Saliency	CLIP	Zero-shot classification					Linear probing
Saliency	CLIP	PG $\uparrow$	E-PG $\uparrow$	AP $\uparrow$	PA $\uparrow$	IoU $\uparrow$	PG $\uparrow$	E-PG $\uparrow$	AP $\uparrow$	PA $\uparrow$	IoU $\uparrow$
	w/o AFT	25.26	29.21	31.65	61.15	12.19	28.05	30.95	33.10	61.96	13.37
SG	w/ AFT 4 (ours)	56.55	46.22	49.95	68.71	24.43	57.83	46.99	50.67	68.97	24.87
	w/ AFT 4 (original)	56.61	46.09	49.98	68.75	24.47	57.87	46.84	50.68	69.00	24.90
	w/o AFT	26.56	27.16	31.58	59.29	10.53	67.10	59.45	61.56	69.97	31.46
GC	w/ AFT 4 (ours)	57.57	49.59	54.22	67.29	26.35	68.68	61.39	64.81	68.90	32.85

Table 1: Evaluation of localisation ability using PG ( Zhang et al., 2018 ) , E-PG ( Wang et al., 2020 ) , and segmentation metrics (PA, AP and Mask IoU) on the ImageNet-Segmentation validation dataset. Absolute values differ from those reported by Gong et al. (2025) owing to undocumented evaluation-protocol choices (Appendix G ); see text.

	w/o AFT		w/ AFT 1		w/ AFT 4
Metric	Mean	Median	Mean	Median	Mean	Median
Soft-WPMI	0.2264	0.2023	0.2610	0.2307	0.2817	0.2558
Cos	0.2852	0.4437	0.3046	0.4726	0.3065	0.4996
Cos-cubed	0.2369	0.2205	0.3051	0.2853	0.3289	0.3124

Table 2: Mean and median neuron-concept alignment scores for different AFT strengths.

Model	Detectors	% neurons
CLIP w/o AFT	160/768	20.8%
AFT 1 (ours)	279/768	36.3%
AFT 4 (ours)	390/768	50.8%
AFT 4 (original)	390/768	50.8%

Table 3: Network Dissection on the Broden dataset (768 neurons,

\text{IoU}\geq 0.04

threshold).

ImageNet fine-tuning only
	ISIC (zero-shot)			Kvasir-SEG (zero-shot)			COCO linear probe
Model	SG PG	GE PG	SG IoU	SG PG	GE PG	SG IoU	SG PG	GC PG	SG IoU
CLIP (w/o AFT)	0.107	0.397	0.091	0.058	0.085	0.062	0.140	0.226	0.076
AFT 4 (original)	0.428	0.460	0.282	0.051	0.020	0.121	0.333	0.244	0.145
AFT 1 (ours)	0.457	0.469	0.249	0.072	0.043	0.137	0.324	0.230	0.139
AFT 4 (ours)	0.424	0.426	0.274	0.056	0.028	0.127	0.328	0.239	0.144

Table 4: Localisation against ground-truth masks in the transfer settings of Figure 1 (c,d): zero-shot on ISIC and Kvasir-SEG, and the 6-class COCO linear probe. 1,000 samples per setting, seed 42, protocol of Section 4.1.1 . SG = Simple Gradient, GE = Grad-ECLIP, GC = Grad-CAM; Grad-ECLIP is not defined for the linear probe, so the COCO columns report Grad-CAM. Full five-metric tables in Appendix J . Bold marks the best value per column; italics mark the best among the ImageNet-only arms.

$L_{\infty}$	Model	CIFAR-10	STL-10	EuroSAT	FGVC Air.	Imagenette	Average
clean	CLIP	89.4	98.1	40.5	22.0	99.58	69.92
	AFT 1 (ours)	86.4	97.7	9.5	21.4	99.26	62.85
	AFT 4 (ours)	66.2	91.4	10.5	12.2	98.60	55.78
	AFT 4 (original)	63.2	90.4	11.9	12.0	98.22	55.14
	COCO AFT ( $w_{s}{=}0$ )	55.7	83.4	8.3	13.7	95.20	51.26
2/255	CLIP	0.0	0.0	0.0	0.0	0.00	0.00

Table 5: Adversarial Robustness and Accuracy Results. (ours) denotes retrained models; (original) denotes original checkpoints provided by authors. Imagenette values are means over five evaluation seeds; all other columns use the fixed seed 42, except the COCO AFT (

w_{s}{=}0

) row, which reports five-seed means throughout. Per-cell 95% confidence intervals for every dataset are given in Appendix I .

Panel of Figure 2 — Figure 2: ROAR on Imagenette (a: Simple Gradient, b: Grad-CAM) and Caltech256 (c: Simple Gradient, d: Grad-CAM). Each panel compares saliency maps from the reference AFT 4 model against maps from standard CLIP; curves show the validation accuracy of a ResNet-50 retrained from scratch at each removal level. Imagenette curves are means over three seeds; Caltech256 uses a single run.

Claim	Verdict	Primary source of discrepancy
1a: Simple Gradient	Directional	Protocol choices shift absolute values; none reorder models
1b: Grad-CAM	Inconclusive	Unstable on CLIP ViT-B/16 ( $14\times 14$ grid); metrics still improve
2: Concept alignment	Trend	Detector counts set by undocumented IoU threshold ( $9\times$ range)
3: Transferability	Directional	Quantitative in 2/3 settings; Kvasir-SEG uninformative
4: Accuracy trade-off	Reproduced ∗	EuroSAT at chance for all AFT models

Table 7: Reproduction outcome for the four claims of Gong et al. (2025) . References given in text.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Usage	#images
ImageNet 2012	Adversarial fine-tuning	1.28M
COCO 2017	Saliency-guided fine-tuning, linear probing, EPG evaluations	118K
ImageNet-S	Saliency localisation evaluation	12.4K
Caltech256	ROAR analysis	30.6K
Flickr30k	Retrieval evaluation	31K
ISIC 2024	OOD saliency evaluation	401K

Appendix

Table 8: Datasets used in our reproduction study.

Figure 4: Saliency map comparison between CLIP (middle row) and CLIP with AFT (bottom row) for the original AFT 4 backbone (left column, as in Figure 1 ) and the AFT 4 backbone with saliency supervision (right column). Images (a) and (c) are Simple Gradient maps for different text prompts, images (b) and (d) are Grad-ECLIP visualisations for the same images. Saliency mass is visibly more concentrated on the referenced object and contains fewer off-object artefacts than the corresponding AFT 4 maps in Figure 1 .

Figure 5: Grad-CAM sanity check. Triplets of input image, ResNet-50 Grad-CAM, and CLIP ViT-B/16 Grad-CAM for 16 Imagenette validation images, computed with the identical pipeline. The CNN yields stable, object-centred maps; CLIP ViT-B/16 yields coarse, fragmented, and spatially unstable maps.

Model	$\varepsilon$	Background $\Phi$	Foreground $\Phi$	BG/FG
AFT 4 (no EPG)	2/255	0.0036	0.0029	1.253
+ EPG $w_{s}{=}0.2$	2/255	0.0074	0.0079	0.936
+ EPG $w_{s}{=}0.4$	2/255	0.0070	0.0088	0.787
+ EPG $w_{s}{=}0.6$	2/255	0.0066	0.0097	0.677
AFT 4 (no EPG)	4/255	0.0123	0.0095	1.293
+ EPG $w_{s}{=}0.2$	4/255	0.0175	0.0188	0.933

Appendix

Table 9: Feature-consistency damage

\Phi

under region-restricted PGD attacks on COCO. Background attacks perturb only pixels outside the object mask, while foreground attacks perturb only object pixels. Ratios below one indicate that the representation is less sensitive to background perturbations than to foreground perturbations, as predicted by Proposition 1 . The plain-AFT row is our ImageNet-trained AFT 4 reproduction; the EPG rows share an identical COCO fine-tuning recipe and differ only in

w_{s}

	Simple Gradient				Grad-CAM				Grad-ECLIP
radius	clip	base	ours	gap	clip	base	ours	gap	clip	base	ours	gap
0	0.261	0.556	0.559	$+0.003$	0.270	0.595	0.571	$-0.025$	0.667	0.655	0.661	$+0.006$
5	0.294	0.660	0.663	$+0.004$	0.341	0.713	0.685	$-0.028$	0.781	0.767	0.779	$+0.012$
15	0.388	0.783	0.783	$+0.000$	0.463	0.856	0.829	$-0.026$	0.908	0.902	0.906	$+0.005$

Appendix

Table 10: Pointing Game under varying hit radius (invariant to topk). clip = non-fine-tuned CLIP, base = author-released AFT 4 , ours = our retrained AFT 4 ; gap = ours

-

base. 3,000 ImageNet-S samples.

ImageNet fine-tuning only
Model	IoU 0.02	IoU 0.04	IoU 0.06
CLIP (w/o AFT)	572	160	64
AFT 1 (ours)	662	279	122
AFT 4 (ours)	721	390	183
AFT 4 (original)	721	390	197
COCO fine-tuning

Appendix

Table 11: Detector counts across the IoU-threshold sweep (activation quantile 0.005, final token). The main-text operating point is IoU 0.04.

Figure 6: Distribution of neuron-concept alignment scores under three similarity metrics for CLIP with and without AFT.

$L_{\infty}$	Model	CIFAR-10	STL-10	EuroSAT	FGVC Air.	Imagenette
clean	AFT 4 (original)	64.08 $\pm$ 1.58	89.96 $\pm$ 0.73	10.88 $\pm$ 0.89	12.84 $\pm$ 0.98	98.22 $\pm$ 0.36
	AFT 1 (ours)	87.56 $\pm$ 0.90	97.38 $\pm$ 0.32	10.50 $\pm$ 1.18	20.82 $\pm$ 1.78	99.26 $\pm$ 0.17
	AFT 4 (ours)	65.80 $\pm$ 1.45	90.88 $\pm$ 0.59	10.88 $\pm$ 0.89	12.26 $\pm$ 0.90	98.60 $\pm$ 0.18
	COCO AFT ( $w_{s}{=}0$ )	55.74 $\pm$ 1.71	83.42 $\pm$ 1.08	8.26 $\pm$ 0.60	13.68 $\pm$ 1.40	95.20 $\pm$ 0.28
	+ EPG $w_{s}{=}0.2$	52.96 $\pm$ 1.75	82.12 $\pm$ 1.09	5.36 $\pm$ 0.52	14.20 $\pm$ 1.32	95.10 $\pm$ 0.33
	+ EPG $w_{s}{=}0.4$	51.26 $\pm$ 0.99	80.22 $\pm$ 1.01	5.20 $\pm$ 0.80	13.70 $\pm$ 1.20	94.86 $\pm$ 0.41

Appendix

Table 12: Zero-shot accuracy, mean

\pm

95% CI over five evaluation seeds.

		T $\rightarrow$ I (%)		I $\rightarrow$ T (%)
$L_{\infty}$	Vision Encoder	R@1	R@5	R@1	R@5
clean	CLIP baseline	24.95	45.37	44.41	68.61
	AFT 1 ( ours )	27.20	48.52	43.02	67.09
	AFT 4 ( original )	19.53	37.50	26.38	47.54
	AFT 4 ( ours )	19.53	37.46	26.33	47.78
2/255	CLIP baseline	1.83	4.74	2.25	5.66

Appendix

Table 13: Cross-modal retrieval results (Recall@k, %) on the Flickr30k dataset under different perturbation budgets.

\mathrm{AFT}^{i}

denotes models trained with perturbation

\varepsilon=i/255

. (ours) denotes retrained models; (original) denotes original checkpoints provided by authors. Bold marks the best value per column within each budget.

ImageNet fine-tuning only
	Simple Gradient					Grad-CAM					Grad-ECLIP
Model	PG	E-PG	AP	PA	IoU	PG	E-PG	AP	PA	IoU	PG	E-PG	AP	PA	IoU
CLIP (w/o AFT)	0.107	0.238	0.253	0.609	0.091	0.190	0.261	0.286	0.622	0.103	0.397	0.308	0.425	0.663	0.196
AFT 4 (original)	0.428	0.441	0.523	0.714	0.282	0.346	0.367	0.386	0.464	0.281	0.460	0.395	0.537	0.700	0.272
AFT 1 (ours)	0.457	0.401	0.484	0.699	0.249	0.382	0.352	0.373	0.550	0.223	0.469	0.364	0.498	0.691	0.246
AFT 4 (ours)	0.424	0.436	0.511	0.709	0.274	0.290	0.344	0.372	0.558	0.246	0.426	0.377	0.516	0.691	0.255

Appendix

Table 14: Zero-shot localisation on ISIC (dermoscopy) and Kvasir-SEG (colonoscopy polyps; small targets). The CLIP and AFT rows are fine-tuned on ImageNet only; the EPG rows are fine-tuned on COCO with COCO instance masks as supervision. Both domains are out-of-distribution for either training set, so the in-domain confound noted for Table 15 does not apply here. Bold marks the best value per column; italics mark the best among the ImageNet-only arms.

ImageNet fine-tuning only
	Simple Gradient					Grad-CAM
Model	PG	E-PG	AP	PA	IoU	PG	E-PG	AP	PA	IoU
CLIP (w/o AFT)	0.140	0.156	0.166	0.701	0.076	0.226	0.212	0.243	0.666	0.119
AFT 4 (original)	0.333	0.241	0.261	0.738	0.145	0.244	0.228	0.254	0.656	0.128
AFT 1 (ours)	0.324	0.227	0.254	0.735	0.139	0.230	0.216	0.250	0.654	0.120
AFT 4 (ours)	0.328	0.240	0.260	0.738	0.144	0.239	0.227	0.257	0.680	0.129

Appendix

Table 15: COCO 6-class linear-probe localisation (person, car, dog, cat, bird, chair). Grad-ECLIP is not defined for the linear probe. The CLIP and AFT rows are fine-tuned on ImageNet only; the EPG rows are fine-tuned on COCO with COCO instance masks as supervision, so on this COCO-derived task part of their margin may reflect in-domain exposure rather than the EPG term itself. Bold marks the best value per column; italics mark the best among the ImageNet-only arms.

Figure 7: ROAR after EPG fine-tuning. Validation accuracy of a ResNet-50 retrained from scratch on data with the top- $k\%$ most salient pixels removed, for the COCO $w_{s}{=}0$ control (green) and the $w_{s}{=}0.6$ model (purple). SG = Simple Gradient, GC = Grad-CAM.

ImageNet fine-tuning only
	Imagenette		ImageNet-val
Model	Del gap	Ins gap	Del gap	Ins gap
AFT 4 (original)	$+0.042$	$+\textit{0.133}$	$+0.012$	$+0.130$
AFT 1 (ours)	$+\textbf{{0.143}}$	$+0.069$	$+\textbf{{0.039}}$	$+0.131$
AFT 4 (ours)	$+0.045$	$+0.129$	$+0.014$	$+\textbf{{0.133}}$
COCO fine-tuning with EPG supervision

Appendix

Table 16: SmoothGrad separation from the random control. Deletion gap = random

-

SmoothGrad (positive is good); insertion gap = SmoothGrad

-

random (positive is good). Zeros deletion baseline. The AFT rows are fine-tuned on ImageNet only; the EPG rows are fine-tuned on COCO with COCO instance masks as supervision. Bold marks the best value per column; italics mark the best among the ImageNet-only arms.

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 268 papers

Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
May 15, 2026Fabian Morelli, Arnas Uselis, Ankit Sonthalia +1Distribution Shift Robustness VLM Robustness
University of Tübingen ∗ · University of Tübingen
Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models
Apr 13, 2026Songlong Xing, Weijie Wang, Zhengyu Zhao +3VLM Robustness
University of Trento, Italy · Fondazione Bruno Kessler, Italy · Xi’an Jiaotong University, China +1
LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder
Oct 2, 2026Anh-Khoa Dinh-Duc, Duc-Tai Dinh, Tam V. Nguyen +1Referring Expression Comprehension VLM Adaptation
University of Science, Ho Chi Minh City, Vietnam · Viet Nam National University, Ho Chi Minh City, Vietnam · University of Dayton, USA

May 15, 2026J/K move · Enter open · S save

Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models

Fabian Morelli, Arnas Uselis, Ankit Sonthalia, Seong Joon Oh

University of Tübingen ∗ · University of Tübingen

Distribution Shift Robustness VLM Robustness VLM Adaptation VLM Interpretability Sparse Autoencoders

Open paper AskPDF arXiv

Large-scale pre-trained vision-language models like CLIP demonstrate remarkable zero-shot performance across diverse tasks. However, fine-tuning these models to improve downstream performance often degrades robustness against distribution shifts. Recent approaches have attempted to mitigate this trade-off, but often rely on computationally expensive text-guidance. We propose a novel method for robust fine-tuning, SAE-FT, which operates only on the model's visual representations. SAE-FT regularizes changes to these representations by penalizing the addition and removal of semantically meaningful features identified by a Sparse Autoencoder trained on the pre-trained model. This constraint prevents catastrophic forgetting and makes the fine-tuning process interpretable, enabling direct analysis of semantic changes. SAE-FT is both mechanistically transparent and computationally efficient, matching or exceeding state-of-the-art performance on ImageNet and its associated distribution shift benchmarks. Code is publicly available at: https://github.com/Fabian-Mor/sae-ft.

Abstract

VLM Interpretability

Explore similar work

Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models

Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder