Large Pretraining Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification

Jaedong Hwang; Brian Cheung; Zhang-Wei Hong; Akhilan Boopathy; Pulkit Agrawal; Ila Fiete

cs.CVOct 28, 2024

Large Pretraining Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification

Authors: Jaedong Hwang, Brian Cheung, Zhang-Wei Hong, Akhilan Boopathy, Pulkit Agrawal, Ila Fiete

Organizations: Massachusetts Institute of Technology

Abstract

Large-scale pretrained models are widely leveraged as foundations for learning new specialized tasks via fine-tuning, with the goal of maintaining the general performance of the model while allowing it to gain new skills. A valuable goal for all such models is robustness: the ability to perform well on out-of-distribution (OOD) tasks. We assess whether fine-tuning preserves the overall robustness of the pretrained model in image classification, and observed that models pretrained on large datasets exhibited strong catastrophic forgetting and loss of OOD generalization. To systematically assess robustness preservation in fine-tuned models, we propose the Robustness Inheritance Benchmark (ImageNet-RIB). The benchmark, which can be applied to any pretrained model, consists of a set of related but distinct OOD (downstream) tasks and involves fine-tuning on one of the OOD tasks in the set then testing on the rest. We find that though continual learning methods help, fine-tuning reduces robustness across pretrained models. Surprisingly, models pretrained on the largest and most diverse datasets (e.g., LAION-2B) exhibit both larger robustness losses and lower absolute robustness after fine-tuning on small datasets, relative to models pretrained on smaller datasets. We observe this collapse in contrastively pretrained (CLIP) models and their fine-tuned variants, where it grows with pretraining scale; the supervised models we test do not exhibit it. These findings suggest that starting with the strongest foundation model is not necessarily the best approach for performance on specialist tasks. https://jd730.github.io/projects/ImageNet-RIB

Figures & tables

Figure 1 : Schematic: How fine-tuning changes the robustness of pretrained models. A model pretrained on the dataset $D_{\text{pre}}$ (purple solid) has a measure of robustness, generalizing to some out-of-distribution data (purple dashed, $f_{\text{pre}}$ ). Dotted gray line: volume ( $\mathbb{D}$ ) containing a number of related OOD datasets (dark gray ellipsoids). Fine-tuning on one of these datasets ( $D_{\text{FT}}$ ) shifts $f_{\text{pre}}$ to $f_{\text{FT}}$ (blue dashed ellipsoid), increasing performance on $D_{\text{FT}}$ and some OOD tasks in $\mathbb{D}$ but possibly reducing performance on others (red), thus making the model less robust.

Figure 2 : OOD accuracy (robustness) of a ViT-B/16 model pretrained on two different datasets (LAION-2B, IN-21K with AugReg), before and after fine-tuning on ImageNet-R.

Figure 3 : ImageNet-RIB benchmarking process. (1) We define a set $\mathbb{D}$ of ImageNet OOD datasets. We select one for fine-tuning, $D_{\text{FT}}$ , then assess the performance of the pretrained model on $\mathbb{D}\setminus D_{\text{FT}}$ . (2) After fine-tuning the pretrained model on $D_{\text{FT}}$ , we (3) re-assess its performance on $\mathbb{D}\setminus D_{\text{FT}}$ and compute the robustness change. (4) This process is repeated until each dataset in $\mathbb{D}$ has been chosen once as the fine-tuning dataset, ensuring a detailed evaluation of fine-tuning’s impact on robustness.

Pretraining Dataset	ImageNet-1K	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
IN-1K + AugReg	79.2	66.4	15.0	38.0	28.0	33.0	66.2	39.1	56.0
IN-21K	81.8	71.4	32.0	47.3	35.8	42.5	69.4	44.1	58.3
IN-21K + AugReg	84.5	74.0	43.2	56.8	43.2	48.4	75.1	54.9	66.5
OpenAI	85.3	75.7	47.3	65.9	50.9	50.7	76.3	55.7	62.6
LAION-2B	85.5	75.6	41.5	68.8	55.4	51.1	78.2	58.4	63.0

Table 1 : Average robustness of the ViT-B/16 model pretrained on various datasets, assessed on the datasets in ImageNet-RIB set

\mathbb{D}

. LAION-2B pretraining exhibits the highest robustness.

Method	$mRI$	RI on specific $D_{\text{FT}}$
Method	$mRI$	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
FT	1.3	2.9	-4.0	2.8	4.4	-2.7	0.6	0.4	5.9
Linear Probing	0.7	0.1	-0.1	0.8	1.2	0.3	0.2	0.1	3.2
Visual Prompt	-4.5	-2.3	-9.1	-4.9	-1.6	-11.2	-3.9	-4.3	1.7
LoRA	0.9	0.2	0.4	1.1	2.6	0.3	-0.1	1.3	1.1
EWC	2.8	2.9	-0.2	5.2	4.4	1.4	1.6	2.8	4.3

Table 2 :

RI

and

mRI

of ViT-B/16 pretrained on ImageNet-1K and AugReg, fine-tuned on each of the datasets in the ImageNet-RIB set

\mathbb{D}

Figure 4 : Severe robustness loss from fine-tuning models pretrained on LAION-2B and OpenAI relative to fine-tuning models pretrained only smaller datasets. The average accuracy on OOD datasets before (blue) and after (red) vanilla fine-tuning (see Figure 8 for other methods). The red bars are calculated by evaluating fine-tuned pretrained models on

\mathbb{D}

. The blue bars are the pretrained models’ mean accuracy on

\mathbb{D}

. Fine-tuning models pretrained on LAION-2B and OpenAI causes severe robustness loss, leading to worse absolute performance on

\mathbb{D}

than after fine-tuning an AugReg model pretrained on ImageNet-1K. Conversely, a model pretrained on ImageNet-1K with AugReg actually exhibits improved robustness after fine-tuning. Note that the difference between red and blue bars is

mRI

Architecture $\rightarrow$	ViT-B/16					ViT-B/32				ViT-S/16		ViT-S/32	ViT-L/16	ResNet-18	ResNet-50
Method	IN-1K + AugReg	IN-21K	IN-21K + AugReg	OpenAI	LAION-2B	IN-1K + AugReg	IN-21K + AugReg	OpenAI	LAION-2B	IN-1K + AugReg	IN-21K + AugReg	IN-21K + AugReg	IN-21K + AugReg	IN-1K	IN-1K
FT	1.3	-0.1	-5.5	-38.0	-38.1	-0.0	-0.1	-28.7	-31.6	-3.2	-2.3	-2.9	-2.1	-5.2	-5.2
Linear Probing	0.7	0.4	-0.3	-2.0	-2.0	1.1	0.3	-1.3	-1.4	0.3	-0.2	-0.1	-1.3	-7.3	-11.2
Visual Prompt	-4.5	-9.4	-8.8	-8.4	-8.2	-5.4	-8.4	-8.0	-8.4	-7.4	-9.2	-9.6	-12.9	-8.3	-6.5
LoRA	0.9	-0.3	-2.1	-3.6	-3.6	0.9	0.9	-1.8	-1.9	0.9	-1.5	0.4	1.0	-	-
EWC	2.8	1.4	0.6	-12.7	-12.5	1.3	1.6	-7.0	-10.0	1.6	1.6	1.0	1.1	-5.7	-8.9

Table 3 : Mean Robustness Improvement (

mRI

) after fine-tuning with different fine-tuning methods.

Method	ViT-B/16			ViT-B/32				ConvNeXt	ResNet-50			SigLip	SigLip2
Method	LAION 400M	OpenAI	LAION 2B	LAION 100M	LAION 400M	OpenAI	LAION 2B	LAION-2B	CC 12M	YFCC 15M	OpenAI	WebLI	WebLI
FT	-32.5	-31.2	-37.4	-7.3	-23.8	-27.0	-33.0	-16.9	-7.1	-2.6	-24.1	-21.3	-25.7
FYLP	-26.0	-30.9	-36.6	-4.6	-24.5	-26.9	-32.8	-17.5	-4.4	-0.7	-26.1	-24.0	-28.7
Visual Prompt	-7.6	-11.0	-10.2	-11.6	-9.6	-7.7	-10.4	-53.5	-8.9	-6.3	-10.9	-26.7	-25.6
LoRA	-47.8	-49.2	-52.5	-35.2	-41.3	-41.1	-19.4	-	-	-	-	-47.4	-20.7
EWC	-8.1	-8.8	-13.1	-1.5	-5.4	-7.3	-10.5	-5.5	-8.8	-5.8	-17.2	-1.9	-3.1

Table 4 :

mRI

of ViT-B/16, ViT-B/32, ConvNeXt-Base, and ResNet-50 CLIP models, and SigLip-B/16, SigLip2-B/16 pretrained on various datasets using zero-shot classifier ( Radford et al., 2021 ) .

Architecture	Pretraining Dataset	IN-1K	Avg. OOD	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	LAION-400M	67.1	50.2	59.6	33.1	77.9	52.4	46.1	56.1	36.4	40.2
	OpenAI	64.4	48.8	57.8	44.4	73.5	44.3	50.5	48.2	33.3	38.2
	LAION-2B	70.2	53.5	62.3	38.0	80.6	56.1	50.8	59.2	37.5	43.0
ViT-B/32	LAION-100M	52.5	38.6	44.5	14.6	64.5	39.8	30.0	43.9	44.5	27.3
	LAION-400M	60.2	42.6	52.4	19.6	70.8	46.4	38.9	48.4	29.4	35.1
	OpenAI	59.6	41.4	52.9	28.3	67.1	40.4	31.6	46.1	29.1	35.9

Table 5 : Average zero-shot accuracy on ImageNet-1K validation set and each OOD dataset using pretrained ViT-B/16 and ResNet-50 CLIP models. All pretrained weights are acquired from open_clip library.

Figure 5 : Fine-tuning LAION-2B and OpenAI pretrained models causes severe robustness loss while learning slower than ImageNet-21K pretrained AugReg model. The average accuracy on fine-tuning datasets (left) and the average accuracy on OOD datasets (right) while fine-tuning on the downstream dataset using vanilla fine-tuning method (FT) with ViT-B/16. Although these models learn slower than other methods, they suffer from a huge robustness drop even in the early period of fine-tuning.

Figure 6 : Fine-tuning on small dataset leads to severe accuracy degradation both in- and out-of distribution. Accuracy on ImageNet-1K validation set and OOD datasets after fine-tuning LAION-2B pretrained ViT-B/16 CLIP model with zero-shot classifier on a small number of images per class of ImageNet-1K training set. The dashed line denotes the accuracy of the pretrained model on each dataset.

Pretraining Dataset	IN-1K	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
IN-1K + AugReg	97.5 ( +18.3 )	66.9 ( +0.4 )	23.3 ( +8.3 )	40.9 ( +2.9 )	29.5 ( +1.5 )	37.2 ( +4.2 )	71.1 ( +4.9 )	41.0 ( +1.9 )	59.5 ( +3.5 )
IN-1K + SAM	87.3 ( +7.1 )	69.4 ( +1.2 )	17.7 ( +8.7 )	41.8 ( +1.7 )	30.1 ( +2.4 )	38 ( +3.8 )	72.1 ( +5.2 )	42.9 ( +0.6 )	56.9 ( +2.3 )
IN-21K	94.7 ( +12.9 )	71.6 ( +0.2 )	38.5 ( +6.5 )	49.9 ( +2.6 )	36.7 ( +0.9 )	45.2 ( +2.7 )	73.9 ( +4.5 )	44.1 (0.0)	59.8 ( +1.5 )
IN-21K-P	96.9 ( +12.6 )	73.0 ( -1.0 )	41.4 ( +7.3 )	51.5 (0.0)	39.8 ( -0.4 )	45.8 ( -0.9 )	76.4 ( +2.9 )	44.3 ( -0.8 )	61.7 ( +0.3 )
IN-21K + AugReg	99.9 ( +15.4 )	70.6 ( -3.4 )	42.2 ( -1.0 )	54.1 ( -2.7 )	39.4 ( -3.8 )	47.9 ( -0.5 )	84.5 ( +9.4 )	55.5 ( +0.6 )	69.7 ( +3.2 )
OpenAI	99.9 ( +14.6 )	59.9 ( -15.8 )	13.9 ( -33.4 )	34.9 ( -31.0 )	19.7 ( -31.2 )	30.5 ( -20.2 )	75.0 ( -1.3 )	33.4 ( -22.3 )	45.7 ( -16.9 )

Table 8 : Accuracy on each dataset of ViT-B/16 pretrained models after fine-tuning on ImageNet-1K validation set. The parenthesis denotes the difference with pretrained models.

Figure 7 : Representational shifts from fine-tuning on ImageNet-R, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured and column indicates different part of transformer demonstrated in Figure 9 . ImageNet-R (downstream), ImageNet-1K (pretraining), and ImageNet-A (OOD dataset) are used as evaluation datasets. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Pretraining Dataset	ImageNet-1K	AvgOOD	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
LAION-100M (Pretrained)	52.5	38.6	44.5	14.6	64.5	39.8	30	43.9	44.5	27.3
LAION-100M (Fine-tuned)	43.9	28.6	36.2	8.7	53.9	29.7	23.2	34.1	22.0	20.8
LAION-400M (Pretrained)	59.6	42.6	52.4	19.6	70.8	46.4	38.9	48.4	29.4	35.1
LAION-400M (Fine-tuned)	14.2	9.2	12.5	4.0	19.6	7.3	9.3	9.8	5.3	5.8
OpenAI (Pretrained)	60.2	41.4	52.9	28.3	67.1	40.4	31.6	46.1	29.1	35.9
OpenAI (Fine-tuned)	5.7	3.9	4.6	2.1	8.7	2.3	5.1	3.9	2.1	2.6

Table 9 : Average accuracy on ImageNet-1K validation set and each OOD dataset using ViT-B/32 CLIP with zero-shot classifier before and after fine-tuning on the StanfordCars Dataset. Bold and denotes the best performance of pretrained and fine-tuned models, respectively.

Appendix figures & tables57 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8 : The average accuracy on OOD datasets before (blue) and after (red) fine-tuning with each method on fine-tuning datasets. The red bar is calculated directly by evaluating pretrained models on OOD datasets while the blue bar is calculated by adding

mRI

of each method to the pretrained models’ accuracy. Note that it is identical to the average accuracy on OOD datasets after fine-tuning on each dataset (

mRI+\frac{1}{n}\sum_{i}^{n}A^{(i)}_{\text{pre}}=\frac{1}{n}\sum_{j}\frac{1}{n-1}\sum_{i,i\neq j}^{n}A^{(i)}_{\text{down}}

). Fine-tuning LAION-2B and OpenAI pretrained models on the fine-tuning datasets causes severe robustness loss leading to worse performance than ImageNet-1K with AugReg pretrained model. Conversely, ImageNet-1K with the AugReg pretrained model improves robustness after fine-tuning. Note that the difference between red and blue bars is

mRI

Figure 9 : Transformer Architecture of ViT-B/16. We describe only the weighted layers, i.e. , fully connected layers and layer normalization layers.

Panel of Figure 10 — Figure 10 : Optimal Transport Dataset Distances (OTDD) in the feature space aligns with each dataset design. Pairwise OTDD (up) and Normalized Compression Distance (NCD) (down) between datasets using images (left) and features extracted by ImageNet-1K with AugReg pretrained ViT-B/16 on each dataset (right), respectively.

Figure 11 : Relationship between post fine-tuning ImageNet-1K accuracy and the distance between ImageNet-1K and the fine-tuning dataset. As the distance increases, accuracy generally decreases across fine-tuning methods. We exclude synthetic datasets made from ImageNet-1K validation set to avoid interference.

Method	FT	LinearProbing	Visual Prompt	LoRA	EWC	LwF	LP-FT	WiSE-FT	Model Soup
IN-1K + AugReg	-0.64	-0.22	-0.91	-0.63	-0.57	-0.49	-0.59	-0.46	-0.54
IN-21K	-0.77	-0.36	-0.92	-0.25	-0.88	-0.56	-0.69	-0.92	-0.89
IN-21K + AugReg	-0.68	0.10	-0.86	-0.63	-0.91	-0.38	-0.39	-0.52	-0.51
LAION-2B	-0.67	-0.19	-0.74	-0.31	-0.32	-0.44	-0.56	-0.31	-0.13

Appendix

Table 10 : Pearson correlation coefficient between the accuracy on ImageNet-1K and the dataset distance between ImageNet-1K and each fine-tuning dataset. There is a negative correlation between accuracy and dataset distance. Notably, FT and Prompter consistently exhibit a strong negative correlation across different pretrained models.

Figure 12 : Fine-tuning on a small dataset leads to severe accuracy degradation in both in- and out-of-distribution. Average accuracy on fine-tuning datasets and OOD datasets after fine-tuning pretrained ViT-B/16 models on a small number of images per class of fine-tuning datasets. The dashed line denotes the average accuracy of the pretrained models.

Method	IN-1K + AugReg		IN-21K + AugReg		LAION-2B
Method	$mRI$	Best Epoch	$mRI$	Best Epoch	$mRI$	Best Epoch
FT	2.8 ( +1.5 )	4.2	-5.6 ( -0.1 )	7.1	-40.4 ( -2.3 )	12.1
Linear Probing	1.6 ( +0.9 )	17.8	-0.7 ( -0.4 )	11.5	-2.3 ( -0.3 )	14.1
Visual Prompt	-4.1 ( +0.4 )	22.4	-9.1 ( -0.3 )	22.4	-8.9 ( -0.7 )	20.6
LoRA	2.6 ( +1.7 )	20.1	1.2 ( +3.3 )	16.5	-2.9 ( +0.7 )	15.6
EWC	3.6 ( +0.8 )	18.1	0.5 ( -0.1 )	19.8	-14.1 ( -1.6 )	24.5

Appendix

Table 11 :

mRI

values obtained using the best validation accuracy for each model on the fine-tuning datasets. Parentheses indicate the accuracy difference compared to models fine-tuned for 10 epochs without splitting the training and validation sets. The average number of epochs needed to achieve the highest validation accuracy on each fine-tuning dataset.

\dagger

: WiSE-FT and Model Soup are post-hoc weight interpolation methods and do not involve training.

Method	IN-1K + AugReg	IN-21K + AugReg	LAION-2B
FT	-17.6	-18.5	-31.7
linear Probing	-11.4	-9.2	-9.7
Visual Prompt	-13.6	-12.8	-10.6
LoRA	-3.7	-5.4	-0.3
EWC	-11.5	-11.2	-19.5
LwF	-15.3	-13.4	-29.3

Appendix

Table 12 : mean effective robustness improvement across OOD datasets obtained using the best validation accuracy for each model on the fine-tuning datasets. Parentheses indicate the accuracy difference compared to models fine-tuned for 10 epochs without splitting the training and validation sets. The average number of epochs needed to achieve the highest validation accuracy on each fine-tuning dataset.

Architecture	Pretraining Dataset	Library	Weight Name
ViT-B/16	IN-1K + AugReg	timm	vit_base_patch16_224.augreg_in1k
	IN-1K + SAM	timm	vit_base_patch16_224.sam_in1k
	IN-21K	timm	vit_base_patch16_224.orig_in21k_ft_in1k
	IN-21K + AugReg	timm	vit_base_patch16_224.augreg_in21k_ft_in1k
	IN-21K-P	timm	vit_base_patch16_224_miil.in21k_ft_in1k
	LAION-2B	timm	vit_base_patch16_clip_224.laion2b_ft_in1k

Appendix

Table 13 : Python libraries and the names of network weights for each pretrained model.

Figure 13 :

mRI

is much lower than ImageNet-21K with AugReg model, even with a small learning rate. Average accuracy on the fine-tuning datasets and

mRI

while fine-tuning pretrained ViT-B/16 models.

Weight Decay	0	0.0001	0.0005	0.001	0.005	0.01	0.1
$mRI$	-38.1	-39.1	-44.2	-39.4	-40.9	-49.4	-59.3

Appendix

Table 14 :

mRI

of ViT-B/16 pre-trained on LAION-2B with various weight decay used in fine-tuning. We use vanilla fine-tuning (FT).

Model	Pretraining	AdamW-0.001		AdamW-0.00001
Model	Pretraining	$mRI$	Down.Acc	$mRI$	Down.Acc
Fine-tuned on ImageNet-1K	IN-1K + AugReg	-30.5	99.7	1.7	97.1
	IN-21K + AugReg	-49.2	99.7	-2.2	99.7
	OpenAI	-58.6	49.9	-5.7	99.7
	LAION-2B	-56.1	78.7	-4.8	99.7
Zero-shot CLIP	LAION-400M	-49.2	27.9	-2.2	99.6

Appendix

Table 15 :

mRI

and mean accuracy on the fine-tuning dataset (Down.Acc) of ViT-B/16 under AdamW with learning rates 0.001 and 0.00001, for four pretrainings fine-tuned on ImageNet-1K and three zero-shot CLIP models. Only AdamW with learning rate 0.00001 both fits the target and retains robustness.

Method	ImageNet 1K + AugReg	ImageNet 21K + AugReg	LAION-2B
FT	1.4 $\pm$ 0.0	-5.4 $\pm$ 0.1	-38.2 $\pm$ 0.0
Linear Probing	0.8 $\pm$ 0.0	-0.3 $\pm$ 0.0	-2.0 $\pm$ 0.1
EWC	2.9 $\pm$ 0.1	0.8 $\pm$ 0.1	-12.9 $\pm$ 0.4
LwF	3.2 $\pm$ 0.1	-0.8 $\pm$ 0.1	-33.5 $\pm$ 0.2
LP-FT	2.0 $\pm$ 0.2	-2.7 $\pm$ 0.2	-37.2 $\pm$ 0.1
Wise-FT	3.7 $\pm$ 0.1	1.9 $\pm$ 0.1	-21.5 $\pm$ 0.3

Appendix

Table 16 :

mRI

of ViT-B/16 pre-trained on three different datasets after fine-tuning with different fine-tuning methods. We run with three random seeds and calculate the mean and standard error.

Pretraining Dataset	WiSE-FT ( $\alpha=0.5$ )	WiSE-FT (best $\alpha$ )	best $\alpha$
IN-1K + AugReg	3.6	4.7	0.4
IN-1K + SAM	3.6	4.1	0.3
IN-21K	2.5	2.5	0.5
IN-21K-P	3.0	3.0	0.5
IN-21K + AugReg	1.7	2.3	0.7
OpenAI	-18.1	-1.6	0.9

Appendix

Table 17 :

mRI

of ViT-B/16 pretrained on various datasets using WiSE-FT with default ratio (0.5) and the best ratio

Arch.	Pretraining Dataset	ImageNet-1K	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	IN-1K + AugReg	79.2	66.4	15.0	38.0	28.0	33.0	66.2	39.1	56.0
	IN-1K + SAM	80.2	68.2	9.0	40.1	27.7	34.2	66.9	42.3	54.6
	IN-21K	81.8	71.4	32.0	47.3	35.8	42.5	69.4	44.1	58.3
	IN-21K-P	84.3	74.0	34.1	51.5	40.2	46.7	73.5	45.1	61.4
	IN-21K + AugReg	84.5	74.0	43.2	56.8	43.2	48.4	75.1	54.9	66.5
	OpenAI	85.3	75.7	47.3	65.9	50.9	50.7	76.3	55.7	62.6

Appendix

Table 18 : The average accuracy of various pretrained models on ImageNet-1K validation set and OOD datasets.

Architecture	ViT-B/16							ViT-B/32					ViT-S/16		ViT-S/32	ViT-L/16	ResNet-18	ResNet-50
Method	IN-1K + AugReg	IN-1K + SAM	IN-21K	IN-21K-P	IN-21K + AugReg	OpenAI	LAION-2B	IN-1K + AugReg	IN-1K + SAM	IN-21K + AugReg	OpenAI	LAION-2B	IN-1K + AugReg	IN-21K + AugReg	IN-21K + AugReg	IN-21K + AugReg	IN-1K	IN-1K
FT	0.6	1.3	0.6	0.8	-1.0	-28.0	-28.5	0.5	1.7	0.4	-27.5	-31.7	-1.3	-3.0	-2.2	1.5	-7.9	-7.3
Linear Probing	1.8	1.5	1.5	1.6	1.6	0.0	0.3	2.7	2.4	2.2	-0.4	-0.3	1.4	1.3	1.6	1.9	-13.6	-19.4
Visual Prompt	-6.9	-7.6	-8.1	-7.3	-6.0	-4.5	-4.3	-9.2	-9.0	-9.3	-6.8	-6.7	-9.3	-9.0	-13.7	-8.2	-16.1	-6.4
LoRA	-0.1	-0.1	-0.6	-0.1	-2.7	-2.7	-2.5	-0.4	-0.1	-0.2	-2.8	-2.5	-1.5	-0.3	-0.3	0.1	-	-
EWC	-0.7	0.0	-0.3	-0.6	-1.5	-7.5	-6.7	-1.3	-0.1	-0.9	-7.1	-9.2	-1.2	-2.0	-2.0	-0.6	-12.4	-16.6

Appendix

Table 19 : Average backward transfer on the ImageNet-1K validation set for each method, evaluated across different architectures and pretraining datasets. Bold indicates the highest backward transfer for each model.

Arch.	Pretraining Dataset	Method	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	IN-1K + AugReg	FT	96.3	97.5	98.4	96.0	97.7	97.5	97.6	100.0
		Linear Probing	71.5	42.4	60.4	58.8	57.8	76.1	61.2	89.7
		Visual Prompt	66.6	28.8	58.9	45.8	46.3	72.7	64.8	64.4
		LoRA	66.5	18.7	41.8	39.0	36.7	69.7	62.4	58.6
		EWC	72.0	54.3	65.3	50.3	50.8	76.2	70.3	67.7
		LwF	95.8	95.5	97.3	95.1	95.1	96.7	96.5	100.0

Appendix

Table 20 : Training accuracy on downstream datasets after fine-tuning with each method using ViT-B/16. FT and LP-FT generally achieve the highest performance, while Visual Prompt and LoRA show the lowest.

Arch.	Pretraining Dataset	Method	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/32	IN-1K + AugReg	FT	94.6	94.0	97.3	95.3	95.9	96.3	96.3	100.0
		Linear Probing	68.7	34.9	63.4	62.1	54.7	75.1	62.6	90.9
		Visual Prompt	59.6	15.1	54.0	41.4	38.3	68.3	60.1	59.3
		LoRA	61.1	10.1	42.1	31.9	31.0	67.3	53.1	66.3
		EWC	65.9	33.4	59.3	45.6	42.1	71.3	64.3	62.7
		LwF	94.0	91.2	95.7	94.2	92.6	95.5	95.0	99.9

Appendix

Table 21 : Training accuracy on downstream datasets after fine-tuning with each method using various ViTs. FT and LP-FT generally achieve the highest performance, while Visual Prompt and LoRA show the lowest.

Arch.	Pretraining Dataset	Method	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ResNet-18	IN-1K	FT	98.3	98.7	99.1	95.7	97.6	97.4	97.1	100.0
		Linear Probing	59.9	6.5	47.0	33.3	34.2	65.1	48.3	50.8
		Visual Prompt	52.4	5.8	35.9	22.9	29.1	46.8	28.2	28.1
		EWC	63.0	17.5	50.8	37.9	38.0	66.0	54.0	42.6
		LwF	97.0	97.6	98.3	94.7	96.7	96.2	95.8	99.9
		LP-FT	98.5	98.5	98.9	95.8	97.4	97.7	97.3	100.0

Appendix

Table 22 : Training accuracy on downstream datasets after fine-tuning with each method. FT and LP-FT generally achieve the highest performance, while EWC shows the lowest.

Architecture	Method	$mRI$	$RI$
Architecture	Method	$mRI$	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	FT	1.3	2.9	-4.0	2.8	4.4	-2.7	0.6	0.4	5.9
	Linear Probing	0.7	0.1	-0.1	0.8	1.2	0.3	0.2	0.1	3.2
	Visual Prompt	-4.5	-2.3	-9.1	-4.9	-1.6	-11.2	-3.9	-4.3	1.7
	LoRA	0.9	0.2	0.4	1.1	2.6	0.3	-0.1	1.3	1.1
	EWC	2.8	2.9	-0.2	5.2	4.4	1.4	1.6	2.8	4.3

Appendix

Table 23 :

RI

and

mRI

of ImageNet-1K with AugReg pretrained models with different fine-tuning methods and downstream datasets on each dataset in ImageNet-RIB.

Architecture	Method	$mRI$	$RI$
Architecture	Method	$mRI$	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	FT	2.5	3.4	-3.2	5.8	5.7	-2.3	1.7	1.8	7.3
	Linear Probing	0.8	0.1	0.3	0.5	1.1	0.0	0.0	0.4	4.0
	Visual Prompt	-6.1	-4.1	-12.4	-6.4	-6.3	-10.6	-3.4	-4.9	-0.2
	LoRA	0.9	0.1	0.4	0.8	1.1	0.3	0.1	0.5	3.6
	EWC	1.6	0.7	0.3	3.7	2.4	1.2	1.0	2.0	1.3

Appendix

Table 24 :

RI

and

mRI

of ImageNet-1K with SAM pretrained models with different fine-tuning methods and downstream datasets on each dataset in ImageNet-RIB.

Architecture	Method	$mRI$	$RI$
Architecture	Method	$mRI$	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	FT	-0.1	1.5	-2.8	0.3	2.7	-4.0	-1.2	-1.8	4.2
	Linear Probing	0.4	0.3	0.4	0.2	0.0	0.8	-0.0	1.0	0.5
	Visual Prompt	-9.4	-7.7	-12.7	-11.1	-7.7	-14.8	-8.4	-10.0	-3.3
	LoRA	-0.3	0.2	0.5	-1.6	-0.5	0.9	-0.4	0.6	-1.9
	EWC	1.4	1.5	0.2	2.4	2.9	0.1	0.5	1.2	2.6

Appendix

Table 25 :

RI

and

mRI

of ImageNet-21K pretrained models with different fine-tuning methods and downstream datasets on each dataset in ImageNet-RIB.

Architecture	Method	$mRI$	$RI$
Architecture	Method	$mRI$	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	FT	-5.5	-1.4	-9.1	-5.4	-5.3	-11.1	-3.6	-5.7	-2.5
	Linear Probing	-0.3	-0.4	-0.4	-0.9	-1.3	-0.5	0.4	0.6	0.2
	Visual Prompt	-8.0	-6.1	-9.2	-9.3	-6.1	-16.6	-7.1	-7.0	-3.0
	LoRA	-2.1	0.7	0.8	2.6	2.9	0.8	0.8	1.6	-27.4
	EWC	0.6	2.0	-2.0	2.3	3.5	-3.5	0.4	0.5	1.7

Appendix

Table 26 :

RI

and

mRI

of ImageNet-21K with AugReg pretrained models with different fine-tuning methods and downstream datasets on each dataset in ImageNet-RIB.

Architecture	Method	$mRI$	$RI$
Architecture	Method	$mRI$	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	FT	-0.5	0.7	-3.5	1.6	3.0	-4.4	-1.4	-2.2	2.3
	Linear Probing	0.2	0.2	0.4	0.5	1.1	0.3	0.2	0.3	-1.0
	Visual Prompt	-10.1	-8.0	-11.5	-9.9	-7.8	-19.9	-8.8	-11.1	-3.6
	LoRA	0.4	0.1	0.3	0.5	1.2	0.5	-0.2	0.9	-0.1
	EWC	1.3	1.3	0.6	1.1	3.0	0.8	0.6	0.6	2.1

Appendix

Table 27 :

RI

and

mRI

of ImageNet-21K-P pretrained models with different fine-tuning methods and downstream datasets on each dataset in ImageNet-RIB.

Architecture	Method	$mRI$	$RI$
Architecture	Method	$mRI$	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	FT	-38.1	-39.4	-53.7	-36.9	-46.9	-49.2	-29.0	-36.6	-12.9
	Linear Probing	-2.0	-0.6	-0.9	-1.5	-0.8	-1.9	-2.4	-5.5	-2.0
	Visual Prompt	-8.2	-6.4	-8.1	-8.1	-6.7	-16.7	-8.3	-8.0	-2.9
	LoRA	-3.6	-0.8	-1.3	-1.6	-1.2	-3.0	-2.8	-5.8	-12.3
	EWC	-12.5	-16.1	-27.2	-2.4	-17.3	-19.0	-6.4	-9.9	-1.7

Appendix

Table 28 :

RI

and

mRI

of LAION-2B pretrained models with different fine-tuning methods and downstream datasets on each dataset in ImageNet-RIB.

Architecture	Method	$mRI$	$RI$
Architecture	Method	$mRI$	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
ViT-B/16	FT	-38.0	-38.3	-51.6	-35.4	-48.5	-50.3	-28.9	-35.8	-15.3
	Linear Probing	-2.0	-0.5	-0.8	-1.3	-1.3	-1.2	-3.4	-5.6	-1.8
	Visual Prompt	-8.4	-7.4	-8.1	-7.6	-6.3	-16.3	-9.4	-9.9	-2.7
	LoRA	-3.6	-0.6	-1.0	-1.9	-1.0	-2.8	-4.0	-6.4	-11.3
	EWC	-12.7	-14.4	-20.9	-2.4	-24.8	-19.9	-7.5	-10.8	-0.8

Appendix

Table 29 :

RI

and

mRI

of OpenAI CLIP models with different fine-tuning methods and downstream datasets on each dataset in ImageNet-RIB.

Figure 14 : Representational shifts while fine-tuning on ImageNet-R, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured and column indicates different part of transformer demonstrated in Figure 9 . Darker and higher-contrast shades denote later epochs in the fine-tuning process. ImageNet-R (downstream), ImageNet-1K (pretraining), and ImageNet-A (OOD dataset) are used as evaluation datasets. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Figure 15 : Training accuracy versus OOD robustness across pretraining datasets (ViT-B/32). Each panel corresponds to one pretraining dataset. Each point is one fine-tuned model, i.e., a (fine-tuning method, fine-tuning dataset) pair. The

x

-axis shows the training accuracy on the fine-tuning dataset and the

y

-axis shows the mean accuracy on the seven held-out OOD datasets. The dashed line indicates the Pareto envelope over all points. Vanilla fine-tuning (FT) lies on the envelope for ImageNet-pretrained models but falls below it for CLIP-pretrained models (OpenAI, LAION-2B), where linear probing and weight-averaging methods span the envelope.

Fine-Tuning Dataset	Figure Index
ImageNet-V2	Figure S1
ImageNet-A	Figure S2
ImageNet-R	Figure S3
ImageNet-Sketch	Figure S4
ObjectNet	Figure S5
ImageNet-Cartoon	Figure S6

Appendix

Table 30 : Reference for the Figures regarding Centered Kernel Alignment.

Architecture	$D_{\text{pre}}$	Accuracy on OOD datasets	Accuracy on ImageNet-C
ViT-B/16	IN-1K + AugReg	Table S1	Table S18
	IN-1K + SAM	Table S2	Table S19
	IN-21K	Table S3	Table S20
	IN-21K-P	Table S4	Table S21
	IN-21K + AugReg	Table S5	Table S22
	LAION-2B	Table S6	Table S23

Appendix

Table 31 : Reference for the tables showing accuracy of pre-trained models on OOD datasets (left) and ImageNet-C corruptions (right).

Figure S1 : Representational shifts from fine-tuning on ImageNet-V2, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured, and column indicates different parts of the transformer demonstrated in Figure 9 in the main paper. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Figure S2 : Representational shifts from fine-tuning on ImageNet-A, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured, and column indicates different parts of the transformer demonstrated in Figure 9 in the main paper. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Figure S3 : Representational shifts from fine-tuning on ImageNet-R, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured, and column indicates different parts of the transformer demonstrated in Figure 9 in the main paper. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Figure S4 : Representational shifts from fine-tuning on ImageNet-Sketch, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured, and column indicates different parts of the transformer demonstrated in Figure 9 in the main paper. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Figure S5 : Representational shifts from fine-tuning on ObjectNet, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured, and column indicates different parts of the transformer demonstrated in Figure 9 in the main paper. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Figure S6 : Representational shifts from fine-tuning on ImageNet-Cartoon, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured, and column indicates different parts of the transformer demonstrated in Figure 9 in the main paper. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Figure S7 : Representational shifts from fine-tuning on ImageNet-Drawing, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured, and column indicates different parts of the transformer demonstrated in Figure 9 in the main paper. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Figure S8 : Representational shifts from fine-tuning on ImageNet-C, analyzed by Centered Kernel Alignment (CKA), in ViT-B/16 models pretrained on various datasets. Rows indicate datasets where CKA is measured, and column indicates different parts of the transformer demonstrated in Figure 9 in the main paper. Models pretrained on OpenAI and LAION-2B show distinct CKA patterns across layers compared to others.

Method	Fine-Tuning Dataset	IN-1K	IN-V2	IN-A	IN-R	IN-Sketch	ObjNet	IN-Cartoon	IN-Drawing	IN-C
Pretrained		85.3	75.7	47.3	65.9	50.9	50.7	76.3	55.7	62.6
FT	IN-V2	60.8	-	9.7	24.2	10.1	22.0	36.4	13.0	25.7
	IN-A	29.8	23.3	-	10.5	4.1	11.4	14.3	4.1	8.9
	IN-R	47.9	37.9	7.9	-	29.4	19.3	34.4	20.1	22.2
	IN-Sketch	24.4	18.0	1.9	34.6	-	8.2	17.3	7.5	7.2
	ObjNet	36.0	26.7	4.9	12.5	5.4	-	18.3	3.7	10.6

Appendix

Table S7 : The accuracy on each OOD dataset after fine-tuning OpenAI CLIP ViT-B/16 on the downstream datasets with various methods. Note that ImageNet-Drawing, ImageNet-Cartoon, and ImageNet-C are generated from the ImageNet validation set. Green and red indicate relative performance increases and decreases, respectively, compared to the pretrained model. Bold indicates the best performance on each evaluation dataset.

Method	Fine-Tuning Dataset	Avg.	Noise			Blur				Weather				Digital
Method	Fine-Tuning Dataset	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		56.0	57	54	54	49	42	53	46	48	55	61	74	56	59	67	66
FT	IN-V2	57.4	56	54	53	51	40	55	46	53	59	65	74	59	58	68	67
	IN-A	53.5	53	51	50	50	38	52	39	50	56	57	70	56	51	65	64
	IN-R	52.0	52	50	49	46	44	49	37	49	55	57	66	53	51	62	61
	IN-Sketch	53.8	55	53	52	46	39	49	43	51	56	58	70	55	55	63	62

Appendix

Table S18 : Accuracy of ImageNet-1K with AugReg pretrained ViT-B/16 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		54.6	53	50	51	50	48	55	47	47	51	51	73	46	64	68	67
FT	IN-V2	56.8	51	49	48	54	49	59	50	51	53	59	75	52	65	69	68
	IN-A	51.3	42	39	36	56	45	56	43	46	46	56	71	52	58	61	61
	IN-R	53.9	49	48	47	53	51	54	42	50	54	55	70	52	58	62	64
	IN-Sketch	55.7	53	52	51	52	46	55	49	50	52	59	72	52	60	66	66

Appendix

Table S19 : Accuracy of ImageNet-1K with SAM pretrained ViT-B/16 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		58.3	55	53	53	56	50	60	52	53	54	57	75	52	64	72	69
FT	IN-V2	58.5	51	49	48	60	49	62	53	54	53	62	76	56	63	72	70
	IN-A	55.0	47	45	43	56	45	60	50	51	50	57	73	55	58	69	66
	IN-R	52.9	48	47	46	51	49	53	44	48	52	53	70	50	56	65	63
	IN-Sketch	55.5	50	49	48	53	47	56	50	51	54	57	72	53	59	69	66

Appendix

Table S20 : Accuracy of ImageNet-21K pretrained ViT-B/16 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		61.4	60	59	58	57	46	60	53	55	55	65	79	69	64	70	71
FT	IN-V2	62.1	58	57	55	62	48	63	55	57	55	67	79	69	62	73	73
	IN-A	60.5	56	55	52	63	48	63	53	54	51	64	77	69	60	71	72
	IN-R	57.5	55	55	53	56	50	57	47	52	55	59	73	57	58	66	68
	IN-Sketch	59.3	57	57	56	54	44	58	52	54	56	61	76	65	60	70	70

Appendix

Table S21 : Accuracy of ViT-B/16 pretrained on ImageNet-21K-P, using different fine-tuning methods and downstream datasets on each ImageNet-C corruption For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		66.5	67	66	66	63	54	67	59	66	62	69	80	65	66	75	74
FT	IN-V2	64.4	64	62	63	61	56	64	55	64	61	66	77	63	63	73	73
	IN-A	56.3	57	56	56	52	48	56	46	53	51	56	70	53	55	67	68
	IN-R	55.7	54	53	53	53	51	54	43	52	56	57	69	54	56	65	65
	IN-Sketch	54.8	54	53	52	52	44	53	43	52	55	57	70	55	53	64	65

Appendix

Table S22 : Accuracy of ImageNet-21K with AugReg pretrained ViT-B/16 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		63.0	59	59	59	58	49	64	52	64	61	70	81	70	61	67	71
FT	IN-V2	24.5	18	17	16	15	18	19	18	19	21	35	45	33	30	31	32
	IN-A	8.5	8	7	7	5	5	6	6	6	6	12	17	11	11	11	10
	IN-R	21.4	16	15	14	15	18	18	16	15	20	29	38	31	24	26	25
	IN-Sketch	9.7	9	8	8	3	4	5	6	9	11	15	21	16	9	8	14

Appendix

Table S23 : Accuracy of LAION-2B pretrained ViT-B/16 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		62.6	58	58	58	57	46	63	54	63	59	69	80	70	62	71	70
FT	IN-V2	25.7	18	17	16	15	21	21	19	20	23	36	45	33	34	33	36
	IN-A	8.9	7	7	5	5	6	6	6	7	7	13	17	12	11	12	12
	IN-R	22.2	17	16	15	16	18	19	16	16	22	28	38	29	26	28	28
	IN-Sketch	7.2	7	7	6	2	3	4	4	7	9	11	14	10	7	7	12

Appendix

Table S24 : Accuracy of OpenAI CLIP ViT-B/16 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		53.5	55	54	54	46	43	49	41	43	53	54	69	52	57	67	64
FT	IN-V2	53.4	53	51	51	48	43	50	39	47	55	56	70	54	57	66	64
	IN-A	47.5	48	46	46	45	38	46	32	40	48	48	62	49	48	58	58
	IN-R	47.0	47	46	45	42	43	44	31	42	50	46	61	44	49	57	57
	IN-Sketch	51.1	53	52	51	45	41	45	39	45	52	51	65	50	53	63	61

Appendix

Table S25 : Accuracy of ImageNet-1K with AugReg pretrained ViT-B/32 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		48.8	49	48	47	45	47	46	41	35	45	36	65	35	59	68	65
FT	IN-V2	51.1	48	46	45	50	48	50	42	41	47	45	68	41	61	67	67
	IN-A	43.1	36	35	33	50	42	45	32	36	38	34	59	35	53	59	59
	IN-R	49.2	49	49	48	49	47	46	34	42	48	44	62	44	55	61	62
	IN-Sketch	50.1	49	48	47	48	46	47	42	41	46	46	64	42	57	65	64

Appendix

Table S26 : Accuracy of ImageNet-1K with SAM pretrained ViT-B/32 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		60.5	61	60	60	56	49	61	50	54	56	60	75	59	62	72	70
FT	IN-V2	60.2	59	57	58	58	48	62	50	55	57	62	75	61	60	72	70
	IN-A	56.9	54	52	53	56	46	59	46	51	53	58	71	59	56	69	68
	IN-R	55.3	54	53	53	54	50	55	43	50	55	56	68	55	55	66	65
	IN-Sketch	57.0	58	57	56	54	44	56	48	52	55	58	71	56	56	68	66

Appendix

Table S27 : Accuracy of ImageNet-21K with AugReg pretrained ViT-B/32 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		57.5	56	55	55	53	42	57	46	52	51	61	75	64	59	68	69
FT	IN-V2	21.7	20	19	18	11	15	15	15	14	20	23	36	25	28	31	36
	IN-A	7.0	8	7	7	4	5	4	5	4	5	7	12	8	8	10	11
	IN-R	20.9	20	19	18	14	17	17	15	12	20	22	33	25	23	28	31
	IN-Sketch	9.7	10	10	9	4	5	5	6	7	11	11	17	13	9	11	17

Appendix

Table S28 : Accuracy of LAION-2B pretrained ViT-B/32 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		57.9	58	56	56	53	41	57	46	52	53	61	76	64	60	68	68
FT	IN-V2	25.9	23	22	21	16	20	20	19	17	23	29	39	30	33	37	39
	IN-A	10.1	10	9	10	7	8	8	7	6	7	10	15	12	13	15	15
	IN-R	23.1	21	20	20	17	20	19	17	15	23	23	37	26	26	30	32
	IN-Sketch	11.1	12	11	11	4	5	6	6	8	12	12	21	14	11	14	20

Appendix

Table S29 : Accuracy of OpenAI CLIP ViT-B/32 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		53.2	54	52	52	46	37	50	42	48	52	58	73	53	55	63	63
FT	IN-V2	51.2	49	47	46	46	36	50	38	50	52	58	70	52	52	60	62
	IN-A	44.1	43	40	40	42	31	44	31	42	43	47	62	44	44	51	57
	IN-R	42.7	40	39	37	39	37	41	31	39	44	47	58	44	44	50	52
	IN-Sketch	45.5	44	42	42	38	31	42	35	42	46	50	64	49	46	53	55

Appendix

Table S30 : Accuracy of ImageNet-1K with AugReg pretrained ViT-S/16 with with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		58.0	58	56	56	54	43	59	48	55	55	60	75	56	59	69	68
FT	IN-V2	56.8	54	52	53	54	44	58	46	54	55	62	74	58	56	65	67
	IN-A	52.7	49	47	48	51	39	55	43	51	50	57	69	54	51	63	64
	IN-R	50.6	48	46	46	49	45	50	39	46	51	53	66	50	51	58	60
	IN-Sketch	50.7	48	47	47	48	36	51	41	48	50	54	67	52	50	59	60

Appendix

Table S31 : Accuracy of ImageNet-21K with AugReg pretrained ViT-S/16 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		52.0	54	53	53	47	40	51	39	43	47	50	69	48	56	65	64
FT	IN-V2	50.1	49	47	47	47	40	50	40	43	47	51	66	47	52	62	62
	IN-A	44.0	43	41	41	44	37	46	34	34	39	43	59	42	44	56	56
	IN-R	43.4	42	41	40	42	40	42	31	36	43	44	57	42	45	53	54
	IN-Sketch	45.5	46	45	45	43	34	45	35	39	43	45	60	44	46	56	56

Appendix

Table S32 : Accuracy of ImageNet-21K with AugReg pretrained ViT-S/32 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption. For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		72.2	73	73	73	69	62	73	67	72	69	73	82	70	71	79	78
FT	IN-V2	71.0	72	71	72	67	62	71	64	70	70	72	81	69	69	79	78
	IN-A	70.4	72	71	71	67	61	71	64	70	69	71	80	68	68	78	77
	IN-R	65.4	68	67	67	61	61	64	55	61	64	65	76	60	64	74	74
	IN-Sketch	67.6	69	68	68	65	58	67	61	67	66	68	78	64	65	75	74

Appendix

Table S33 : Accuracy of ImageNet-21K with AugReg pretrained ViT-L/16 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		31.7	23	21	18	28	23	30	30	24	28	34	59	31	40	42	46
FT	IN-V2	29.9	21	20	16	26	22	27	26	25	28	31	56	27	37	41	46
	IN-A	19.9	13	13	10	16	14	17	14	16	18	23	39	20	24	30	31
	IN-R	25.6	21	21	17	20	21	22	20	21	26	26	45	22	30	35	37
	IN-Sketch	19.2	16	15	13	11	12	14	12	17	22	20	37	18	21	28	32

Appendix

Table S34 : Accuracy of ImageNet-1K pretrained ResNet-50 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption For each corruption, accuracy is averaged across 5 levels of severity.

Dataset	Method	Avg.	Noise			Blur				Weather				Digital
Dataset	Method	Avg.	Gauss.	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
Pretrained		46.6	41	39	36	41	27	42	43	40	45	58	71	57	47	50	59
FT	IN-V2	47.0	41	40	36	40	28	43	42	42	48	59	72	55	47	51	62
	IN-A	46.3	43	42	40	40	27	41	39	44	47	58	68	56	44	48	58
	IN-R	43.7	40	39	36	34	33	38	36	39	48	52	65	48	45	48	55
	IN-Sketch	23.3	20	19	16	13	11	17	16	26	28	33	41	23	22	29	35

Appendix

Table S35 : Accuracy of ImageNet-1K pretrained ResNet-50 with different fine-tuning methods and downstream datasets on each ImageNet-C corruption For each corruption, accuracy is averaged across 5 levels of severity.

Explore similar work

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
May 12, 2026Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer +2Model Fine-Tuning Retention
Carnegie Mellon University
Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
May 15, 2026Fabian Morelli, Arnas Uselis, Ankit Sonthalia +1Contrastive Language-Image Pre-Training Model Model Fine-Tuning
University of Tübingen ∗ · University of Tübingen
Robustness of Vision Foundation Models to Common Perturbations
Apr 16, 2026Hongbin Liu, Zhengyuan Jiang, Cheng Hong +1Recent Vision Foundation Models Vision Foundation Models
Duke University · Ant Group

May 12, 2026, cs.LGJ/K move · Enter open · S save

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer, Ziqian Zhong+1

Carnegie Mellon University

Model Fine-Tuning Retention Forgetting Early Layers Exposure

Open paper AskPDF arXiv

How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabilities, we study how upstream training choices - that is, the manner in which a capability is acquired - shape how robustly that capability is retained. We investigate this question in a controlled three-stage language-model pipeline: pretraining, post-training to acquire a target capability, and downstream fine-tuning on a new objective. Across 135M and 1B models, two post-training domains, and two downstream fine-tuning tasks, we find that immediate post-training performance does not reliably predict retention after subsequent fine-tuning: training recipes that look equivalent immediately after post-training can retain the target capability very differently after subsequent fine-tuning. In particular, early exposure - mixing post-training data into pretraining - consistently improves the frontier between retained upstream performance and downstream performance. In compute-matched experiments, where the target data must be allocated between pretraining and post-training, we find that the optimum lies at neither extreme. Together with our other empirical and theoretical findings, this supports the view that post-training drives immediate specialization while early exposure improves robustness to later forgetting. Replay and dropout, typically used to mitigate forgetting as it occurs during fine-tuning, provide complementary gains to early exposure when applied during post-training. Our findings suggest that robustness to subsequent fine-tuning should be treated as a first-class objective of upstream training, addressed preventatively through choices like early exposure rather than reactively during fine-tuning itself.

Abstract

Explore similar work

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models

Robustness of Vision Foundation Models to Common Perturbations