Restoring Neural Network Plasticity for Faster Transfer Learning

Xander Coetzer; Arné Schreuder; Anna Sergeevna Bosman

cs.CVMar 21, 2026

Restoring Neural Network Plasticity for Faster Transfer Learning

Authors: Xander Coetzer, Arné Schreuder, Anna Sergeevna Bosman

Organizations: Department of Computer Science, University of Pretoria, Pretoria, South Africa

Abstract

Transfer learning with models pretrained on ImageNet has become a standard practice in computer vision. Transfer learning refers to fine-tuning pretrained weights of a neural network on a downstream task, typically unrelated to ImageNet. However, pretrained weights can become saturated and may yield insignificant gradients, failing to adapt to the downstream task. This hinders the ability of the model to train effectively, and is commonly referred to as loss of neural plasticity. Loss of plasticity may prevent the model from fully adapting to the target domain, especially when the downstream dataset is atypical in nature. While this issue has been widely explored in continual learning, it remains relatively understudied in the context of transfer learning. In this work, we propose the use of a targeted weight re-initialization strategy to restore neural plasticity prior to fine-tuning. Our experiments show that both convolutional neural networks (CNNs) and vision transformers (ViTs) benefit from this approach, yielding higher test accuracy with faster convergence on several image classification benchmarks. Our method introduces negligible computational overhead and is compatible with common transfer learning pipelines.

Figures & tables

Dataset	Number of Classes	Images per Class	Total Images
DTD	47	120	5640
Brain Tumor	4	1600-2000	7150
Fruit25	25	450	11250

Table 1: The number of classes, number of images per class and total number of images within each dataset.

Model	Number of Parameters
SimpleNet	0.5 million
ResNet-50	25.6 million
ViT-B16	86.6 million

Table 2: Total number of trainable parameters per model.

Dataset	Experimental Case	Accuracy $\pm$	Std	Epochs $\pm$	Std
	10 M	60.355 $\pm$	1.148	96.000 $\pm$	9.110
DTD	10 MN	59.734 $\pm$	1.684	88.000 $\pm$	20.736
	5 MN	59.220 $\pm$	0.251	88.500 $\pm$	6.364
	Base	57.926 $\pm$	1.463	89.400 $\pm$	5.899
	Base	96.087 $\pm$	0.781	169.167 $\pm$	61.411
Brain Tumor	10 MN	95.304 $\pm$	0.583	107.200 $\pm$	18.102

Table 3: A comparison between plasticity-restoring experimental cases and the base case on the SimpleNet model. The specific experimental cases are listed using the notation defined in Section 3.1 . The results are ordered by accuracy, and the base case is indicated in bold.

Dataset	Experimental Case	Accuracy $\pm$	Std	Epochs $\pm$	Std
	5 M	72.030 $\pm$	0.564	129.000 $\pm$	58.387
DTD	Base	71.738 $\pm$	1.915	218.400 $\pm$	69.984
	10 MN	71.223 $\pm$	1.083	139.400 $\pm$	26.557
	10 M	70.626 $\pm$	1.242	133.667 $\pm$	35.698
	10 M	95.584 $\pm$	0.581	113.800 $\pm$	14.822
Brain Tumor	25 M	95.312 $\pm$	1.434	115.167 $\pm$	44.640

Table 4: A comparison between plasticity-restoring experimental cases and the base case on the ResNet-50 model. The specific experimental cases are listed using the notation defined in Section 3.1 . The results are ordered by accuracy, and the base case is indicated in bold.

Dataset	Experimental Case	Accuracy $\pm$	Std	Epochs $\pm$	Std
	5 M	71.809 $\pm$	0.862	95.600 $\pm$	15.010
DTD	Base	71.649 $\pm$	0.887	97.800 $\pm$	18.185
	5 MN	71.596 $\pm$	0.563	89.400 $\pm$	5.814
	25 MN	71.241 $\pm$	0.437	90.000 $\pm$	6.745
	25 M	98.001 $\pm$	0.390	70.100 $\pm$	17.559
Brain Tumor	5 MN	97.918 $\pm$	0.415	65.200 $\pm$	15.959

Table 5: A comparison between plasticity-restoring experimental cases and the base case on the ViT-B16 model. The specific experimental cases are listed using the notation defined in Section 3.1 . The results are ordered by accuracy, and the base case is indicated in bold.

Dataset	SimpleNet	ResNet-50	ViT-B16
DTD	$[2.4;+7]$	[0.3;-90]	[0.15;-2]
Brain Tumor	$[-0.7;-62]$	[0.5;-5]	$[0.4;+5]$
Fruit25	[0.5;-18]	[0.9;-9]	$[0.7;+4]$

Table 6: Best runs across all datasets and architectures, presented as tuples. The first element in the tuple is the gain or loss in the best accuracy for a model, and the second element is the corresponding gain or loss in epochs for that best experimental case. Tuples with both a gain in plasticity and decrease in epoch (improved convergence rate) are indicated in bold.

Panel of Figure 1 — Figure 1: Comparison between 25 $M$ (a) and 25 $N$ (b) experimental cases. The blue histogram is the total weight distribution after training on the base case. The green histogram represents the total weight distribution after fine-tuning in the experimental cases. The red histogram shows the absolute difference between the blue and green histograms and the final histogram overlays the blue and green histograms for easy comparison between base and experimental cases.

Explore similar work

Vision Transformers that Never Stop Learning
Mar 8, 2026Caihao Sun, Mingqi Yuan, Shiyuan Wang +1Loss of Plasticity Vision Transformer
University of Hong Kong · The Hong Kong Polytechnic University · Technische Universität Dresden +1
A Transfer Learning Evaluation of Deep Neural Networks for Image Classification
May 12, 2026Nermeen Abou Baker, Nico Zengeler, Uwe HandmannFine-Tuning Cross-Domain Transfer Learning
Computer Science Institute, Ruhr West University of Applied Sciences, 46236 Bottrop, Germany
Grow, Don't Overwrite: Fine-tuning Without Forgetting
Mar 9, 2026Dyah Adila, Hanna Mazzawi, Benoit Dherin +1Catastrophic Forgetting Parameter-Efficient Fine-Tuning
University of Wisconsin-Madison · Google Research

Mar 8, 2026J/K move · Enter open · S save

Vision Transformers that Never Stop Learning

Caihao Sun, Mingqi Yuan, Shiyuan Wang, Jiayu Chen

University of Hong Kong · The Hong Kong Polytechnic University · Technische Universität Dresden+1

Loss of Plasticity Vision Transformer

Open paper AskPDF arXiv

Loss of plasticity refers to the progressive inability of a model to adapt to new tasks and poses a fundamental challenge for continual learning. While this phenomenon has been extensively studied in homogeneous neural architectures, such as multilayer perceptrons, its mechanisms in structurally heterogeneous, attention-based models such as Vision Transformers (ViTs) remain underexplored. In this work, we present a systematic investigation of loss of plasticity in ViTs, including a fine-grained diagnosis using local metrics that capture parameter diversity and utilization. Our analysis reveals that stacked attention modules exhibit increasing instability that exacerbates plasticity loss, while feed-forward network modules suffer even more pronounced degradation. Furthermore, we evaluate several approaches for mitigating plasticity loss. The results indicate that methods based on parameter re-initialization fail to recover plasticity in ViTs, whereas approaches that explicitly regulate the update process are more effective. Motivated by this insight, we propose ARROW, a geometry-aware optimizer that preserves plasticity by adaptively reshaping gradient directions using an online curvature estimate for the attention module. Extensive experiments show that ARROW effectively improves plasticity and maintains better performance on newly encountered tasks.

Abstract

Explore similar work

Vision Transformers that Never Stop Learning

A Transfer Learning Evaluation of Deep Neural Networks for Image Classification

Grow, Don't Overwrite: Fine-tuning Without Forgetting