Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models

Elif Ceren Gok Yildirim; Murat Onur Yildirim; Joaquin Vanschoren

cs.LGMar 22, 2026

Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models

Authors: Elif Ceren Gok Yildirim, Murat Onur Yildirim, Joaquin Vanschoren

Organizations: AMOR/e Lab, Eindhoven University of Technology

Abstract

The continual learning literature has rapidly shifted from traditional class incremental learning (CIL) techniques to foundation model (FM)-based CIL methods without a clear understanding of how these newer approaches compare to strong, lightweight convolutional baselines. This abrupt transition has created a substantial methodological gap, making it difficult to assess whether recent FM-based CIL progress reflects genuine advances or merely the absence of rigorous baselines. To address this gap, we introduce Pruned Adaptation Modules (PAM), a simple yet effective method that freezes the vast majority of the pre-trained ResNet while enabling scalable continual adaptation through sparse task-specific layers. PAM yields up to a ~5x reduction in trainable parameters and a ~6x reduction in total parameters, significantly reducing the cost of continual updates. Across diverse benchmarks, PAM consistently mitigates catastrophic forgetting and outperforms state-of-the-art FM-based CIL approaches. Our findings position PAM as a strong and transparent baseline that helps bridge the gap between traditional and FM-based CIL, guiding future research for a more accurate assessment of true progress in continual adaptation. The code can be found at: https://github.com/ElifCerenGokYildirim/PAM.

Figures & tables

Figure 1: PAM is a simple yet powerful bridge that challenges the progress in FM–based CIL. It achieves better accuracy with ResNets, which significantly reduces runtime and parameters.

Figure 2: PAM freezes the first three layers of a pre-trained ResNet to preserve general knowledge while dynamically adding a task-specific last layer for each new task. To improve parameter efficiency, each last layer is structurally pruned to become ‘slim’ before training on its corresponding task. After training, the weights are frozen to prevent forgetting.

Method	CIFAR B0 Inc5		CUB B0 Inc10		IN-R B0 Inc20		Cars B0 Inc10
Method	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$
Finetune	60.65 ± 5.6	48.12 ± 3.3	55.78 ± 2.8	33.13 ± 3.3	59.09 ± 3.7	49.46 ± 3.3	41.90 ± 1.0	19.47 ± 2.7
SimpleCIL	86.48 ± 0.8	81.28 ± 0.1	91.58 ± 1.3	86.73 ± 0.1	61.31 ± 0.4	54.55 ± 0.1	54.95 ± 0.8	35.43 ± 0.0
L2P	84.90 ± 1.2	80.06 ± 1.4	73.22 ± 1.8	61.55 ± 1.7	75.92 ± 0.7	70.88 ± 0.7	42.06 ± 2.0	30.07 ± 0.8
DualPrompt	85.61 ± 1.3	79.92 ± 0.4	81.36 ± 1.8	70.51 ± 1.1	71.48 ± 0.5	66.09 ± 1.3	45.30 ± 1.1	30.15 ± 0.9
CODA-Prompt	87.64 ± 0.4	81.46 ± 0.3	77.65 ± 1.0	68.44 ± 1.0	76.25 ± 0.3	71.39 ± 0.3	36.22 ± 0.6	25.44 ± 0.3

Table 1: Average and final accuracy [%] with methods using ViT-B/16-IN21K and PAM using ResNets .

Figure 3: Parameter size vs. accuracy: The left panel shows that PAM challenges existing and future FM-based methods; and the right panel presents the parameter count for different methods after completing all sessions.

Panel of Figure 4 — Figure 4: Ablations of different components for the PAM method. (a) Effect of pruning timing on the performance. (b) Impact of different sparsity levels on performance. (c) Comparison of task-specific adaptation module selection strategies during inference.

Panel of Figure 5 — Figure 5: Analysis of PAM across different evaluation settings. (a) Final incremental accuracy after all tasks, (b) per-task training accuracy illustrating the effect of initialization strategies for pruned adaptation modules, and (c) performance of weighted ensemble strategies compared to the single best module. Ratios show the relative contribution of the most confident module and the remaining modules (e.g., 0.9/0.1).

Panel of Figure 6 — Figure 6: PAM achieves noticeably more stable accuracy throughout long-horizon experiments with challenging ImageNet-R (IN-R) dataset, reflecting its robustness against performance degradation as tasks accumulate.

Panel of Figure 7 — Figure 7: Correct module alignment of the PAM: (a) The confusion matrix shows that PAM reliably selects the correct adaptation module. (b) Comparing CIL and TIL results illustrates that providing true task identifiers yields a marginal improvement, indicating that PAM’s performance is already close to the TIL upper bound.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure A: Analysis of the effect of threshold values on the adaptive initialization of pruned adaptation modules. As the threshold increases from (a) $0.70$ to (d) $0.77$ , the model reuses existing modules more frequently, thereby reducing the total number of newly instantiated modules.

Panel of Figure 10 — Figure B: Incremental accuracy flow of each method over sequential tasks where comparison methods utilize ViT-B/16 and PAM employs ResNet152. PAM maintains a more stable SOTA performance across tasks.

Storage-Efficient Continual Learning

See all 11 papers

Parameter Importance-Driven Continual Learning for Foundation Models
Nov 19, 2025Lingxiang Wang, Hainan Zhang, Zhiming ZhengContinual Learning for LLMs Parameter-Efficient Fine-Tuning
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · School of Artificial Intelligence, Beihang University
One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning
Aug 31, 2026Yunxiang Fu, Meng Lou, Yizhou YuClass-Incremental Learning Incremental Learning
School of Computing and Data Science The University of Hong Kong
Unifying Detection and Adaptation in Task-Free Continual Learning
Aug 27, 2026Dezheng Han, Anbang Zhang, Zhihao Zhu +1Task-Free Continual Learning Continual Learning
School of Control Science and Engineering, Shandong University, Jinan, China · Department of ECE, The Hong Kong University of Science and Technology, Hong Kong · National Medicine-Engineering Interdisciplinary Industry-Education Integration Innovation Platform, Shandong University, Jinan, China +1

Nov 19, 2025J/K move · Enter open · S save

Parameter Importance-Driven Continual Learning for Foundation Models

Lingxiang Wang, Hainan Zhang, Zhiming Zheng

Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · School of Artificial Intelligence, Beihang University

Continual Learning for LLMs Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Domain-specific post-training often causes catastrophic forgetting, making foundation models lose their general reasoning ability and limiting their adaptability to dynamic real-world environments. Preserving general capabilities while acquiring downstream domain knowledge is a central challenge for large language and multimodal models. Traditional continual learning methods, such as regularization, replay and architectural isolation, suffer from poor downstream performance, reliance on inaccessible historical data, or additional parameter overhead. While recent parameter-efficient tuning (PET) methods can alleviate forgetting, their effectiveness strongly depends on the choice of parameters and update strategies. In this paper, we introduce PIECE, a Parameter Importance Estimation-based Continual Enhancement method that preserves general ability while efficiently learning domain knowledge without accessing prior training data or increasing model parameters. PIECE selectively updates only 0.1% of core parameters most relevant to new tasks, guided by two importance estimators: PIECE-F based on Fisher Information, and PIECE-S based on a second-order normalization that combines gradient and curvature information. Experiments across three language models and two multimodal models show that PIECE maintains general capabilities and achieves state-of-the-art continual learning performance across diverse downstream tasks. Our results highlight a practical path to scalable, domain-adaptive foundation models without catastrophic forgetting.

Abstract

Storage-Efficient Continual Learning

Explore similar work

Parameter Importance-Driven Continual Learning for Foundation Models

One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning

Unifying Detection and Adaptation in Task-Free Continual Learning