Critical Patch-Aware Sparse Prompting with Decoupled Training for Continual Learning on the Edge

Wonseon Lim; Jaesung Lee; Dae-Won Kim

cs.LGApr 8, 2026

Critical Patch-Aware Sparse Prompting with Decoupled Training for Continual Learning on the Edge

Authors: Wonseon Lim, Jaesung Lee, Dae-Won Kim

Organizations: School of Computer Science and Engineering, Chung-Ang University · Department of Artificial Intelligence, Chung-Ang University

Abstract

Continual learning (CL) on edge devices requires not only high accuracy but also training-time efficiency to support on-device adaptation under strict memory and computational constraints. While prompt-based continual learning (PCL) is parameter-efficient and achieves competitive accuracy, prior work has focused mainly on accuracy or inference-time performance, often overlooking the memory and computational costs of on-device training. In this paper, we propose CPS-Prompt, a critical patch-aware sparse prompting framework that explicitly targets training-time memory usage and computational cost by integrating critical patch sampling (CPS) for task-aware token reduction and decoupled prompt and classifier training (DPCT) to reduce backpropagation overhead. Experiments on three public benchmarks and real edge hardware show that CPS-Prompt improves peak memory, training time, and energy efficiency by about 1.6x over the balanced CODA-Prompt baseline, while maintaining accuracy within 2% of the state-of-the-art C-Prompt on average and remaining competitive with CODA-Prompt in accuracy. The code is available at https://github.com/laymond1/cps-prompt.

Figures & tables

Figure 1: Comparison of accuracy and training-time efficiency on CIFAR-100 using PCL and token reduction methods, with efficiency metrics measured on a Jetson Orin Nano. Our method exhibits a more balanced accuracy–efficiency trade-off under edge-device constraints.

Figure 2: Overview of the CPS-Prompt framework. Left : CPS selects a small subset of task-relevant patches to preserve accuracy while reducing memory usage. Right : DPCT mitigates representation mismatch through decoupled training, where the prompt is optimized with sparse patches and the classifier with full patches.

Method	CIFAR-100		ImageNet-R		CUB-200
Method	$\boldsymbol{ACC}_{\text{T}}$ ( $\uparrow$ )	$\boldsymbol{FGT}_{\text{T}}$ ( $\downarrow$ )	$\boldsymbol{ACC}_{\text{T}}$ ( $\uparrow$ )	$\boldsymbol{FGT}_{\text{T}}$ ( $\downarrow$ )	$\boldsymbol{ACC}_{\text{T}}$ ( $\uparrow$ )	$\boldsymbol{FGT}_{\text{T}}$ ( $\downarrow$ )
Upper-Bound	83.73 $\pm$ 0.17	-	63.04 $\pm$ 0.63	-	75.38 $\pm$ 1.45	-
SGD	0 9.91 $\pm$ 0.29 $\blacktriangledown$	94.70 $\pm$ 0.50 $\blacktriangledown$	0 7.50 $\pm$ 0.09 $\blacktriangledown$	76.56 $\pm$ 0.37 $\blacktriangledown$	0 8.72 $\pm$ 0.34 $\blacktriangledown$	80.41 $\pm$ 0.54 $\blacktriangledown$
LwF [ 13 ]	10.98 $\pm$ 0.32 $\blacktriangledown$	93.29 $\pm$ 0.38 $\blacktriangledown$	0 7.64 $\pm$ 0.23 $\blacktriangledown$	74.20 $\pm$ 0.42 $\blacktriangledown$	10.26 $\pm$ 0.54 $\blacktriangledown$	80.00 $\pm$ 0.45 $\blacktriangledown$
ER [ 2 ]	38.83 $\pm$ 3.04 $\blacktriangledown$	62.51 $\pm$ 3.37 $\blacktriangledown$	19.57 $\pm$ 0.52 $\blacktriangledown$	67.96 $\pm$ 0.69 $\blacktriangledown$	46.29 $\pm$ 1.35 $\blacktriangledown$	33.14 $\pm$ 1.08 $\blacktriangledown$
L2P [ 29 ]	62.96 $\pm$ 0.98 $\blacktriangledown$	15.93 $\pm$ 1.67 $\blacktriangledown$	45.08 $\pm$ 0.45 $\blacktriangledown$	0 8.28 $\pm$ 0.92 $\vartriangle$	49.79 $\pm$ 1.14 $\blacktriangledown$	11.68 $\pm$ 1.13

Table 1: Comparison of CL methods on three datasets.

\blacktriangledown

\vartriangle

indicate that the corresponding method is statistically worse/better than the proposed method, based on a paired

t

-test (

p<0.05

). Higher is better for Accuracy (

\uparrow

), while lower is better for Forgetting (

\downarrow

). All results are averaged over 10 runs.

Figure 3: Comparison of memory usage, training time, and energy consumption between our method and other PCL methods on three datasets on the Jetson Orin Nano. For clarity, method names are abbreviated (e.g., CODA = CODA-Prompt, CP = C-Prompt, OS++ = OS-Prompt++, OS = OS-Prompt).

Figure 4: Comparison of accuracy and memory usage between CPS-Prompt and other token reduction methods based on the CODA-Prompt baseline under varying reduction ratios on the Jetson Orin Nano.

Figure 5: Comparison of training time and memory usage between CPS-Prompt and other token reduction methods based on the CODA-Prompt baseline under varying reduction ratios on the Jetson Orin Nano.

Modules	$\boldsymbol{ACC}_{N}$ $(\uparrow)$	Memory ( $\downarrow$ )	Train Time ( $\downarrow$ )
CODA-Prompt	50.24	440 MB	1,788 s/task
w/ PD	45.32	253 MB	1,388 s/task
w/ CPS	47.16	253 MB	1,389 s/task
w/ PD + DPCT	47.96	253 MB	1,126 s/task
w/ CPS + DPCT	49.28	253 MB	1,126 s/task

Table 2: Ablation results on ImageNet-R at a reduction ratio of 0.5. We compare combinations of CPS and DPCT, using PD as the baseline for CPS.

Figure 6: Effect of temperature and phase ratio on CUB-200. Accuracy is averaged over reduction ratios of 0.2, 0.4, 0.6, and 0.8.

Figure 7: Comparison between stochastic and deterministic sampling on CUB-200. Accuracy is averaged over reduction ratios of 0.2, 0.4, 0.6, and 0.8.

Figure 8: Qualitative comparison between deterministic top- $k$ and stochastic sampling at a reduction ratio of 0.5 on CUB-200.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Information-Theoretic Decoupled Prompt Tuning for Continual Learning
Aug 13, 2026Yunfei Zhang, Wen Wen, Tieliang Gong +1Continual Learning for VLMs Continual Learning for LLMs
National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
Leveraging ECRAM for Edge Continual Learning
Jul 22, 2026Nabila Tasnim, Haoran Liu, Qing Cao +1Compute-in-Memory Continual Learning
University of Illinois Urbana-Champaign
Online Continual Learning on Intel Loihi 2 via a Co-designed Spiking Neural Network
Nov 3, 2025Elvin Hajizada, Danielle Rager, Timothy Shea +5Online Continual Learning Incremental Learning
Institute of Informatics, University of Munich (LMU), Munich, Germany · Intel Labs, Intel Corporation, Santa Clara, CA, USA · Southwest Research Institute, San Antonio, TX, USA +3

Aug 13, 2026J/K move · Enter open · S save

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Yunfei Zhang, Wen Wen, Tieliang Gong, Weizhan Zhang

National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China

Continual Learning for VLMs Continual Learning for LLMs VLM Adaptation

Open paper AskPDF arXiv

Continual learning (CL) aims to incrementally acquire knowledge from sequential data while avoiding catastrophic forgetting. Recently, prompt tuning has attracted increasing attention as an efficient approach for adapting pre-trained models to CL tasks. However, existing prompt design paradigms commonly suffer from retrieval dependence and classifier bias, which make model adaptation sensitive to prompt selection and bias predictions toward newly arrived classes. To address these challenges, we propose Decoupled Prompt Tuning for Continual Learning (DPT4CL), which decouples the CLIP textual prompt into a task-shared prompt distribution and class-specific prompts. The task-shared prompt distribution is derived by optimizing an Information Bottleneck objective to facilitate cross-task knowledge transfer and alleviate classifier bias, while class-specific prompts enhance inter-class separability without relying on explicit prompt retrieval. Furthermore, we establish a unified excess risk bound from an information-theoretic perspective, providing theoretical support for the robust generalization and forgetting mitigation of the proposed framework. Extensive experiments on standard CL benchmarks demonstrate that DPT4CL achieves state-of-the-art performance. The source code is available at https://github.com/Cloudfly-Z/DPT4CL

Abstract

Continual Learning

Explore similar work

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Leveraging ECRAM for Edge Continual Learning

Online Continual Learning on Intel Loihi 2 via a Co-designed Spiking Neural Network