Parameter Efficient Continual Learning for Sparse Event-Based Transformers

Vaishnavi Nagabhushana; Kartikay Agrawal; Ayon Borthakur

cs.CVAug 27, 2026

Parameter Efficient Continual Learning for Sparse Event-Based Transformers

Authors: Vaishnavi Nagabhushana, Kartikay Agrawal, Ayon Borthakur

Organizations: SustainAI Lab, MFSDS&AI IIT Guwahati

Abstract

Robotic and edge intelligence systems operate in dynamic environments where data arrives continuously, requiring models to adapt while preserving previously learned knowledge under strict memory and energy constraints. While parameter-efficient fine-tuning has shown promise for continual learning with vision transformers, conventional architectures rely on dense computation and remain costly for real-world deployment. Sparse event-based vision transformers provide energy-efficient event-driven computation, yet their continual learning capabilities remain largely unexplored. We here introduce sLoTh, a parameter-efficient continual learning framework for pretrained sparse event-based (spiking) vision transformers. sLoTh freezes the backbone and restricts plasticity to scalable-efficient low-rank attention updates (seLoRA) and shared neuronal threshold modulation, enabling adaptation without replay buffers by updating less than 1% of model parameters. Experiments across CIFAR-100, Tiny-ImageNet, ImageNet-100, and ImageNet-R with up to 100 tasks demonstrate competitive rehearsal-free performance in class-incremental learning and online continual learning, while enabling approximately 6.5x lower energy consumption than conventional dense vision transformers.

Figures & tables

Figure 1: Comparison of Continual Learning Paradigms. In Task-Incremental Learning ( TIL ), task boundaries are known , allowing for task-specific components during evaluation. Class-Incremental Learning ( CIL ) increases difficulty by requiring the model to distinguish between all seen classes without task IDs at inference. Online Continual Learning ( OCL ) is a challenging scenario , as it combines the expanding output space of CIL with the constraint that data is streamed only once.

Methods	T=10	T=20	T=50	Total Param(M)	$\#$ updates(M)
L2P Wang et al. (2022b)	83.18 $\pm$ 1.20	79.51 $\pm$ 0.67	67.95 $\pm$ 2.12	172	0.12
DualPrompt Wang et al. (2022a)	81.48 $\pm$ 0.86	80.44 $\pm$ 1.38	72.5 $\pm$ 1.08	172	0.86
CODA-Prompt Smith et al. (2023)	86.31 $\pm$ 0.12	81.36 $\pm$ 0.88	73.77 $\pm$ 0.98	172	4.6
InfLoRA Liang and Li (2024)	86.75 $\pm$ 0.35	80.97 $\pm$ 0.74	70.68 $\pm$ 1.26	172	0.51
SD-LoRA Wu et al. (2025)	88.01 $\pm$ 0.31	OOM	OOM	172	0.39
Full sLoTh	81.23 $\pm$ 1.74	84.65 $\pm$ 1.46	87.22 $\pm$ 1.59	64.32	0.44

Table 1: CIFAR100 CIL comparison between pretrained ViT with PEFT Continual Learning methods v/s pretrained QKFormer on Full sLoTh for different task granularity, i.e.,

T

represents the number of tasks. Parameters are in millions. First and Second best models are highlighted.

Method	Imagenet-R ( $T=5$ )	Imagenet-R ( $T=10$ )	Imagenet-R ( $T=20$ )
Full Fine tuning	64.92 $\pm$ 0.87	60.57 $\pm$ 1.06	49.95 $\pm$ 1.31
L2P Wang et al. (2022b)	73.04 $\pm$ 0.71	71.26 $\pm$ 0.44	68.97 $\pm$ 0.51
DualPrompt Wang et al. (2022a)	69.99 $\pm$ 0.57	68.22 $\pm$ 0.20	65.23 $\pm$ 0.45
CODA-Prompt Smith et al. (2023)	76.63 $\pm$ 0.27	74.05 $\pm$ 0.41	69.38 $\pm$ 0.33
InfLoRA Liang and Li (2024)	76.95 $\pm$ 0.23	74.75 $\pm$ 0.64	69.89 $\pm$ 0.56
SD-LoRA Wu et al. (2025)	79.15 $\pm$ 0.20	77.34 $\pm$ 0.35	75.26 $\pm$ 0.37

Table 2: CIL under domain shift on ImageNet-R.

T

denotes the number of task splits. Accuracy is reported as mean

\pm

std over 3 runs. First and Second best models are highlighted.

	CIFAR100 (T=10)		Tiny-ImageNet (T=100)
Method	Avg Acc	Forgetting	Avg Acc	Forgetting
ER Rolnick et al. (2019)	39.41 $\pm$ 1.81	13.29	24.71 $\pm$ 2.52	40.77
OCM Guo et al. (2022)	42.22 $\pm$ 1.06	3.76	31.94 $\pm$ 1.19	15.92
OnPro Wei et al. (2023)	41.59 $\pm$ 1.38	6.72	26.38 $\pm$ 2.18	20.32
OCM-CCLDC Wang et al. (2024)	51.43 $\pm$ 1.37	3.99	39.25 $\pm$ 0.88	15.56
OnPro-CCLDC Wang et al. (2024)	50.01 $\pm$ 0.85	10.55	38.18 $\pm$ 1.02	16.17

Table 3: Online Continual Learning comparison under single-pass streaming. Replay-based results are taken from S6MOD under the largest buffer sizes (CIFAR100: M=5k, Tiny-ImageNet: M=10k). sLoTh operates without any replay buffer has only per-class mean prototype. Accuracy is reported as mean

\pm

std over 3 runs. First and Second best models are highlighted.

Figure 2: Performance gap between boundary-aware OCL v/s boundary-free OCL setting on Tiny-Imagenet. Here, sLoTh narrows the gap between these two settings compared to other methods without relying on replay.

Methods	Avg Acc ( $\uparrow$ )	Forgetting ( $\downarrow$ )	# Updates	Energy
AGEM Chaudhry et al. (2018b)	12.67 $\pm$ 1.87	82.51 $\pm$ 2.27	Full	254.84
ER Rolnick et al. (2019)	44.85 $\pm$ 1.83	44.67 $\pm$ 4.29	Full	254.84
EWC++ Chaudhry et al. (2018a)	10.61 $\pm$ 0.74	84.10 $\pm$ 1.11	Full	254.84
MIR Aljundi et al. (2019a)	48.36 $\pm$ 3.11	43.41 $\pm$ 1.02	Full	254.84
PCR Lin et al. (2023)	48.48 $\pm$ 0.15	46.23 $\pm$ 1.29	Full	254.84
LODE (DER++) Liang and Li (2023)	44.29 $\pm$ 1.48	45.54 $\pm$ 3.32	Full	254.84

Table 4: Comparison of OCL on Split-CIFAR-100 with 10 tasks with ViT-B/16 backbone(86 Million parameters) Dosovitskiy et al. (2020) v/s QKFormer(64 Million parameters) Zhou et al. (2024a) as backbone for our method. We consider the closest comparable model sizes for a fair comparison, based on parameter count in millions and energy consumption in mJ. First and Second best models are highlighted.

	ViT			QKFormer
Method	L2P	Online LoRA	seLoRA	Online LoRA	Full sLoTh	Threshold only	seLoRA only
Accuracy	56.25 $\pm$ 0.45	60.38 $\pm$ 2.59	55.68 $\pm$ 0.97	61.87 $\pm$ 1.63	62.14 $\pm$ 1.30	66.91 $\pm$ 1.56	70.00 $\pm$ 3.85

Table 5: Comparison of SOTA methods on ImageNet-100 with 50 tasks for OCL.

Variant	CIL Avg Acc	OCL Avg Acc	param updates
seLoRA only (w/o Threshold)	72.63 $\pm$ 1.24	63.88 $\pm$ 1.03	0.402
Threshold only (w/o seLoRA)	69.39 $\pm$ 1.01	60.25 $\pm$ 0.98	$\mathbf{0.037_{(\ 10x\downarrow)}}$
Full sLoTh (seLoRA + Threshold)	81.23 $\pm$ 1.74	55.12 $\pm$ 1.18	0.439

Table 6: Component interaction under CIL and OCL (CIFAR-100, 10 tasks).

Components	Non-zeros	$\lambda_{kl}=0$	$\lambda_{\text{local}}=0$	$\lambda_{\text{global}}=0$
Accuracy	55.12 $\pm$ 1.18	10.23 $\pm$ 1.10	21.64 $\pm$ 2.38	20.11 $\pm$ 1.09

Table 7: Component analysis of Loss function on CIFAR-100 dataset in OCL setting

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	CIFAR-100	Imagenet-R	Imagenet-100	Tiny-IM
Batch size	128	64	64	128
Time steps (T)	4	4	4	4
Learning rate	1e-4	3e-4	3e-4	1e-4
Epochs	50	30	30	30
Optimizer	AdamW	AdamW	AdamW	AdamW

Appendix

Table 8: Hyperparameters used across datasets and continual learning settings (OCL, CIL, and TIL).

Method	Low-Rank Adaptation Hu et al. (2022)	Adapters Houlsby et al. (2019)	Spiking Adapters	Thresholds
Accuracy	80.89	87.21	79.82	70.56

Appendix

Table 9: PEFT feasibility on pretrained QKFormer evaluated on the CIFAR-100.

Tasks	Tiny-ImageNet	CIFAR-100	ImageNet-R	Imagenet-100
10	85.23 $\pm$ 2.94	87.38 $\pm$ 1.63	86.77 $\pm$ 1.08	89.33 $\pm$ 1.78
20	87.17 $\pm$ 1.38	86.84 $\pm$ 2.54	88.45 $\pm$ 2.79	90.92 $\pm$ 1.63
50	96.50 $\pm$ 1.25	91.20 $\pm$ 2.71	93.61 $\pm$ 1.81	94.5 $\pm$ 1.95

Appendix

Table 10: Task-Incremental Learning results for sLoTh across different datasets. Accuracy reported as mean

\pm

std over 3 runs.

Methods	T=10	T=20	T=50	$\#$ Total Param.	$\#$ updates
L2P Wang et al. (2022b)	83.18 $\pm$ 1.20	79.51 $\pm$ 0.67	67.95 $\pm$ 2.12	172	0.12
DualPrompt Wang et al. (2022a)	81.48 $\pm$ 0.86	80.44 $\pm$ 1.38	72.5 $\pm$ 1.08	172	0.86
CODA-Prompt Smith et al. (2023)	86.31 $\pm$ 0.12	81.36 $\pm$ 0.88	73.77 $\pm$ 0.98	172	4.6
InfLoRA Liang and Li (2024)	86.75 $\pm$ 0.35	80.97 $\pm$ 0.74	70.68 $\pm$ 1.26	172	0.51
SD-LoRA Wu et al. (2025)	88.01 $\pm$ 0.31	OOM	OOM	172	0.39
Full sLoTh	81.23 $\pm$ 1.74	84.65 $\pm$ 1.46	87.22 $\pm$ 1.59	64.32	0.44

Appendix

Table 11: CIFAR100 CIL comparison between pretrained ViT with PEFT Continual Learning methods v/s pretrained QKFormer & SpikingFormer on Full sLoTh for different task granularity, i.e.,

T

represents the number of tasks. Parameters are in millions. First and Second best models are highlighted.

Online Continual Learning

Online continual learning adapts models to sequential data streams while preserving useful knowledge and limiting forgetting.

See all 106 papers

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree
Jun 12, 2025Yu-Yang Qian, Yuan-Ze Xu, Zhen-Yu Zhang +2Continual Learning for LLMs Low-Rank Adaptation
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · RIKEN AIP, Tokyo, Japan
CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework
Sep 15, 2026Yunxiang Fu, Meng Lou, Zicheng Liao +1Class-Incremental Learning Continual Learning
School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center
Energy-Structured Low-Rank Adaptation for Continual Learning
May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual Learning Low-Rank Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Jun 12, 2025J/K move · Enter open · S save

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree

Yu-Yang Qian, Yuan-Ze Xu, Zhen-Yu Zhang, Peng Zhao+1

National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · RIKEN AIP, Tokyo, Japan

Continual Learning for LLMs Low-Rank Adaptation Online Continual Learning Incremental Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Many real-world applications collect data in a streaming environment, where learning tasks are encountered sequentially. This necessitates continual learning (CL) to update models online, enabling adaptation to new tasks while preserving past knowledge to prevent catastrophic forgetting. Nowadays, with the flourish of large pre-trained models (LPMs), efficiency has become increasingly critical for CL, due to their substantial computational demands and growing parameter sizes. In this paper, we introduce TreeLoRA (K-D Tree of Low-Rank Adapters), a novel approach that constructs layer-wise adapters by leveraging hierarchical gradient similarity to enable efficient CL, particularly for LPMs. To reduce the computational burden of task similarity estimation, we employ bandit techniques to develop an algorithm based on lower confidence bounds to efficiently explore the task structure. Furthermore, we use sparse gradient updates to facilitate parameter optimization, making the approach better suited for LPMs. Theoretical analysis is provided to justify the rationale behind our approach, and experiments on both vision transformers (ViTs) and large language models (LLMs) demonstrate the effectiveness and efficiency of our approach across various domains, including vision and natural language processing tasks.

Abstract

Online Continual Learning

Explore similar work

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree

CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

Energy-Structured Low-Rank Adaptation for Continual Learning