LSA-PAST: Layer-Specific Adaptation and Prototype Transport for Compositional Incremental Learning

Kyle Stein; Andrew Arash Mahyari; Guillermo Francia; Eman El-Sheikh; Jiangpeng He

cs.AIOct 19, 2026

LSA-PAST: Layer-Specific Adaptation and Prototype Transport for Compositional Incremental Learning

Authors: Kyle Stein, Andrew Arash Mahyari, Guillermo Francia, Eman El-Sheikh, Jiangpeng He

Organizations: University of West Florida, Pensacola, FL, USA · Florida Institute for Human and Machine Cognition, Pensacola, FL, USA · Indiana Univeristy, Bloomington, IN, USA

Abstract

Class-Incremental Learning (CIL) enables models to recognize new classes over time while retaining prior knowledge, but many real-world problems are compositional, where visual concepts arise from combinations of primitives such as states (i.e., red, shiny) and objects (i.e., shoe, car). Standard CIL methods treat each composition as an independent class, ignoring shared primitives and providing no mechanism to stabilize their representations as new compositions arrive. This causes parameter updates to entangle primitives and induce drift that degrades performance on earlier compositions. Compositional Incremental Learning (composition-IL) formalizes this setting by introducing new state-object compositions across tasks. In this paper, we propose Layer-Specific Adaptation with Prototype-Anchored Adapter-Space Transport (LSA-PAST), a parameter-efficient method tailored to composition-IL. LSA integrates Low-Rank Adaptation (LoRA) into a frozen ViT with depth-specialized adapters for states, objects, and compositions. PAST then maps past, anchored prototypes into the current adapter space through a rehearsal-free affine transform fit on current-task data, enabling old prototypes to remain compatible with new adapters. Extensive experiments demonstrate that LSA-PAST achieves state-of-the-art (SOA) accuracy-forgetting trade-off performance for composition-IL.

Figures & tables

Figure 1: Accuracy–forgetting trade-off on composition-IL benchmarks for adapter- and prompt-based methods.

Figure 2: (A) We adapt a frozen ViT with depth-specialized LoRA feeding three heads. (B) Post-training, empirical prototypes are blended with primitive directions (PGS) and stored. (C) Subsequent tasks instantiate fresh adapters for new compositions. (D) Using current-task images as transient correspondences, AST fits an affine map $\phi_{t^{\prime}\rightarrow t}$ to transport past prototypes into the current adapter space without rehearsal. (E) Inference ensembles cosine similarities across all adapter spaces, recognizing both new and previously learned compositions.

Method type	Method	%Params	Split-Clothing		Split-UT-Zappos				Split-C-GQA
			5 tasks		5 tasks		10 tasks		5 tasks
			Avg Acc % ( $\uparrow$ )	FTT % ( $\downarrow$ )	Avg Acc % ( $\uparrow$ )	FTT % ( $\downarrow$ )	Avg Acc % ( $\uparrow$ )	FTT % ( $\downarrow$ )	Avg Acc % ( $\uparrow$ )	FTT % ( $\downarrow$ )
Classic CIL	EWC [ 20 ]	100	$47.89\text{\scriptsize$ \pm 0.87 $}$	$52.75\text{\scriptsize$ \pm 0.44 $}$	$37.59\text{\scriptsize$ \pm 2.06 $}$	$55.70\text{\scriptsize$ \pm 2.76 $}$	$24.63\text{\scriptsize$ \pm 0.94 $}$	$61.31\text{\scriptsize$ \pm 2.29 $}$	$38.69\text{\scriptsize$ \pm 1.39 $}$	$38.98\text{\scriptsize$ \pm 1.53 $}$
	LwF [ 28 ]	100	$49.96\text{\scriptsize$ \pm 0.68 $}$	$44.22\text{\scriptsize$ \pm 0.53 $}$	$40.15\text{\scriptsize$ \pm 0.43 $}$	$49.61\text{\scriptsize$ \pm 0.68 $}$	$30.38\text{\scriptsize$ \pm 1.41 $}$	$58.15\text{\scriptsize$ \pm 0.20 $}$	$45.95\text{\scriptsize$ \pm 0.72 $}$	$23.21\text{\scriptsize$ \pm 1.05 $}$
	iCaRL [ 39 ]	100	$68.65\text{\scriptsize$ \pm 0.41 $}$	$31.74\text{\scriptsize$ \pm 1.89 $}$	$37.78\text{\scriptsize$ \pm 2.14 $}$	$55.06\text{\scriptsize$ \pm 3.50 $}$	$31.40\text{\scriptsize$ \pm 1.96 $}$	$59.65\text{\scriptsize$ \pm 2.40 $}$	$33.40\text{\scriptsize$ \pm 0.40 $}$	$53.32\text{\scriptsize$ \pm 0.90 $}$
Prompt-based	L2P [ 45 ]	0.2	$83.16\text{\scriptsize$ \pm 0.50 $}$	$10.97\text{\scriptsize$ \pm 2.02 $}$	$35.26\text{\scriptsize$ \pm 0.81 $}$	$28.17\text{\scriptsize$ \pm 0.05 $}$	$31.65\text{\scriptsize$ \pm 0.16 $}$	$31.02\text{\scriptsize$ \pm 1.62 $}$	$39.39\text{\scriptsize$ \pm 0.24 $}$	$27.27\text{\scriptsize$ \pm 1.67 $}$

Table 1: Average accuracy and forgetting on Split-Clothing, Split-UT-Zappos under 5-task and 10-task, and Split-C-GQA. %Params denotes cumulative additional parameters relative to the ViT-B/16 backbone. Best and Second-Best results highlighted.

Type	Method	Split-Clothing		Split-C-GQA
Type	Method	State %	Object %	State %	Object %
Classic	EWC [ 20 ]	$86.49\text{\scriptsize$ \pm 0.97 $}$	$52.72\text{\scriptsize$ \pm 1.30 $}$	$52.91\text{\scriptsize$ \pm 0.28 $}$	$60.42\text{\scriptsize$ \pm 1.91 $}$
	LwF [ 28 ]	$87.11\text{\scriptsize$ \pm 0.66 $}$	$54.57\text{\scriptsize$ \pm 0.69 $}$	$\mathbf{58.33}\text{\scriptsize$ \pm 0.42 $}$	$66.32\text{\scriptsize$ \pm 0.31 $}$
	iCaRL [ 39 ]	$91.21\text{\scriptsize$ \pm 1.05 $}$	$71.70\text{\scriptsize$ \pm 0.99 $}$	$46.96\text{\scriptsize$ \pm 0.27 $}$	$55.61\text{\scriptsize$ \pm 0.45 $}$
Prompt	L2P [ 45 ]	$85.41\text{\scriptsize$ \pm 0.23 $}$	$95.42\text{\scriptsize$ \pm 0.42 $}$	$51.65\text{\scriptsize$ \pm 1.54 $}$	$63.53\text{\scriptsize$ \pm 1.04 $}$
	Dual-Prompt [ 44 ]	$86.89\text{\scriptsize$ \pm 0.37 $}$	$95.87\text{\scriptsize$ \pm 0.53 $}$	$49.52\text{\scriptsize$ \pm 0.69 $}$	$64.15\text{\scriptsize$ \pm 0.37 $}$

Table 2: State and Object Accuracy (%) on Split-Clothing and Split-C-GQA (5 tasks). Best and Second-Best results highlighted.

			Split-Clothing		Split-UT-Zappos
LSA	PGS	Inference	Avg Acc % ( $\uparrow$ )	FTT % ( $\downarrow$ )	Avg Acc % ( $\uparrow$ )	FTT % ( $\downarrow$ )
✓	✓	Unaligned Ens.	$75.71\text{\scriptsize$ \pm 0.39 $}$	$28.32\text{\scriptsize$ \pm 0.53 $}$	$29.92\text{\scriptsize$ \pm 0.37 $}$	$53.21\text{\scriptsize$ \pm 1.51 $}$
✓	✗	Native	$90.29\text{\scriptsize$ \pm 0.65 $}$	$4.97\text{\scriptsize$ \pm 0.48 $}$	$40.85\text{\scriptsize$ \pm 0.93 $}$	$17.56\text{\scriptsize$ \pm 1.71 $}$
✓	✓	Native	$90.74\text{\scriptsize$ \pm 0.65 $}$	$3.91\text{\scriptsize$ \pm 0.37 $}$	$41.67\text{\scriptsize$ \pm 0.57 $}$	$14.81\text{\scriptsize$ \pm 1.85 $}$
✓	✗	AST Ens. (ours)	$90.81\text{\scriptsize$ \pm 0.27 $}$	$4.89\text{\scriptsize$ \pm 0.87 $}$	$44.58\text{\scriptsize$ \pm 1.56 $}$	$18.63\text{\scriptsize$ \pm 3.17 $}$
✓	✓	LDC Ens. [ 11 ]	$91.41\text{\scriptsize$ \pm 0.40 $}$	$3.49\text{\scriptsize$ \pm 0.73 $}$	$43.73\text{\scriptsize$ \pm 0.77 $}$	$15.09\text{\scriptsize$ \pm 1.74 $}$

Table 3: Component and inference ablation. Native scores each class only in its originating adapter space, while ensemble (Ens.) variants additionally aggregate available cross-space evidence.

Figure 3: PGS success and failure cases on Split-UT-Zappos and Split-C-GQA. Boxes show $\Delta$ forgetting, with green indicating reduced forgetting and red indicating no improvement.

	Split-Clothing				Split-C-GQA
Task Gap	Uncorrected	LDC [ 11 ]	DPCR [ 17 ]	AST	Uncorrected	LDC [ 11 ]	DPCR [ 17 ]	AST
1	$0.740$	$\mathbf{0.898}$	$0.863$	$0.880$	$0.784$	$0.942$	$\mathbf{0.945}$	$0.941$
2	$0.750$	$0.873$	$0.830$	$\mathbf{0.897}$	$0.786$	$0.928$	$0.934$	$\mathbf{0.945}$
3	$0.704$	$0.844$	$0.812$	$\mathbf{0.886}$	$0.798$	$0.910$	$0.920$	$\mathbf{0.948}$
4	$0.662$	$0.819$	$0.799$	$\mathbf{0.905}$	$0.774$	$0.881$	$0.894$	$\mathbf{0.952}$
Overall	$0.728$	$0.872$	$0.836$	$\mathbf{0.889}$	$0.786$	$0.925$	$0.932$	$\mathbf{0.944}$

Table 4: Cosine similarity to oracle current-space prototypes.

Panel of Figure 4 — Figure 4: Top: effect of AST transport-pair budget on accuracy and forgetting. Bottom: LSA ablation with explicit state (S), object (O), and composition (C) layer ranges.

Incremental Learning

Incremental learning studies how models update from sequential data while retaining earlier knowledge as classes, tasks, or domains change.

See all 146 papers

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning
Nov 14, 2025Lan Li, Tao Hu, Da-Wei Zhou +3Class-Incremental Learning VLM Adaptation
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning
Sep 30, 2026Hongwei Zhao, Rui Liu, Yansong LiuClass-Incremental Learning Prototype-Based Classification
School of Computer Science and Engineering, Beihang University
Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning
May 28, 2026Xiang Tan, Run He, Yawen Cui +6Class-Incremental Learning Incremental Learning
South China University of Technology · Hong Kong Polytechnic University · Agency for Science, Technology and Research +3

Nov 14, 2025J/K move · Enter open · S save

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

Lan Li, Tao Hu, Da-Wei Zhou, Jia-Qi Yang+2

School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China

Class-Incremental Learning VLM Adaptation Incremental Learning

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge. Vision-language models such as CLIP offer strong transferable representations via multi-modal supervision, making them promising for CIL. However, applying CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities. To address these issues, we propose BOFA (Bridge-layer Orthogonal Fusion for Adaptation), a novel framework for CIL. BOFA confines all model adaptation exclusively to CLIP's existing cross-modal bridge-layer, thereby adding no extra parameters or inference cost. To prevent forgetting within this layer, it leverages Orthogonal Low-Rank Fusion, a mechanism that constrains parameter updates to a low-rank ``safe subspace" mathematically constructed to be orthogonal to past task features. This ensures stable knowledge accumulation without data replay. Furthermore, BOFA employs a cross-modal hybrid prototype that synergizes stable textual prototypes with visual counterparts derived from our stably adapted bridge-layer, enhancing classification performance. Extensive experiments on standard benchmarks show that BOFA achieves superior accuracy and efficiency compared to existing methods.

Abstract

Incremental Learning

Explore similar work

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning