Organizations: Department of Computer Science University of Pisa · Department of AI, Data and Decision Sciences Luiss Guido Carli University · Institute of Information Science and Technologies “Alessandro Faedo” (ISTI) National Research Council of Italy (CNR)
Continual learning suffers from catastrophic forgetting, leading to the loss of past knowledge and unstable adaptation across task streams. Existing approaches typically address this issue through replay buffers, architectural changes, or additional regularization terms. We argue that forgetting is a consequence of unstable learning dynamics, which may benefit from an optimizer-level solution. The main challenge is that learning dynamics are time-dependent, as demonstrated by empirical phenomena such as the stability gap. First, we formally prove that, for distance-based classifiers, bounded parameter drift guarantees bounded margin erosion under a local score-stability condition. Based on this insight, we propose STABLE, a primal-dual optimizer with adaptive drift control. STABLE enforces a drift budget by increasing the dual variable when the budget is violated and decreasing it otherwise. This results in high stability without penalizing plasticity. Experiments on long-horizon unimodal streams and CLIP-based multimodal streams show that vanilla PEFT baselines trained with STABLE match or outperform specialized continual learning methods. In the multimodal setting, STABLE further preserves zero-shot transfer, matching or exceeding dedicated multimodal continual learning methods. Code is available at https://github.com/atlas-luiss/STABLE.
Table 2: CLIP multimodal results (10 tasks). 3-seed means; std in Appendix Table 10 . Bold : best, underlined : second. CLAP4CLIP uses replay; all other methods are replay-free.
Method
ImageNet-R
CIFAR-100
Food101
Pets
Flowers
DTD
Cars
Food101
Pets
Flowers
DTD
Cars
Pre-trained CLIP
88.38
88.01
66.03
44.31
64.42
88.38
88.01
66.03
44.31
64.42
Joint LoRA + Adam
88.32
89.76
68.35
44.75
63.10
86.35
88.16
68.68
44.29
61.61
Vanilla LoRA + Adam
74.76
75.80
52.51
38.67
48.90
58.68
62.66
33.85
32.94
38.20
MindTheGap
85.97
87.40
65.43
43.78
60.32
87.85
86.21
68.81
44.72
61.73
RAPF
47.01
69.91
32.06
24.77
23.90
18.71
26.02
10.73
15.05
6.17
Table 3: CLIP zero-shot accuracy (%) after ImageNet-R/CIFAR-100 streams. 3-seed means ( ± std and ImageNet-A-100 in Appendix Tables 11 – 13 ); Bold : best, underlined : second.
Figure 1: STABLE drift-control dynamics on ImageNet-R 100-task LoRA. STABLE distance from θt⋆ (red), Adam distance (gray). When drift exceeds the budget τ (green dashed), λ (blue) rises; once the trajectory returns, λ decays.
Figure 2: Local margin stability and boundary forgetting on ImageNet-R 100-task LoRA. Top: worst-case projected margin erosion vs. distance from θt−1⋆ inside the drift budget (dashed: empirical bound from end-of-epoch probes). Bottom: projected margin erosion vs. old-task forgetting (one point per post-switch epoch).
Figure 3: Zero-shot transfer retention on ImageNet-R 10-task CLIP LoRA. Task 0: pre-trained CLIP; Tasks 1–10: stream tasks. Top: accuracy; bottom: mean margin.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Local score and margin stability inside the drift budget. Top: worst-case old-task score changes versus distance from the inherited solution. Bottom: worst-case projected margin erosion versus distance. Dashed bounds are estimated from end-of-epoch probes and evaluated on held-out intermediate-step probes.
Method
Moment handling
Final avg. acc.
Final avg. forgetting
Vanilla LoRA + Adam
Carried across tasks
0.66
60.22
Vanilla LoRA + Adam reset
Reset per task
0.77
89.87
Vanilla LoRA + STABLE
Controlled drift
73.12
10.17
Appendix
Table 4: Adam moment-reset control on ImageNet-R 100-task LoRA. Entries report final average accuracy and forgetting (%).
c
Final avg. acc.
Avg. acc. over tasks
Final avg. forgetting
Max forgetting
0.01
1.29
7.47
59.19
100.00
0.1
36.12
45.37
55.91
100.00
0.3
42.66
61.60
41.73
90.00
1
56.38
63.40
11.09
60.00
3
51.93
59.06
11.26
58.33
10
52.18
60.80
11.06
50.00
Appendix
Table 5: Fixed L2 pullback sweep on ImageNet-A-100 (50 tasks). Entries report accuracy and forgetting (%) for static coefficients c .
Method
Coefficient / rule
ImageNet-A-100
ImageNet-R
50 tasks
100 tasks
Fixed pullback
c=1 (best sweep)
56.38
38.84
STABLE
Task-1 budget calibration
62.34
73.12
Appendix
Table 6: Fixed L2 pullback transfer test. Entries report final average accuracy (%) on ImageNet-A-100 and ImageNet-R.
Method
Drift Budget
Final avg. acc.
Final avg. forgetting
Hard projection
0.01
53.90
14.73
0.03
54.61
21.11
0.06
22.57
57.65
0.09
21.31
62.35
0.10
14.42
71.08
0.30
28.93
61.65
Appendix
Table 7: Hard-projection baseline on ImageNet-A-100 (50 tasks). Entries report accuracy and forgetting (%).
10 tasks
100 tasks
Method
Optimizer
AˉT
FˉT
AˉT
FˉT
None
STABLE
75.74
6.09
73.12
10.17
EWC
Adam
59.95
29.30
0.62
58.47
STABLE
75.95
6.67
71.57
13.58
LwF
Adam
71.74
13.57
0.44
67.29
STABLE
75.76
4.35
66.24
16.46
Appendix
Table 8: STABLE with regularization and replay on ImageNet-R (vanilla LoRA). Entries report accuracy and forgetting (%) for 10- and 100-task streams. Bold : best per column.
Method
Opt.
ImageNet-R (100 tasks)
CIFAR-100 (50 tasks)
ImageNet-A-100 (50 tasks)
AˉT (%)
FˉT (%)
AˉT (%)
FˉT (%)
AˉT (%)
FˉT (%)
Joint LoRA
Adam
79.85±0.20
–
92.32±0.18
–
72.68±0.39
–
Joint Prompts
Adam
72.23±0.67
–
88.86±0.15
–
61.34±0.52
–
LoRA-based methods
O-LoRA [ 41 ]
AdamW
57.87±2.62
33.87±2.41
74.09±1.34
25.90±1.39
26.92±2.21
59.37±1.56
HiDeLoRA [ 40 ]
Adam
72.36±0.62
6.16 ± 0.44
79.92±2.70
2.29 ± 0.06
60.37±0.97
10.62±1.20
Appendix
Table 9: Long-horizon unimodal continual learning corresponding to Table 1 . Metric entries are mean ± std over 3 seeds for AˉT and FˉT . Joint rows are offline upper bounds; best in bold , second underlined .
Method
Opt.
ImageNet-R
CIFAR-100
ImageNet-A-100
AˉT (%)
FˉT (%)
AˉT (%)
FˉT (%)
AˉT (%)
FˉT (%)
Joint LoRA
Adam
86.06±0.17
–
87.45±0.11
–
74.28±0.51
–
Vanilla LoRA
Adam
64.34±1.68
25.74±2.10
56.84±1.47
44.68±1.74
28.15±1.18
55.03±0.38
MindTheGap [ 18 ]
Adam
81.50 ± 0.08
6.94±0.14
80.40±0.10
9.62±0.15
65.98±0.78
6.59±0.21
RAPF [ 17 ]
Adam
80.41±0.12
3.30 ± 0.26
80.41 ± 0.11
6.99 ± 0.11
61.07±1.06
16.57±1.97
MoE4CL [ 47 ]
AdamW
80.37±0.18
5.48±0.08
79.08±0.06
9.64±0.09
58.20±0.14
5.75 ± 0.17
Appendix
Table 10: CLIP multimodal continual learning corresponding to Table 2 . Metric entries are mean ± std over 3 seeds for AˉT and FˉT . The joint row is an offline upper bound; best in bold , second underlined .
Method
Food101
Pets
Flowers
DTD
Cars
Pre-trained CLIP
88.38
88.01
66.03
44.31
64.42
Joint LoRA + Adam
88.32±0.06
89.76±0.33
68.35±0.23
44.75±0.31
63.10±0.07
Vanilla LoRA + Adam
74.76±1.32
75.80±2.06
52.51±0.07
38.67±0.70
48.90±0.95
MindTheGap
85.97±0.55
87.40 ± 0.40
65.43 ± 2.21
43.78 ± 0.44
60.32 ± 1.19
RAPF
47.01±2.35
69.91±0.74
32.06±4.32
24.77±0.97
23.90±2.55
MoE4CL
85.12±0.21
84.45±0.13
61.66±0.31
43.62±0.44
59.69±0.59
Appendix
Table 11: CLIP zero-shot accuracy (%) after the ImageNet-R stream. Entries are mean ± std over 3 seeds using the standard image-text head. Pre-trained CLIP and joint rows are references; best in bold , second underlined .
Method
Food101
Pets
Flowers
DTD
Cars
Pre-trained CLIP
88.38
88.01
66.03
44.31
64.42
Joint LoRA + Adam
86.35±0.07
88.16±0.54
68.68±0.56
44.29±0.36
61.61±0.33
Vanilla LoRA + Adam
58.68±3.01
62.66±3.18
33.85±0.66
32.94±1.11
38.20±3.98
MindTheGap
87.85 ± 0.31
86.21 ± 1.13
68.81 ± 0.38
44.72 ± 0.13
61.73 ± 0.31
RAPF
18.71±0.64
26.02±5.15
10.73±1.15
15.05±1.05
6.17±0.37
MoE4CL
86.35±0.53
80.43±0.21
66.88±1.43
41.35±0.62
55.48±0.75
Appendix
Table 12: CLIP zero-shot accuracy (%) after the CIFAR-100 stream. Entries are mean ± std over 3 seeds using the standard image-text head. Pre-trained CLIP and joint rows are references; best in bold , second underlined .
Method
Food101
Pets
Flowers
DTD
Cars
Pre-trained CLIP
88.38
88.01
66.03
44.31
64.42
Joint LoRA + Adam
87.70±0.06
87.94±0.62
67.86±0.54
44.04±0.23
62.58±0.36
Vanilla LoRA + Adam
36.58±4.31
40.86±2.83
17.01±5.58
23.40±1.33
26.86±2.82
MindTheGap
88.17 ± 0.03
87.16 ± 0.36
65.84 ± 0.37
43.62±0.05
63.54 ± 0.09
RAPF
8.77±2.61
13.89±2.95
7.67±0.63
9.72±0.46
5.20±1.09
MoE4CL
87.98±0.02
87.43 ± 0.05
66.12 ± 0.03
43.87 ± 0.09
63.42 ± 0.07
Appendix
Table 13: CLIP zero-shot accuracy (%) after the ImageNet-A-100 stream. Entries are mean ± std over 3 seeds using the standard image-text head. Pre-trained CLIP and joint rows are references; best in bold , second underlined .
Method
Opt.
ImageNet-R
CIFAR-100
ImageNet-A-100
AˉT (%)
FˉT (%)
AˉT (%)
FˉT (%)
AˉT (%)
FˉT (%)
Joint LoRA
Adam
83.71±0.16
–
87.43±0.24
–
73.00±0.18
–
Vanilla LoRA
Adam
64.34±1.68
25.74±2.10
56.84±1.47
44.68±1.74
28.15±1.18
55.03±0.38
MindTheGap [ 18 ]
Adam
79.63±0.32
8.46±0.49
79.76±0.46
15.76±0.44
69.63 ± 0.19
7.07±0.69
RAPF [ 17 ]
Adam
48.09±0.49
42.36±0.48
48.46±0.26
51.29±0.14
34.64±0.91
52.54±1.17
MoE4CL [ 47 ]
AdamW
80.21 ± 0.16
6.41±0.11
79.85 ± 0.10
15.89±0.14
68.23±0.08
6.44 ± 0.09
Appendix
Table 14: Common-head CLIP multimodal continual learning. Metric entries are mean ± std over 3 seeds for AˉT and FˉT , with all methods evaluated by W-NCM. The joint row is an offline upper bound; best in bold , second underlined .
η
0.01
0.03 (default)
0.06
0.1
0.3
0.6
Final avg. acc.
72.62
73.12
73.61
73.18
73.96
73.92
Avg. forgetting
10.48
10.17
9.39
10.27
9.72
9.64
Appendix
Table 15: Sensitivity to the dual step size η on ImageNet-R (100 tasks, vanilla LoRA). Entries report final average accuracy and forgetting (%).
Stream
Method
Linear head
W-NCM head
ImageNet-R, 100 tasks
L2P
34.43±0.55
68.60±0.23
DualPrompt
27.44±1.08
70.39±0.71
InfLoRA
38.90±0.14
72.54±1.56
CIFAR-100, 50 tasks
L2P
63.72±1.30
85.32±0.31
DualPrompt
68.63±0.50
86.25±0.95
InfLoRA
58.13±4.90
84.93±3.63
Appendix
Table 16: Baselines with their native linear head and with the W-NCM head of Table 1 . Entries report final average accuracy (%), mean ± std over 3 seeds.
ETH AI Center, Department of Computer Science, ETH Zürich · Mila - Quebec AI Institute, School of Computer Science, McGill University · Mila - Quebec AI Institute +1