STABLE: A Continual Learning Optimizer with Adaptive Drift Control
Organizations: Department of Computer Science University of Pisa · Department of AI, Data and Decision Sciences Luiss Guido Carli University · Institute of Information Science and Technologies “Alessandro Faedo” (ISTI) National Research Council of Italy (CNR)
Abstract
Continual learning suffers from catastrophic forgetting, leading to the loss of past knowledge and unstable adaptation across task streams. Existing approaches typically address this issue through replay buffers, architectural changes, or additional regularization terms. We argue that forgetting is a consequence of unstable learning dynamics, which may benefit from an optimizer-level solution. The main challenge is that learning dynamics are time-dependent, as demonstrated by empirical phenomena such as the stability gap. First, we formally prove that, for distance-based classifiers, bounded parameter drift guarantees bounded margin erosion under a local score-stability condition. Based on this insight, we propose STABLE, a primal-dual optimizer with adaptive drift control. STABLE enforces a drift budget by increasing the dual variable when the budget is violated and decreasing it otherwise. This results in high stability without penalizing plasticity. Experiments on long-horizon unimodal streams and CLIP-based multimodal streams show that vanilla PEFT baselines trained with STABLE match or outperform specialized continual learning methods. In the multimodal setting, STABLE further preserves zero-shot transfer, matching or exceeding dedicated multimodal continual learning methods. Code is available at https://github.com/atlas-luiss/STABLE.
Figures & tables
| Method | Opt. | ImageNet-R | CIFAR-100 | ImageNet-A-100 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Params | (%) | (%) | Params | (%) | (%) | Params | (%) | (%) | ||
| Joint LoRA | Adam | 0.44M | 79.85 | – | 0.44M | 92.32 | – | 0.44M | 72.68 | – |
| Joint Prompts | Adam | 0.37M | 72.23 | – | 0.37M | 88.86 | – | 0.37M | 61.34 | – |
| LoRA-based methods | ||||||||||
| O-LoRA [ 41 ] | AdamW | 44.2M | 57.87 | 33.87 | 22.1M | 74.09 | 25.90 | 22.1M | 26.92 | 59.37 |
| HiDeLoRA [ 40 ] | Adam | 10.04M | 72.36 | 6.16 | 6.20M | 79.92 | 2.29 | 6.20M | 60.37 | 10.62 |
| Method | Opt. | Params | ImageNet-R | CIFAR-100 | ImageNet-A-100 | |||
|---|---|---|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (%) | (%) | |||
| Joint LoRA | Adam | 0.98M | 86.06 | – | 87.45 | – | 74.28 | – |
| Vanilla LoRA | Adam | 0.98M | 64.34 | 25.74 | 56.84 | 44.68 | 28.15 | 55.03 |
| MindTheGap [ 18 ] | Adam | 0.49M | 81.50 | 6.94 | 80.40 | 9.62 | 65.98 | 6.59 |
| RAPF [ 17 ] | Adam | 0.26M | 80.41 | 3.30 | 80.41 | 6.99 | 61.07 | 16.57 |
| MoE4CL [ 47 ] | AdamW | 4.03M | 80.37 | 5.48 | 79.08 | 9.64 | 58.20 | 5.75 |
| Method | ImageNet-R | CIFAR-100 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Food101 | Pets | Flowers | DTD | Cars | Food101 | Pets | Flowers | DTD | Cars | |
| Pre-trained CLIP | 88.38 | 88.01 | 66.03 | 44.31 | 64.42 | 88.38 | 88.01 | 66.03 | 44.31 | 64.42 |
| Joint LoRA + Adam | 88.32 | 89.76 | 68.35 | 44.75 | 63.10 | 86.35 | 88.16 | 68.68 | 44.29 | 61.61 |
| Vanilla LoRA + Adam | 74.76 | 75.80 | 52.51 | 38.67 | 48.90 | 58.68 | 62.66 | 33.85 | 32.94 | 38.20 |
| MindTheGap | 85.97 | 87.40 | 65.43 | 43.78 | 60.32 | 87.85 | 86.21 | 68.81 | 44.72 | 61.73 |
| RAPF | 47.01 | 69.91 | 32.06 | 24.77 | 23.90 | 18.71 | 26.02 | 10.73 | 15.05 | 6.17 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Moment handling | Final avg. acc. | Final avg. forgetting |
|---|---|---|---|
| Vanilla LoRA + Adam | Carried across tasks | 0.66 | 60.22 |
| Vanilla LoRA + Adam reset | Reset per task | 0.77 | 89.87 |
| Vanilla LoRA + STABLE | Controlled drift | 73.12 | 10.17 |
| Final avg. acc. | Avg. acc. over tasks | Final avg. forgetting | Max forgetting | |
|---|---|---|---|---|
| 0.01 | 1.29 | 7.47 | 59.19 | 100.00 |
| 0.1 | 36.12 | 45.37 | 55.91 | 100.00 |
| 0.3 | 42.66 | 61.60 | 41.73 | 90.00 |
| 1 | 56.38 | 63.40 | 11.09 | 60.00 |
| 3 | 51.93 | 59.06 | 11.26 | 58.33 |
| 10 | 52.18 | 60.80 | 11.06 | 50.00 |
| Method | Coefficient / rule | ImageNet-A-100 | ImageNet-R |
|---|---|---|---|
| 50 tasks | 100 tasks | ||
| Fixed pullback | (best sweep) | 56.38 | 38.84 |
| STABLE | Task-1 budget calibration | 62.34 | 73.12 |
| Method | Drift Budget | Final avg. acc. | Final avg. forgetting |
|---|---|---|---|
| Hard projection | 0.01 | 53.90 | 14.73 |
| 0.03 | 54.61 | 21.11 | |
| 0.06 | 22.57 | 57.65 | |
| 0.09 | 21.31 | 62.35 | |
| 0.10 | 14.42 | 71.08 | |
| 0.30 | 28.93 | 61.65 |
| 10 tasks | 100 tasks | ||||
| Method | Optimizer | ||||
| None | STABLE | 75.74 | 6.09 | 73.12 | 10.17 |
| EWC | Adam | 59.95 | 29.30 | 0.62 | 58.47 |
| STABLE | 75.95 | 6.67 | 71.57 | 13.58 | |
| LwF | Adam | 71.74 | 13.57 | 0.44 | 67.29 |
| STABLE | 75.76 | 4.35 | 66.24 | 16.46 | |
| Method | Opt. | ImageNet-R (100 tasks) | CIFAR-100 (50 tasks) | ImageNet-A-100 (50 tasks) | |||
|---|---|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (%) | (%) | ||
| Joint LoRA | Adam | – | – | – | |||
| Joint Prompts | Adam | – | – | – | |||
| LoRA-based methods | |||||||
| O-LoRA [ 41 ] | AdamW | ||||||
| HiDeLoRA [ 40 ] | Adam | 6.16 0.44 | 2.29 0.06 | ||||
| Method | Opt. | ImageNet-R | CIFAR-100 | ImageNet-A-100 | |||
|---|---|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (%) | (%) | ||
| Joint LoRA | Adam | – | – | – | |||
| Vanilla LoRA | Adam | ||||||
| MindTheGap [ 18 ] | Adam | 81.50 0.08 | |||||
| RAPF [ 17 ] | Adam | 3.30 0.26 | 80.41 0.11 | 6.99 0.11 | |||
| MoE4CL [ 47 ] | AdamW | 5.75 0.17 | |||||
| Method | Food101 | Pets | Flowers | DTD | Cars |
|---|---|---|---|---|---|
| Pre-trained CLIP | 88.38 | 88.01 | 66.03 | 44.31 | 64.42 |
| Joint LoRA + Adam | |||||
| Vanilla LoRA + Adam | |||||
| MindTheGap | 87.40 0.40 | 65.43 2.21 | 43.78 0.44 | 60.32 1.19 | |
| RAPF | |||||
| MoE4CL |
| Method | Food101 | Pets | Flowers | DTD | Cars |
|---|---|---|---|---|---|
| Pre-trained CLIP | 88.38 | 88.01 | 66.03 | 44.31 | 64.42 |
| Joint LoRA + Adam | |||||
| Vanilla LoRA + Adam | |||||
| MindTheGap | 87.85 0.31 | 86.21 1.13 | 68.81 0.38 | 44.72 0.13 | 61.73 0.31 |
| RAPF | |||||
| MoE4CL |
| Method | Food101 | Pets | Flowers | DTD | Cars |
|---|---|---|---|---|---|
| Pre-trained CLIP | 88.38 | 88.01 | 66.03 | 44.31 | 64.42 |
| Joint LoRA + Adam | |||||
| Vanilla LoRA + Adam | |||||
| MindTheGap | 88.17 0.03 | 87.16 0.36 | 65.84 0.37 | 63.54 0.09 | |
| RAPF | |||||
| MoE4CL | 87.43 0.05 | 66.12 0.03 | 43.87 0.09 | 63.42 0.07 |
| Method | Opt. | ImageNet-R | CIFAR-100 | ImageNet-A-100 | |||
|---|---|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (%) | (%) | ||
| Joint LoRA | Adam | – | – | – | |||
| Vanilla LoRA | Adam | ||||||
| MindTheGap [ 18 ] | Adam | 69.63 0.19 | |||||
| RAPF [ 17 ] | Adam | ||||||
| MoE4CL [ 47 ] | AdamW | 80.21 0.16 | 79.85 0.10 | 6.44 0.09 | |||
| 0.01 | 0.03 (default) | 0.06 | 0.1 | 0.3 | 0.6 | |
|---|---|---|---|---|---|---|
| Final avg. acc. | 72.62 | 73.12 | 73.61 | 73.18 | 73.96 | 73.92 |
| Avg. forgetting | 10.48 | 10.17 | 9.39 | 10.27 | 9.72 | 9.64 |
| Stream | Method | Linear head | W-NCM head |
|---|---|---|---|
| ImageNet-R, 100 tasks | L2P | ||
| DualPrompt | |||
| InfLoRA | |||
| CIFAR-100, 50 tasks | L2P | ||
| DualPrompt | |||
| InfLoRA |
Continual Learning
Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.
See all 323 papers