Incremental Learning of Vision-Language Models via Task Subspace Projection and Dynamic LoRA
Organizations: College of Computer Science and Technology, Zhejiang University, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi · School of Computer Science and Technology, Zhejiang University of Science and Technology, China · School of Computer and Computing Science, Hangzhou City University, China
Abstract
Recent pre-trained vision-language models often encounter a Multi-Domain Task-Incremental Learning (MTIL) scenario in practice, where a set of classes of multi-modal tasks arrive incrementally. Due to privacy concerns and memory constraints, MTIL with pre-trained models encounters forgetting of knowledge from old tasks, degradation of zero-shot transfer capability, and underfitting of new-task knowledge. To overcome these challenges, previous MTIL methods attempt to learn a discriminative cross-task identification (CTI) module and an effective new-task adaptation (NTA) module. However, current CTI modules suffer from severe task confusion between seen and unseen tasks, and NTA modules cannot adaptively balance the performance and parameter cost while incorporating task-specific knowledge. To alleviate the above dilemmas, we propose an effective and efficient TSP-DLoRA method for MTIL, which consists of Task Subspace Projection (TSP) and Dynamic Low Rank Adapter (DLoRA) modules. Specifically, our TSP module includes a task identifier classifier based on task-specific subspaces and a feature projection strategy that can determine the identifier associated with samples from both seen and unseen tasks. Our DLoRA improves the knowledge adaptation from new tasks by dynamically assigning Low Rank Adapter (LoRA) across transformer layers based on the task distributions. Experimental evaluations across 11 datasets, using three performance metrics, demonstrate the effectiveness of our proposed method. Code is available at https://github.com/FerdinandZJU/TSP-DLoRA.
Figures & tables
| Method | Extra. | Param. | Aircraft Maji et al. (2013) | Caltech101 Fei-Fei et al. (2004) | CIFAR100 Krizhevsky et al. (2009) | DTD Cimpoi et al. (2014) | EuroSAT Helber et al. (2019) | Flowers Nilsback and Zisserman (2008) | Food Bossard et al. (2014) | MNIST Deng (2012) | OxfordPet Parkhi et al. (2012) | StanfordCars Krause et al. (2013) | SUN397 Xiao et al. (2010) | Average | |
| CLIP | Zero-shot | ✗ | - | 24.3 | 88.4 | 68.2 | 44.6 | 54.9 | 71.0 | 88.5 | 59.4 | 89.0 | 64.7 | 65.2 | 65.3 |
| Full Fine-tune | ✗ | 211M | 62.0 | 95.1 | 89.6 | 79.5 | 98.9 | 97.5 | 92.7 | 99.6 | 94.7 | 89.6 | 81.8 | 89.2 | |
| Transfer | Continual-FT | ✓ | 211M | - | 67.1 | 46.0 | 32.1 | 35.6 | 35.0 | 57.7 | 44.1 | 60.8 | 20.5 | 46.6 | 44.6 |
| LwF Li and Hoiem (2017) | ✓ | 211M | - | 74.5 | 56.9 | 39.1 | 51.1 | 52.6 | 72.8 | 60.6 | 75.1 | 30.3 | 55.9 | 58.9 | |
| iCaRL Rebuffi et al. (2017) | ✓ | 211M | - | 56.6 | 44.6 | 32.7 | 39.3 | 46.6 | 68.0 | 46.0 | 77.4 | 31.9 | 60.5 | 50.4 | |
| LwF-VR Ding et al. (2022) | ✓ | 211M | - | 77.1 | 61.0 | 40.5 | 45.3 | 54.4 | 74.6 | 47.9 | 76.7 | 36.3 | 58.6 | 57.2 |
| Method | Extra. | Param. | Cars Krause et al. (2013) | Food Bossard et al. (2014) | MNIST Deng (2012) | OxfordPet Parkhi et al. (2012) | Flowers Nilsback and Zisserman (2008) | SUN397 Xiao et al. (2010) | Aircraft Maji et al. (2013) | Caltech101 Fei-Fei et al. (2004) | DTD Cimpoi et al. (2014) | EuroSAT Helber et al. (2019) | CIFAR100 Krizhevsky et al. (2009) | Average | |
| CLIP | Zero-shot | ✗ | - | 64.7 | 88.5 | 59.4 | 89.0 | 71.0 | 65.2 | 24.3 | 88.4 | 44.6 | 54.9 | 68.2 | 65.3 |
| Full Fine-tune | ✗ | 211M | 89.6 | 92.7 | 99.6 | 94.7 | 97.5 | 81.8 | 62.0 | 95.1 | 79.5 | 98.9 | 89.6 | 89.2 | |
| Transfer | Continual-FT | ✓ | 211M | 89.5 | 59.6 | 57.9 | 40.0 | 46.7 | 11.1 | 70.0 | 30.5 | 26.6 | 37.7 | 46.6 | |
| LwF Li and Hoiem (2017) | ✓ | 211M | - | 87.8 | 58.5 | 71.9 | 46.6 | 57.3 | 12.8 | 81.4 | 34.5 | 34.5 | 46.8 | 53.2 | |
| iCaRL Rebuffi et al. (2017) | ✓ | 211M | - | 86.1 | 51.8 | 67.6 | 50.4 | 57.9 | 11.0 | 72.3 | 31.2 | 32.7 | 48.1 | 50.9 | |
| LwF-VR Ding et al. (2022) | ✓ | 211M | - | 88.2 | 57.0 | 71.4 | 50.0 | 58.0 | 13.0 | 82.0 | 34.4 | 29.3 | 47.6 | 53.1 |
| Method | Extra. | Param. | Aircraft | Caltech101 | CIFAR100 | DTD | Flowers | Food | StanfordCars | SUN397 | Average | |
| CLIP | Zero-shot | ✗ | - | 24.8 | 92.9 | 68.4 | 43.8 | 71.4 | 85.8 | 65.8 | 62.6 | 64.4 |
| Full Fine-tune | ✗ | 211M | 62.0 | 96.2 | 89.6 | 79.5 | 97.5 | 92.7 | 89.6 | 81.8 | 86.1 | |
| Transfer | ZSCL Zheng et al. (2023) | ✓ | 211M | 87.3 | 67.7 | 45.4 | 67.8 | 86.6 | 59.7 | 63.4 | 68.3 | |
| L2P Wang et al. (2022c) | ✗ | 0.5M | 66.7 | 54.3 | 30.6 | 47.3 | 71.5 | 54.6 | 52.4 | 53.9 | ||
| DualPrompt Wang et al. (2022b) | ✗ | 1.8M | 78.8 | 64.4 | 32.0 | 51.7 | 77.5 | 49.4 | 51.3 | 57.9 | ||
| S-Prompts Wang et al. (2022a) | ✗ | 0.5M | 70.3 | 52.7 | 31.5 | 54.8 | 74.0 | 55.4 | 50.0 | 55.5 |
| Dataset | Classes | Train | Test |
| Aircraft Maji et al. (2013) | 100 | 3334 | 3333 |
| Caltech101 Fei-Fei et al. (2004) | 101 | 6212 | 2465 |
| CIFAR100 Krizhevsky et al. (2009) | 100 | 50000 | 10000 |
| DTD Cimpoi et al. (2014) | 47 | 2068 | 1692 |
| EuroSAT Helber et al. (2019) | 10 | 18800 | 8100 |
| Flowers Nilsback and Zisserman (2008) | 102 | 4706 | 2463 |
| Method | Trans . | Avg. | Lst. |
| DIKI* | 67.4 | 75.7 | 85.1 |
| DIKI*+TSP | 68.9 | 76.3 | 85.3 |
| LoRA+TSP | 69.3 | 76.5 | 85.4 |
| DLoRA+TSP | 69.4 | 77.5 | 86.5 |
| Method | Time | GPU | Train | Infer. |
| MoE-Adapter | 5.3h | 48 GB | 1.58s/it | 0.63s/it |
| DIKI | 2.6h | 24 GB | 0.36s/it | 0.33s/it |
| Ours | 2.4h | 24 GB | 0.38s/it | 0.23s/it |
| Threshold | |||||||||||
| 0.89 | 0.90 | 0.91 | 0.92 | 0.93 | 0.94 | 0.95 | 0.96 | 0.97 | 0.98 | ||
| Energy | 0.89 | 0.889 | 0.912 | 0.921 | 0.909 | 0.867 | 0.800 | 0.699 | 0.589 | 0.498 | 0.462 |
| 0.90 | 0.869 | 0.897 | 0.918 | 0.923 | 0.899 | 0.850 | 0.762 | 0.646 | 0.525 | 0.466 | |
| 0.91 | 0.846 | 0.878 | 0.908 | 0.926 | 0.920 | 0.884 | 0.811 | 0.699 | 0.559 | 0.474 | |
| 0.92 | 0.815 | 0.850 | 0.885 | 0.914 | 0.930 | 0.915 | 0.862 | 0.767 | 0.614 | 0.492 | |
| 0.93 | 0.779 | 0.815 | 0.853 | 0.888 | 0.921 | 0.930 | 0.900 | 0.827 | 0.682 | 0.520 | |