Inherited Adapter Experts for Replay-Free Domain-Incremental Learning
Organizations: Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea
Abstract
Replay-free domain-incremental learning trains a model on the same classes in successive domains without keeping earlier images. Per-domain learners freeze a pre-trained ViT and train an additional expert for each new domain: the small set of parameters and the classification head learned on that domain. Test images carry no domain label, so a router selects one expert for each image. Recent work trains this router. However, a training-free Mahalanobis router costs the analyzed learners under 2.7 points of final accuracy relative to always selecting each image's own-domain expert. The experts, not this router, limit accuracy. Our learner, Lamarck, therefore uses this router and strengthens the experts. Its adapters, small trainable layers that correct every image token after each deep ViT block, add 4.4 points of final accuracy on the deepfake benchmark CDDB-Hard. In addition, each expert is initialized from its predecessor at no extra cost: the domains share their classes, so an image misrouted to a later expert loses less. Against twelve frozen-backbone baselines under one protocol, Lamarck is first on CDDB-Hard by 5.2 points, second on DomainNet and OfficeHome, and highest in mean accuracy.
Figures & tables
| DN | OH | CDDB | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Expert bank | DRA | Mah. | Proto. | SE | DRA | Mah. | Proto. | SE | DRA | Mah. | Proto. | SE |
| S-iPrompt | ||||||||||||
| SOYO | ||||||||||||
| Lamarck, prompts only | ||||||||||||
| Lamarck (Ours) | ||||||||||||
| Method | Venue | DN | OH | CDDB | Mean | ||||
|---|---|---|---|---|---|---|---|---|---|
| AIA | ACC | AIA | ACC | AIA | ACC | AIA | ACC | ||
| Joint linear probe | – | – | – | – | – | ||||
| Shared-pool learners: one prompt pool and classifier shared by all domains | |||||||||
| CODA-Prompt [ 25 ] | CVPR 2023 | ||||||||
| OS-Prompt [ 13 ] | ECCV 2024 | ||||||||
| OS-Prompt++ [ 13 ] | ECCV 2024 | ||||||||
| Method | Optimizer | LR | Schedule | WD | Prompt length / pool / module | Auxiliary phases (epochs) | Other settings |
| Joint linear probe | Adam | cosine | 0 | – | – | linear head on the pooled TRAIN domains | |
| SLCA | SGD | 0.01 | multi-step | fine-tuned backbone | classifier alignment (5) | LR step at epoch 12; backbone LR 0.01; alignment on 256 sampled features per class with logit normalization 0.1; prediction modulo over the per-domain heads | |
| DUCT | SGD | 0.1 | multi-step ‡ | fine-tuned backbone | classifier retraining (5; LR ) | backbone LR 0.01; merge ratio 0.25 (OfficeHome, DomainNet) / 0.5 (CDDB-Hard) and optimal-transport regularization 0.1 (OfficeHome, CDDB-Hard) / 0.01 (DomainNet), as in the released configurations; prediction modulo over the per-domain heads; classes without training images left out of the transport; log-domain Sinkhorn solver | |
| CODA-Prompt | Adam | cosine | 0 | pool 100, length 8, blocks 1–5 | – | orthogonality weight 0 (released script) | |
| OS-Prompt | Adam | cosine | 0 | pool 100, length 8, blocks 1–5 | – | query regularizer 0 | |
| OS-Prompt++ | Adam | cosine | 0 | pool 100, length 8, blocks 1–5 | – | query regularizer |
| Benchmark | Block | Entropy/ | Largest weight | TV mean | TV max |
|---|---|---|---|---|---|
| DomainNet | 1 | 99.14% | 0.0169 | 0.0000 | 0.0000 |
| 2 | 99.01% | 0.0177 | 0.0008 | 0.0027 | |
| 3 | 98.47% | 0.0179 | 0.0039 | 0.0090 | |
| 4 | 99.56% | 0.0156 | 0.0081 | 0.0162 | |
| 5 | 98.96% | 0.0211 | 0.0075 | 0.0200 | |
| OfficeHome | 1 | 99.94% | 0.0117 | 0.0000 | 0.0000 |
| VAL AIA | VAL ACC | TEST AIA | TEST ACC | |
|---|---|---|---|---|
| 0.03 | (1/5) | (1/5) | (2/5) | (4/5) |
| 0.1 | (2/5) | (3/5) | (4/5) | (3/5) |
| 0.3 | (4/5) | (2/5) | (2/5) | (4/5) |
| Method | DomainNet | OfficeHome | CDDB-Hard | |||
|---|---|---|---|---|---|---|
| AIA | ACC | AIA | ACC | AIA | ACC | |
| SLCA [ 40 ] | ||||||
| DUCT [ 43 ] | ||||||
| Lamarck (Ours) | ||||||
| Aggregate accuracy | Per-domain accuracy | ||||||||
| Method | Equal | Images | Equal w/o SAN | Images w/o SAN | GauGAN (800) | BigGAN (320) | Wild (827) | WhichFaceIsReal (160) | SAN (33) |
| Joint linear probe | |||||||||
| SLCA [ 40 ] (fine-tuned backbone) | |||||||||
| DUCT [ 43 ] (fine-tuned backbone) | |||||||||
| CODA-Prompt [ 25 ] | |||||||||
| OS-Prompt [ 13 ] | |||||||||
| Domain | EASE [ 41 ] | CONEC-LoRA [ 20 ] | Lamarck (Ours) |
|---|---|---|---|
| Clipart | |||
| Infograph | |||
| Painting | |||
| QuickDraw | |||
| Real | |||
| Sketch |
| Method | Inference | Training step | ||||||
|---|---|---|---|---|---|---|---|---|
| Passes | GFLOPs | Param. (M) | img/s | Mem. (GB) | GFLOPs | img/s | Mem. (GB) | |
| CODA-Prompt [ 25 ] | 2 | 70.3 | 4.11 | 159.1 | 0.90 | 105.9 | 100.9 | 6.15 |
| OS-Prompt [ 13 ] | 1 | 35.1 | 3.72 | 314.8 | 0.86 | 70.8 | 146.3 | 6.09 |
| OS-Prompt++ [ 13 ] | 1 | 35.1 | 3.72 | 314.6 | 1.18 | 105.9 | 100.5 | 6.45 |
| KA-Prompt [ 39 ] | 2 | 88.1 | 6.76 | 130.0 | 1.45 | 283.0 | 38.2 | 15.68 |
| CPS-Prompt [ 17 ] | 2 | 70.3 | 4.11 | 159.7 | 0.97 | 69.7 | 151.2 | 3.09 |
| Benchmark | Adapters | VAL ACC | TEST ACC | Later-expert cost |
|---|---|---|---|---|
| DomainNet | 31.43 | |||
| 31.47 | ||||
| 31.34 | ||||
| Blocks 1–12 | – | 31.46 | ||
| OfficeHome | 4.79 | |||
| 5.04 |
| DomainNet | OfficeHome | CDDB-Hard | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Routing | Mis. | reset | indep. | Mis. | reset | indep. | Mis. | reset | indep. |
| Domain-label routing | 0.0 | 0.0 | 0.0 | ||||||
| Mahalanobis router | 11.9 | 16.2 | 3.6 | ||||||
| Prototype router | 19.5 | 36.4 | 10.5 | ||||||
| Last expert only | 88.2 | 72.1 | 98.5 | ||||||
| OfficeHome | CDDB-Hard | ||||||
| Difference | Extra errors to | ||||||
| Lamarck reset adapters | later experts | (4/5) | (5/5) | (5/5) | (3/5) | (5/5) | (5/5) |
| any wrong expert | (4/5) | (5/5) | (5/5) | (3/5) | (5/5) | (5/5) | |
| earlier experts | (4/5) | (4/5) | (4/5) | (3/5) | (4/5) | (3/5) | |
| Lamarck independent experts | later experts | (5/5) | (5/5) | (5/5) | (4/5) | (5/5) | (5/5) |
| any wrong expert | (5/5) | (5/5) | (5/5) | (4/5) | (5/5) | (5/5) | |
| OfficeHome | CDDB-Hard | ||||||||||||
| Router | reset | Own | Later | indep. | reset | Own | Later | indep. | |||||
| Mahalanobis | all | 17.47 | 9.65 | (4/5) | (5/5) | 6.45 | 1.28 | (3/5) | (4/5) | ||||
| -shot | 256 | 35.88 | 18.72 | (5/5) | (5/5) | 13.42 | 6.44 | (5/5) | (4/5) | ||||
| 128 | 39.59 | 20.56 | (5/5) | (5/5) | 18.61 | 9.36 | (5/5) | (5/5) | |||||
| 64 | 36.93 | 19.82 | (5/5) | (5/5) | 15.80 | 7.27 | (5/5) | (5/5) | |||||
| 32 | 40.04 | 21.73 | (5/5) | (5/5) | 17.08 | 7.88 | (5/5) | (5/5) | |||||
| Original order (GauGAN to SAN) | Reversed order (SAN to GauGAN) | |||||
|---|---|---|---|---|---|---|
| Metric | Reset adapters | Lamarck (Ours) | Reset adapters | Lamarck (Ours) | ||
| ACC, Mahalanobis router | (3/5) | (2/5) | ||||
| ACC, prototype router | (5/5) | (3/5) | ||||
| ACC, last expert only | (5/5) | (5/5) | ||||
| DRA | (2/5) | (1/5) | ||||
| Later-expert accuracy | (5/5) | (5/5) | ||||
Domain-Incremental Learning
Domain-incremental learning studies how models adapt to changing input distributions while retaining performance on previously encountered domains.
See all 61 papers