Pretrained vision models provide strong general-purpose representations, but in Domain-Incremental Learning (DIL), where the input distribution changes sequentially, adapting to new domains often degrades performance on previously learned ones. We observe that forgetting is not uniform across domains and classes and is not explained by training order alone. This motivates us to track each past domain-class concept individually. We use representation drift, defined as the deviation of a past concept's current representation from a fixed anchor, as a measurable surrogate for forgetting risk. The measured drift then serves two complementary roles. A consolidation loss assigns stronger preservation to high-drift pairs, while prototype construction down-weights highly drifted domains to reduce their influence on the class reference. Preservation is thus allocated by measured drift rather than assumed importance. To obtain this signal during training without storing past raw data, we repurpose a Joint-Embedding Predictive Architecture (JEPA). For each domain-class pair, we store only a compact context summary in the backbone feature space and rerun it through the learned context-to-target relation to measure its drift from the fixed anchor. Across DomainNet, Office-Home, CORe50, and CDDB-Hard, our method sustains a strong accuracy-forgetting balance on every benchmark, using substantially less training memory than accuracy-competitive methods and the smallest storage among compared methods.
Figures & tables
Figure 1: Per-domain forgetting when DINOv2 is sequentially fine-tuned on DomainNet. Example images of the same class (e.g., crab) illustrate the visual variation across domains.
Figure 2: Overview of our framework. Evidence Decomposition splits each input image into a target and a context. Context-to-Target Prediction predicts the target from the context and performs classification. Drift-Adaptive Consolidation measures the drift from the fixed anchors using stored past-domain summaries while learning a new domain, and uses this signal to modulate the preservation strength and to weight each domain’s contribution when forming the class prototypes.
Model
DomainNet
CORe50
OfficeHome
CDDB-Hard
AvgAcc ↑
AvgF ↓
AvgAcc ↑
AvgF ↓
AvgAcc ↑
AvgF ↓
AvgAcc ↑
AvgF ↓
DIDD [ 15 ]
9.42 ± 3.23
4.75 ± 2.87
55.9 ± 2.07
22.6 ± 1.85
29.5 ± 2.60
3.9 ± 1.60
54.5 ± 1.54
1.7 ± 0.87
CODA-Prompt [ 19 ]
53.6 ± 2.33
21.5 ± 3.27
99.1 ± 0.15
0.8 ± 0.16
87.2 ± 0.13
4.7 ± 0.26
67.8 ± 1.01
12.9 ± 0.92
OS-Prompt [ 7 ]
52.1 ± 1.59
23.3 ± 1.13
99.2 ± 0.08
0.8 ± 0.08
86.6 ± 0.17
5.5 ± 0.14
66.9 ± 0.73
20.2 ± 0.51
OS-Prompt++ [ 7 ]
51.4 ± 1.18
24.3 ± 1.87
99.2 ± 0.03
0.8 ± 0.02
86.3 ± 0.47
5.2 ± 0.46
67.5 ± 1.76
17.9 ± 2.14
MoE [ 25 ]
63.6 ± 0.82
17.9 ± 2.23
99.3 ± 0.02
0.5 ± 0.08
87.5 ± 0.05
1.8 ± 0.27
81.4 ± 0.48
8.3 ± 1.53
Table 1: Experiment results. For each benchmark we report the AvgAcc ( ↑ ) and AvgF ( ↓ ), in percent. Bold marks the best and underline the second best in each column.
Model
DIDD
CODA-Prompt
OS-Prompt
OS-Prompt++
MoE
GraphKeeper
SOYO
DUCT
Ours
VRAM (GB)
6.38
19.99
19.79
20.21
32.00
2.00
15.23
14.07
3.00
Table 2: Peak GPU memory usage on DomainNet (GB).
Figure 3: Cumulative storage (MB, log scale) on DomainNet as the number of learned domains grows. Numbers at the right denote the final storage after all six domains.
DomainNet
CDDB-Hard
Variant
AvgAcc ↑
AvgF ↓
AvgAcc ↑
AvgF ↓
Full (Ours)
61.80
4.12
79.40
6.70
w/o Lcons
58.18
14.50
76.82
7.74
w/o Lpred
59.83
5.53
76.28
8.22
w/o adaptive cons. ( γ=0 )
58.25
8.56
78.46
6.88
w/o robust proto. ( ρ=0 )
58.80
7.07
77.71
7.28
Table 3: Ablation study. Each variant removes one component of the full model. AvgAcc and AvgF are the final average accuracy and forgetting (%).
Figure 4: Mean drift over all past domains at each training stage on DomainNet, with and without the consolidation loss Lcons .
Figure 5: Polar visualization of drift with and without Lcons , where all class anchors are placed at the center ( ★ ) and the radius denotes the drift magnitude.
Table 4: Benchmark statistics. Domains are presented sequentially in the listed order.
DomainNet
CORe50
Office-Home
CDDB-Hard
Epochs per domain
10
10
30
20
wcons
300
3
300
30
γ
2
1
1
2
Appendix
Table 5: Per-benchmark training settings. All other hyperparameters are shared across benchmarks.
γ\wcons
10
30
100
300
γ=0
60.59 / 11.52
60.63 / 11.23
61.47 / 9.87
58.25 / 8.56
γ=1
59.80 / 12.78
60.51 / 11.63
61.06 / 10.52
60.51 / 6.40
γ=2
59.90 / 12.66
60.53 / 11.61
61.19 / 9.85
61.80 / 4.12
Appendix
Table 6: Sensitivity to the consolidation weight wcons on DomainNet. Each cell reports AvgAcc / AvgF in percent; bold marks the adopted configuration ( wcons=300 , γ=2 ).
γ\wcons
10
30
100
300
γ=0
87.12 / 1.82
86.83 / 2.14
87.75 / 0.21
87.68 / 0.50
γ=1
84.93 / 4.09
87.16 / 0.99
87.51 / 0.80
87.80 / 0.22
γ=2
86.60 / 2.32
86.80 / 2.18
87.43 / 0.65
86.15 / 0.53
Appendix
Table 7: Sensitivity to the consolidation weight wcons on Office-Home. Each cell reports AvgAcc / AvgF in percent; bold marks the adopted configuration ( wcons=300 , γ=1 ).
Figure 6: CLS-to-patch attention of the frozen backbone for the airplane class across the six DomainNet domains. Top rows: input images; bottom rows: attention maps.