Learning When to Recur: Token-Adaptive Recursion for Imbalanced Ophthalmic Domain Incremental Learning
Authors: Nanxi Yu, Kang Li, Ye Du, Xiaowei Hu, Weihua Yang, Shujun Wang
Organizations: School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Sichuan, China · Department of Biomedical Engineering and Sports Technology, The Hong Kong Polytechnic University, Hong Kong SAR, China · School of Future Technology, South China University of Technology, Guangdong, China · Shenzhen Eye Hospital, Shenzhen Eye Medical Center, Southern Medical University, Guangdong, China
Domain incremental learning is essential for adapting ophthalmic deep learning models to sequential clinical domains while preserving diagnostic expertise. Existing domain incremental learning methods predominantly address the domain shift induced by style variations. However, they often overlook the severe class imbalance inherent in real-world clinical scenarios, such as clinical referral systems. Institutions in these systems encounter drastic fluctuations in class priors, resulting in label distribution shift, a critical form of domain shift that triggers severe catastrophic forgetting. To address these challenges, we propose ToRe, a rehearsal-free and parameter-efficient framework that leverages frozen ophthalmic foundation models for robust incremental adaptation. ToRe employs a parameter isolation strategy to decouple domain-specific optimization paths, thereby helping mitigate catastrophic forgetting driven by both label distribution shift and style variations. Simultaneously, it introduces token-adaptive recursion that adaptively allocates additional computational depth across tokens, allowing simple tokens to exit the recursion loop early while subjecting complex tokens, such as those associated with lesions, to deeper recursive processing. This mechanism enhances the feature representations for minority classes, thereby supporting generalization throughout the domain incremental learning process. Extensive evaluations on nine heterogeneous datasets demonstrate that ToRe consistently outperforms state-of-the-art methods in overall performance across the three benchmarks, while maintaining near-zero forgetting. Together, these results support the applicability of ToRe to dynamic and imbalanced clinical environments. The code is available at https://github.com/Nancyolo/ToRe
Figures & tables
Fig. 1: Conceptual overview of ToRe for imbalanced ophthalmic DIL. (A) Challenges : Fluctuating class priors across screening and referral institutions create severe imbalance and label distribution shift (LDS), leading to severe catastrophic forgetting. (B) Parameter Isolation : Domain-specific paths are optimized independently to reduce interference driven by LDS and style variations. (C) Token-Adaptive Recursion : Breaking the uniform depth constraint of standard Vision Transformers, this mechanism adaptively increases computational depth for complex tokens, such as those associated with small lesions, to support representation learning for minority classes.
Fig. 2: Training Process of ToRe. (Top) Sequential Domain Stream: The model learns from a sequence of domains ( D1…Dt ) via parameter isolation . For each new domain Dt , a specific set of modules (Gating Module Gt , LoRA Lt , and Classification head Ct ) is initialized and trained, whereas previous modules remain frozen. After training, the domain-specific module and its domain key are registered in the module bank. (Middle) Training at Dt : The left panel illustrates the model architecture for domain Dt . The right panel unrolls a single recursive block with recursion steps 1–3 ( Kmax=3 ). (Bottom) Visualization of token dynamics in a single recursive block: At each recursion step, the gating module selects tokens for recursive processing. Selected tokens receive recursive feature updates, while unselected tokens retain their current states.
Fig. 3: Inference Flow with Domain Identification. At inference, a query vector is extracted by the frozen backbone fΦ and matched against pre-computed domain keys ( k1…kT ) in the module bank via cosine similarity. The module ( Θt^={Gt^,Lt^,Ct^} ) corresponding to the highest similarity score is retrieved and injected to perform the final diagnostic prediction.
Per-Domain Accuracy (%)
Overall Metrics
Method
APTOS
Messidor-2
IDRiD
Avg.ACC (%) ↑
Avg. F1 (%) ↑
Avg. BAcc (%) ↑
Avg. MCC ↑
Forgetting (%) ↓
FT-Seq
60.79 (1.48)
66.22 (0.48)
52.56 (2.42)
59.86 (0.30)
46.40 (1.45)
46.75 (1.31)
0.40 (0.01)
13.25 (1.07)
EWC ( Kirkpatrick et al., 2017 )
64.24 (5.02)
63.44 (1.81)
52.56 (7.22)
60.08 (4.19)
44.12 (0.44)
44.16 (0.41)
0.41 (0.04)
10.49 (3.41)
LwF ( Li and Hoiem, 2017 )
68.18 (7.27)
65.02 (1.33)
49.68 (3.89)
60.96 (3.74)
41.71 (3.83)
42.80 (3.68)
0.41 (0.04)
10.78 (0.51)
L2P ( Wang et al., 2022c )
72.67 (0.95)
59.95 (0.61)
51.28 (1.47)
61.30 (0.46)
37.40 (1.06)
37.80 (0.93)
0.36 (0.02)
3.16 (0.26)
DualPrompt ( Wang et al., 2022b )
64.21 (3.99)
62.42 (0.94)
55.13 (2.42)
60.59 (1.99)
41.04 (3.18)
42.05 (2.28)
0.37 (0.03)
7.26 (2.18)
Table 2: Performance comparison on the DR Grading benchmark. Per-domain accuracy and overall metrics are reported as mean (std) over three seeds. Best results are in bold , second best are underlined .
Benchmark
Dataset
Total used
Training
Validation
Testing
DR
APTOS 2019
3,662
2,048
514
1,100
Messidor-2
1,744
972
246
526
IDRiD
516
329
84
103
AMD
ODIR
2,497
1,798
199
500
ADAM
800
361
39
400
HYAMD
1,214
866
95
253
Table 3: Numbers of samples in the training, validation, and test partitions. Counts denote fundus images for DR and AMD and individual B-scans for OCT.
Per-Domain Accuracy (%)
Overall Metrics
Method
ODIR
ADAM
HYAMD
Avg.ACC (%) ↑
Avg. F1 (%) ↑
Avg. BAcc (%) ↑
Avg. MCC ↑
Forgetting (%) ↓
FT-Seq
86.87 (4.44)
81.67 (3.45)
67.06 (2.97)
78.53 (0.65)
61.35 (15.44)
67.12 (14.88)
0.29 (0.25)
8.57 (0.62)
EWC ( Kirkpatrick et al., 2017 )
90.13 (4.82)
84.67 (3.21)
65.88 (5.16)
80.23 (2.67)
62.62 (7.15)
61.82 (5.46)
0.34 (0.08)
4.47 (2.97)
LwF ( Li and Hoiem, 2017 )
92.87 (1.33)
84.00 (4.39)
66.80 (1.04)
81.22 (1.73)
66.23 (11.86)
65.01 (9.08)
0.41 (0.15)
5.38 (2.63)
L2P ( Wang et al., 2022c )
90.60 (0.20)
82.42 (3.51)
66.93 (0.82)
79.98 (1.35)
55.72 (4.72)
56.23 (3.06)
0.21 (0.07)
1.46 (1.44)
DualPrompt ( Wang et al., 2022b )
91.53 (0.12)
81.08 (3.61)
64.95 (0.60)
79.19 (1.42)
51.89 (5.98)
54.11 (3.21)
0.15 (0.07)
1.37 (1.08)
Table 4: Performance comparison on the AMD Grading benchmark. Per-domain accuracy and overall metrics are reported as mean (std) over three seeds. Best results are in bold , second best are underlined .
Fig. 4: Per-domain Macro-F1 scores on the DR and AMD benchmarks, evaluated after the final task. (Left) Comparison on the DR domains. (Right) Comparison on the AMD domains.
Setting
Avg. ACC (%) ↑
Avg. F1 (%) ↑
Avg. BAcc (%) ↑
Avg. MCC ↑
w/o gate, 1 step
65.34 (1.27)
49.13 (2.43)
48.49 (2.43)
0.47 (0.02)
w/o gate, 3 steps
66.61 (2.16)
48.26 (2.09)
47.85 (1.58)
0.49 (0.03)
w/ gate, 3 steps
67.55 (1.63)
53.55 (2.40)
52.60 (2.30)
0.50 (0.02)
Full ToRe (+IE)
68.98 (1.68)
56.43 (1.68)
55.25 (1.93)
0.52 (0.02)
Table 6: Ablation study of token-adaptive recursion on the DR test set. Results are reported as mean (std) over three seeds. IE denotes iteration embedding. Best means are in bold ; second-best means are underlined .
Kmax
Avg. ACC (%) ↑
Avg. F1 (%) ↑
Avg. BAcc (%) ↑
Avg. MCC ↑
1
66.77 (0.67)
52.48 (0.50)
51.74 (0.86)
0.49 (0.01)
3
68.98 (1.68)
56.43 (1.68)
55.25 (1.93)
0.52 (0.02)
5
67.98 (2.50)
53.82 (3.65)
53.88 (3.78)
0.51 (0.03)
10
67.66 (0.60)
52.91 (0.88)
52.79 (0.98)
0.50 (0.01)
Table 7: Sensitivity to maximum recursion depth Kmax on the DR test set. Results are reported as mean (std) over three seeds. Best means are in bold ; second-best means are underlined .
Fig. 5: Sensitivity of Avg. F1 and Avg. BAcc to LoRA rank on the DR test set over three seeds. Shaded bands show one standard deviation.
Fig. 6: Sensitivity of Avg. F1 and Avg. BAcc to insertion depth on the DR test set over three seeds. Shaded bands show one standard deviation.
Fig. 7: Row-normalized domain-routing confusion matrices (%) on the (a) DR, (b) AMD, and (c) OCT test sets. Rows indicate true domains, and columns indicate predicted domains, in the same domain order within each panel.
Fig. 8: Evolution of active-token selection across recursive steps. Rows (a) and (b) show two examples. Columns 1–3 show active-token overlays at Steps 1–3, with the number of selected tokens above each image; Columns 4–6 show the corresponding gating activation maps.
Benchmark
Misrouted (%)
Pred. Acc. (%)
True Acc. (%)
Δ (pp)
DR
7.80
63.21 (1.13)
63.70 (1.48)
0.49 (2.60)
AMD
13.70
89.87 (1.27)
90.30 (0.73)
0.42 (0.73)
OCT
1.33
56.48 (11.23)
85.19 (3.21)
28.70 (11.23)
Table 8: Diagnostic performance on misrouted test images under the predicted and true-domain routes. Results are reported as mean (std) over three seeds. Misrouted (%) indicates the percentage of each benchmark’s test set assigned to an incorrect domain. Pred. and True denote the predicted and true-domain routes, respectively. Acc. denotes ACC for DR and AMD and subset accuracy for OCT, which requires all three biomarker labels to be correct. Δ denotes true-domain minus predicted-route accuracy in percentage points.
Benchmark
All tokens
Selected tokens
DR
1.37 (0.32)
2.70 (0.08)
AMD
1.15 (0.08)
2.69 (0.08)
OCT
1.27 (0.45)
2.61 (0.21)
Table 9: Mean recursion depth on the DR, AMD, and OCT test sets. Results are reported as mean (standard deviation) over three seeds.
Method
Params
FLOPs
Throughput
Memory
FT-Seq
303.31
119.61
131.36
1.890
EWC
303.31
119.61
103.82
2.649
LwF
303.31
119.61
125.97
1.896
L2P
0.12
279.27
51.35
3.441
DualPrompt
0.97
247.01
52.78
3.440
S-Prompts
0.09
259.79
49.57
1.487
Table 10: Parameter and computational cost comparison. Computational costs are means over three seeds, averaged equally across DR, AMD, and OCT; † indicates averages over DR and AMD only. Params denotes the reported trainable parameters (M), FLOPs is estimated per image (G/image), Throughput is reported in images/s, and Memory denotes peak allocated GPU memory (GiB).
Domain Incremental Learning is a critical scenario that requires models to continuously adapt to new data domains without retraining. However, domain shifts often cause severe performance degradation. To address this, we propose Hybrid Energy-Distance Prompt, a domain-incremental framework inspired by Helmholtz free energy. HEDP introduces an energy regularization loss to enhance the separability of domain representations and a hybrid energy-distance weighted mechanism that fuses energy-based and distance-based cues to improve domain selection and generalization. Experiments on multiple benchmarks, including CORe50, show that HEDP achieves superior performance on unseen domains with a 2.57% accuracy gain, effectively mitigating catastrophic forgetting and enhancing open-world adaptability. Our code is available here.
Yu Feng, Zhen Tian, Haoran Luo +8
China Mobile Research Institute, China · Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore +1
In this work we introduce a novel approach to domain incremental learning, adapting models over time to evolving, non-stationary data. In contrast to other works, we do not attempt to avoid catastrophic forgetting, but rather allow it and exploit it. Our model combines a main task head with a self-supervised masked autoencoder (MAE) head. We then learn domain-specific LoRA adapters during incremental training. Each adapter specializes to its domain, naturally inducing forgetting on other domains in both heads. At inference, we perform online test-time training on the self-supervised MAE head to identify which LoRAs best matches the current input, so the model can `remember' the domain again. Our scheme is especially well-suited to real-world streaming data, such as video, where consecutive samples are highly correlated and domain shifts are gradual. We demonstrate our method on domain-incremental action recognition and semantic segmentation tasks.
Deep neural networks excel in various tasks but struggle to generalize across evolving data distributions, leading to significant performance degradation under domain shifts. Domain incremental learning (DIL) addresses this challenge by enabling models to continuously adapt while retaining prior knowledge. Among existing DIL approaches, the parameter-isolation paradigm achieves state-of-the-art performance. However, these methods often adopt a one-size-fits-all approach to adapt to new domains, resulting in either insufficient learning capacity or redundant parameters. In this work, we propose BPG, a unified framework that addresses both challenges through two complementary components: BPG-Adapter, which dynamically determines each domain's adapter hidden dimension based on domain-specific feature separability, and BPG-Inference, a soft domain mixture strategy that integrates multiple domain-specific models at test time, mitigating domain ID misselection. Experimental results on DomainNet, CDDB, and CORe50 demonstrate that BPG consistently outperforms uniform adapter-based approaches and hard domain selection strategies, achieving state-of-the-art average accuracy while reducing forgetting to as low as 0.22% on DomainNet.
Qiang Wang, Songlin Dong, Shaokun Wang +5
College of Artificial Intelligence, Xi’an Jiaotong University, Xi’an, China · Shenzhen University of Advanced Technology, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China +1