Learning When to Recur: Token-Adaptive Recursion for Imbalanced Ophthalmic Domain Incremental Learning

Nanxi Yu; Kang Li; Ye Du; Xiaowei Hu; Weihua Yang; Shujun Wang

cs.LGSep 26, 2026

Learning When to Recur: Token-Adaptive Recursion for Imbalanced Ophthalmic Domain Incremental Learning

Authors: Nanxi Yu, Kang Li, Ye Du, Xiaowei Hu, Weihua Yang, Shujun Wang

Organizations: School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Sichuan, China · Department of Biomedical Engineering and Sports Technology, The Hong Kong Polytechnic University, Hong Kong SAR, China · School of Future Technology, South China University of Technology, Guangdong, China · Shenzhen Eye Hospital, Shenzhen Eye Medical Center, Southern Medical University, Guangdong, China

Abstract

Domain incremental learning is essential for adapting ophthalmic deep learning models to sequential clinical domains while preserving diagnostic expertise. Existing domain incremental learning methods predominantly address the domain shift induced by style variations. However, they often overlook the severe class imbalance inherent in real-world clinical scenarios, such as clinical referral systems. Institutions in these systems encounter drastic fluctuations in class priors, resulting in label distribution shift, a critical form of domain shift that triggers severe catastrophic forgetting. To address these challenges, we propose ToRe, a rehearsal-free and parameter-efficient framework that leverages frozen ophthalmic foundation models for robust incremental adaptation. ToRe employs a parameter isolation strategy to decouple domain-specific optimization paths, thereby helping mitigate catastrophic forgetting driven by both label distribution shift and style variations. Simultaneously, it introduces token-adaptive recursion that adaptively allocates additional computational depth across tokens, allowing simple tokens to exit the recursion loop early while subjecting complex tokens, such as those associated with lesions, to deeper recursive processing. This mechanism enhances the feature representations for minority classes, thereby supporting generalization throughout the domain incremental learning process. Extensive evaluations on nine heterogeneous datasets demonstrate that ToRe consistently outperforms state-of-the-art methods in overall performance across the three benchmarks, while maintaining near-zero forgetting. Together, these results support the applicability of ToRe to dynamic and imbalanced clinical environments. The code is available at https://github.com/Nancyolo/ToRe

Figures & tables

Fig. 1: Conceptual overview of ToRe for imbalanced ophthalmic DIL. (A) Challenges : Fluctuating class priors across screening and referral institutions create severe imbalance and label distribution shift (LDS), leading to severe catastrophic forgetting. (B) Parameter Isolation : Domain-specific paths are optimized independently to reduce interference driven by LDS and style variations. (C) Token-Adaptive Recursion : Breaking the uniform depth constraint of standard Vision Transformers, this mechanism adaptively increases computational depth for complex tokens, such as those associated with small lesions, to support representation learning for minority classes.

Fig. 2: Training Process of ToRe. (Top) Sequential Domain Stream: The model learns from a sequence of domains ( $\mathcal{D}_{1}\dots\mathcal{D}_{t}$ ) via parameter isolation . For each new domain $\mathcal{D}_{t}$ , a specific set of modules (Gating Module $\mathcal{G}_{t}$ , LoRA $\mathcal{L}_{t}$ , and Classification head $\mathcal{C}_{t}$ ) is initialized and trained, whereas previous modules remain frozen. After training, the domain-specific module and its domain key are registered in the module bank. (Middle) Training at $\mathcal{D}_{t}$ : The left panel illustrates the model architecture for domain $\mathcal{D}_{t}$ . The right panel unrolls a single recursive block with recursion steps 1–3 ( $K_{\max}=3$ ). (Bottom) Visualization of token dynamics in a single recursive block: At each recursion step, the gating module selects tokens for recursive processing. Selected tokens receive recursive feature updates, while unselected tokens retain their current states.

Fig. 3: Inference Flow with Domain Identification. At inference, a query vector is extracted by the frozen backbone $f_{\Phi}$ and matched against pre-computed domain keys ( $k_{1}\dots k_{T}$ ) in the module bank via cosine similarity. The module ( $\Theta_{\hat{t}}=\{\mathcal{G}_{\hat{t}},\mathcal{L}_{\hat{t}},\mathcal{C}_{\hat{t}}\}$ ) corresponding to the highest similarity score is retrieved and injected to perform the final diagnostic prediction.

	Per-Domain Accuracy (%)			Overall Metrics
Method	APTOS	Messidor-2	IDRiD	Avg.ACC (%) $\uparrow$	Avg. F1 (%) $\uparrow$	Avg. BAcc (%) $\uparrow$	Avg. MCC $\uparrow$	Forgetting (%) $\downarrow$
FT-Seq	60.79 (1.48)	66.22 (0.48)	52.56 (2.42)	59.86 (0.30)	46.40 (1.45)	46.75 (1.31)	0.40 (0.01)	13.25 (1.07)
EWC ( Kirkpatrick et al., 2017 )	64.24 (5.02)	63.44 (1.81)	52.56 (7.22)	60.08 (4.19)	44.12 (0.44)	44.16 (0.41)	0.41 (0.04)	10.49 (3.41)
LwF ( Li and Hoiem, 2017 )	68.18 (7.27)	65.02 (1.33)	49.68 (3.89)	60.96 (3.74)	41.71 (3.83)	42.80 (3.68)	0.41 (0.04)	10.78 (0.51)
L2P ( Wang et al., 2022c )	72.67 (0.95)	59.95 (0.61)	51.28 (1.47)	61.30 (0.46)	37.40 (1.06)	37.80 (0.93)	0.36 (0.02)	3.16 (0.26)
DualPrompt ( Wang et al., 2022b )	64.21 (3.99)	62.42 (0.94)	55.13 (2.42)	60.59 (1.99)	41.04 (3.18)	42.05 (2.28)	0.37 (0.03)	7.26 (2.18)

Table 2: Performance comparison on the DR Grading benchmark. Per-domain accuracy and overall metrics are reported as mean (std) over three seeds. Best results are in bold , second best are underlined .

Benchmark	Dataset	Total used	Training	Validation	Testing
DR	APTOS 2019	3,662	2,048	514	1,100
	Messidor-2	1,744	972	246	526
	IDRiD	516	329	84	103
AMD	ODIR	2,497	1,798	199	500
	ADAM	800	361	39	400
	HYAMD	1,214	866	95	253

Table 3: Numbers of samples in the training, validation, and test partitions. Counts denote fundus images for DR and AMD and individual B-scans for OCT.

	Per-Domain Accuracy (%)			Overall Metrics
Method	ODIR	ADAM	HYAMD	Avg.ACC (%) $\uparrow$	Avg. F1 (%) $\uparrow$	Avg. BAcc (%) $\uparrow$	Avg. MCC $\uparrow$	Forgetting (%) $\downarrow$
FT-Seq	86.87 (4.44)	81.67 (3.45)	67.06 (2.97)	78.53 (0.65)	61.35 (15.44)	67.12 (14.88)	0.29 (0.25)	8.57 (0.62)
EWC ( Kirkpatrick et al., 2017 )	90.13 (4.82)	84.67 (3.21)	65.88 (5.16)	80.23 (2.67)	62.62 (7.15)	61.82 (5.46)	0.34 (0.08)	4.47 (2.97)
LwF ( Li and Hoiem, 2017 )	92.87 (1.33)	84.00 (4.39)	66.80 (1.04)	81.22 (1.73)	66.23 (11.86)	65.01 (9.08)	0.41 (0.15)	5.38 (2.63)
L2P ( Wang et al., 2022c )	90.60 (0.20)	82.42 (3.51)	66.93 (0.82)	79.98 (1.35)	55.72 (4.72)	56.23 (3.06)	0.21 (0.07)	1.46 (1.44)
DualPrompt ( Wang et al., 2022b )	91.53 (0.12)	81.08 (3.61)	64.95 (0.60)	79.19 (1.42)	51.89 (5.98)	54.11 (3.21)	0.15 (0.07)	1.37 (1.08)

Table 4: Performance comparison on the AMD Grading benchmark. Per-domain accuracy and overall metrics are reported as mean (std) over three seeds. Best results are in bold , second best are underlined .

Fig. 4: Per-domain Macro-F1 scores on the DR and AMD benchmarks, evaluated after the final task. (Left) Comparison on the DR domains. (Right) Comparison on the AMD domains.

Setting	Avg. ACC (%) $\uparrow$	Avg. F1 (%) $\uparrow$	Avg. BAcc (%) $\uparrow$	Avg. MCC $\uparrow$
w/o gate, 1 step	65.34 (1.27)	49.13 (2.43)	48.49 (2.43)	0.47 (0.02)
w/o gate, 3 steps	66.61 (2.16)	48.26 (2.09)	47.85 (1.58)	0.49 (0.03)
w/ gate, 3 steps	67.55 (1.63)	53.55 (2.40)	52.60 (2.30)	0.50 (0.02)
Full ToRe (+IE)	68.98 (1.68)	56.43 (1.68)	55.25 (1.93)	0.52 (0.02)

Table 6: Ablation study of token-adaptive recursion on the DR test set. Results are reported as mean (std) over three seeds. IE denotes iteration embedding. Best means are in bold ; second-best means are underlined .

$K_{\max}$	Avg. ACC (%) $\uparrow$	Avg. F1 (%) $\uparrow$	Avg. BAcc (%) $\uparrow$	Avg. MCC $\uparrow$
1	66.77 (0.67)	52.48 (0.50)	51.74 (0.86)	0.49 (0.01)
3	68.98 (1.68)	56.43 (1.68)	55.25 (1.93)	0.52 (0.02)
5	67.98 (2.50)	53.82 (3.65)	53.88 (3.78)	0.51 (0.03)
10	67.66 (0.60)	52.91 (0.88)	52.79 (0.98)	0.50 (0.01)

Table 7: Sensitivity to maximum recursion depth

K_{\max}

on the DR test set. Results are reported as mean (std) over three seeds. Best means are in bold ; second-best means are underlined .

Fig. 5: Sensitivity of Avg. F1 and Avg. BAcc to LoRA rank on the DR test set over three seeds. Shaded bands show one standard deviation.

Fig. 6: Sensitivity of Avg. F1 and Avg. BAcc to insertion depth on the DR test set over three seeds. Shaded bands show one standard deviation.

Fig. 7: Row-normalized domain-routing confusion matrices (%) on the (a) DR, (b) AMD, and (c) OCT test sets. Rows indicate true domains, and columns indicate predicted domains, in the same domain order within each panel.

Fig. 8: Evolution of active-token selection across recursive steps. Rows (a) and (b) show two examples. Columns 1–3 show active-token overlays at Steps 1–3, with the number of selected tokens above each image; Columns 4–6 show the corresponding gating activation maps.

Benchmark	Misrouted (%)	Pred. Acc. (%)	True Acc. (%)	$\Delta$ (pp)
DR	7.80	63.21 (1.13)	63.70 (1.48)	0.49 (2.60)
AMD	13.70	89.87 (1.27)	90.30 (0.73)	0.42 (0.73)
OCT	1.33	56.48 (11.23)	85.19 (3.21)	28.70 (11.23)

Table 8: Diagnostic performance on misrouted test images under the predicted and true-domain routes. Results are reported as mean (std) over three seeds. Misrouted (%) indicates the percentage of each benchmark’s test set assigned to an incorrect domain. Pred. and True denote the predicted and true-domain routes, respectively. Acc. denotes ACC for DR and AMD and subset accuracy for OCT, which requires all three biomarker labels to be correct.

\Delta

denotes true-domain minus predicted-route accuracy in percentage points.

Benchmark	All tokens	Selected tokens
DR	1.37 (0.32)	2.70 (0.08)
AMD	1.15 (0.08)	2.69 (0.08)
OCT	1.27 (0.45)	2.61 (0.21)

Table 9: Mean recursion depth on the DR, AMD, and OCT test sets. Results are reported as mean (standard deviation) over three seeds.

Method	Params	FLOPs	Throughput	Memory
FT-Seq	303.31	119.61	131.36	1.890
EWC	303.31	119.61	103.82	2.649
LwF	303.31	119.61	125.97	1.896
L2P	0.12	279.27	51.35	3.441
DualPrompt	0.97	247.01	52.78	3.440
S-Prompts	0.09	259.79	49.57	1.487

Table 10: Parameter and computational cost comparison. Computational costs are means over three seeds, averaged equally across DR, AMD, and OCT;

\dagger

indicates averages over DR and AMD only. Params denotes the reported trainable parameters (M), FLOPs is estimated per image (G/image), Throughput is reported in images/s, and Memory denotes peak allocated GPU memory (GiB).

Ophthalmic Image Analysis

See all 42 papers

HEDP: A Hybrid Energy-Distance Prompt-based Framework for Domain Incremental Learning
May 7, 2026Yu Feng, Zhen Tian, Haoran Luo +8Prompt Learning Domain-Incremental Learning
China Mobile Research Institute, China · Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore +1
Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams
May 29, 2026Jonathan Swinnen, Tinne TuytelaarsDomain-Incremental Learning Test-Time Training
ESAT, KU Leuven, Leuven, Belgium
BPG: Balancing Plasticity and Generalization for Domain Incremental Learning
Aug 11, 2026Qiang Wang, Songlin Dong, Shaokun Wang +5Domain-Incremental Learning Incremental Learning
College of Artificial Intelligence, Xi’an Jiaotong University, Xi’an, China · Shenzhen University of Advanced Technology, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China +1

May 7, 2026J/K move · Enter open · S save

HEDP: A Hybrid Energy-Distance Prompt-based Framework for Domain Incremental Learning

Yu Feng, Zhen Tian, Haoran Luo, Xie Yu+7

China Mobile Research Institute, China · Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore+1

Prompt Learning Domain-Incremental Learning Cross-Domain Generalization Incremental Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Domain Incremental Learning is a critical scenario that requires models to continuously adapt to new data domains without retraining. However, domain shifts often cause severe performance degradation. To address this, we propose Hybrid Energy-Distance Prompt, a domain-incremental framework inspired by Helmholtz free energy. HEDP introduces an energy regularization loss to enhance the separability of domain representations and a hybrid energy-distance weighted mechanism that fuses energy-based and distance-based cues to improve domain selection and generalization. Experiments on multiple benchmarks, including CORe50, show that HEDP achieves superior performance on unseen domains with a 2.57% accuracy gain, effectively mitigating catastrophic forgetting and enhancing open-world adaptability. Our code is available here.

Abstract

Ophthalmic Image Analysis

Explore similar work

HEDP: A Hybrid Energy-Distance Prompt-based Framework for Domain Incremental Learning

Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams

BPG: Balancing Plasticity and Generalization for Domain Incremental Learning