Brain lesion segmentation is a fundamental task in medical image analysis, playing a critical role in diagnosis, treatment planning, and longitudinal disease monitoring. Yet existing models still struggle to meet the demands of real clinical use, where deployments contain data distribution shifts, arising from differences in scanner hardware, imaging protocol (varying MRI modality sets), and new pathologies. We present BrainIAC (Brain lesion Interactive Adaptive Continuously learning segmentation), a unified framework that integrates (i) a multi-modal backbone network trained to segment multiple types of brain lesions and handle heterogeneous sets of modalities via zero-filling and random modality dropping; (ii) 3D interactive segmentation with bounding-box and click prompts that preserves fully automatic prediction when no prompt is given; and (iii) an online adaptation mechanism combining Mid-Interaction adaptation and Post-Interaction adaptation, supervised by the network's own predictions as pseudo labels and guided by an extra Click-Centered Gaussian loss. To our knowledge, this is one of the first 3D online adaptation methods for interactive segmentation, and the first to combine handling of heterogeneous modality sets with online adaptation. Experiments across seven brain MRI datasets demonstrate that the proposed components provide complementary and synergistic benefits. The method consistently outperforms existing approaches and generalizes well across heterogeneous imaging modalities, including those unseen during training, as well as previously unseen brain pathology types. The code and a 3D Slicer plug-in will be released at https://github.com/WenTXuL/BrainIAC upon publication.
Figures & tables
Figure 1: Overview of BrainIAC.Pretraining : Simulated clicks and BBoxes are encoded in channels concatenated with the image for model training. Each modality is placed in its corresponding channel, and during training, modalities are randomly dropped to improve robustness across modality combinations. Inference and adaptation operate on a sequence of samples. Each sample includes multiple modalities. For each sample, the user iteratively provides one BBox for initialization and multiple clicks to correct the segmentation (one prompt per iteration), until the user is satisfied and the final prediction Pfinal is obtained. In our experiments, the number of clicks T is fixed for simplicity. The adaptation procedures follow our previous work ( Xu et al., 2026 ) , applied here to 3D segmentation. In mid-interaction adaptation , the prediction Pt′ obtained with the new corrective click lt supplies pseudo supervision for the pre-correction prediction Pt−1 . In post-interaction adaptation , the final corrected segmentation Pfinal supervises two fine-tuning stages.
Figure 2: Illustration of how the predicted segmentation from BrainIAC evolves as the BBox and interaction clicks are provided.The examples are, from top to bottom, MSSEG, TBI, ISLES and VES-SEG. From left to right: ground truth, automated prediction, BBox only, BBox+1 click, BBox+6 click, and the final prediction. Positive clicks are shown in yellow, negative clicks in purple, and clicks are accumulated across interactions. Regions in red denote the current prediction, while regions highlighted in green indicate improvements resulting from the current prompt. The results demonstrate progressive refinement of the segmentation as more prompts are provided.
Main settings
Click-only
Method
Modalities
No Prompt
BBox
BBox+1
BBox+6
Pts-1
Pts-6
BrainIAC (ours) - No Online
FLAIR
53.6
57.3
60.7
68.5
56.2
67.5
BrainIAC (ours) - Online
FLAIR
64.7
66.2
68.1
72.8
66.3
72.5
BrainIAC (ours) - No Online
FLAIR, T1
55.0
59.1
62.0
69.6
57.3
68.2
BrainIAC (ours) - Online
FLAIR, T1
66.1
66.9
68.7
73.3
67.2
72.9
TSCA
FLAIR
60.8
63.1
66.0
72.0
63.8
71.5
Table 1: Results on MSSEG in the held-out setting, where MSSEG is excluded from training. Dice score (%). The main interaction setting starts from automatic prediction and is followed by bounding-box initialization and iterative correction ( No Prompt , BBox , BBox+1 , BBox+6 ). Click-only results ( Pts-1 , Pts-6 ) are also reported for comparison with methods that require click-based initialization. “BrainIAC (ours) - No Online” refers to the backbone 3D interactive model which does not include online adaptation, while “BrainIAC (ours) - Online” includes online adaptation. The lower block reports the Top-5 BBox setting, in which the five largest lesion components are each given a bounding box at initialization. Best result in each column is shown in bold .
Method
Modalities
No Prompt
BBox
BBox+1
BBox+6
FLAIR-visible lesions:
BrainIAC (ours) – No Online
FLAIR
37.9
39.5
41.1
47.9
BrainIAC (ours) – Online
FLAIR
44.3
50.5
54.3
62.2
BrainIAC (ours) – No Online
FLAIR, T1, T2
41.5
43.7
46.1
52.4
BrainIAC (ours) – Online
FLAIR, T1, T2
46.6
52.6
55.8
62.7
TSCA
FLAIR
37.3
41.3
45.2
55.3
Table 2: Results on TBI. Dice score (%). FLAIR-visible lesions : only lesions visible on FLAIR are evaluated, providing the fairest comparison since all methods receive FLAIR as input. All lesions : lesions visible on SWI are also included, although SWI is not provided as an input modality, making the task more challenging. Best result in each column (within each setting) is shown in bold .
Figure 3: TBI sample and three-dimensional visualisation of the progressive refinement of the predicted segmentation with a BBox and 12 subsequent interaction clicks, which alternate between the largest global segmentation error and the largest remaining unsegmented SWI lesion component. The top row shows results obtained with our model using T1, T2, and FLAIR as inputs during online adaptation. The bottom row shows the corresponding results when SWI (highlighted as blue in ground truth) is additionally incorporated through an agnostic input channel. Positive clicks in yellow, negative clicks in purple, clicks accumulated across interactions. Red denotes current prediction. Blue highlights correctly predicted SWI components. Green highlights correction of predicted FLAIR label at current interaction step. This iterative interaction strategy progressively refines the segmentation and improves lesion segmentation.
Method
Channels
Auto
BBox
BBox+1
BBox+6
BrainIAC (ours) – Online
FLAIR
42.0
46.9
50.6
58.0
BrainIAC (ours) – No Online
FLAIR, T1, T2
38.4
38.5
40.5
46.4
BrainIAC (ours) – Online
FLAIR, T1, T2
43.6
49.2
52.2
58.7
BrainIAC (ours) + Extra Ch. – Online
FLAIR,SWI
43.7
44.0
49.0
57.7
BrainIAC (ours) + Extra Ch. – Online
FLAIR,T1,T2,SWI
44.1
49.2
52.5
58.7
BrainIAC (ours) + Agn. – Online
FLAIR,SWI
44.6
49.4
53.8
61.0
Table 3: Evaluation on unseen modalities for TBI. Dice score (%). This table reports results evaluated with labels containing both lesions visible on FLAIR and lesions visible on SWI, while using SWI as an extra input to the model. It also reports results from using multiple nnInteractive models for different modalities and combining the result. Best result in each section is shown in bold . (Extra Ch.: Extra channel initiated, Agn.: Agnostic channel initiated)
Method
Automated
BBox
BBox+1
BBox+6
BBox+12
Standard 1 BBox+6 clicks (largest error)
BrainIAC (ours) – Online
35.4(43.6)
36.3(49.2)
37.5(52.2)
38.6(58.7)
-
BrainIAC (ours) + Extra Ch. – Online
35.9(44.1)
36.5(49.2)
37.3(52.5)
38.8(58.7)
-
BrainIAC (ours) + Agn. – Online
47.9 (46.3)
48.0 (50.4)
48.4 (53.5)
49.8 (61.0)
-
1 BBox+12 alternate clicks (largest error then largest unpredicted SWI only lesion)
BrainIAC (ours) – Online
46.5(45.8)
47.3(50.4)
47.6(52.5)
50.5(56.6)
53.1(59.2)
Table 4: Percentage of detected SWI components across the entire TBI dataset for different click strategies (Overall Dice score (%) in bracket). All experiments used T1, T2 and FLAIR. SWI was included in experiments involving the additional channel or the pretrained modality-agnostic channel. Best results highlighted in bold for each click strategy. (Extra Ch.: Extra channel initiated, Agn.: Agnostic channel initiated)
Figure 4: Results on VES-SEG T1C (Dice %) along the interaction protocol: automatic prediction (No Prompt), bounding-box initialisation (BBox), and one to six correction clicks (BBox +1 to BBox +6 ). Online adaptation lifts the entire trajectory far above the no-adaptation variant and the TSCA and IA+SA online-adaptation baselines.
Figure 5: Results on ISLES (Dice %, FLAIR) along the interaction protocol (No Prompt, BBox, and BBox +1 to BBox +6 correction clicks). BrainIAC with online adaptation consistently surpasses the no-adaptation variant and the TSCA and IA+SA baselines across the whole trajectory.
Method
Interact.
MSSEG
TBI
WMH
ATLAS
BraTS
Ours
No Prompt
74.4
59.9
78.0
58.2
92.5
BBox
74.3
63.6
78.1
68.0
92.6
BBox+1
75.0
64.5
78.4
70.5
92.7
BBox+6
77.7
66.9
79.6
75.1
93.2
nnU-Net
No Prompt
74.2
59.0
81.0
60.8
92.4
nnInter.
BBox+6
–
–
56.9
71.5
–
Table 5: In-distribution results (5-DB training, C=6 modality channels, no online adaptation). Dice (%). Best results for each database in bold .
MI
PI
No Prompt
BBox
BBox+1
BBox+6
✗
✗
37.9
39.5
41.1
47.9
✓
✗
43.9
49.1
53.2
61.6
✗
✓
42.2
46.8
50.9
59.1
✓
✓
44.3
50.5
54.3
62.2
Table 6: Ablation study of online adaptation components on TBI (FLAIR-visible lesions). MI = Mid-Interaction adaptation, PI = Post-Interaction adaptation. Dice score (%). Best result in bold .
Avg clicks
Reached threshold (%)
Dataset
w/ Online
w/o Online
w/ Online
w/o Online
MSSEG
3.81
6.66
96.2
71.7
VES-SEG
2.67
12.81
90.1
31.8
ISLES
3.43
5.46
85.7
64.3
TBI
6.78
10.35
69.8
39.9
Table 7: Click efficiency at fixed Dice thresholds. For each database we report the fraction of cases that reach the target Dice within a 15-click budget and the average number of correction clicks per case (cases that never reach the threshold are counted as 15 clicks). Thresholds: MSSEG = 0.70, VES-SEG T1C = 0.80, ISLES = 0.65, TBI FLAIR = 0.65. Best result in each row is shown in bold .
Metric
w/ Online
w/o Online
10-click budget (max 10 clicks/case)
Avg. undetected – pre-clicks
8.26
23.75
Avg. undetected – post-clicks
2.51
13.87
Avg. clicks used
4.00
7.96
Unlimited clicks (until all detected)
Avg. undetected – pre-clicks
5.75
23.75
Table 8: Undetected lesion-component analysis on the held-out MSSEG dataset (53 subjects). Each correction click is placed inside a currently undetected ground-truth component. We report the average number of undetected lesion components pre and post-clicks, together with the average clicks used under both a 10-click budget and under an unlimited-click setting. With unlimited clicks every component is eventually recovered (zero undetected) for both models, but online adaptation reaches this with far fewer clicks.
Figure 6: Online-adaptation trajectory on (a) VES-SEG and (b) TBI. The x -axis is the number of adaptation images processed so far, and the y -axis is the average Dice on the held-out evaluation split
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
Accurate brain lesion segmentation in MRI is vital for effective clinical diagnosis and treatment planning. Due to high annotation costs and strict data privacy regulations, universal models require employing Continual Learning (CL) to adapt to evolving clinical tasks without losing previously acquired knowledge. However, existing CL paradigms often suffer from capacity limits or redundant parameter growth, and even advanced dynamic methods rely mostly on image-perception strategies that struggle to handle the substantial pathological and multimodal heterogeneity inherent in brain imaging. To address this issue, we propose Concept-Reasoning Expansion (CoRE) framework, which establishes a joint decision-making mechanism by integrating visual features with structured concepts. Through the alignment of image tokens with a hierarchical concept library, CoRE simulates clinical reasoning to guide both interpretable expert routing and demand-based model growth. This collaborative process ensures model evolution is grounded in clinical priors, preventing redundant parameter expansion while maximizing knowledge reuse. Extensive evaluations across 12 sequential brain lesion MRI tasks demonstrate that CoRE achieves state-of-the-art performance and provides a high knowledge starting point for efficient future adaptation. Its superior few-shot transferability and clinical interpretability further validate its effectiveness in managing non-stationary clinical data streams. Our code will be released soon.
Qianqian Chen, Anglin Liu, Jingyang Zhang +1
Southeast University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Different MRI modalities reveal different parts of a brain tumor. Segmentation models combine information from these modalities to outline tumor regions. Many models use a separate encoder for each modality. However, exchanging features among these encoders at several stages can increase model complexity. We introduce the Block-Wise Shared Adapter (BWSA), which combines compressed features from the modality encoders in a shared path and returns adapted features to each stream. In its fixed-blending form, BWSA adds the adapted features to the original features. Its trainable-gated variant (TG-BWSA) instead learns separate weights for the original and adapted features at each modality and encoder stage. We evaluate both variants on BraTS 2020 and BraTS 2023 with four segmentation backbones. On the official BraTS 2023 online validation set, BWSA increases average region-wise Dice by 1.3--7.6 percentage points relative to the corresponding unmodified backbones. TG-BWSA increases it by 2.8--7.7 points. Both variants add approximately 0.85--3.73% parameters and reduce average HD95 relative to the unmodified backbones.
School of Computer Engineering, Iran University of Science and Technology (IUST), Tehran, Iran · Computer Vision Laboratory, ETH Zurich, 8092 Zurich, Switzerland
Multimodal MRI provides complementary information for brain tumor segmentation, but high accuracy alone does not reveal whether a model relies on clinically appropriate imaging evidence. Radiologists use different sequences for different findings, such as T1ce for enhancing tumor and FLAIR for peritumoral edema. Yet most 3D segmentation networks fuse MRI sequences through fixed connections, regardless of patient, region, or depth. We frame multimodal segmentation as grounded multimodal reasoning and introduce (i) a model agnostic probing framework combining per region modality ablation, Dominant Sensitivity Enhancement (DSE), and gate trajectory analysis to reveal sequence reliance, and (ii) evidence that adaptive Hyper-Connections (HC) induce clinically aligned modality reliance without modality supervision. Across nnU Net, SwinUNETR, and VT-UNet on BraTS 2021, most HC configurations increase reliance on clinically appropriate sequences. This interpretable pattern recurs across all three backbones, suggesting it arises from adaptive connections rather than a single architecture. HC also improves segmentation accuracy at negligible parameter cost. On validation, the best HC configuration increases mean Dice by 0.43 to 0.81 percentage points across the three backbones. On the held-out test set, it reduces mean HD95 by 1.17 to 1.78 mm (approximately 20 to 27 percent) for nnU-Net and VT UNet, adding at most 0.53 percent parameters. Our framework offers an inspectable measure of clinical grounding and extends naturally to multimodal medical foundation models.