cs.CVSep 19, 2026
SaveBrainIAC: Interactive 3D Brain Lesion Segmentation across Heterogeneous MRI Modalities with Online Adaptation
Organizations: Department of Engineering Science, University of Oxford, UK
Abstract
Brain lesion segmentation is a fundamental task in medical image analysis, playing a critical role in diagnosis, treatment planning, and longitudinal disease monitoring. Yet existing models still struggle to meet the demands of real clinical use, where deployments contain data distribution shifts, arising from differences in scanner hardware, imaging protocol (varying MRI modality sets), and new pathologies. We present BrainIAC (Brain lesion Interactive Adaptive Continuously learning segmentation), a unified framework that integrates (i) a multi-modal backbone network trained to segment multiple types of brain lesions and handle heterogeneous sets of modalities via zero-filling and random modality dropping; (ii) 3D interactive segmentation with bounding-box and click prompts that preserves fully automatic prediction when no prompt is given; and (iii) an online adaptation mechanism combining Mid-Interaction adaptation and Post-Interaction adaptation, supervised by the network's own predictions as pseudo labels and guided by an extra Click-Centered Gaussian loss. To our knowledge, this is one of the first 3D online adaptation methods for interactive segmentation, and the first to combine handling of heterogeneous modality sets with online adaptation. Experiments across seven brain MRI datasets demonstrate that the proposed components provide complementary and synergistic benefits. The method consistently outperforms existing approaches and generalizes well across heterogeneous imaging modalities, including those unseen during training, as well as previously unseen brain pathology types. The code and a 3D Slicer plug-in will be released at https://github.com/WenTXuL/BrainIAC upon publication.
Figures & tables
Figure 1: Overview of BrainIAC.Pretraining : Simulated clicks and BBoxes are encoded in channels concatenated with the image for model training. Each modality is placed in its corresponding channel, and during training, modalities are randomly dropped to improve robustness across modality combinations. Inference and adaptation operate on a sequence of samples. Each sample includes multiple modalities. For each sample, the user iteratively provides one BBox for initialization and multiple clicks to correct the segmentation (one prompt per iteration), until the user is satisfied and the final prediction is obtained. In our experiments, the number of clicks is fixed for simplicity. The adaptation procedures follow our previous work ( Xu et al., 2026 ) , applied here to 3D segmentation. In mid-interaction adaptation , the prediction obtained with the new corrective click supplies pseudo supervision for the pre-correction prediction . In post-interaction adaptation , the final corrected segmentation supervises two fine-tuning stages.
Figure 2: Illustration of how the predicted segmentation from BrainIAC evolves as the BBox and interaction clicks are provided.The examples are, from top to bottom, MSSEG, TBI, ISLES and VES-SEG. From left to right: ground truth, automated prediction, BBox only, BBox+1 click, BBox+6 click, and the final prediction. Positive clicks are shown in yellow, negative clicks in purple, and clicks are accumulated across interactions. Regions in red denote the current prediction, while regions highlighted in green indicate improvements resulting from the current prompt. The results demonstrate progressive refinement of the segmentation as more prompts are provided.
| Main settings | Click-only | ||||||
| Method | Modalities | No Prompt | BBox | BBox+1 | BBox+6 | Pts-1 | Pts-6 |
| BrainIAC (ours) - No Online | FLAIR | 53.6 | 57.3 | 60.7 | 68.5 | 56.2 | 67.5 |
| BrainIAC (ours) - Online | FLAIR | 64.7 | 66.2 | 68.1 | 72.8 | 66.3 | 72.5 |
| BrainIAC (ours) - No Online | FLAIR, T1 | 55.0 | 59.1 | 62.0 | 69.6 | 57.3 | 68.2 |
| BrainIAC (ours) - Online | FLAIR, T1 | 66.1 | 66.9 | 68.7 | 73.3 | 67.2 | 72.9 |
| TSCA | FLAIR | 60.8 | 63.1 | 66.0 | 72.0 | 63.8 | 71.5 |
Table 1: Results on MSSEG in the held-out setting, where MSSEG is excluded from training. Dice score (%). The main interaction setting starts from automatic prediction and is followed by bounding-box initialization and iterative correction ( No Prompt , BBox , BBox+1 , BBox+6 ). Click-only results ( Pts-1 , Pts-6 ) are also reported for comparison with methods that require click-based initialization. “BrainIAC (ours) - No Online” refers to the backbone 3D interactive model which does not include online adaptation, while “BrainIAC (ours) - Online” includes online adaptation. The lower block reports the Top-5 BBox setting, in which the five largest lesion components are each given a bounding box at initialization. Best result in each column is shown in bold .
| Method | Modalities | No Prompt | BBox | BBox+1 | BBox+6 |
| FLAIR-visible lesions: | |||||
| BrainIAC (ours) – No Online | FLAIR | 37.9 | 39.5 | 41.1 | 47.9 |
| BrainIAC (ours) – Online | FLAIR | 44.3 | 50.5 | 54.3 | 62.2 |
| BrainIAC (ours) – No Online | FLAIR, T1, T2 | 41.5 | 43.7 | 46.1 | 52.4 |
| BrainIAC (ours) – Online | FLAIR, T1, T2 | 46.6 | 52.6 | 55.8 | 62.7 |
| TSCA | FLAIR | 37.3 | 41.3 | 45.2 | 55.3 |
Table 2: Results on TBI. Dice score (%). FLAIR-visible lesions : only lesions visible on FLAIR are evaluated, providing the fairest comparison since all methods receive FLAIR as input. All lesions : lesions visible on SWI are also included, although SWI is not provided as an input modality, making the task more challenging. Best result in each column (within each setting) is shown in bold .
Figure 3: TBI sample and three-dimensional visualisation of the progressive refinement of the predicted segmentation with a BBox and 12 subsequent interaction clicks, which alternate between the largest global segmentation error and the largest remaining unsegmented SWI lesion component. The top row shows results obtained with our model using T1, T2, and FLAIR as inputs during online adaptation. The bottom row shows the corresponding results when SWI (highlighted as blue in ground truth) is additionally incorporated through an agnostic input channel. Positive clicks in yellow, negative clicks in purple, clicks accumulated across interactions. Red denotes current prediction. Blue highlights correctly predicted SWI components. Green highlights correction of predicted FLAIR label at current interaction step. This iterative interaction strategy progressively refines the segmentation and improves lesion segmentation.
| Method | Channels | Auto | BBox | BBox+1 | BBox+6 |
| BrainIAC (ours) – Online | FLAIR | 42.0 | 46.9 | 50.6 | 58.0 |
| BrainIAC (ours) – No Online | FLAIR, T1, T2 | 38.4 | 38.5 | 40.5 | 46.4 |
| BrainIAC (ours) – Online | FLAIR, T1, T2 | 43.6 | 49.2 | 52.2 | 58.7 |
| BrainIAC (ours) + Extra Ch. – Online | FLAIR,SWI | 43.7 | 44.0 | 49.0 | 57.7 |
| BrainIAC (ours) + Extra Ch. – Online | FLAIR,T1,T2,SWI | 44.1 | 49.2 | 52.5 | 58.7 |
| BrainIAC (ours) + Agn. – Online | FLAIR,SWI | 44.6 | 49.4 | 53.8 | 61.0 |
Table 3: Evaluation on unseen modalities for TBI. Dice score (%). This table reports results evaluated with labels containing both lesions visible on FLAIR and lesions visible on SWI, while using SWI as an extra input to the model. It also reports results from using multiple nnInteractive models for different modalities and combining the result. Best result in each section is shown in bold . (Extra Ch.: Extra channel initiated, Agn.: Agnostic channel initiated)
| Method | Automated | BBox | BBox+1 | BBox+6 | BBox+12 |
| Standard 1 BBox+6 clicks (largest error) | |||||
| BrainIAC (ours) – Online | 35.4(43.6) | 36.3(49.2) | 37.5(52.2) | 38.6(58.7) | - |
| BrainIAC (ours) + Extra Ch. – Online | 35.9(44.1) | 36.5(49.2) | 37.3(52.5) | 38.8(58.7) | - |
| BrainIAC (ours) + Agn. – Online | 47.9 (46.3) | 48.0 (50.4) | 48.4 (53.5) | 49.8 (61.0) | - |
| 1 BBox+12 alternate clicks (largest error then largest unpredicted SWI only lesion) | |||||
| BrainIAC (ours) – Online | 46.5(45.8) | 47.3(50.4) | 47.6(52.5) | 50.5(56.6) | 53.1(59.2) |
Table 4: Percentage of detected SWI components across the entire TBI dataset for different click strategies (Overall Dice score (%) in bracket). All experiments used T1, T2 and FLAIR. SWI was included in experiments involving the additional channel or the pretrained modality-agnostic channel. Best results highlighted in bold for each click strategy. (Extra Ch.: Extra channel initiated, Agn.: Agnostic channel initiated)
Figure 4: Results on VES-SEG T1C (Dice %) along the interaction protocol: automatic prediction (No Prompt), bounding-box initialisation (BBox), and one to six correction clicks (BBox to BBox ). Online adaptation lifts the entire trajectory far above the no-adaptation variant and the TSCA and IA+SA online-adaptation baselines.
Figure 5: Results on ISLES (Dice %, FLAIR) along the interaction protocol (No Prompt, BBox, and BBox to BBox correction clicks). BrainIAC with online adaptation consistently surpasses the no-adaptation variant and the TSCA and IA+SA baselines across the whole trajectory.
| Method | Interact. | MSSEG | TBI | WMH | ATLAS | BraTS |
| Ours | No Prompt | 74.4 | 59.9 | 78.0 | 58.2 | 92.5 |
| BBox | 74.3 | 63.6 | 78.1 | 68.0 | 92.6 | |
| BBox+1 | 75.0 | 64.5 | 78.4 | 70.5 | 92.7 | |
| BBox+6 | 77.7 | 66.9 | 79.6 | 75.1 | 93.2 | |
| nnU-Net | No Prompt | 74.2 | 59.0 | 81.0 | 60.8 | 92.4 |
| nnInter. | BBox+6 | – | – | 56.9 | 71.5 | – |
Table 5: In-distribution results (5-DB training, modality channels, no online adaptation). Dice (%). Best results for each database in bold .
| MI | PI | No Prompt | BBox | BBox+1 | BBox+6 |
| ✗ | ✗ | 37.9 | 39.5 | 41.1 | 47.9 |
| ✓ | ✗ | 43.9 | 49.1 | 53.2 | 61.6 |
| ✗ | ✓ | 42.2 | 46.8 | 50.9 | 59.1 |
| ✓ | ✓ | 44.3 | 50.5 | 54.3 | 62.2 |
Table 6: Ablation study of online adaptation components on TBI (FLAIR-visible lesions). MI = Mid-Interaction adaptation, PI = Post-Interaction adaptation. Dice score (%). Best result in bold .
| Avg clicks | Reached threshold (%) | |||
| Dataset | w/ Online | w/o Online | w/ Online | w/o Online |
| MSSEG | 3.81 | 6.66 | 96.2 | 71.7 |
| VES-SEG | 2.67 | 12.81 | 90.1 | 31.8 |
| ISLES | 3.43 | 5.46 | 85.7 | 64.3 |
| TBI | 6.78 | 10.35 | 69.8 | 39.9 |
Table 7: Click efficiency at fixed Dice thresholds. For each database we report the fraction of cases that reach the target Dice within a 15-click budget and the average number of correction clicks per case (cases that never reach the threshold are counted as 15 clicks). Thresholds: MSSEG = 0.70, VES-SEG T1C = 0.80, ISLES = 0.65, TBI FLAIR = 0.65. Best result in each row is shown in bold .
| Metric | w/ Online | w/o Online |
| 10-click budget (max 10 clicks/case) | ||
| Avg. undetected – pre-clicks | 8.26 | 23.75 |
| Avg. undetected – post-clicks | 2.51 | 13.87 |
| Avg. clicks used | 4.00 | 7.96 |
| Unlimited clicks (until all detected) | ||
| Avg. undetected – pre-clicks | 5.75 | 23.75 |
Table 8: Undetected lesion-component analysis on the held-out MSSEG dataset (53 subjects). Each correction click is placed inside a currently undetected ground-truth component. We report the average number of undetected lesion components pre and post-clicks, together with the average clicks used under both a 10-click budget and under an unlimited-click setting. With unlimited clicks every component is eventually recovered (zero undetected) for both models, but online adaptation reaches this with far fewer clicks.
Figure 6: Online-adaptation trajectory on (a) VES-SEG and (b) TBI. The -axis is the number of adaptation images processed so far, and the -axis is the average Dice on the held-out evaluation split
More in this topic
Medical Image Segmentation
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
See all 788 papers