Authors: Philip Matthias Winter, Maria Wimmer, Bernhard Schäfl, David Major, Dimitrios Lenis, Astrid Berg, Theresa Neubauer, Ariharasudhan Muthusami, Johannes Novotny, Georg Zotti, Katja Bühler
Organizations: VRVis GmbH - Vienna Research Center for Visual Computing, Vienna, Austria
In open-world recognition, machine learning models not only make predictions on unknown structures ("generalized open-set"), but also continually learn and improve from those predictions over time. Interactive segmentation, despite its access to user feedback, has so far remained confined to the open-set setting. In this work we introduce MEOW, the first framework that turns non-interactive segmentation backbones into interactive open-world learners without updating learnable model parameters. MEOW combines parameter-free transductive reasoning with memory consolidation to accumulate knowledge and thereby improve future performance. Focusing on medical data, we show that MEOW consistently improves its own predictions on classes never seen during training, and transfers zero-shot to unfamiliar domains, including micro-CT data and other imaging modalities.
Figures & tables
Fig. 1: We propose MEOW, a framework for open-world interactive segmentation. a) Flexible properties for Interactive Segmentation. b) Segmenting an unknown structure and learning from own prediction. c) Improving segmentation performance by leveraging previously learned structures.
Method
modular / flexible
interaction- agnostic
OS
OW
real-time learning
U-Net SuPreM [ 15 ]
✗
-
✗
✗
✗
TotalSegmentator [ 16 ]
✗
-
✗
✗
✗
PARMESAN [ 14 ]
✓
-
✗
✗
✓
SAM-Med3D [ 35 ]
✗
✗
✓
✗
✗
VISTA3D [ 31 ]
✗
✗
✓
✗
✗
nnInteractive [ 30 ]
✗
✓
✓
✗
✗
TABLE I: Qualitative comparison of MEOW to baselines and semantic segmentation methods.
Fig. 2: Data flow diagram of our OW-IS approach MEOW: A memory module M accumulates knowledge over time and is used in combination with a frozen feature extractor F and a parameter-free transduction module G for making predictions. A user U interacts with the system by providing guidance to improve results. Final results may be learned via memory consolidation. Learned samples can instantly be leveraged to improve performance for future predictions.
Fig. 3: After the initialization phase, the alternating interaction phase starts by presenting several visual components to U : the query image xq with an overlay of the initial prediction y (in blue), as well as the weak label z+ at the center (in green). A crosshair indicates the current cursor position on the canvas. U can navigate through the volume via the mouse wheel or via keyboard buttons. On a control panel, interactive hyperparameters ι can be adjusted via sliders. A memory module suited for the task at hand can be selected. After every user action (i.e., clicks, brushes, or hyperparameter changes), G is called to update the prediction.
Fig. 4: Interactive segmentation with MEOW. The three processing steps of a call to G include an interactive base prediction (red), optional merging of a memory prediction (blue), and hierarchical message passing (green). a) A feature pyramid (h)ql is calculated once by applying a feature extractor F to a query input xq . b) Weak user labels z± for model guidance (positive: blue, negative: red) determine labeled parts (μ)ql of (h)ql . c) Approximate, hierarchical k -NN w.r.t. (μ)ql illustrated for 4 exemplary query voxels (in different colors), followed by label retrieval and reduction to obtain a base prediction yb . d) Combining yb with an optional memory prediction yM yields a raw prediction yr . e) Hierarchical message passing for smooth and consistent segmentations.
Method
Learnable Params
Training Volumes
Training Masks
Memory Samples
SAM-Med3D
100.51 M
22 K
143 K
-
VISTA3D
217. 96 M
11.5 K
> 1.34 M
-
nnInteractive
102.36 M
∼ 61.3 K
∼ 681 K
-
MEOW ( F1&M )
*19.07 M
2.1 K
∼ 52 K
≤128
MEOW ( F2&M )
*31.21 M
1.2 K
< 141 K
≤128
MEOW ( F3&M )
*217. 26 M
11.5 K
> 1.34 M
≤128
TABLE II: Model size and training data efficiency of different methods. For MEOW feature extractors are frozen (marked with *).
ID
Method
Ablation
Original → Ours
DSCcr8↑(ER↑)
A0
F1&M
PARMESAN
all original
0.805(0.106)
A1
F1&M
compression
trim → linear AE
0.851(0.234)
A2
F1&M
kernel
box → cross + flexi
0.859(0.170)
A3
F1&M
similarity
ω=1→0.90
0.814(0.096)
A4
F1&M
balancing
γ=0→−0.99
0.808(0.106)
A5
F1&M
matcher
cos → reg. cos
0.813(0.125)
TABLE III: Results for ablation studies, demonstrating various improvements when applying modifications to PARMESAN.
Lmp
Unperturbed
Blur
Noise
Translate
Rotate
Shear
0
0.814
0.655
0.403
0.531
0.364
0.492
1
0.830
0.712
0.828
0.578
0.394
0.563
2
0.840
0.731
0.829
0.644
0.448
0.639
3
0.840
0.746
0.823
0.710
0.521
0.691
0
0.833
0.707
0.487
0.554
0.361
0.499
1
0.856
0.703
0.851
0.612
0.390
0.571
TABLE IV: Results for ablation studies (top block: DSCcr8 bottom block: DSCr8 ), demonstrating the effectiveness of HMP under strong (mass-conserving) perturbations of 3D base predictions yb preceding HMP.
Fig. 5: Qualitative results demonstrating the effectiveness of hierarchical message passing (HMP) under strong (mass-conserving) perturbations of 3D base predictions yb preceding HMP. Ground truth is indicated in contours.
ID
Method
NoCr90↓
DSCr8↑
DSCcr1↑
DSCcr8↑
B0.0
nnInteractive
2.518
0.940
0.854
0.920
B0.1
VISTA3D
5.296
0.844
0.702
0.797
B0.2
SAM-Med3D
4.100
0.847
0.805
0.842
B0.3
MEOW ( F1&M )
4.184
0.858
0.792
0.831
B0.4
MEOW ( F1 )
5.321
0.795
0.637
0.740
B0.5
MEOW ( F2&M )
4.529
0.838
0.767
0.817
TABLE V: Closed-set IS performance results for KKCs on medical datasets (top: TotSeg Organ, bottom: MSD+KiTS23) for organ classes liver, kidney, pancreas, and spleen.
ID
Method
NoCr90↓
DSCr8↑
DSCcr1↑
DSCcr8↑
B2.0
nnInteractive
4.880
0.894
0.650
0.826
B2.1
VISTA3D
7.990
0.537
0.166
0.422
B2.2
SAM-Med3D
7.698
0.673
0.485
0.651
B2.3
MEOW ( F1 )
7.760
0.747
0.495
0.668
B2.4
MEOW ( F2 )
7.990
0.694
0.340
0.583
B2.5
MEOW ( F3 )
7.990
0.710
0.474
0.622
TABLE VI: Open-set IS performance results for UKCs for unknown CT datasets (top: Mouse μ CT, bottom: InsectAnatomy). We use all organ classes for Mouse μ CT and the single brain class for InsectAnatomy. Note that we do not use M for MEOW.
Fig. 6: Qualitative demonstrations of user interactions (blue: model prediction, green/red: weak positive/negative labels z± , contours: ground truth). With MEOW, clicks and brushes can be mixed flexibly across slices and axes.
Fig. 7: Qualitative demonstrations of open-domain IS capabilities of MEOW (predictions in colors). We use U-Net SuPreM pre-trained on human abdomen CT data (i.e., F1 without M ) to segment various unknown objects under extreme domain shifts, as well as different data modalities [ 109 , 126 , 127 , 128 ] . Individual results were obtained in ≈1 minute of human user interaction. For the brain tumor example, prediction and ground truth are visualized in blue and yellow, respectively ( DSC4=0.823 ).
ID
Method
tinf [ms]
tlearn [ms]
B0.0
nnInteractive
289.750±4.610
-
B0.1
VISTA3D
267.430±6.647
-
B0.2
SAM-Med3D
8.980±0.099
-
B0.3
MEOW( F1&M )
279.470±12.488
9.62±0.11
B0.4
MEOW( F1 )
519.360±15.613
-
B0.5
MEOW( F2&M )
501.315±13.018
7.61±0.18
TABLE VII: Mean and standard deviation of inference speed tinf (per model prediction) and learning speed tlearn (per sample) at a resolution of 1283 voxels using an RTX 3090 GPU. MEOW is real-time capable w.r.t. inference and learning using frozen feature extractors.
Fig. 8: Demonstration of OW-learning for IS via memory consolidation. MEOW is continually trained by filling M with self-generated pseudo-labels, which are then used for making future predictions via transductive reasoning. OW-learning typically leads to increased performance (black arrows), even with small memory modules.
ID
Method
m (#SoIs)
DSCcr1↑
DSCcr8↑
C0.0
nnInteractive
-
0.667±0.007
0.831±0.0
C0.1
VISTA3D
-
0.167±0.003
0.425±0.006
C0.2
SAM-Med3D
-
0.488±0.0
0.650±0.001
C0.3
MEOW ( F1&M )
0
0.507±0.001
0.672±0.001
→32
0.542±0.010
0.653±0.009
→64
0.600±0.007
0.681±0.005
TABLE VIII: OW-IS results for OW-learning capabilities on Mouse μ CT data (top) and InsectAnatomy data (bottom) with different number m of pseudo-labeled memory samples ( UKCs ).
ID
Ablation
DSCcr1↑
DSCcr8↑
C0.3
-
0.625±0.006
0.691±0.004
C0.4
-
0.515±0.005
0.574±0.001
C0.5
-
0.440±0.003
0.608±0.007
C0.6
zp→zgt in M ( F1 )
0.682±0.020
0.750±0.009
C0.7
zp→zgt in M ( F2 )
0.671±0.009
0.718±0.005
C0.8
zp→zgt in M ( F3 )
0.550±0.003
0.702±0.001
TABLE IX: OW-IS results for ablation studies on Mouse μ CT data (top) and InsectAnatomy data (bottom). We analyze using ground truth labels, using untrained feature extractors (marked with r ), and using no class metadata.
ID
Method
m (#SoIs)
DSCcr1↑
DSCcr8↑
C2.0
VISTA3D
-
0.291±0.0
0.653±0.0
C2.1
MEOW ( F1&M )
0
0.555±0.006
0.650±0.004
→8
0.635±0.013
0.715±0.009
→16
0.642±0.014
0.717±0.007
→32
0.640±0.001
0.718±0.007
→64
0.643±0.003
0.719±0.006
TABLE X: OW-IS results when learning within unknown MRI domain for brain tumors. All F were pre-trained on human CT data.
Fig. 9: a) Query patch containing 2d query nodes (in 2D for simplicity) that are used for similarity search. Green nodes indicate query nodes of interest. b) Original box-kernel with 2d nodes, where a common kernel region is shared across all query nodes in the query patch. c) Newly introduced cross -kernel with 1+2⋅d nodes, used for dense labels (e.g., message passing). Kernel positions (with anchor nodes in blue) adapt locally depending on relative query node positions in the patch. d) Newly introduced flexi -kernel, used for sparse labels (e.g., weak user-labels). Kernel orientations adapt locally depending on query node positions. From the total of 2d+d nodes within the kernel region, 2d nodes are being kept while preferring labeled nodes as well as prioritizing nodes within the reference frame.
ID
Data
F
Mem
τ
ω
γ
ξ−
log(βnn)
βreg
αfuse
σmp
λmp
Lmp
log(βmp)
ηmp
TTA
H0
TotSeg Organ
F1
✓
0.873
0.900
-0.850
0.120
-1.000
0.000
0.000
1.000
0.957
2
0.462
12
✗
H1
TotSeg Organ
F1
✗
0.821
0.900
-0.990
0.500
3.000
-
-
22.583
1.000
2
-1.000
12
✗
H2
MSD+KiTS23
F1
✓
0.863
0.900
-0.990
0.462
1.308
0.000
0.000
25.000
1.000
2
0.805
12
✗
H3
MSD+KiTS23
F1
✗
0.812
0.900
-0.990
0.025
2.516
-
-
21.581
1.000
3
1.000
12
✗
H4
TotSeg Organ
F2
✓
0.842
0.900
-0.179
0.177
0.113
-1.000
0.000
1.000
0.718
2
1.000
6
✗
H5
TotSeg Organ
F2
✗
0.818
0.900
-0.990
0.089
1.516
-
-
25.000
0.441
3
0.310
6
✗
TABLE XI: Results for Bayesian hyperparameter optimization using KKC organ classes liver, pancreas, spleen, and kidney.
Interactive medical image segmentation uses clinician prompts and iterative model feedback to refine anatomical masks and reduce manual annotation effort.
Open world image segmentation aims to achieve precise segmentation and semantic understanding of targets within images by addressing the infinitely open set of object categories encountered in the real world. However, traditional closed-set segmentation approaches struggle to adapt to complex open world scenarios, while foundation segmentation models such as SAM exhibit notable discrepancies between their strong segmentation capabilities and relatively weaker semantic understanding. To bridge these discrepancies, we propose WOW-Seg, a Word-free Open World Segmentation model for segmenting and recognizing objects from open-set categories. Specifically, WOW-Seg introduces a novel visual prompt module, Mask2Token, which transforms image masks into visual tokens and ensures their alignment with the VLLM feature space. Moreover, we introduce the Cascade Attention Mask to decouple information across different instances. This approach mitigates inter-instance interference, leading to a significant improvement in model performance. We further construct an open world region recognition test benchmark: the Region Recognition Dataset (RR-7K). With 7,662 classes, it represents the most extensive category-rich region recognition dataset to date. WOW-Seg attains strong results on the LVIS dataset, achieving a semantic similarity of 89.7 and a semantic IoU of 82.4. This performance surpasses the previous SOTA while using only one-eighth the parameter count. These results underscore the strong open world generalization capabilities of WOW-Seg. The code and related resources are available at https://github.com/AAwcAA/WOW-Seg-Meta.
Danyang Li, Tianhao Wu, Bin Li +5
2VCIP, CS, Nankai University · 1NKIARI, Shenzhen Futian · 3AAIS, Nankai University +2
Interactive segmentation enables clinicians to guide annotation, but existing zero-shot models like nnInteractive fail to consistently reach expert-level performance across diverse medical imaging tasks. Because annotation campaigns produce a growing stream of task-specific labelled data, online adaptation of the segmentation model is a natural complement to zero-shot inference. We propose CLoPA, a continual adaptation strategy that tunes a small fraction of nnInteractive's parameters on the annotation cache, triggered by lightweight episode scheduling. CLoPA requires no new parameters or changes to the inference pipeline, and operates entirely within the existing annotation workflow. Across eight Medical Segmentation Decathlon tasks spanning diverse anatomical targets and imaging characteristics, CLoPA rapidly elevates performance to expert-level, even for tasks where nnInteractive previously failed, with the majority of gains realised after a single training episode. We show that the benefits of tuning different parameter groups depends on task characteristics and data regimes. Also, that for targets with complex geometries (e.g., hepatic vessels), instance normalisation and low-level feature tuning saturates, suggesting a need for deeper feature-representation alignment in the most challenging scenarios.
Parhom Esmaeili, Chayanin Tangwiriyasakul, Eli Gibson +2
School of Biomedical Engineering and Imaging Sciences, KCL, UK · Siemens Healthineers, Princeton NJ, USA
Medical image segmentation is vital for clinical diagnosis and quantitative analysis, yet remains challenging due to the heterogeneity of imaging modalities and the high cost of pixel-level annotations. Although general interactive segmentation models like SAM have achieved remarkable progress, their transfer to medical imaging still faces two key bottlenecks: (i) the lack of adaptive mechanisms for modality- and anatomy-specific tasks, which limits generalization in out-of-distribution medical scenarios; and (ii) current medical adaptation methods fine-tune on large, heterogeneous datasets without selection, leading to noisy supervision, higher cost, and negative transfer. To address these issues, we propose SegMoTE, an efficient and adaptive framework for medical image segmentation. SegMoTE preserves SAM's original prompt interface, efficient inference, and zero-shot generalization while introducing only a small number of learnable parameters to dynamically adapt across modalities and tasks. In addition, we design a progressive prompt tokenization mechanism that enables fully automatic segmentation, significantly reducing annotation dependence. Trained on MedSeg-HQ, a curated dataset less than 1% of existing large-scale datasets, SegMoTE achieves SOTA performance across diverse imaging modalities and anatomical tasks. It represents the first efficient, robust, and scalable adaptation of general segmentation models to the medical domain under extremely low annotation cost, advancing the practical deployment of foundation vision models in clinical applications.
Yujie Lu, Jingwen Li, Sibo Ju +5
Sichuan University · Xinjiang University · Fuzhou University +1