Authors: Philip Matthias Winter, Maria Wimmer, Bernhard Schäfl, David Major, Dimitrios Lenis, Astrid Berg, Theresa Neubauer, Ariharasudhan Muthusami, Johannes Novotny, Georg Zotti, Katja Bühler
Organizations: VRVis GmbH - Vienna Research Center for Visual Computing, Vienna, Austria
In open-world recognition, machine learning models not only make predictions on unknown structures ("generalized open-set"), but also continually learn and improve from those predictions over time. Interactive segmentation, despite its access to user feedback, has so far remained confined to the open-set setting. In this work we introduce MEOW, the first framework that turns non-interactive segmentation backbones into interactive open-world learners without updating learnable model parameters. MEOW combines parameter-free transductive reasoning with memory consolidation to accumulate knowledge and thereby improve future performance. Focusing on medical data, we show that MEOW consistently improves its own predictions on classes never seen during training, and transfers zero-shot to unfamiliar domains, including micro-CT data and other imaging modalities.
Figures & tables
Fig. 1: We propose MEOW, a framework for open-world interactive segmentation. a) Flexible properties for Interactive Segmentation. b) Segmenting an unknown structure and learning from own prediction. c) Improving segmentation performance by leveraging previously learned structures.
Method
modular / flexible
interaction- agnostic
OS
OW
real-time learning
U-Net SuPreM [ 15 ]
✗
-
✗
✗
✗
TotalSegmentator [ 16 ]
✗
-
✗
✗
✗
PARMESAN [ 14 ]
✓
-
✗
✗
✓
SAM-Med3D [ 35 ]
✗
✗
✓
✗
✗
VISTA3D [ 31 ]
✗
✗
✓
✗
✗
nnInteractive [ 30 ]
✗
✓
✓
✗
✗
TABLE I: Qualitative comparison of MEOW to baselines and semantic segmentation methods.
Fig. 2: Data flow diagram of our OW-IS approach MEOW: A memory module M accumulates knowledge over time and is used in combination with a frozen feature extractor F and a parameter-free transduction module G for making predictions. A user U interacts with the system by providing guidance to improve results. Final results may be learned via memory consolidation. Learned samples can instantly be leveraged to improve performance for future predictions.
Fig. 3: After the initialization phase, the alternating interaction phase starts by presenting several visual components to U : the query image xq with an overlay of the initial prediction y (in blue), as well as the weak label z+ at the center (in green). A crosshair indicates the current cursor position on the canvas. U can navigate through the volume via the mouse wheel or via keyboard buttons. On a control panel, interactive hyperparameters ι can be adjusted via sliders. A memory module suited for the task at hand can be selected. After every user action (i.e., clicks, brushes, or hyperparameter changes), G is called to update the prediction.
Fig. 4: Interactive segmentation with MEOW. The three processing steps of a call to G include an interactive base prediction (red), optional merging of a memory prediction (blue), and hierarchical message passing (green). a) A feature pyramid (h)ql is calculated once by applying a feature extractor F to a query input xq . b) Weak user labels z± for model guidance (positive: blue, negative: red) determine labeled parts (μ)ql of (h)ql . c) Approximate, hierarchical k -NN w.r.t. (μ)ql illustrated for 4 exemplary query voxels (in different colors), followed by label retrieval and reduction to obtain a base prediction yb . d) Combining yb with an optional memory prediction yM yields a raw prediction yr . e) Hierarchical message passing for smooth and consistent segmentations.
Method
Learnable Params
Training Volumes
Training Masks
Memory Samples
SAM-Med3D
100.51 M
22 K
143 K
-
VISTA3D
217. 96 M
11.5 K
> 1.34 M
-
nnInteractive
102.36 M
∼ 61.3 K
∼ 681 K
-
MEOW ( F1&M )
*19.07 M
2.1 K
∼ 52 K
≤128
MEOW ( F2&M )
*31.21 M
1.2 K
< 141 K
≤128
MEOW ( F3&M )
*217. 26 M
11.5 K
> 1.34 M
≤128
TABLE II: Model size and training data efficiency of different methods. For MEOW feature extractors are frozen (marked with *).
ID
Method
Ablation
Original → Ours
DSCcr8↑(ER↑)
A0
F1&M
PARMESAN
all original
0.805(0.106)
A1
F1&M
compression
trim → linear AE
0.851(0.234)
A2
F1&M
kernel
box → cross + flexi
0.859(0.170)
A3
F1&M
similarity
ω=1→0.90
0.814(0.096)
A4
F1&M
balancing
γ=0→−0.99
0.808(0.106)
A5
F1&M
matcher
cos → reg. cos
0.813(0.125)
TABLE III: Results for ablation studies, demonstrating various improvements when applying modifications to PARMESAN.
Lmp
Unperturbed
Blur
Noise
Translate
Rotate
Shear
0
0.814
0.655
0.403
0.531
0.364
0.492
1
0.830
0.712
0.828
0.578
0.394
0.563
2
0.840
0.731
0.829
0.644
0.448
0.639
3
0.840
0.746
0.823
0.710
0.521
0.691
0
0.833
0.707
0.487
0.554
0.361
0.499
1
0.856
0.703
0.851
0.612
0.390
0.571
TABLE IV: Results for ablation studies (top block: DSCcr8 bottom block: DSCr8 ), demonstrating the effectiveness of HMP under strong (mass-conserving) perturbations of 3D base predictions yb preceding HMP.
Fig. 5: Qualitative results demonstrating the effectiveness of hierarchical message passing (HMP) under strong (mass-conserving) perturbations of 3D base predictions yb preceding HMP. Ground truth is indicated in contours.
ID
Method
NoCr90↓
DSCr8↑
DSCcr1↑
DSCcr8↑
B0.0
nnInteractive
2.518
0.940
0.854
0.920
B0.1
VISTA3D
5.296
0.844
0.702
0.797
B0.2
SAM-Med3D
4.100
0.847
0.805
0.842
B0.3
MEOW ( F1&M )
4.184
0.858
0.792
0.831
B0.4
MEOW ( F1 )
5.321
0.795
0.637
0.740
B0.5
MEOW ( F2&M )
4.529
0.838
0.767
0.817
TABLE V: Closed-set IS performance results for KKCs on medical datasets (top: TotSeg Organ, bottom: MSD+KiTS23) for organ classes liver, kidney, pancreas, and spleen.
ID
Method
NoCr90↓
DSCr8↑
DSCcr1↑
DSCcr8↑
B2.0
nnInteractive
4.880
0.894
0.650
0.826
B2.1
VISTA3D
7.990
0.537
0.166
0.422
B2.2
SAM-Med3D
7.698
0.673
0.485
0.651
B2.3
MEOW ( F1 )
7.760
0.747
0.495
0.668
B2.4
MEOW ( F2 )
7.990
0.694
0.340
0.583
B2.5
MEOW ( F3 )
7.990
0.710
0.474
0.622
TABLE VI: Open-set IS performance results for UKCs for unknown CT datasets (top: Mouse μ CT, bottom: InsectAnatomy). We use all organ classes for Mouse μ CT and the single brain class for InsectAnatomy. Note that we do not use M for MEOW.
Fig. 6: Qualitative demonstrations of user interactions (blue: model prediction, green/red: weak positive/negative labels z± , contours: ground truth). With MEOW, clicks and brushes can be mixed flexibly across slices and axes.
Fig. 7: Qualitative demonstrations of open-domain IS capabilities of MEOW (predictions in colors). We use U-Net SuPreM pre-trained on human abdomen CT data (i.e., F1 without M ) to segment various unknown objects under extreme domain shifts, as well as different data modalities [ 109 , 126 , 127 , 128 ] . Individual results were obtained in ≈1 minute of human user interaction. For the brain tumor example, prediction and ground truth are visualized in blue and yellow, respectively ( DSC4=0.823 ).
ID
Method
tinf [ms]
tlearn [ms]
B0.0
nnInteractive
289.750±4.610
-
B0.1
VISTA3D
267.430±6.647
-
B0.2
SAM-Med3D
8.980±0.099
-
B0.3
MEOW( F1&M )
279.470±12.488
9.62±0.11
B0.4
MEOW( F1 )
519.360±15.613
-
B0.5
MEOW( F2&M )
501.315±13.018
7.61±0.18
TABLE VII: Mean and standard deviation of inference speed tinf (per model prediction) and learning speed tlearn (per sample) at a resolution of 1283 voxels using an RTX 3090 GPU. MEOW is real-time capable w.r.t. inference and learning using frozen feature extractors.
Fig. 8: Demonstration of OW-learning for IS via memory consolidation. MEOW is continually trained by filling M with self-generated pseudo-labels, which are then used for making future predictions via transductive reasoning. OW-learning typically leads to increased performance (black arrows), even with small memory modules.
ID
Method
m (#SoIs)
DSCcr1↑
DSCcr8↑
C0.0
nnInteractive
-
0.667±0.007
0.831±0.0
C0.1
VISTA3D
-
0.167±0.003
0.425±0.006
C0.2
SAM-Med3D
-
0.488±0.0
0.650±0.001
C0.3
MEOW ( F1&M )
0
0.507±0.001
0.672±0.001
→32
0.542±0.010
0.653±0.009
→64
0.600±0.007
0.681±0.005
TABLE VIII: OW-IS results for OW-learning capabilities on Mouse μ CT data (top) and InsectAnatomy data (bottom) with different number m of pseudo-labeled memory samples ( UKCs ).
ID
Ablation
DSCcr1↑
DSCcr8↑
C0.3
-
0.625±0.006
0.691±0.004
C0.4
-
0.515±0.005
0.574±0.001
C0.5
-
0.440±0.003
0.608±0.007
C0.6
zp→zgt in M ( F1 )
0.682±0.020
0.750±0.009
C0.7
zp→zgt in M ( F2 )
0.671±0.009
0.718±0.005
C0.8
zp→zgt in M ( F3 )
0.550±0.003
0.702±0.001
TABLE IX: OW-IS results for ablation studies on Mouse μ CT data (top) and InsectAnatomy data (bottom). We analyze using ground truth labels, using untrained feature extractors (marked with r ), and using no class metadata.
ID
Method
m (#SoIs)
DSCcr1↑
DSCcr8↑
C2.0
VISTA3D
-
0.291±0.0
0.653±0.0
C2.1
MEOW ( F1&M )
0
0.555±0.006
0.650±0.004
→8
0.635±0.013
0.715±0.009
→16
0.642±0.014
0.717±0.007
→32
0.640±0.001
0.718±0.007
→64
0.643±0.003
0.719±0.006
TABLE X: OW-IS results when learning within unknown MRI domain for brain tumors. All F were pre-trained on human CT data.
Fig. 9: a) Query patch containing 2d query nodes (in 2D for simplicity) that are used for similarity search. Green nodes indicate query nodes of interest. b) Original box-kernel with 2d nodes, where a common kernel region is shared across all query nodes in the query patch. c) Newly introduced cross -kernel with 1+2⋅d nodes, used for dense labels (e.g., message passing). Kernel positions (with anchor nodes in blue) adapt locally depending on relative query node positions in the patch. d) Newly introduced flexi -kernel, used for sparse labels (e.g., weak user-labels). Kernel orientations adapt locally depending on query node positions. From the total of 2d+d nodes within the kernel region, 2d nodes are being kept while preferring labeled nodes as well as prioritizing nodes within the reference frame.
ID
Data
F
Mem
τ
ω
γ
ξ−
log(βnn)
βreg
αfuse
σmp
λmp
Lmp
log(βmp)
ηmp
TTA
H0
TotSeg Organ
F1
✓
0.873
0.900
-0.850
0.120
-1.000
0.000
0.000
1.000
0.957
2
0.462
12
✗
H1
TotSeg Organ
F1
✗
0.821
0.900
-0.990
0.500
3.000
-
-
22.583
1.000
2
-1.000
12
✗
H2
MSD+KiTS23
F1
✓
0.863
0.900
-0.990
0.462
1.308
0.000
0.000
25.000
1.000
2
0.805
12
✗
H3
MSD+KiTS23
F1
✗
0.812
0.900
-0.990
0.025
2.516
-
-
21.581
1.000
3
1.000
12
✗
H4
TotSeg Organ
F2
✓
0.842
0.900
-0.179
0.177
0.113
-1.000
0.000
1.000
0.718
2
1.000
6
✗
H5
TotSeg Organ
F2
✗
0.818
0.900
-0.990
0.089
1.516
-
-
25.000
0.441
3
0.310
6
✗
TABLE XI: Results for Bayesian hyperparameter optimization using KKC organ classes liver, pancreas, spleen, and kidney.
Interactive medical image segmentation uses clinician prompts and iterative model feedback to refine anatomical masks and reduce manual annotation effort.