SAMI3D-DW: Interactive Segmentation of Any 3D Medical Images

Ping Gong; Shiyuan Su; Fandong Zhang; Xinchen Han; Haowei Sun; Yiming Li; Yizhou Yu

cs.CVSep 22, 2026

SAMI3D-DW: Interactive Segmentation of Any 3D Medical Images

Authors: Ping Gong, Shiyuan Su, Fandong Zhang, Xinchen Han, Haowei Sun, Yiming Li, Yizhou Yu

Organizations: Artificial Intelligence Laboratory, Deepwise Healthcare · The University of Hong Kong

Abstract

Interactive segmentation of 3D medical images supports quantitative analysis of anatomical structures and disease while allowing users to specify and refine their targets. Despite substantial progress by nnInteractive and VISTA3D, reliable segmentation across diverse clinical targets remains challenging, particularly for complex anatomical structures and the heterogeneous, long-tailed spectrum of pathology. We present SAMI3D-DW V1 (hereafter SAMI3D-DW), an interactive 3D segmentation model trained on Deepwise's large-scale proprietary medical image datasets. We evaluate the model under simulated user interactions on a CT/MR benchmark comprising 4,326 cases from 219 source datasets, spanning 107 anatomical and pathological categories, organized by a medical taxonomy and evaluated with a category-balanced DSC score. SAMI3D-DW achieves the highest category-macro Dice among evaluated methods in both interaction modes. With one point, it scores 0.5756 versus 0.5316 for nnInteractive, the strongest baseline, rising to 0.7771 versus 0.7495 with five points. With bounding-box initialization, the scores are 0.7129 versus 0.6530. After five corrective clicks, SAMI3D-DW reaches 0.8004 versus 0.7868. For radiologists and clinicians, SAMI3D-DW enables segmentation of complex anatomical structures, including intracranial vessel trees on CT and MR angiography, with a few clicks. In a preliminary in-house comparison involving neurofibromatosis type 1 (NF1), SAMI3D-DW-assisted tumor annotation took minutes per case and approximately one-fifteenth of the time required for manual annotation, highlighting its potential to support volumetric treatment-response assessment.

Figures & tables

Figure 1 : Benchmark-wide performance with point-only and box-initialized interaction. Top: initial prompts. Bottom: five total points or five corrective points after a BBox. Bar endpoints show category-macro Dice on shared 0–1 axes; colors identify methods. Gains over SOTA are shown in percentage points (pp) at the top-right of each panel.

Figure 2 : Two major trends in interactive medical segmentation: A , from spatial prompts to richer spatial, semantic, and free-text interfaces; B , from 2D images to 3D volumetric segmentation and correction, the direction pursued by SAMI3D-DW. Arrows indicate conceptual transitions or selected adaptations, not performance rankings or direct architectural inheritance. Dates follow the cited publication or preprint version; grouped branches are not a uniform timeline.

Figure 3 : Training data curation. FN and FP denote false-negative and false-positive annotation errors.

Grade	Criteria
5: Excellent	Accurate boundaries with fine anatomical detail preserved.
4: Good	Minor focal contour deviations.
3: Acceptable	Overall anatomical structures preserved; localized errors requiring correction.
2: Poor	Major contour errors or substantial anatomical distortion.
1: Unusable	Absent required annotations, invalid masks, or image–mask mismatch.

Table 1 : Contour-quality grading. Cases graded 1–2 were excluded.

Figure 4 : From dataset-specific labels to medical categories. Dashed boxes distinguish labels determined by dataset assets (left) from categories pre-defined using medical knowledge (right). Specific Targets provide shared, standardized definitions across datasets and bridge the two spaces. Medical categories are organized under domains A (anatomy), B (pathology), and C (other).

Code	Taxonomy family	Categories	Instances
A: Anatomical and physiological structures
Scope: Normal human anatomical structures and physiological constituents. Examples: Liver, kidneys, heart, vascular trees, vertebrae, and muscles.
A01	Nervous system	4	224
A02	Cardiovascular system	15	1,107
A03	Respiratory system	2	463
A04	Digestive system	9	1,263

Table 2 : Medical taxonomy and coverage of the scored test set. Domain definitions and illustrative examples describe the classification scope; counts refer only to represented families. Instance counts are deduplicated within each category; a physical object can contribute to more than one category.

Figure 5 : Anatomical, pathological, and modality coverage of the benchmark. Medical illustrations show representative anatomical structures for all 11 represented anatomical families (A). The pathology panel (B) lists seven families, with bar lengths indicating category counts; the Other domain (C) retains non-human structures for complete accounting. Category-level instance counts are deduplicated within each category; a physical instance can contribute to more than one category. The overall count and modality shares use the 13,826 unique physical instances. Illustrations are cropped and arranged from Servier Medical Art , licensed under CC BY 4.0 ; they are not to scale and do not imply exhaustive organ coverage.

Method	1 point	3 points	5 points	Valid objects	Categories
SAMI3D-DW	0.5756	0.7396	0.7771	13,826	107
nnInteractive	0.5316	0.6917	0.7495	13,826	107
MedSAM2	0.2790	0.3236	0.3657	13,715	107
SAM-Med3D	0.3409	0.3930	0.4057	13,823	107
SegVol	0.2774	0.2972	0.3138	13,785	107
VISTA3D (CVPR’25)	0.3692	0.4185	0.4297	13,747	107

Table 3 : Point-only prompt comparison. Scores are category-macro Dice; invalid object scores are excluded rather than replaced with zero. VISTA3D’s published formulation incorporates target-class information in addition to spatial prompts.

Method	BBox	$+1$	$+3$	$+5$	Valid objects	Categories
SAMI3D-DW	0.7129	0.7605	0.7854	0.8004	13,826	107
nnInteractive	0.6530	0.7153	0.7639	0.7868	13,826	107
MedSAM2	0.3764	0.3779	0.4101	0.4352	13,420	107
SAM-Med3D	0.3404	0.3792	0.4009	0.4077	13,823	107
SegVol	0.2402	0.2522	0.2578	0.2612	13,540	107

Table 4 : BBox-initialized point-refinement comparison. Scores are category-macro Dice computed from each method’s valid object scores. Object and category coverage is identical across the four displayed stages; invalid scores are excluded, not replaced with zero.

Taxonomy	Category	Samples (instances)	SAMI3D-DW		nnInteractive
			BBox	$+5$	BBox	$+5$
	Mean		0.6862	0.7914	0.6321	0.7784
A01 Nervous System	Brain	72	0.6170	0.6767	0.7260	0.8049
	Ventricular & Cerebrospinal Fluid System	22	0.2870	0.4918	0.1857	0.3331
	Spinal Cord	89	0.6410	0.8487	0.6227	0.8381
	Cranial Nerves	41	0.4998	0.6043	0.3197	0.4840

Table 5 : Anatomical and Physiological Structures: BBox and BBox + 5 Clicks (Dice)

Taxonomy	Category	Samples (instances)	SAMI3D-DW		nnInteractive
			BBox	$+5$	BBox	$+5$
	Mean		0.7545	0.8185	0.6775	0.7954
B01 Neoplastic Lesion	Brain and Intracranial Tumor	318	0.7942	0.8340	0.6950	0.7801
	Parotid Tumor	33	0.8381	0.8686	0.7922	0.8662
	Lung Cancer or Pulmonary Mass	238	0.7213	0.7913	0.6599	0.7798
	Thymic Cancer	70	0.7540	0.8380	0.6504	0.7475

Table 6 : Pathological Findings and Abnormalities: BBox and BBox + 5 Clicks (Dice)

Modality	Categories (used/mapped)	Objects	SAMI3D-DW	nnInteractive	MedSAM2	SAM-Med3D	SegVol
CT	85/85	9,289	0.7445 / 0.8302	0.6870 / 0.8195	0.3769 / 0.4161	0.3578 / 0.4367	0.2285 / 0.2497
MR	69/69	4,537	0.6405 / 0.7455	0.5997 / 0.7433	0.3384 / 0.4080	0.3378 / 0.4162	0.2382 / 0.2741

Table 7 : Category-macro Dice by 3D imaging modality. Each method cell reports BBox /

+5

corrections. Categories denotes the common denominator of categories with valid scores for all five methods; bold marks the best value at each stage within a row. The cohort is restricted to CT and MR.

Size by target voxels	Categories (used/mapped)	Objects	SAMI3D-DW	nnInteractive	MedSAM2	SAM-Med3D	SegVol
Small ( $<10{,}000$ )	97/97	7,855	0.6746 / 0.7650	0.6123 / 0.7551	0.3638 / 0.4728	0.2939 / 0.3223	0.3428 / 0.3589
Medium ( $10{,}000$ – $99{,}999$ )	90/90	4,009	0.7443 / 0.8270	0.7089 / 0.8231	0.4134 / 0.4285	0.4400 / 0.5255	0.1644 / 0.2127
Large ( $\geq 100{,}000$ )	68/69	1,962	0.7523 / 0.8392	0.7237 / 0.8358	0.4782 / 0.4181	0.4711 / 0.6068	0.0555 / 0.1128

Table 8 : Category-macro Dice by reference-object size. Each method cell reports BBox /

+5

corrections. Categories denotes the common denominator of categories with valid scores for all five methods; bold marks the best value at each stage within a row.

Maximum spacing	Categories (used/mapped)	Objects	SAMI3D-DW	nnInteractive	MedSAM2	SAM-Med3D	SegVol
Thin ( $<1$ mm)	46/47	981	0.7826 / 0.8403	0.7282 / 0.8343	0.3616 / 0.3402	0.3810 / 0.4828	0.1127 / 0.1064
Intermediate ( $1$ – $<5$ mm)	104/104	9,657	0.7168 / 0.8028	0.6518 / 0.7850	0.3637 / 0.4151	0.3370 / 0.4075	0.2204 / 0.2371
Thick ( $\geq 5$ mm)	70/70	3,188	0.6803 / 0.7768	0.6338 / 0.7688	0.4425 / 0.5400	0.3851 / 0.4355	0.3421 / 0.3709

Table 9 : Category-macro Dice by maximum voxel spacing. Each method cell reports BBox /

+5

corrections. Categories denotes the common denominator of categories with valid scores for all five methods; bold marks the best value at each stage within a row.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Taxonomy	Category	Combined Targets	Samples (instances)	SAMI3D-DW
				BBox	$+1$	$+3$	$+5$
Mean				0.7044	0.7521	0.7770	0.7978
A01 Nervous System	Brain	Brain	14	0.9121	0.9063	0.9174	0.9279
		Brainstem	14	0.7552	0.8059	0.8275	0.8416
		Cerebral Gray Matter	22	0.4740	0.6228	0.4513	0.4913
		Cerebral White Matter	22	0.4841	0.3970	0.5267	0.5973

Appendix

Table 10 : Anatomical and Physiological Targets: BBox + Point Segmentation (Dice)

Taxonomy	Category	Combined Targets	Samples (instances)	SAMI3D-DW
				BBox	$+1$	$+3$	$+5$
Mean				0.7483	0.7846	0.8043	0.8132
B01 Neoplastic Lesion	Brain and Intracranial Tumor	Meningioma	4	0.8394	0.9101	0.8987	0.9131
		Glioma	68	0.6727	0.7063	0.7334	0.7524
		Pituitary Tumor	10	0.8247	0.8522	0.8647	0.8689
		Brain Tumor, Type Unspecified	207	0.8282	0.8451	0.8542	0.8558

Appendix

Table 11 : Pathological and Abnormal Targets: BBox + Point Segmentation (Dice)

Explore similar work

SLIP: Segmentation with Low-latency Interactive Prompting for 3D Medical Images
Jul 24, 2026Baptiste Podvin, Alexandre Ancel, Flavio Milana +8Interaction Data Slipsense
IRCAD France, Strasbourg, France · Division of Hepatobiliary and General Surgery, Department of Surgery, IRCCS Humanitas Research Hospital, Milan, Italy · Department of Biomedical Sciences, Humanitas University, Milan, Italy +2
SAMRI-3D: Adapting SAM2 for 3D MRI Segmentation with Global Volume Tokens
Jul 20, 2026Zhao Wang, Wei Dai, Hongfu Sun +23D Medical Image Segmentation Magnetic Resonance Imaging
School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, Australia · School of Engineering, College of Engineering, Science and Environment, University of Newcastle, Australia · School of Human Movement and Nutrition Sciences, The University of Queensland, Australia
MedSAM3: Delving into Segment Anything with Medical Concepts
Nov 24, 2025Anglin Liu, Xu R. Cao, Yifan Shen +4Segment Anything Model Medical Imaging
The Hong Kong University of Science and Technology (Guangzhou) · University of Illinois Urbana-Champaign · Southeast University +1

Jul 24, 2026, cs.CVJ/K move · Enter open · S save

SLIP: Segmentation with Low-latency Interactive Prompting for 3D Medical Images

Baptiste Podvin, Alexandre Ancel, Flavio Milana, Chiara Innocenzi+7

IRCAD France, Strasbourg, France · Division of Hepatobiliary and General Surgery, Department of Surgery, IRCCS Humanitas Research Hospital, Milan, Italy · Department of Biomedical Sciences, Humanitas University, Milan, Italy+2

Interaction Data Slipsense

Open paper AskPDF arXiv

Interactive deep image segmentation enables efficient medical image annotation by iteratively refining predictions from user prompts, such as positive and negative clicks. Recent patch-based methods, including nnInteractive, achieve strong segmentation performance but remain limited in annotation workflows by high interaction latency, limited responsiveness to successive interactions, and the lack of support for reversible prompting. Furthermore, evaluation relies predominantly on simulated rather than controlled real-user interaction studies. We present SLIP, an end-to-end trainable framework for interactive 3D medical image segmentation that decouples image encoding from prompt-guided refinement. Image features are computed once and reused, while a lightweight patch memory bank maintains an interaction-aware segmentation state shared across patches. This representation enables prediction updates by propagating interaction context throughout the image, supports reversible prompting without recomputing image features, and substantially reduces interaction latency. By separating image representation from interactive reasoning, SLIP remains compatible with a wide range of image encoders. We train a single SLIP model for general interactive segmentation across diverse anatomical structures and imaging modalities. Beyond standard simulated evaluation, we conduct a controlled prospective user study comparing manual segmentation, nnInteractive, and SLIP across three clinical annotation tasks, six expert participants, and subjective usability measures, addressing the limited human validation of interactive segmentation methods. SLIP achieves SOTA interactive segmentation performance across 13 public datasets while providing lower interaction latency, greater responsiveness, support for reversible prompting, and higher user preference than existing approaches.

Abstract

Explore similar work

SLIP: Segmentation with Low-latency Interactive Prompting for 3D Medical Images

SAMRI-3D: Adapting SAM2 for 3D MRI Segmentation with Global Volume Tokens

MedSAM3: Delving into Segment Anything with Medical Concepts