Seeing Through the Tool: A Controlled Benchmark for Occlusion Robustness in Foundation Segmentation Models

Nhan Ho; Luu Le; Thanh-Huy Nguyen; Thien Nguyen; Xiaofeng Liu; Ulas Bagci

cs.CVApr 13, 2026

Seeing Through the Tool: A Controlled Benchmark for Occlusion Robustness in Foundation Segmentation Models

Authors: Nhan Ho, Luu Le, Thanh-Huy Nguyen, Thien Nguyen, Xiaofeng Liu, Ulas Bagci

Organizations: Stony Brook University, USA · AIMA Research Lab · Carnegie Mellon University, USA · Yale University, USA ·

Abstract

Occlusion, where target structures are partially hidden by surgical instruments or overlapping tissues, remains a critical yet underexplored challenge for foundation segmentation models in clinical endoscopy. We introduce OccSAM-Bench, a benchmark designed to systematically evaluate SAM-family models under controlled, synthesized surgical occlusion. Our framework simulates two occlusion types (i.e., surgical tool overlay and cutout) across three calibrated severity levels on three public polyp datasets. We propose a novel three-region evaluation protocol that decomposes segmentation performance into full, visible-only, and invisible targets. This metric exposes behaviors that standard amodal evaluation obscures, revealing two distinct model archetypes: Occluder-Aware models (SAM, SAM 2, SAM 3, MedSAM3), which prioritize visible tissue delineation and reject instruments, and Occluder-Agnostic models (MedSAM, MedSAM2), which confidently predict into occluded regions. SAM-Med2D aligns with neither and underperforms across all conditions. Ultimately, our results demonstrate that occlusion robustness is not uniform across architectures, and model selection must be driven by specific clinical intent-whether prioritizing conservative visible-tissue segmentation or the amodal inference of hidden anatomy.

Figures & tables

Figure 1: OccSAM-Bench: evaluating segmentation under surgical occlusion. Clinical endoscopy often involves partial occlusion of target anatomy by surgical instruments. OccSAM-Bench simulates such conditions by overlaying surgical tools onto polyp images to create controlled occlusions. Models are evaluated using a three-region protocol that separates visible , invisible (occluded), and full masks, revealing behaviors that standard full-mask evaluation alone cannot capture.

Figure 2: OccSAM-Bench overview. Left: Surgical-tool occlusions are simulated by pasting instrument masks onto polyp images from three colonoscopy datasets. Right: SAM-family models exhibit two behaviors under occlusion: Occluder-Aware models segment visible tissue while rejecting the instrument, whereas Occluder-Agnostic models predict through the occluder. Both can achieve similar full-mask scores, showing that standard evaluation obscures these differences.

Dataset	Modality	Object	Masks
CVC-300 [ 26 ]	Endoscopy	Polyp	60
CVC-ColonDB [ 24 ]	Endoscopy	Polyp	380
ETIS-LaribPolypDB [ 22 ]	Endoscopy	Polyp	196

Table 1: Summary of polyp segmentation datasets used in our benchmark.

Setting	Model	Year	Backbone / Initialization	Training Scale	Data Type
General-purpose	SAM [ 11 ]	2023	ViT-H	11M images, 1B masks	Natural 2D images
	SAM 2 [ 19 ]	2024	Hiera-L	50.9K videos, 642K masklets	Natural images + videos
	SAM 3 [ 1 ]	2026	VLP-based encoder	5.2M HQ images, 52.5K videos	Natural images + videos (open-vocab)
Medical-adapted	MedSAM [ 14 ]	2023	SAM (ViT-H) fine-tuned	1.6M medical images	Medical 2D images
	SAM-Med2D [ 3 ]	2023	SAM (ViT-H) + PEFT	4.6M images, 19.7M masks	Medical 2D images
	MedSAM2 [ 29 ]	2024	SAM2-Tiny fine-tuned	455K 3D pairs, 76K frames	Medical 3D images + video

Table 2: Summary of evaluated models. All models are assessed in a zero-shot manner, i.e., without fine-tuning or adaptation on the target endoscopy datasets.

	CVC-300					CVC-ColonDB					ETIS
Model	Clean	Low	Med	High	$\Delta$ %	Clean	Low	Med	High	$\Delta$ %	Clean	Low	Med	High	$\Delta$ %
SAM	.925	.849	.754	.583	37.0	.884	.800	.599	.567	35.9	.910	.807	.662	.552	39.4
SAM 2	.913	.865	.766	.692	24.2	.907	.844	.731	.651	28.2	.907	.838	.736	.621	31.6
SAM 3	.950	.869	.776	.618	35.0	.928	.868	.762	.631	32.0	.935	.845	.762	.603	35.6
MedSAM	.742	.635	.602	.640	13.8	.709	.664	.573	.607	14.5	.778	.724	.622	.588	24.5
SAM-Med2D	.903	.845	.699	.554	38.7	.854	.762	.568	.545	36.2	.837	.780	.610	.502	40.0

Table 3: Full (amodal) DSC under surgical tool occlusion, box prompts. MedSAM2 appears the most robust model under this metric. However, as shown in Sec. 4.2 , this ranking is an artifact: high full-mask scores can result from models that segment through the occluder rather than around it.

\Delta

% (lower is better) is relative degradation from clean to high severity. Best and second best highlighted per column per dataset.

	CVC-300					CVC-ColonDB					ETIS
Model	Clean	Low	Med	High	$\Delta$ %	Clean	Low	Med	High	$\Delta$ %	Clean	Low	Med	High	$\Delta$ %
SAM	.925	.886	.914	.388	58.1	.884	.844	.635	.505	42.9	.910	.842	.783	.554	39.1
SAM 2	.913	.910	.894	.662	27.5	.907	.887	.824	.730	19.5	.907	.877	.863	.725	20.1
SAM 3	.950	.913	.931	.724	23.8	.928	.912	.876	.748	19.3	.935	.890	.887	.801	14.3
MedSAM	.742	.669	.579	.302	59.3	.709	.659	.462	.281	60.4	.778	.759	.598	.395	49.2
SAM-Med2D	.903	.889	.780	.345	61.8	.854	.801	.544	.385	54.9	.837	.818	.654	.491	41.3

Table 4: Visible-only DSC under surgical tool occlusion, box prompts. SAM 3 achieves the highest visible-region DSC in the majority of columns, with MedSAM2 and MedSAM3 competitive across datasets. MedSAM and SAM-Med2D show the steepest degradation under increasing occlusion severity.

\Delta

% (lower is better) is relative degradation from clean to high severity. Best and second best highlighted per column per dataset.

	CVC-300			CVC-ColonDB			ETIS
Model	Low	Med	High	Low	Med	High	Low	Med	High
SAM	.039	.002	.588	.052	.285	.501	.035	.155	.442
SAM 2	.120	.190	.569	.114	.278	.410	.122	.158	.395
SAM 3	.048	.056	.336	.082	.168	.304	.035	.106	.185
MedSAM	.387	.587	.823	.492	.643	.787	.222	.546	.695
SAM-Med2D	.065	.297	.657	.133	.419	.629	.098	.304	.478

Table 5: Invisible-region DSC under surgical tool occlusion (box prompts). Higher values indicate prediction inside the occluded region. At high severity, scores may be inflated by the large tool footprint and should be interpreted together with visible DSC (Table 4 ). Best and second best are highlighted per column.

Figure 3: Visible-only DSC comparison between Cutout and Surgical Tool occlusions across 3 datasets under increasing severity (Clean, Low, Medium, High) with Box prompts. Left : General SAMs (SAM, SAM 2, SAM 3); Right : Medical SAMs (MedSAM, SAM-Med2D, MedSAM2, MedSAM3). Differences between occlusion types are minimal at low-to-moderate levels but diverge at High severity, where surgical occlusion induces substantially larger degradation.

Figure 4: Visible-mask DSC comparison between Box and Single-Point prompts under increasing occlusion levels (Clean, Low, Medium, High), with surgical tool occlusions. Top row : General SAMs (SAM, SAM 2, SAM 3); bottom row : Medical SAMs (SAM-Med2D, MedSAM2, MedSAM3). Columns correspond to CVC-300, CVC-ColonDB, and ETIS-LaribPolypDB. Box prompting demonstrates consistently higher DSC and more consistent performance across datasets under all occlusion levels.

Model	Prompt	Clean Avg	Occluded Avg
MedSAM	Box	0.7431	0.4872
MedSAM	Point	0.0207	0.0172
MedSAM3	Point	0.8429	0.6840
SAM 3	Point	0.8217	0.7046

Table 6: Visible-only DSC under box vs. point prompts. Clean Avg is performance on clean images; Occluded Avg averages Low, Medium, and High occlusion severities across both occlusion types. MedSAM collapses under point prompting but remains functional with box prompts.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.93	10.05	0.85	38.30	0.75	62.17	0.58	73.95
	SAM 2	0.91	8.48	0.87	35.34	0.77	61.64	0.69	65.53
	SAM 3	0.95	4.98	0.87	35.51	0.78	62.42	0.62	60.11
	MedSAM	0.74	24.43	0.63	53.34	0.60	68.74	0.64	64.27
	SAM-Med2D	0.90	12.78	0.84	36.47	0.70	66.66	0.55	75.47

Appendix

Table 7: Segmentation performance on CVC-300 under Surgical Tool occlusion using Box prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.93	10.05	0.89	17.48	0.69	35.73	0.61	36.59
	SAM 2	0.91	8.48	0.92	15.81	0.77	31.93	0.68	31.15
	SAM 3	0.95	4.98	0.94	6.95	0.89	19.67	0.87	27.55
	MedSAM	0.74	24.43	0.68	30.31	0.60	37.29	0.64	32.04
	SAM-Med2D	0.90	12.78	0.87	20.81	0.75	30.07	0.68	32.14

Appendix

Table 8: Segmentation performance on CVC-300 under Cutout occlusion using Box prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.49	165.54	0.23	468.78	0.18	517.32	0.39	206.37
	SAM 2	0.83	44.03	0.66	130.22	0.59	144.94	0.49	130.24
	SAM 3	0.93	6.71	0.73	105.00	0.52	112.44	0.50	129.06
	MedSAM	0.02	89.81	0.00	180.03	0.01	177.41	0.02	150.37
	SAM-Med2D	0.68	40.67	0.57	79.31	0.38	99.08	0.51	99.78

Appendix

Table 9: Segmentation performance on CVC-300 under Surgical Tool occlusion using Point prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.49	165.54	0.29	227.06	0.34	146.29	0.38	122.88
	SAM 2	0.83	44.03	0.78	33.04	0.54	55.09	0.52	50.47
	SAM 3	0.93	6.71	0.80	30.49	0.75	41.28	0.68	46.37
	MedSAM	0.02	89.81	0.03	81.26	0.01	82.60	0.02	72.76
	SAM-Med2D	0.68	40.67	0.54	38.21	0.45	45.54	0.46	51.26

Appendix

Table 10: Segmentation performance on CVC-300 under Cutout occlusion using Point prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.88	20.48	0.80	86.41	0.60	110.62	0.57	95.72
	SAM 2	0.91	16.24	0.84	67.70	0.73	80.37	0.65	71.77
	SAM 3	0.93	13.51	0.87	62.71	0.76	72.79	0.63	71.57
	MedSAM	0.71	36.73	0.66	88.09	0.57	90.08	0.61	77.75
	SAM-Med2D	0.85	28.24	0.76	77.80	0.57	97.07	0.54	79.34

Appendix

Table 11: Segmentation performance on CVC-ColonDB under Surgical Tool occlusion using Box prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.88	20.48	0.82	30.02	0.66	49.17	0.58	59.09
	SAM 2	0.91	16.24	0.88	23.22	0.77	39.30	0.66	45.02
	SAM 3	0.93	13.51	0.91	14.53	0.90	29.74	0.68	34.97
	MedSAM	0.71	36.73	0.65	40.02	0.58	45.92	0.65	42.88
	SAM-Med2D	0.85	28.24	0.81	31.89	0.71	41.60	0.61	46.97

Appendix

Table 12: Segmentation performance on CVC-ColonDB under Cutout occlusion using Box prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.32	248.89	0.26	496.00	0.24	439.15	0.34	305.62
	SAM 2	0.74	72.05	0.56	241.01	0.43	248.51	0.45	177.26
	SAM 3	0.78	50.32	0.65	132.18	0.49	170.44	0.43	155.95
	MedSAM	0.02	133.46	0.02	265.20	0.02	252.97	0.02	194.77
	SAM-Med2D	0.53	67.91	0.40	155.28	0.35	172.58	0.45	133.59

Appendix

Table 13: Segmentation performance on CVC-ColonDB under Surgical Tool occlusion using Point prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.32	248.89	0.27	238.48	0.29	189.87	0.46	106.36
	SAM 2	0.74	72.05	0.66	73.48	0.55	80.03	0.51	89.63
	SAM 3	0.78	50.32	0.58	24.94	0.50	90.58	0.58	154.00
	MedSAM	0.02	133.46	0.02	125.80	0.02	116.01	0.02	118.44
	SAM-Med2D	0.53	67.91	0.43	71.74	0.35	79.62	0.39	81.40

Appendix

Table 14: Segmentation performance on CVC-ColonDB under Cutout occlusion using Point prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.91	33.13	0.81	45.30	0.66	84.53	0.55	91.15
	SAM 2	0.91	33.18	0.84	48.94	0.74	70.69	0.62	82.04
	SAM 3	0.94	22.15	0.84	32.71	0.76	66.51	0.60	66.48
	MedSAM	0.78	47.40	0.72	46.89	0.62	71.77	0.59	76.70
	SAM-Med2D	0.84	52.78	0.78	46.87	0.61	77.94	0.50	87.52

Appendix

Table 15: Segmentation performance on ETIS-LaribPolypDB under Surgical Tool occlusion using Box prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.91	33.13	0.82	50.09	0.68	79.46	0.63	92.08
	SAM 2	0.91	33.18	0.88	43.65	0.75	70.49	0.66	79.74
	SAM 3	0.94	22.15	0.92	31.75	0.86	60.73	0.77	69.34
	MedSAM	0.78	47.40	0.70	55.99	0.67	69.33	0.67	68.51
	SAM-Med2D	0.84	52.78	0.79	57.14	0.72	71.58	0.62	81.02

Appendix

Table 16: Segmentation performance on ETIS-LaribPolypDB under Cutout occlusion using Box prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.46	405.77	0.28	503.74	0.28	414.99	0.32	296.23
	SAM 2	0.69	225.01	0.57	220.11	0.44	245.45	0.40	201.71
	SAM 3	0.75	125.54	0.66	133.30	0.48	161.10	0.44	164.78
	MedSAM	0.02	232.68	0.03	180.45	0.03	175.92	0.02	192.56
	SAM-Med2D	0.67	112.44	0.58	102.98	0.42	132.24	0.44	134.33

Appendix

Table 17: Segmentation performance on ETIS-LaribPolypDB under Surgical Tool occlusion using Point prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Eval Mode	Model	Clean		Low		Medium		High
Eval Mode	Model	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Full (Amodal)	SAM	0.46	405.77	0.40	382.44	0.32	377.71	0.45	209.90
	SAM 2	0.69	225.01	0.65	165.76	0.48	193.81	0.49	188.75
	SAM 3	0.75	125.54	0.78	98.36	0.64	140.81	0.61	153.30
	MedSAM	0.02	232.68	0.03	198.13	0.03	207.02	0.02	226.62
	SAM-Med2D	0.67	112.44	0.58	103.03	0.46	139.07	0.49	136.29

Appendix

Table 18: Segmentation performance on ETIS-LaribPolypDB under Cutout occlusion using Point prompts across clean, low, medium, and high severity levels. Bold and underline denote the best and second best per column, respectively.

Amodal Segmentation

See all 11 papers

Robustness Evaluation of a Foundation Segmentation Model Under Simulated Domain Shifts in Abdominal CT: Implications for Health Digital Twin Deployment
Apr 28, 2026Sanghati BasuDistribution Shift Robustness Image Segmentation
Healthcare Informatics, University of Illinois Springfield
Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)
Aug 9, 2026Nakul Poudel, Richard Simon, Cristian A. LinteImage Segmentation Text-Guided Image Segmentation
Center for Imaging Science, Rochester Institute of Technology, Rochester, NY 14623, USA · Biomedical Engineering, Rochester Institute of Technology, Rochester, NY 14623, USA
RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation
Apr 10, 2026Jieru Li, Matthew Chen, Micky C. Nnamdi +3Medical Image Segmentation Medical Imaging Foundation Models
Georgia Institute of Technology

Apr 28, 2026J/K move · Enter open · S save

Robustness Evaluation of a Foundation Segmentation Model Under Simulated Domain Shifts in Abdominal CT: Implications for Health Digital Twin Deployment

Sanghati Basu

Healthcare Informatics, University of Illinois Springfield

Distribution Shift Robustness Image Segmentation Medical Imaging Foundation Models Image Corruption Robustness Medical Image Segmentation Computed Tomography+1

Open paper AskPDF arXiv

Foundation segmentation models such as the Segment Anything Model (SAM) have demonstrated strong generalization across natural images; however, their robustness under clinically realistic medical imaging domain shifts remains insufficiently quantified. We present a systematic slice-level robustness audit of SAM (ViT-B) for spleen segmentation in abdominal CT using 1,051 nonempty slices from 41 volumes in the Medical Segmentation Decathlon. A standardized ground-truth-derived bounding-box protocol was used to isolate encoder robustness from prompt uncertainty. Controlled perturbations simulating inter-scanner variability, including Gaussian noise, blur, contrast scaling, gamma correction, and resolution mismatch, were applied across ten conditions. The clean baseline achieved a mean Dice score of 0.9145 (95% CI: [0.909, 0.919]) with a failure rate of 0.67%. Across all perturbations, the absolute mean ΔDice remained below 0.01. Paired Wilcoxon signed-rank tests with Benjamini-Hochberg false discovery rate correction identified statistically significant but small-magnitude changes under selected conditions, while McNemar analysis showed no significant increase in failure probability. These findings indicate that SAM exhibits stable segmentation behavior under moderate CT domain shifts, supporting its role as a robust foundation baseline for medical image segmentation research. As health digital twins increasingly incorporate foundation segmentation models for anatomical modeling and organ-level monitoring, formal characterization of robustness under real-world imaging variability is a necessary step toward trustworthy deployment.

Abstract

Amodal Segmentation

Explore similar work

Robustness Evaluation of a Foundation Segmentation Model Under Simulated Domain Shifts in Abdominal CT: Implications for Health Digital Twin Deployment

Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)

RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation