Prompting with the human-touch: evaluating model-sensitivity of foundation models for musculoskeletal CT segmentation

Caroline Magg; Maaike A. ter Wee; Johannes G. G. Dobbe; Geert J. Streekstra; Leendert Blankevoort; Clara I. Sánchez; Hoel Kervadec

cs.CVMar 11, 2026

Prompting with the human-touch: evaluating model-sensitivity of foundation models for musculoskeletal CT segmentation

Authors: Caroline Magg, Maaike A. ter Wee, Johannes G. G. Dobbe, Geert J. Streekstra, Leendert Blankevoort, Clara I. Sánchez, Hoel Kervadec

Organizations: Quantitative Healthcare Analysis (QurAI) Group, University of Amsterdam, Science Park 900, Amsterdam, 1098 XH, The Netherlands · Department Biomedical Engineering and Physics, Amsterdam UMC, Meibergdreef 9, Amsterdam, 1105 AZ, The Netherlands · Department Orthopaedics, Amsterdam UMC, Meibergdreef 9, Amsterdam, 1105 AZ, The Netherlands

Abstract

Promptable Foundation Models (FMs), initially introduced for natural image segmentation, have also revolutionized medical image segmentation. The increasing number of models, along with evaluations varying in datasets, metrics, and compared models, makes direct performance comparison between models difficult and complicates the selection of the most suitable model for specific clinical tasks. In our study, 11 promptable FMs are tested using non-iterative 2D and 3D prompting strategies on a private and public dataset focusing on bone and implant segmentation in four anatomical regions (wrist, shoulder, hip and lower leg). The Pareto-optimal models are identified and further analyzed using human prompts collected through a dedicated observer study. Our findings are: 1) The segmentation performance varies a lot between FMs and prompting strategies; 2) The Pareto-optimal models in 2D are SAM and SAM2.1, in 3D nnInteractive and Med-SAM2; 3) Localization accuracy and rater consistency vary with anatomical structures, with higher consistency for simple structures (wrist bones) and lower consistency for complex structures (pelvis, tibia, implants); 4) The segmentation performance drops using human prompts, suggesting that performance reported on "ideal" prompts extracted from reference labels might overestimate the performance in a human-driven setting; 5) All models were sensitive to prompt variations. While two models demonstrated intra-rater robustness, it did not scale to inter-rater settings. We conclude that the selection of the most optimal FM for a human-driven setting remains challenging, with even high-performing FMs being sensitive to variations in human input prompts. Our code base for prompt extraction and model inference is available: https://github.com/CarolineMagg/segmentation-FM-benchmark/

Figures & tables

Model	Arch.	Dim.	Data		Box	Point	Pt+Box	Slice	Vol.
			Domain	[ 35 ]	(1/ $N_{P}$ )	(1/ $N_{P}$ )	(1/ $N_{P}$ )	(1/ $N_{S}$ )	Limits
Sam [ 1 ]	ViT	2D	N	✗	✓/ ✓	✓/ ✓	✓/ ✓	-	-
Sam 2 2D [ 6 ]	Hiera	2D	N	✗	✓/ ✓	✓/ ✓	✓/ ✓	-	-
Med- Sam [ 8 ]	Sam	2D	M	✓	✓/ ✓	✗ / ✗	✗ / ✗	-	-
Sam -Med2D [ 9 ]	Sam	2D	M	✓	✓/ ✓	✓/ ✓	✓/ ✓	-	-
ScribblePrompt-U [ 10 ]	UNet	2D	M	(✓)*	✓/ ✓	✓/ ✓	✓/ ✓	-	-

Table 1: Overview of promptable FMs: Model backbone architecture, prediction dimensionality (2D vs. 3D), training data domain (Medical vs. Natural) and the supported prompting strategies. The prompting strategies are: single (1) or multiple (

N_{P}

) boxes, points, and their combinations, for single (1) or multiple (

N_{S}

) slices, with or without volumetric limitations (for 3D predictions). Boxed settings are our default settings, as they are possible across different models. (✓)* denotes that authors explicitly stated that the test set of [ 35 ] was excluded from training.

N_{P}

prompts denotes that multiple prompts (in our work, up to 5 prompts) per initial slice were used.

N_{S}

slices denotes that multiple initial slices (in our work, all selected slices) were used.

Panel of Figure 1 — Figure 1: Prompting strategies in 2D consist of prompt primitives, i.e., bounding box (a) and/or center point (b), and component selections, i.e., including prompts from either the largest component (blue prompt) or all components (white and blue prompts). The 3D prompting strategies extend this concept with slice selection (c).

Panel of Figure 2 — Figure 2: Dataset Overview: Our dataset consists of four subsets, i.e., Wrist, Lower Leg, Shoulder, and Hip [ 35 ] . A subset of 404 axial slices was extracted based on constraints ensuring data coverage, diversity and comparability.

Annotation accuracy in % $\uparrow$
Metrics	Bounding Box	Center Point
IoU	90.56 (83.4-94.5)	-
Localization error in mm $\downarrow$
Eucl. distance	0.49 (0.0-1.4)	1.50 (0.7-3.0)
$\|\Delta x\|$	0.00 (0.0-0.9)	0.98 (0.3-1.9)
$\|\Delta y\|$	0.00 (0.0-0.9)	0.97 (0.3-1.9)

Table 2: Annotation performance for human bounding box and center point variations, reported as median (IQR).

Agreement in % $\uparrow$
Metrics	Bounding Box		Point
	intra	inter	intra	inter
IoU	93.04 (88.5-96.1)	90.11 (84.2-94.0)	-	-
Variability in mm $\downarrow$
Eucl. distance	0.49 (0.0-1.4)	0.73 (0.3-1.5)	0.98 (0.5-1.9)	1.37 (0.7-2.6)
$\|\Delta x\|$	0.33 (0.0-0.9)	0.33 (0.0-1.0)	0.49 (0.0-1.5)	0.83 (0.0-1.6)

Table 3: Intra-rater and inter-rater annotation consistency for human bounding box and center point, reported as median (IQR).

Panel of Figure 3 — Figure 3: Spatial distribution of prompt placement per annotator per dataset. Each point corresponds to one annotator. It represents the mean deviations (mm) in the x- ( $\Delta x$ ) and y-directions ( $\Delta y$ ) of the center point (i.e., human (a) or extracted from the bounding box (b)), relative to the reference prompt at the origin (0,0). The same-colored (more transparent) ellipse (a) and rectangles (b) represent each annotator’s intra-rater consistency ((a): ( $\Delta x$ , $\Delta y$ ), (b): ( $\Delta w$ , $\Delta h$ )). Wrist shows the least localization errors and highest consistency, while Lower Leg and Hip show high localization errors and low intra- and inter-rater consistency.

Figure 4: Accumulated annotation time per annotator for all projects.

Panel of Table 4 — Table 4: Segmentation performance with reference prompts of Pareto-optimal models per prompt type (i.e., bounding box, center point, combination) and category (2D vs. 3D; medical vs. natural). The Pareto-optimal models with the least parameters per category are highlighted in bold (selected for further analysis with human prompts). Gray-shaded cells and prompt symbols next to the model names indicate the smallest Pareto-optimal models per prompt type. No Pareto-optimal results are omitted in this Table and can be found in Table 13 .

Panel of Figure 5 — Figure 5: DSC vs. HD95 (mm) performance of all models (color-encoded) and three prompt types (symbol-encoded) for perfect prompts. Larger symbols highlight the smallest Pareto-optimal models.

Panel of Table 5 — Table 5: Segmentation performance and intra-rater segmentation consistency with human prompts – grouped by prompt type (bounding box, center point, combination). The best values per prompt type are highlighted in bold. The best performing models also showed the highest consistency. The performance difference to perfect prompts is shown in Table 18 .

Panel of Table 6 — Table 6: Spearman’s rank correlation coefficients for each metric ( $\rho_{\text{DSC}}$ , $\rho_{\text{NSD}}$ , $\rho_{\text{HD95}}$ ) between intra-rater annotation variability and segmentation consistency. Asterisks ( $*$ ) denote statistical significance. Positive values for HD95 and negative values for DSC and NSD indicate that increased prompt variability significantly reduces segmentation consistency.

Panel of Table 7 — Table 7: Spearman’s rank correlation coefficients for each metric ( $\rho_{\text{DSC}}$ , $\rho_{\text{NSD}}$ , $\rho_{\text{HD95}}$ ) between inter-rater annotation variability and segmentation consistency. Annotator rows are ordered by euclidean distance (mm), starting with the lowest-variance rater. Due to iterative search, not all inter-rater variabilities are tested (Table 12 ). The first column indicates the order of tests. Asterisks ( $*$ ) denote statistical significance. Positive values for HD95 and negative values for DSC and NSD indicate that increased prompt variability significantly reduces segmentation consistency.

Panel of Figure 6 — Figure 6: Model sensitivity to input variations visualized as intra-rater annotation variability (euclidean distance or IoU) vs. segmentation consistency (DSC). Each point represent the mean prompt variability and mean segmentation consistency for one sample. Dotted lines represent ordinary least squares (OLS) linear regression trends. Shaded areas denote the $95\%$ confidence intervals ( $\alpha=0.05$ ).

Panel of Figure 7 — Figure 7: Visual examples for model sensitivity to input prompt variations: reference mask, predicted mask with reference prompt, predicted mask with 1st set of human prompt and with 2nd set of the same annotator. The reference prompt is drawn as black point or box. The human prompt is drawn as colored point or box.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8: Euclidean distances (mm) of human center points compared to reference center points measured as median and IQR.

Table 16

Panel of Figure 9 — Figure 9: Spatial distribution of mean $\Delta x$ and $\Delta y$ per annotator per class label. The same-colored (more transparent) ellipse represent each annotator’s intra-rater consistency ( $\Delta x$ , $\Delta y$ ).

Panel of Figure 10 — Figure 10: Examples for center point annotations: Center points with low euclidean distance (mm) (top row) and high values (bottom row) per data subset. Black dots are automatically extracted reference annotation, annotators’ annotations are color-encoded.

Table 10: IoU (%) of human bounding boxes compared to referenc bounding boxes measured as median and IQR.

Table 21

Panel of Figure 12 — Figure 12: Spatial distribution of the mean $\Delta x$ and $\Delta y$ per annotator per class label. The same-colored (more transparent) rectangle represents each annotator’s intra-rater consistency ( $\Delta w$ , $\Delta h$ ).

Panel of Figure 13 — Figure 13: Examples for bounding box annotations: Boxes with high IoU (%) (top row) and low values (bottom row) per data subset. Black dots are automatically extracted reference annotation, annotators’ annotations are color-encoded.

Table 12: Ranking of inter-rater variability, measured by averaged euclidean distance (mm), per annotator, starting with the lowest variability. The euclidean distance (mm) is averaged for all comparisons of one annotator to all other annotators. For the combination prompt, the euclidean distance, averaged from center point and bounding box analysis, is used for the ranking, because it considers both prompts. Annotators highlighted in bold have been used in the iterative search approach.

Panel of Table 13 — Table 13: Segmentation performance of all 2D and 3D models per prompt type. Gray-shaded cells indicate the smallest 2D and 3D Pareto-optimal models per prompt type. Omitted results (-) mean that the experiment was not performed, since it was not supported (see Table 1 ).

Panel of Figure 14 — Figure 14: Axial slice of Wrist with lowest DSC value (69.9%) across 2D models. The predictions are binary and were combined for visualization; as a result, some predicted regions may not appear because each pixel can only be assigned a single label.

Panel of Figure 15 — Figure 15: Axial slice of Lower Leg with lowest DSC value (62.1%) across 2D models. The predictions are binary and were combined for visualization; as a result, some predicted regions may not appear because each pixel can only be assigned a single label.

Panel of Figure 16 — Figure 16: Axial slice of Shoulder with lowest DSC value (75.1%) across 2D models. The predictions are binary and were combined for visualization; as a result, some predicted regions may not appear because each pixel can only be assigned a single label.

Panel of Table 14 — Table 14: Comparison of 2D segmentation performance of all model sizes of SAM2 and SAM2.1 per prompt type. $\nearrow$ indicates that all metrics improve, whereas – denotes no consistent trend across metrics. Asterisk ( $*$ ) marks statistically significant differences between models ( $p$ -value $<0.05/12=0.0042$ .

Panel of Table 15 — Table 15: Comparison of volumetric prediction without (default setting) and with propagation limitation, per prompt type.

Panel of Table 16 — Table 16: Comparison of volumetric prediction with a single initial slice (default setting) or all initial slices, per prompt type. $\nearrow$ indicates that all metrics improve, whereas $\searrow$ indicates that all metrics deteriorate. Asterisk ( $*$ ) marks statistically significant differences between models.

Panel of Table 17 — Table 17: Comparison of volumetric prediction with a single (default setting) or multiple (up to 5) prompts, per prompt type. $\nearrow$ indicates that all metrics improve, $\searrow$ indicates that all metrics deteriorate, whereas – denotes no consistent trend across metrics. An asterisk (*) marks statistically significant differences between models.

Panel of Table 18 — Table 18: Difference in segmentation performance between reference and human prompts, per prompt type. The models with the least difference per prompt type are highlighted in bold. The selected models are the smallest Pareto-optimal models prompted with reference prompts per category highlighted in bold in Table 4 .

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

MedSAM2-Anatomy: Training-Free Inference-Time Optimization for Musculoskeletal Segmentation
Jul 31, 2026John Garcia Henao, Nicholas Bünger, Benedikt Herzog +11Medical Imaging Foundation Models Inference-Time Optimization
Digital Medicine Unit Team, Balgrist University Hospital, 8008 Zurich, Switzerland · Research Centre for Computational Health, Zurich University of Applied Sciences (ZHAW), 8820 Wädenswil, Switzerland · Hip and Pelvis Surgery Team, Balgrist University Hospital, 8008 Zurich, Switzerland +1
Few-Shot Concept Prompt Learning for Segmentation Foundation Models via Visual Grounding
Aug 3, 2026Rahul Venkataramani, Rachana SathishMedical Imaging Foundation Models Promptable Image Segmentation
Advanced Technology Group, GE HealthCare
Atlas is Your Perfect Context: One-Shot Customization for Generalizable Foundational Medical Image Segmentation
Dec 20, 2025Ziyu Zhang, Yi Yu, Simeng Zhu +4Image Segmentation Medical Imaging Foundation Models
Nanjing University · The Ohio State University · Stanford University

Jul 31, 2026J/K move · Enter open · S save

MedSAM2-Anatomy: Training-Free Inference-Time Optimization for Musculoskeletal Segmentation

John Garcia Henao, Nicholas Bünger, Benedikt Herzog, Cindy Guerrero Toro+10

Digital Medicine Unit Team, Balgrist University Hospital, 8008 Zurich, Switzerland · Research Centre for Computational Health, Zurich University of Applied Sciences (ZHAW), 8820 Wädenswil, Switzerland · Hip and Pelvis Surgery Team, Balgrist University Hospital, 8008 Zurich, Switzerland+1

Medical Imaging Foundation Models Inference-Time Optimization Promptable Image Segmentation Medical Image Segmentation 3D Medical Image Segmentation

Open paper AskPDF arXiv

High-resolution 3D segmentation of hip and shoulder anatomy from CT and MRI is essential for surgical planning, yet frozen segmentation models often fail under domain shift. CNN-based expert models are fully automatic but lack adaptability, whereas promptable foundation models generalize better but require manual prompting. We present MedSAM2-Anatomy, a training-free inference-time optimization framework that improves frozen segmentation models without retraining or human interaction. A frozen expert model generates anatomical priors that are automatically converted into multiple prompt hypotheses for a frozen 3D foundation model. Candidate masks are fused while anatomically implausible priors are rejected. No model weights are updated and no manual prompts are required. TotalSegmentator and MedSAM2 are used as representative expert and foundation models, allowing the contribution of the inference policy to be isolated. Evaluation on the independent Balgrist-V0 CT and MRI cohorts shows that inference-time optimization increases median Dice from 0.71 to 0.92 on hip MRI and from 0.89 to 0.92 on shoulder CT, while reducing median HD95 on hip MRI from 22.0 mm to 5.0 mm. On public TotalSegmentator benchmarks, the expert model remains strongest, indicating that the optimal fusion strategy depends on the reliability of the expert prior. These results demonstrate that training-free inference-time optimization provides a practical strategy for improving frozen segmentation models without manual prompting.

Abstract

Medical Image Segmentation

Explore similar work

MedSAM2-Anatomy: Training-Free Inference-Time Optimization for Musculoskeletal Segmentation

Few-Shot Concept Prompt Learning for Segmentation Foundation Models via Visual Grounding

Atlas is Your Perfect Context: One-Shot Customization for Generalizable Foundational Medical Image Segmentation