cs.CVMar 24, 2026
SaveAutomatic Segmentation of 3D CT scans with SAM2 using a zero-shot approach
Organizations: Department of Computer Science, EPFL, Switzerland
Abstract
Foundation models for image segmentation have shown strong generalization in natural images, yet their applicability to 3D medical imaging remains limited. In this work, we study the zero-shot use of Segment Anything Model 2 (SAM2) for automatic segmentation of volumetric CT data, without any fine-tuning or domain-specific training. We analyze how SAM2 should be applied to CT volumes and identify its main limitation: the lack of inherent volumetric awareness. To address this, we propose a set of inference-alone architectural and procedural modifications that adapt SAM2's video-based memory mechanism to 3D data by treating CT slices as ordered sequences. We conduct a systematic ablation study on a subset of 500 CT scans from the TotalSegmentator dataset to evaluate prompt strategies, memory propagation schemes and multi-pass refinement. Based on these findings, we select the best-performing configuration and report final results on a bigger sample of the TotalSegmentator dataset comprising 2,500 CT scans. Our results show that, even with frozen weights, SAM2 can produce coherent 3D segmentations when its inference pipeline is carefully structured, demonstrating the feasibility of a fully zero-shot approach for volumetric medical image segmentation.
Figures & tables
Fig. 1 : Adaptation of SAM2 to 3D CT volumes . A CT scan is interpreted as a pseudo-video by treating the depth ( ) axis as the temporal dimension. At each slice , the input image is processed by the image encoder to produce an embedding. Embeddings from prompted slices (first last frame in the figure) and a fixed number of neighboring slices are stored in the memory bank and augmented with learned temporal embeddings. Prompt-conditioned frames are assigned a zero temporal offset, while non-conditioned frames use learned temporal embeddings. The memory attention module attends to a fixed window of six frames, producing a fused representation that is passed to the mask decoder to generate the segmentation mask for slice .
Fig. 2 : Multi-axis propagation and fusion strategy . A 3D CT volume is segmented independently along the axial, sagittal, and coronal axes using SAM2. Each axis produces a logit volume, which are reoriented to a common reference frame and merged to obtain the final segmentation.
Fig. 3 : Ablation study on memory design and propagation strategies. (a) Effect of the prompt-conditioned memory threshold, showing improved performance when only nearby prompted slices are retained. (b) (i) Influence of non-conditioned memory size on performance, and (ii) cosine similarity of temporal embeddings, highlighting redundancy among intermediate past frames. (c) Per-category segmentation performance, illustrating variability across anatomical structures.(d) Performance comparison of different volume propagation strategies under varying prompt budgets, including Forward–Backward and Three-Axis variants.
| Method | Dice | IoU | HD |
| NP | 0.594 0.242 | 0.509 0.227 | 11.177 9.366 |
| Baseline | 0.804 0.134 | 0.734 0.148 | 6.240 6.563 |
| SPS | 0.823 0.117 | 0.751 0.134 | 5.236 5.740 |
| IS | 0.822 0.125 | 0.759 0.138 | 5.683 5.178 |
| IS + SPS | 0.841 0.107 | 0.778 0.122 | 4.788 4.930 |
TABLE I : Evaluation on the full dataset (2,500 volumes). NP refers to No Preprocessing , SPS to Structured Prompt Selection (temporal threshold of 0.3), and IS denotes Intelligent Slicing .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 4 : Comparative Runtime analysis across different configurations. Evaluation was performed on a set of 50 bones.
| Anatomical Structure | Specific Labels | Count (Each) | Total Raw Masks |
| Ribs | rib_left_1..12 , rib_right_1..12 | 1,228 | 29,472 |
| Vertebrae | C1..7 , T1..12 , L1..5 , S1 | 1,228 | 30,700 |
| Femur | femur_left , femur_right | 1,228 | 2,456 |
| Humerus | humerus_left , humerus_right | 1,228 | 2,456 |
| Clavicle | clavicula_left , clavicula_right | 1,228 | 2,456 |
| Scapula | scapula_left , scapula_right | 1,228 | 2,456 |
TABLE II: Raw Data Distribution (Pre-Consolidation). Availability of anatomical structures in the TotalSegmentator dataset. Note that all selected structures possess complete annotation coverage across the 1,228 scans.
| Target Class | Aggregation Type | Constituent Raw Labels |
| Vertebrae | Serial Union | |
| Rib | Serial & Bilateral | |
| Hip | Bilateral Union | |
| Humerus | Bilateral Union | |
| Femur | Bilateral Union | |
| Clavicle | Bilateral Union |
TABLE III: Class Aggregation Mapping. The mapping function used to consolidate raw labels into the final ten bone classes. The aggregation logic accounts for lateral symmetry (left/right) and serial structures.
| Bone Class | Consolidated Total Available | Ablation Subset ( ) | Final Evaluation Subset ( ) |
| Vertebrae | 30,700 | 50 | 250 |
| Rib | 29,472 | 50 | 250 |
| Humerus | 2,456 | 50 | 250 |
| Clavicle | 2,456 | 50 | 250 |
| Hip | 2,456 | 50 | 250 |
| Scapula | 2,456 | 50 | 250 |
TABLE IV: Final Balanced Dataset for Ablation. The subset of 500 volumes curated for the ablation study, ensuring equal representation ( ) across the ten aggregated classes.
| Model Scale | Preprocessing | Prompt Strategy | IoU | Dice | Hausdorff |
| Experiment 1: Varying Model Scale | |||||
| Tiny | Windowing Alone | F–M–L | |||
| Base | Windowing Alone | F–M–L | |||
| Small | Windowing Alone | F–M–L | |||
| Large | Windowing Alone | F–M–L | |||
| Experiment 2: Varying Preprocessing | |||||
TABLE V: Ablation study establishing the baseline configuration . We isolate the impact of each component by varying one parameter at a time while fixing the others to their default settings. The Small model is selected as the baseline because it approximates the performance of the Large variant with significantly lower computational cost. For preprocessing, we compare standard intensity windowing against Contrast-Limited Adaptive Histogram Equalization (CLAHE); results show that Windowing Alone yields superior segmentation, improving performance by 30% compared to no preprocessing. Finally, the First–Middle–Last strategy is adopted as the prompt baseline, as it provides the most effective trade-off between performance and annotation budget compared to denser prompting strategies.
| Category | Variant | IoU | Dice |
| Experiment 1: Memory Bank Temporal Stride | |||
| Memory Bank | Stride 1 (Baseline) | ||
| Memory Bank | Stride 2 | ||
| Memory Bank | Stride 4 | ||
| Experiment 2: Confidence Gating Strategies | |||
| Gating | Gate 6 | ||
TABLE VI : Comprehensive ablation study on inference design choices. Stride-based memory sampling shows a small benefit at moderate strides (Stride 4). Confidence gating does not improve performance and generally degrades results compared to the baseline. Reducing memory capacity(without changing temporal embeddings) hurts performance, while intelligent slicing provides the best trade-off. Multi-axis propagation is only beneficial when using a larger prompt budget, and a moderate temporal threshold ( ) yields the best performance.
| Bone Class | NP | Base | SPS | IS | IS + SPS | Dice |
| Vertebrae | ||||||
| Rib | ||||||
| Humerus | ||||||
| Clavicle | ||||||
| Hip | ||||||
| Scapula |
TABLE VII: Per-bone segmentation performance on the final evaluation set (2,500 volumes). Results are reported as Dice / IoU. NP: No Preprocessing; IS: Inteligent Slicing (proposed); SPS: Structured Prompt Selection. Dice indicates absolute change of IS + SPS relative to Base.
Fig. 5 : Vertebra segmentation produced by the baseline model. The proposed configuration (IS), and the ground truth. Each triplet corresponds to a slice of the CT volume. Vertebrae exhibit strong slice-to-slice appearance changes, making consistent tracking across the volume challenging. The proposed method produces more consistent segmentations across slices and successfully tracks the correct vertebra throughout the volume. While some predicted masks remain imprecise, the target structure is preserved. In contrast, the baseline model frequently segments incorrect objects (e.g., slices 100 and 111), likely due to confusion with visually similar structures in distant slices that are included in its memory.