ATLAS: A Multi-Procedure Benchmark and Context-Aware Model for Surgical Anatomy Segmentation
Authors: Ronald L. P. D. de Jong, Tim J. M. Jaspers, Raf A. H. Vervoort, Aron F. H. A. Bakker, Anna L. Dijk, Yiping Li, Jip L. Tolenaar, Jelle P. Ruurda, Willem M. Brinkman, Daan de Geus, Fons van der Sommen, Josien P. W. Pluim, Marcel Breeuwer
Organizations: Department of Biomedical Engineering, Eindhoven University of Technology, Groene Loper 3, 5612 AE Eindhoven, The Netherlands · Department of Electrical Engineering, Eindhoven University of Technology, Groene Loper 3, 5612 AE Eindhoven, The Netherlands · Department of Oncological Urology, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands · Department of Urology, Catharina Hospital, Michelangelolaan 2, 5623 EJ Eindhoven, The Netherlands · Department of Technical Medicine, Delft University of Technology, Mekelweg 2, 2628 CD Delft, The Netherlands · Department of Surgery, Haaglanden Medical Center, Lijnbaan 32, 2512 VA The Hague, The Netherlands · Department of Surgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
Recognizing anatomical structures during minimally invasive surgery (MIS) could support intraoperative navigation and improve patient safety, but progress is limited by small, single-procedure datasets and by segmentation methods developed for natural scenes. This work addresses both limitations through a new dataset and model. First, we introduce ATLAS-120k, a clip-level semantic segmentation dataset of 121,018 annotated frames from 100 surgical videos, covering 14 laparoscopic and robot-assisted procedures and 41 classes. Annotations were created with a pipeline that combines manual labeling, tracker-based propagation, and review by surgeons. Second, we propose ATLAS (Anatomy Recognition with Context Learning using Foundation Representations), a video semantic segmentation model for surgical anatomy recognition. ATLAS builds on in-domain foundation-model embeddings and adds context queries for the procedure and surgical phase, together with query propagation between frames for short-term memory. ATLAS outperforms state-of-the-art natural-image and in-domain foundation models in both segmentation accuracy and temporal consistency, achieving a Dice score of 0.70 with ViT-L while running in real time at 76.2 FPS. Ablations show that the context queries improve accuracy and that query propagation improves temporal consistency. ATLAS also generalizes well, matching a dedicated model zero-shot on CholecSeg8k and improving phase recognition on AutoLaparo over SurgeNet initialization. The model weights, dataset annotations, and annotation tool are publicly available at https://github.com/TimJaspers0801/ATLAS.
Figures & tables
Figure 1: Randomly selected frames from ATLAS-120k. Odd rows show the original frames and even rows the same frames with their annotations overlaid. Each of the 14 procedures and each semantic class is represented in at least one example.
Dataset
Included
#
#
#
#
MIS
RA
procedures
classes
videos
clips
frames
Endoscapes-Seg50 ( Mascagni et al., 2025 )
1
6
50
-
493
✓
CholecSeg8k ( Hong et al., 2020 )
1
12
17
101
8,080
✓
DSAD ( Carstens et al., 2023 )
1
11
32
-
13,195
✓
ATLAS-120k
14
41
100
502
121,018
✓
✓
Table 1: Comparison of surgical video segmentation dataset characteristics. MIS indicates minimally invasive surgery; RA indicates robot-assisted procedures.
Figure 2: ATLAS-120k dataset characteristics.
Figure 3: t-SNE projection of SurgeNet ( Jaspers et al., 2026 ) frame embeddings; each point is one frame, and frames with similar features lie close together. Left: ATLAS-120k compared with Endoscapes-Seg50, CholecSeg8k and DSAD, colored by dataset. Right: ATLAS-120k colored by procedure.
Figure 4: Overview of ATLAS. For each frame, a ViT encoder jointly processes the patch tokens Pt and the query tokens Qt , which consist of segmentation, procedure and phase queries. The output queries Q^t predict the masks and classes, the procedure label and a phase embedding zt . They are then propagated to the next frame as Qt+1=Linear(Q^t)+Q0 . Only past frames are used, so the model runs online. This schematic was created with the assistance of Claude (Anthropic).
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
Inter-observer
0.85 ± 0.10
0.10 ± 0.08
0.96 ± 0.05
0.94 ± 0.08
ATLAS ViT-L
0.70 ± 0.13
0.17 ± 0.11
0.85 ± 0.10
0.79 ± 0.15
Table 2: Inter-observer variability compared with ATLAS ViT-L on the full test set. Values are the mean ± standard deviation over clips.
Figure 5: Dice score between annotations propagated from the first frame without manual correction and the final test-set annotations, as a function of the position in the clip. Results are shown for the tracker without fine-tuning and after fine-tuning on 10–100% of the training set.
Model
Backbone
#PARAMS (M) ↓
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
DINOv2 ( Oquab et al., 2024 )
ViT-B
87
0.49 ± 0.14
0.32 ± 0.13
0.70 ± 0.13
0.59 ± 0.18
DINOv2 ( Oquab et al., 2024 )
ViT-L
304
0.36 ± 0.12
0.38 ± 0.12
0.58 ± 0.15
0.46 ± 0.19
DINOv3 ( Siméoni et al., 2025 )
ViT-B
86
0.56 ± 0.14
0.26 ± 0.12
0.73 ± 0.14
0.64 ± 0.18
DINOv3 ( Siméoni et al., 2025 )
ViT-L
303
0.55 ± 0.14
0.26 ± 0.11
0.72 ± 0.14
0.63 ± 0.19
SAM2-UNet ( Xiong et al., 2026 )
Hiera-L
217
0.54 ± 0.14
0.27 ± 0.11
0.66 ± 0.17
0.55 ± 0.21
SAM3-UNet ( Xiong et al., 2025 )
Hiera-L
449
0.60 ± 0.14
0.24 ± 0.11
0.76 ± 0.14
0.67 ± 0.20
Table 3: Quantitative comparison of foundation models for segmentation. Models are grouped by pretraining domain: natural-image foundation models (top), in-domain pretrained medical models (middle), and our proposed ATLAS models (bottom). Values are the mean ± standard deviation over test clips. For DINOv2 and DINOv3, we train a linear layer attached to the ViT. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value. ∗ Statistically significant improvement over the best baseline for that metric, i.e., the best-performing non-ATLAS model ( p<0.01 , Wilcoxon signed-rank test on paired per-clip scores).
Figure 6: Qualitative segmentation results of ATLAS ViT-L on three test clips: robotic cholecystectomy (top), robot-assisted radical prostatectomy (RARP, middle) and robotic hemicolectomy (bottom). Each example shows five frames sampled every 2 s from the same clip.
Model
#Params (M)
Latency (ms) ↓
FPS ↑
ATLAS ViT-S
23.8
3.79
263.7
ATLAS ViT-B
94.3
6.24
160.3
ATLAS ViT-L
318.3
13.16
76.2
Table 4: Inference speed of ATLAS on a single NVIDIA H100 GPU.
Class
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
Tools/camera
0.94
0.03
0.87
0.81
Lung
0.91
0.04
0.83
0.77
Liver
0.89
0.11
0.87
0.83
Gallbladder
0.86
0.07
0.84
0.79
Non-anatomical
0.83
0.18
0.57
0.47
Ovary
0.74
0.18
0.44
0.29
Table 5: Per-class results of ATLAS ViT-L on the ATLAS-120k test set, sorted by pooled Dice. Pooled Dice is computed over all test frames together; HD95 is averaged over frames; mVC12 and mVC24 are averaged over the test clips in which the class occurs. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
Method
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
Full
0.70 ± 0.13
0.17 ± 0.11
0.85 ± 0.10
0.79 ± 0.15
w/o in-domain pretr.
0.67 ± 0.13
0.20 ± 0.10
0.82 ± 0.11
0.76 ± 0.15
w/o temporal prop.
0.69 ± 0.13
0.18 ± 0.10
0.82 ± 0.12
0.76 ± 0.16
w/o procedure queries
0.69 ± 0.13
0.19 ± 0.11
0.84 ± 0.11
0.79 ± 0.15
w/o phase queries
0.67 ± 0.14
0.23 ± 0.12
0.85 ± 0.10
0.80 ± 0.14
Table 6: Ablation studies showing the effect of in-domain pretraining, temporal propagation, and procedure and phase queries on surgical video segmentation. Values are the mean ± standard deviation over test clips. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
Clip length
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
1 (no temporal prop.)
0.69 ± 0.13
0.18 ± 0.10
0.82 ± 0.12
0.76 ± 0.16
5 (ATLAS)
0.70 ± 0.13
0.17 ± 0.11
0.85 ± 0.10
0.79 ± 0.15
10
0.70 ± 0.13
0.19 ± 0.10
0.85 ± 0.10
0.80 ± 0.14
15
0.69 ± 0.14
0.21 ± 0.12
0.85 ± 0.10
0.80 ± 0.14
Table 7: Effect of the clip length used for temporal propagation on surgical video segmentation. Values are the mean ± standard deviation over test clips. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
Figure 7: Effect of training data size on segmentation accuracy (mDice, HD95) and temporal consistency (mVC12, mVC24). The x-axis shows the fraction of frames from each training clip used for training and ”First 5 frames” corresponds to five annotated frames per clip.
Setup
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
Zero-shot
0.83 ± 0.08
0.12 ± 0.06
0.94 ± 0.03
0.92 ± 0.03
SurgeNet
0.83 ± 0.10
0.13 ± 0.08
0.96 ± 0.02
0.95 ± 0.02
Table 8: Zero-shot transfer of ATLAS to CholecSeg8k compared with training with SurgeNet initialization on CholecSeg8k.
Dataset
Init
Acc ↑
Bal. Acc ↑
F1 ↑
Jaccard ↑
AutoLaparo
SurgeNet
0.82 ± 0.09
0.73 ± 0.07
0.73 ± 0.08
0.63 ± 0.09
ATLAS
0.85 ± 0.07
0.77 ± 0.08
0.76 ± 0.09
0.67 ± 0.10
Table 9: Surgical phase recognition on AutoLaparo when initializing from SurgeNet versus from ATLAS pretraining.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Train
Val
Test
Annotated class
Evaluated as
% frames
Videos
% frames
Videos
% frames
Videos
Tools/camera
97.9
70
98.6
10
99.1
20
Omentum
Fat
41.4
38
60.0
7
57.4
14
Abdominal wall
42.2
43
42.2
8
48.4
12
Liver
31.3
29
56.8
7
50.8
12
Colon/rectum
32.4
26
11.4
3
27.3
8
Appendix
Table 10: Class distribution over the training, validation and test splits of ATLAS-120k, sorted by the total number of annotated frames. For each split, the percentage of annotated frames in which the class occurs and the number of videos that contain it are given. Classes that were merged into a parent class for training and evaluation list that parent class in the second column; classes marked – were relabeled as background.
Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Electrical Engineering, Architectures for Reliable Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands +3
University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom · Vanderbilt University Medical Center Nashville, Tennessee, USA +3