ATLAS: A Multi-Procedure Benchmark and Context-Aware Model for Surgical Anatomy Segmentation
Authors: Ronald L. P. D. de Jong, Tim J. M. Jaspers, Raf A. H. Vervoort, Aron F. H. A. Bakker, Anna L. Dijk, Yiping Li, Jip L. Tolenaar, Jelle P. Ruurda, Willem M. Brinkman, Daan de Geus, Fons van der Sommen, Josien P. W. Pluim, Marcel Breeuwer
Organizations: Department of Biomedical Engineering, Eindhoven University of Technology, Groene Loper 3, 5612 AE Eindhoven, The Netherlands · Department of Electrical Engineering, Eindhoven University of Technology, Groene Loper 3, 5612 AE Eindhoven, The Netherlands · Department of Oncological Urology, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands · Department of Urology, Catharina Hospital, Michelangelolaan 2, 5623 EJ Eindhoven, The Netherlands · Department of Technical Medicine, Delft University of Technology, Mekelweg 2, 2628 CD Delft, The Netherlands · Department of Surgery, Haaglanden Medical Center, Lijnbaan 32, 2512 VA The Hague, The Netherlands · Department of Surgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
Recognizing anatomical structures during minimally invasive surgery (MIS) could support intraoperative navigation and improve patient safety, but progress is limited by small, single-procedure datasets and by segmentation methods developed for natural scenes. This work addresses both limitations through a new dataset and model. First, we introduce ATLAS-120k, a clip-level semantic segmentation dataset of 121,018 annotated frames from 100 surgical videos, covering 14 laparoscopic and robot-assisted procedures and 41 classes. Annotations were created with a pipeline that combines manual labeling, tracker-based propagation, and review by surgeons. Second, we propose ATLAS (Anatomy Recognition with Context Learning using Foundation Representations), a video semantic segmentation model for surgical anatomy recognition. ATLAS builds on in-domain foundation-model embeddings and adds context queries for the procedure and surgical phase, together with query propagation between frames for short-term memory. ATLAS outperforms state-of-the-art natural-image and in-domain foundation models in both segmentation accuracy and temporal consistency, achieving a Dice score of 0.70 with ViT-L while running in real time at 76.2 FPS. Ablations show that the context queries improve accuracy and that query propagation improves temporal consistency. ATLAS also generalizes well, matching a dedicated model zero-shot on CholecSeg8k and improving phase recognition on AutoLaparo over SurgeNet initialization. The model weights, dataset annotations, and annotation tool are publicly available at https://github.com/TimJaspers0801/ATLAS.
Figures & tables
Figure 1: Randomly selected frames from ATLAS-120k. Odd rows show the original frames and even rows the same frames with their annotations overlaid. Each of the 14 procedures and each semantic class is represented in at least one example.
Dataset
Included
#
#
#
#
MIS
RA
procedures
classes
videos
clips
frames
Endoscapes-Seg50 ( Mascagni et al., 2025 )
1
6
50
-
493
✓
CholecSeg8k ( Hong et al., 2020 )
1
12
17
101
8,080
✓
DSAD ( Carstens et al., 2023 )
1
11
32
-
13,195
✓
ATLAS-120k
14
41
100
502
121,018
✓
✓
Table 1: Comparison of surgical video segmentation dataset characteristics. MIS indicates minimally invasive surgery; RA indicates robot-assisted procedures.
Figure 2: ATLAS-120k dataset characteristics.
Figure 3: t-SNE projection of SurgeNet ( Jaspers et al., 2026 ) frame embeddings; each point is one frame, and frames with similar features lie close together. Left: ATLAS-120k compared with Endoscapes-Seg50, CholecSeg8k and DSAD, colored by dataset. Right: ATLAS-120k colored by procedure.
Figure 4: Overview of ATLAS. For each frame, a ViT encoder jointly processes the patch tokens Pt and the query tokens Qt , which consist of segmentation, procedure and phase queries. The output queries Q^t predict the masks and classes, the procedure label and a phase embedding zt . They are then propagated to the next frame as Qt+1=Linear(Q^t)+Q0 . Only past frames are used, so the model runs online. This schematic was created with the assistance of Claude (Anthropic).
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
Inter-observer
0.85 ± 0.10
0.10 ± 0.08
0.96 ± 0.05
0.94 ± 0.08
ATLAS ViT-L
0.70 ± 0.13
0.17 ± 0.11
0.85 ± 0.10
0.79 ± 0.15
Table 2: Inter-observer variability compared with ATLAS ViT-L on the full test set. Values are the mean ± standard deviation over clips.
Figure 5: Dice score between annotations propagated from the first frame without manual correction and the final test-set annotations, as a function of the position in the clip. Results are shown for the tracker without fine-tuning and after fine-tuning on 10–100% of the training set.
Model
Backbone
#PARAMS (M) ↓
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
DINOv2 ( Oquab et al., 2024 )
ViT-B
87
0.49 ± 0.14
0.32 ± 0.13
0.70 ± 0.13
0.59 ± 0.18
DINOv2 ( Oquab et al., 2024 )
ViT-L
304
0.36 ± 0.12
0.38 ± 0.12
0.58 ± 0.15
0.46 ± 0.19
DINOv3 ( Siméoni et al., 2025 )
ViT-B
86
0.56 ± 0.14
0.26 ± 0.12
0.73 ± 0.14
0.64 ± 0.18
DINOv3 ( Siméoni et al., 2025 )
ViT-L
303
0.55 ± 0.14
0.26 ± 0.11
0.72 ± 0.14
0.63 ± 0.19
SAM2-UNet ( Xiong et al., 2026 )
Hiera-L
217
0.54 ± 0.14
0.27 ± 0.11
0.66 ± 0.17
0.55 ± 0.21
SAM3-UNet ( Xiong et al., 2025 )
Hiera-L
449
0.60 ± 0.14
0.24 ± 0.11
0.76 ± 0.14
0.67 ± 0.20
Table 3: Quantitative comparison of foundation models for segmentation. Models are grouped by pretraining domain: natural-image foundation models (top), in-domain pretrained medical models (middle), and our proposed ATLAS models (bottom). Values are the mean ± standard deviation over test clips. For DINOv2 and DINOv3, we train a linear layer attached to the ViT. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value. ∗ Statistically significant improvement over the best baseline for that metric, i.e., the best-performing non-ATLAS model ( p<0.01 , Wilcoxon signed-rank test on paired per-clip scores).
Figure 6: Qualitative segmentation results of ATLAS ViT-L on three test clips: robotic cholecystectomy (top), robot-assisted radical prostatectomy (RARP, middle) and robotic hemicolectomy (bottom). Each example shows five frames sampled every 2 s from the same clip.
Model
#Params (M)
Latency (ms) ↓
FPS ↑
ATLAS ViT-S
23.8
3.79
263.7
ATLAS ViT-B
94.3
6.24
160.3
ATLAS ViT-L
318.3
13.16
76.2
Table 4: Inference speed of ATLAS on a single NVIDIA H100 GPU.
Class
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
Tools/camera
0.94
0.03
0.87
0.81
Lung
0.91
0.04
0.83
0.77
Liver
0.89
0.11
0.87
0.83
Gallbladder
0.86
0.07
0.84
0.79
Non-anatomical
0.83
0.18
0.57
0.47
Ovary
0.74
0.18
0.44
0.29
Table 5: Per-class results of ATLAS ViT-L on the ATLAS-120k test set, sorted by pooled Dice. Pooled Dice is computed over all test frames together; HD95 is averaged over frames; mVC12 and mVC24 are averaged over the test clips in which the class occurs. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
Method
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
Full
0.70 ± 0.13
0.17 ± 0.11
0.85 ± 0.10
0.79 ± 0.15
w/o in-domain pretr.
0.67 ± 0.13
0.20 ± 0.10
0.82 ± 0.11
0.76 ± 0.15
w/o temporal prop.
0.69 ± 0.13
0.18 ± 0.10
0.82 ± 0.12
0.76 ± 0.16
w/o procedure queries
0.69 ± 0.13
0.19 ± 0.11
0.84 ± 0.11
0.79 ± 0.15
w/o phase queries
0.67 ± 0.14
0.23 ± 0.12
0.85 ± 0.10
0.80 ± 0.14
Table 6: Ablation studies showing the effect of in-domain pretraining, temporal propagation, and procedure and phase queries on surgical video segmentation. Values are the mean ± standard deviation over test clips. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
Clip length
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
1 (no temporal prop.)
0.69 ± 0.13
0.18 ± 0.10
0.82 ± 0.12
0.76 ± 0.16
5 (ATLAS)
0.70 ± 0.13
0.17 ± 0.11
0.85 ± 0.10
0.79 ± 0.15
10
0.70 ± 0.13
0.19 ± 0.10
0.85 ± 0.10
0.80 ± 0.14
15
0.69 ± 0.14
0.21 ± 0.12
0.85 ± 0.10
0.80 ± 0.14
Table 7: Effect of the clip length used for temporal propagation on surgical video segmentation. Values are the mean ± standard deviation over test clips. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
Figure 7: Effect of training data size on segmentation accuracy (mDice, HD95) and temporal consistency (mVC12, mVC24). The x-axis shows the fraction of frames from each training clip used for training and ”First 5 frames” corresponds to five annotated frames per clip.
Setup
Dice ↑
HD95 ↓
mVC12 ↑
mVC24 ↑
Zero-shot
0.83 ± 0.08
0.12 ± 0.06
0.94 ± 0.03
0.92 ± 0.03
SurgeNet
0.83 ± 0.10
0.13 ± 0.08
0.96 ± 0.02
0.95 ± 0.02
Table 8: Zero-shot transfer of ATLAS to CholecSeg8k compared with training with SurgeNet initialization on CholecSeg8k.
Dataset
Init
Acc ↑
Bal. Acc ↑
F1 ↑
Jaccard ↑
AutoLaparo
SurgeNet
0.82 ± 0.09
0.73 ± 0.07
0.73 ± 0.08
0.63 ± 0.09
ATLAS
0.85 ± 0.07
0.77 ± 0.08
0.76 ± 0.09
0.67 ± 0.10
Table 9: Surgical phase recognition on AutoLaparo when initializing from SurgeNet versus from ATLAS pretraining.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Train
Val
Test
Annotated class
Evaluated as
% frames
Videos
% frames
Videos
% frames
Videos
Tools/camera
97.9
70
98.6
10
99.1
20
Omentum
Fat
41.4
38
60.0
7
57.4
14
Abdominal wall
42.2
43
42.2
8
48.4
12
Liver
31.3
29
56.8
7
50.8
12
Colon/rectum
32.4
26
11.4
3
27.3
8
Appendix
Table 10: Class distribution over the training, validation and test splits of ATLAS-120k, sorted by the total number of annotated frames. For each split, the percentage of annotated frames in which the class occurs and the number of videos that contain it are given. Classes that were merged into a parent class for training and evaluation list that parent class in the second column; classes marked – were relabeled as background.
Accurate recognition of anatomical structures is essential for safe and effective minimally invasive surgery (MIS), yet it remains underexplored in surgical computer vision due to limited annotated data and methods tailored primarily to natural scenes. In this work, we present a combined dataset and model framework to advance anatomy-aware perception in MIS. First, we introduce ATLAS-120k, a large-scale clip-level semantic segmentation dataset comprising over 120,000 annotated frames from 100 surgical videos spanning 14 procedures and multiple modalities, including laparoscopic and robot-assisted surgery. The dataset captures substantial procedural variability and was created using a scalable annotation pipeline that integrates expert manual labeling, automated propagation, iterative refinement, and surgeon verification to ensure high-quality annotations. Second, we propose ATLAS (Anatomy Recognition with Context Learning using Foundation Representations), a video semantic segmentation model specifically designed for surgical anatomy recognition. Unlike conventional approaches that emphasize object tracking, ATLAS leverages foundation-model embeddings together with lightweight temporal reasoning to incorporate contextual cues such as procedure type, surgical phase, and short-term visual memory. This design enables temporally consistent and accurate predictions while maintaining real-time feasibility. Together, the dataset and model establish a practical foundation for robust surgical scene understanding and support the development of clinically applicable guidance systems for minimally invasive surgery. The models, dataset annotations and annotation platform are publicly available at: https://github.com/TimJaspers0801/ATLAS.
Ronald L. P. D. de Jong, Tim J. M. Jaspers, Raf A. H. Vervoort +9
Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Electrical Engineering, Architectures for Reliable Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands +3
We introduce SurgAtlas, the largest surgical video-language dataset to date, comprising 15,291 videos (2,391 hours) spanning 18 surgical specialties and over 5,000 procedure types, sourced entirely from publicly available YouTube content. SurgAtlas is also the first surgical video-language dataset to include open surgery at scale, with 6,182 open procedure videos alongside over 9,000 minimally invasive recordings, and the first to establish standardized benchmarks for open-surgery video understanding. We additionally provide an expert-validated subset with verified visual question-answer pairs across diverse open and minimally invasive procedures, serving as a clinically grounded benchmark for surgical reasoning. Compared with existing surgical video-language datasets, SurgAtlas provides one of the most diverse annotation schemas, combining segment-level captions, step- and phase-level descriptions, video-level surgical descriptions, and reasoning-oriented question-answer pairs organized within a hierarchical taxonomy. These annotations are constructed through an automated multi-tier pipeline with LLM-based enrichment and a staged VQA generation framework with explicit groundedness verification. The scale and diversity of SurgAtlas enable training surgical foundation models with broad procedural coverage: we finetune Qwen3-VL-8B through a two-stage captioning-then-instruction pipeline and achieve competitive or state-of-the-art results on multiple established surgical benchmarks, including phase recognition, triplet detection, and reasoning question answering. More broadly, SurgAtlas provides a large native public video corpus that can support future large-scale pretraining of multimodal surgical AI systems and contribute to the development of next-generation foundation models for surgery. Dataset publicly available at https://huggingface.co/datasets/filbel/SurgAtlas
Filippos Bellos, Andre S. Gala-Garza, Miaowei Wang +8
University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom · Vanderbilt University Medical Center Nashville, Tennessee, USA +3
Accurate segmentation of anatomical structures in medical images is essential for diagnosis and treatment planning. While recent interactive segmentation foundation models enhance generalization through large-scale multimodal pretraining, they still depend on precise prompts and can fail in underrepresented clinical contexts (e.g., small organs-at-risk). We present AtlasSegFM, an atlas-guided framework that customizes off-the-shelf foundation models to new clinical contexts with a single annotated example. AtlasSegFM 1) performs atlas-query registration to generate context-aware prompts, 2) refines the segmentation with a frozen foundation model, and 3) applies a lightweight adaptive fusion module to combine atlas priors with foundation-model inputs and predictions. Extensive experiments on six public and in-house datasets across radiotherapy and vascular scenarios show consistent gains, with the largest improvements on small and delicate structures. AtlasSegFM provides a lightweight, deployable solution for one-shot customization of segmentation foundation models in real-world clinical workflows.
Ziyu Zhang, Yi Yu, Simeng Zhu +4
Nanjing University · The Ohio State University · Stanford University