cs.CVMar 23, 2026
SaveA vision-language model and platform for temporally mapping surgery from video
Organizations: Halsted AI, Sunnyvale, California, 94086 USA
Abstract
Mapping surgery is fundamental to developing operative guidelines and enabling autonomous robotic surgery. Recent advances in artificial intelligence (AI) have shown promise in mapping the behaviour of surgeons from videos, yet current models remain narrow in scope, capturing limited behavioural components within single procedures, and offer limited translational value, as they remain inaccessible to practising surgeons. Here we introduce Halsted, a vision-language model trained on the Halsted Surgical Atlas (HSA), one of the most comprehensive annotated video libraries grown through an iterative self-labelling framework and encompassing over 650,000 videos across eight surgical specialties. To facilitate benchmarking, we publicly release HSA-27k, a subset of the Halsted Surgical Atlas. Halsted surpasses previous state-of-the-art models in mapping surgical activity while offering greater comprehensiveness and computational efficiency. To bridge the longstanding translational gap of surgical AI, we develop the Halsted web platform (https://halstedhealth.ai/) to provide surgeons anywhere in the world with the previously-unavailable capability of automatically mapping their own procedures within minutes. By standardizing unstructured surgical video data and making these capabilities directly accessible to surgeons, our work brings surgical AI closer to clinical deployment and helps pave the way toward autonomous robotic surgery.
Figures & tables
Figure 1: Halsted maps surgery from video. Halsted is trained on the Halsted Surgical Atlas, a library with 650K+ videos, to generate a comprehensive mapping of surgery with 104 categories of surgical components. We present an example of such a mapping for the components of procedure, steps, anatomy, arm, and instrument.
Figure 2: Halsted learns to comprehensively map surgery across specialties. Halsted reliably (a) maps surgical components at various levels of granularity, from anatomy to procedure type, (b) assesses surgical proficiency irrespective of specialty, and (c) recognizes granular surgical components such as arms used, actions performed, and instruments used. The shaded area and error bars reflect one standard error from the mean.
Figure 3: Halsted learns a nuanced relationship between surgical videos. We present the two-dimensional UMAP embeddings of representations of video clips extracted by VideoMAE (left) and of the features generated by Halsted (right) in the final layer of the transformer decoder when tasked with assessing suturing proficiency. Each colour reflects a distinct procedure and marker size indicates proficiency (large markers indicate low proficiency). Although VideoMAE can distinguish between procedures, Halsted takes this one step further and clusters procedures of the same specialty, as observed with cardiac and gynecology procedures.
Figure 4: Halsted’s performance in mapping micro-activity as a function of decoder size. We train Halsted with a 2-layer decoder or a Llama-3.2 (1B) decoder to perform the micro-activity task, jointly mapping surgical actions and instruments. We report performance using 5-fold Monte-Carlo cross-validation and show that these models perform on par with one another, irrespective of the size of the decoder.
| Accuracy (%) | Relative | ||
|---|---|---|---|
| Method | Random | Actual | Improvement |
| MA-TCN [ 20 ] | 25.9 | 80.9 | 3.1 |
| SAIS [ 10 ] | 14.3 | 59.8 | 4.2 |
| Halsted | 12.5 | 68.6 | 5.5 |
Table 1: Halsted performance on RARP-50 benchmark. We compare the actual accuracy of the model to that of random chance (i.e., randomly guessing an action category) and report the relative improvement. Previous results are borrowed from previous work [ 10 ] . We show that Halsted achieves the greatest improvement when identifying all eight action categories on the RARP-50 test set.
Figure 5: Halsted’s performance improves with a self-learning strategy. We train Halsted on HSA v1, the first version of the Halsted Surgical Atlas without any AI-generated annotations, and HSA v2, the final version of the Halsted Surgical Atlas after incorporating AI-generated annotations. For details on the iterative labelling process, see Methods. Both models are evaluated on the same held-out test sets using 5-fold Monte-Carlo cross-validation.
Figure 6: Overview of the Halsted Platform. The Halsted Platform is a secure web-based platform that enables surgeons to upload a surgical video and receive an automated mapping of their surgical activity within minutes. Upon logging in, surgeons are presented with a high-level summary of their surgical activity. They have the flexibility to select the mapping task (e.g., micro-activity or proficiency) before viewing a detailed breakdown of their activity. Surgeons can also better contextualize this activity through visual overlays pinpointing exactly when the activity occurred during the procedure.
Figure 7: Overview of the Halsted model. Halsted is conditioned on a video clip and a set of task instructions to generate a sequence of surgical component tags. It is a multimodal, multitask, and generative model whose outputs can be controlled via different task instructions. A transformer decoder is presented with a sequence of task instruction embeddings and a video clip embedding to autoregressively generate a sequence of surgical component tags. During inference, we perform greedy decoding at each time-step to select the most likely token (e.g., [25] which corresponds to the word <action>) and retrieve its embedding before feeding it into the model at the next time-step.
| Samples | Annotations | ||||||
| Surgical Procedure / Method | Videos | Task 1 | Task 2 | Task 3 | Task 4 | Total | Manual |
| Bariatric | |||||||
| Gastric Bypass | 5 | 303 | - | 303 | 152 | 758 | 100% |
| Cardiac | |||||||
| Atrial Closure | 6 | 219 | - | 219 | 73 | 511 | 100% |
| IMA Harvest | 2 | 370 | 370 | - | - | 740 | 100% |
Table 2: Summary of annotated videos and samples from each surgical procedure and task. We outline the number of videos and samples in each task for each procedure and specialty. We also present the distribution of manual annotations at the procedure level. For the exact surgical component tags within each task, please refer to the Halsted Surgical Atlas section.
| Task | Tag | Categories | # Categories |
|---|---|---|---|
| 1 | Specialty | Bariatric, Cardiac, Colorectal, General, Gynecology, Hepatobiliary, Thoracic, Urology | 8 |
| Procedure / Method | Gastric Bypass, Atrial Closure, IMA Harvest, Mitral Stitching, Right Colectomy, Total Mesorectal Excision, Laparoscopic Cholecystectomy, Laparoscopic Hernia, Endometriosis, Hysterectomy, Myomectomy, Pancreaticoduodenectomy, Right Middle Lobectomy, Right Upper Lobectomy, Segmentectomy, Prostatectomy | 16 | |
| Step | Suturing, Dissection | 2 | |
| 2 | Action | Assistant, Cold Cut, Cautery, Extraction, Fluorescence, Hot Cut, Hook, Idle, Clip, Camera Move, Mesh, Push/Peel, Retraction, Spread, Sponge, Stapler, Tube, Tug, Other | 19 |
| Arm | Left, Right, Both | 3 | |
| Instrument | Bipolar Dissector, Bipolar Forceps, Bipolar Forceps–Cautery Hook, Bipolar Forceps–Monopolar Scissors, Bipolar Forceps–Vessel Sealer, Bipolar Grasper, Bipolar Grasper–Monopolar Scissors, Cadiere Forceps, Cadiere Forceps–Bipolar Grasper, Cautery Spatula, Cautery Hook, Clip Applier, Clipper, Fenestrated Forceps, Fenestrated Grasper, Fenestrated Grasper–Bipolar Grasper, Grasper, Hook Monopolar, Maryland Grasper, Monopolar Scissors, Needle Driver, Prograsp Forceps, Scissors, Shears, Stapler, Suction, Vessel Sealer | 27 |
Table 3: Breakdown of each task, component tags, and categories. The tags span four tasks: macro-activity, micro-activity, proficiency, and context. Each tag comprises a set of discrete categories. Hyphenated categories indicate items that co-occur.
Figure 8: Self-labelling strategy to curate the Halsted Surgical Atlas. We curated the atlas iteratively, repeating a sequence of four steps. We started with manually-curated video library annotated with component tags, trained a model on the video library, and deployed it on unannotated surgical videos to obtain AI-annotations. We then added the newly-processed videos to the original video library in preparation for training the model on a larger volume dataset. The cycle can repeat with the collection of new surgical videos.
Figure 9: Distribution of durations of video clips in the Halsted Surgical Atlas. We collect full-length surgical videos (1-2 hours in duration) which comprise a sequence of video clips and curate video clips based on task-specific timestamps and annotations (1-20 seconds in duration). We show the distribution of the duration of these video clips across the four tasks and 8 surgical specialties. During training and evaluation of the Halsted model, we ensure the full-length surgical videos only appear in one of the training, validation, or test sets to avoid video contamination of the results.
More in this topic
Surgical Video Understanding
Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.
See all 118 papers