Surgical Video Understanding

Ask about this topic

Overview

The field studies how to turn surgical video into useful descriptions of what is happening, where, and when, including procedural phases, tool-tissue interactions, anatomy, and surgical skill. Common approaches combine video encoders and temporal models such as temporal convolutional networks or state-space models with segmentation, tracking, and vision-language methods; newer systems also use structured scene representations and evidence-grounded reasoning. Evaluation spans phase and action recognition on datasets such as Cholec80 and AutoLaparo, instrument and interaction benchmarks such as EndoVis, and skill assessment settings such as JIGSAWS, alongside growing long-video question-answering tests. A central challenge is making models reliable across procedures, hospitals, and changing visual conditions while ensuring that their temporal conclusions and explanations rely on clinically relevant evidence.

Latest papers

CardsList

118 papers11 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

  1. Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusion

    Mar 26, 2026Nikolo Rohrmoser, Ghazal Ghazaei, Michael Sommersperger +1Surgical Instrument TrackingMedical Image Analysis

    Chair for Computer Aided Medical Procedures, Technical University of Munich, Garching, Germany. · Corporate Research and Technology, Carl Zeiss AG, Munich, Germany.

  2. SurgPhase: Time efficient pituitary tumor surgery phase recognition via an interactive web platform

    Mar 26, 2026Yan Meng, Jack Cook, X. Y. Han +12Surgical Video UnderstandingVideo Action Recognition

    Children’s National Hospital, 111 Michigan Ave NW, Washington, 20010, DC, USA · Surgical Data Science Collective, 2000 Pennsylvania Ave. NW, Suite 7000, Washington, 20006, DC, USA · Booth School of Business, University of Chicago, 5807 S Woodlawn Ave, Chicago, 60637, IL, USA +7

  3. CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition

    Mar 25, 2026Florian Stilz, Vinkle Srivastav, Nassir Navab +1Surgical Video UnderstandingVideo-Language Models

    University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France · IHU Strasbourg, France · Technical University of Munich, Germany

  4. A vision-language model and platform for temporally mapping surgery from video

    Mar 23, 2026Dani KiyassehSurgical Video UnderstandingMedical VLMs

    Halsted AI, Sunnyvale, California, 94086 USA

  5. CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation

    Mar 23, 2026Mohammad Eslami, Dhanvinkumar Ganeshkumar, Saber Kazeminasab +8Medical Image SegmentationSurgical Video Understanding

    Harvard Ophthalmology AI Lab, Massachusetts Eye and Ear, Harvard Medical School, Boston, MA, USA · Thomas Jefferson High School for Science and Technology, Chantilly, Virginia, USA · Massachusetts Eye and Ear, Harvard Medical School, Boston, MA, USA

  6. PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms

    Mar 20, 2026Tuna Gürbüz, Ege Özsoy, Tony Danjun Wang +1Panoptic SegmentationMulti-View Consistency

    Computer Aided Medical Procedures, Technische Universität München, Germany · MCML, Germany

  7. SurgΣΣ: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence

    Mar 17, 2026Zhitao Zeng, Mengya Xu, Jian Jiang +13Multimodal Foundation ModelsSurgical Video Understanding

    NUS · CUHK · SJTU +1

  8. Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery

    Mar 17, 2026Jecia Z. Y. Mao, Francis X. Creighton, Russell H. Taylor +1Surgical Instrument TrackingSurgical Video Understanding

  9. Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning

    Mar 13, 2026Mengya Xu, Daiyun Shen, Jie Zhang +19Surgical Video UnderstandingVideo Action Recognition

    Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Biomedical Engineering, National University of Singapore, Singapore · Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China +13

  10. Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validation

    Mar 12, 2026Jian Jiang, Chenxi Lin, Yiming Gu +24Medical VLMsSurgical Video Understanding

    Global College, Shanghai Jiao Tong University, Shanghai, China · Institute for Medical Engineering & Science, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Biomedical Engineering and Department of Electrical and Computer Engineering, National University of Singapore, Singapore +11

  11. Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization

    Mar 12, 2026Hanyang Hu, Zekai Liang, Florian Richter +1Visual Object TrackingCovariance Matrix Adaptation Evolution Strategy

    Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093 USA.

  12. SurgFed: Language-guided Multi-Task Federated Learning for Surgical Video Understanding

    Mar 10, 2026Zheng Fang, Ziwei Niu, Ziyue Wang +5Surgical Video UnderstandingFederated Learning

    National University of Singapore, Singapore · Spectral AI Team, The Hong Kong University of Science and Technology (Guangzhou), China · Zhejiang University, China +1

  13. TopoOR: A Unified Topological Scene Representation for the Operating Room

    Mar 10, 2026Tony Danjun Wang, Ka Young Kim, Tolga Birdal +2Higher-Order GNNsTopological Representation Learning

    Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Germany · Kyung Hee University, South Korea +1

  14. Training-free Temporal Object Tracking in Surgical Videos

    Mar 8, 2026Subhadeep Koley, Abdolrahim Kadkhodamohammadi, Santiago Barbarisi +2Visual Object TrackingSurgical Video Understanding

    Medtronic plc., London, UK. · UCL Hawkes Institute, University College London, London, UK.

  15. SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning

    Mar 6, 2026Alejandra Perez, Anita Rau, Lee White +4Surgical Video UnderstandingMedical VQA

    Intuitive Surgical Inc., Sunnyvale, United States

  16. From Phase Grounding to Intelligent Surgical Narratives

    Mar 5, 2026Ethan Peterson, Huixin ZhanSurgical Video UnderstandingTemporal Action Segmentation

    New Mexico Institute of Mining and Technology Socorro, New Mexico, USA

  17. Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics

    Feb 27, 2026Omar Mohamed, Edoardo Fazzari, Ayah Al-Naji +5Temporal Action SegmentationSurgical Video Understanding

    Authos are with the Dept. of Robotics, Mohamed bin Zayed University of AI, Abu Dhabi, UAE.

  18. SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video

    Feb 25, 2026Guanyi Qin, Xiaozhen Wang, Zhu Zhuo +7Surgical Video UnderstandingMedical VLMs

    National University of Singapore, Singapore · Guangzhou Research Translation and Innovation Institute, National University of Singapore, China · Southern Medical University, China

  19. SurgAtt-Tracker: Online Surgical Attention Tracking via Temporal Proposal Reranking and Motion-Aware Refinement

    Feb 24, 2026Rulin Zhou, Guankun Wang, An Wang +12Visual AttentionSurgical Video Understanding

    Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · College of Mechatronics and Control Engineering, Shenzhen University, China +2

  20. YOLOv10-Based Multi-Task Framework for Hand Localization and Laterality Classification in Surgical Videos

    Feb 21, 2026Kedi Sun, Le ZhangSurgical Video UnderstandingVideo Object Detection

    School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, UK

  21. Cholec80-port: A Geometrically Consistent Trocar Port Segmentation Dataset for Robust Surgical Scene Understanding

    Feb 19, 2026Shunsuke Kikuchi, Atsushi Kouno, Hiroki MatsuzakiMedical Image SegmentationMedical Image Benchmarks

    Jmees Inc., Kashiwa-city, Chiba, Japan

  22. SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment

    Feb 18, 2026Runlong He, Freweini M. Tesfai, Matthew W. E. Boal +13Surgical Video UnderstandingMultimodal Fusion

    the UCL Hawkes Institute and the Department of Medical Physics & Biomedical Engineering, UCL, UK. · the Griffin Institute, Northwick Park and St Mark’s Hospital, UK. · the UCL Hawkes Institute and the Division of Informatics, Imaging & Data Sciences, The University of Manchester, UK +3

  23. Automated Assessment of Kidney Ureteroscopy Exploration for Training

    Feb 17, 2026Fangjie Li, Nicholas Kavoussi, Charan Mohan +2Surgical Video UnderstandingEndoscopy

    Department of Computer Science, Vanderbilt University, 2301 Vanderbilt Place, Nashville, 37235, TN, USA. · Department of Urology, Vanderbilt University Medical Center, 1211 Medical Center Drive, Nashville, 37232, TN, USA.

  24. SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

    Feb 5, 2026Jinlin Wu, Felix Holm, Chuxi Chen +18Joint-Embedding Predictive ArchitectureSurgical Video Understanding

    Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany +7

  25. State-Change Learning for Prediction of Future Events in Endoscopic Videos

    Oct 14, 2025Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter +1Event Time PredictionSurgical Video Understanding

    University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France · IHU Strasbourg, Strasbourg, France · University Hospital of Strasbourg, France

  26. Federated Learning for Surgical Vision in Appendicitis Classification: Results of the FedSurg EndoVis 2024 Challenge

    Oct 6, 2025Max Kirchner, Hanna Hoffmann, Alexander C. Jenke +16Surgical Video UnderstandingCross-Dataset Generalization

  27. SurgXBench: Explainable Vision-Language Model Benchmark for Surgery

    May 16, 2025Jiajun Cheng, Xianwu Zhao, Sainan Liu +5VLM EvaluationSurgical Video Understanding

    Arizona State University · Intel Labs · University of California, Merced +2

  28. SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis

    Mar 13, 2025Chang Han Low, Ziyue Wang, Tianyi Zhang +4Surgical Video UnderstandingMultimodal CoT Reasoning

    National University of Singapore, Singapore · Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore · Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and the Department of Medical Physics and Biomedical Engineering, University College London, UK