Surgical Video Understanding

Ask about this topic

Overview

The field studies how to turn surgical video into useful descriptions of what is happening, where, and when, including procedural phases, tool-tissue interactions, anatomy, and surgical skill. Common approaches combine video encoders and temporal models such as temporal convolutional networks or state-space models with segmentation, tracking, and vision-language methods; newer systems also use structured scene representations and evidence-grounded reasoning. Evaluation spans phase and action recognition on datasets such as Cholec80 and AutoLaparo, instrument and interaction benchmarks such as EndoVis, and skill assessment settings such as JIGSAWS, alongside growing long-video question-answering tests. A central challenge is making models reliable across procedures, hospitals, and changing visual conditions while ensuring that their temporal conclusions and explanations rely on clinically relevant evidence.

Latest papers

CardsList

118 papers11 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

  1. Shifted Window Temporal Transformer for YOLO-Guided Phase Recognition in Laparoscopic Cholecystectomy Surgery

    Oct 26, 2026Sheekar Banerjee, Md. Shahabuddin, Md. Mahbub Hasan Akash +2Surgical Video UnderstandingLaparoscopic Surgery

    Department of Computer Science and Engineering, IUBAT—International University of Business Agriculture and Technology, 4 Embankment Drive Road, Sector-10, Uttara Model Town, Dhaka, 1230, Bangladesh. · Boston Children’s Hospital, Harvard Medical School, 401 Park Dr, Boston, USA.

  2. MIS-VQA: An Annotation-Driven Surgical Vision-Language Dataset for Complex Surgical Understanding

    Oct 25, 2026Mohamed Hamdy, Iheb Zouari, Fatmaelzahraa Ahmed +6Medical VQASurgical Video Understanding

    Department of Computer Science and Engineering, Qatar University, Doha, Qatar · Hamad Medical Corporation, Doha, Qatar

  3. Semantic-Coupled Perception Framework for Autonomous Laparoscope Control

    Oct 25, 2026Enduo Zhao, Fangzhou Li, Yujie Zhu +4Surgical RoboticsRobotic Perception

    School of Biomedical Engineering, Tsinghua University, Beijing, China. · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.

  4. LapaFM: A Laparoscopic Foundation Model via Multi-Stage Continual Learning

    Oct 23, 2026Xiaobo Zhu, Shiwei Wu, Yu An +21Surgical Video UnderstandingMedical Imaging Foundation Models

    the CAS Key Laboratory of Molecular Imaging and Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, Jiangsu Province, 210094, China +10

  5. Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

    Oct 20, 2026Yiqing Shen, Mathias UnberathSurgical Video UnderstandingReasoning Segmentation

    Department of Computer Science, Johns Hopkins University

  6. SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition

    Oct 18, 2026Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif +1Surgical Video UnderstandingSurgical Robotics

    Electrical and Computer Engineering Department, North Carolina A&T State University, Greensboro, NC, USA · Swinburne University of Technology, Hawthorn, VIC, Australia · Harvard Paulson School of Engineering and Applied Sciences, Harvard University, Allston, MA, USA +3

  7. SurgStreamer: proactive, request-driven monitoring of surgical video streams with structured state and a reusable reasoning layer

    Oct 15, 2026Bowen Xiang, Jon S. Heiselman, Michael I. MigaSurgical Video UnderstandingStreaming Video Understanding

    Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, USA

  8. Augmenting Real-time Visualization in Minimally Invasive Surgery with Particle Filtering and Depth-Aware Occlusion Handling

    Oct 13, 2026Anuska Sarkar, Partha Acharya, Vishnu Nandury +2Surgical Instrument TrackingOcclusion Reasoning

    School of Medical Science and Technology, Indian Institute of Technology Kharagpur, Kharagpur 721302 India · Computer Science Department, Arizona State University, Tempe, Arizona 85287 USA · Narayana Hospital, Kolkata 700107 India

  9. Spatiotemporal Recognition of Abdominal Wall Layers During Optical Trocar Entry

    Oct 10, 2026Shreeram Athreya, Andrew Melehy, Carter Prentice +4Medical Image AnalysisSurgical Video Understanding

    Department of Electrical and Computer Engineering, University of California, Los Angeles, Los Angeles, California, USA. · Department of Surgery, University of California, Los Angeles, Los Angeles, California, USA. · Department of Radiological Sciences, University of California, Los Angeles, Los Angeles, California, USA. +2

  10. HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding

    Oct 7, 2026Leon Mayer, Lucas Luttner, Patrick Godau +42Long-Video UnderstandingSurgical Video Understanding

  11. Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training

    Oct 4, 2026Hanyang Hu, Zekai Liang, Florian Richter +1Self-TrainingSim-to-Real Transfer

    Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093 USA.

  12. Semantic RGB--Depth Based Surgical Skill Assessment in Microscopic Stereo Videos

    Oct 1, 2026Jecia Z. Y. Mao, Sue M. Cho, Francis X. Creighton +3Surgical Video Understanding

    Laboratory of Computational Sensing and Robotics, Johns Hopkins University, MD, USA · Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins University School of Medicine, MD, USA

  13. Structured Reasoning Agentic Framework for Interpretable Critical View of Safety Assessment

    Sep 25, 2026Qing Xu, Yuxiang Luo, Zhen ChenSurgical Video UnderstandingExplainable Medical Image Analysis

    School of Computer Science, University of Nottingham, UK · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR

  14. A Surgical Foundation Model Reveals Task-Dependent Label Efficiency

    Sep 25, 2026Florian Philipp Stilz, Lorenzo Arboit, Vinkle Srivastav +6Surgical Video UnderstandingData-Efficient Learning

    University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France · IHU Strasbourg, Strasbourg, France · Technical University of Munich, Computer Aided Medical Procedures, Munich, Germany +3

  15. PICO: Projection-Informed Consistency Optimisation for 6DoF Surgical Tool Pose Estimation

    Sep 25, 2026Lucy Fothergill, Pietro Valdastri, Dominic Jones +1Object Pose EstimationSurgical Robotics

    School of Computer Science, University of Leeds. · School of Electronic and Electrical Engineering, University of Leeds.

  16. CasCVS-Net: A Staged Multi-Task Cascade for Critical View of Safety Assessment

    Sep 23, 2026Bock-Zien Toh, Yuanchuan Ren, Tay Aw Yu +3Multi-Task LearningSurgical Video Understanding

    Department of Computer Science, University College London, London, UK · UCL Hawkes Institute, University College London, UK

  17. A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery

    Sep 22, 2026Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven +1Video-Language Model EvaluationHierarchical Representation Learning

    Data Science Center HAVA-Lab, University of Amsterdam · Video & Image Sense Lab, Informatics Institute, University of Amsterdam · Amsterdam UMC Location University of Amsterdam, Surgery +2

  18. Video-based Surgical Skill Assessment Using Dynamics-and-Uncertainty-Aware Tree-based Gaussian Process Classifier

    Sep 21, 2026Arefeh Rezaei, Mohammad Javad Ahmadi, Amir Molaei +1Gaussian ProcessesSurgical Video Understanding

    Applied Robotics and AI Solutions (ARAS), Faculties of Electrical and Computer Engineering, K.N. Toosi University of Technology, Tehran, Iran. · Gina Cody School of Engineering and Computer Science, Concordia University, Montreal, Quebec, Canada.

  19. LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation

    Sep 19, 2026Zan Wang, Yunhe Feng, Dong Nie +4Referring Video Object SegmentationSurgical Video Understanding

    University of North Texas, Denton, USA · Meta Inc, USA

  20. RAUL: Reference-Assisted Ureteroscopy Localization for Skill Assessment

    Sep 16, 2026Fangjie Li, Mai Bui, Charan Mohan +43D ReconstructionSurgical Video Understanding

    Department of Biomedical Engineering, Vanderbilt University, Nashville, TN 37240 USA · Department of Computer Science, Vanderbilt University, Nashville, TN 37240 USA · Department of Urology, Vanderbilt University Medical Center, Nashville, TN 37232 USA

  21. TTDF: A Two-Stage Framework for Reliable Surgical Phase Transition Detection

    Sep 13, 2026Yushi Guo, Pietro Valdastri, Duygu SarikayaSurgical Video UnderstandingChange-Point Detection

    School of Computer Science, University of Leeds, Leeds, UK · STORM Lab UK · School of Electronic and Electrical Engineering, University of Leeds, Leeds, UK

  22. S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks

    Sep 13, 2026Jiaming Zhang, Zijian Wu, Mehran Armand +1Point TrackingSurgical Video Understanding

    Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas

  23. OphBiWSSD: Scaling Temporal Action Localization in Ophthalmic Surgeries with Bidirectional Weight-tied State Space Duality

    Sep 11, 2026Yang Liu, Qionghong Ma, Joongwon Chae +10Surgical Video UnderstandingTemporal Action Detection

    Institute of Biopharmaceutics and Health Engineering, Tsinghua Shenzhen International Graduate School, Shenzhen 518055, China · Department of Ophthalmology, The Fifth Affiliated Hospital of Wenzhou Medical University, Wenzhou Medical University, Lishui 323000, China · School of Optometry, The Hong Kong Polytechnic University +4

  24. STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs

    Sep 8, 2026Jing Li, Duygu SarikayaMedical VQAQuestion Answering

    School of Computer Science, University of Leeds, UK

  25. An Integrated Video-AI Platform for Action-Level Microanastomosis Training and Performance Feedback

    Sep 6, 2026Yan Meng, Daniel A. DonohoSurgical Video UnderstandingTemporal Action Segmentation

    Department of Neurosurgery, Children’s National Hospital

  26. Multimodal Shared Latent Representation of Narration, Microscope and iOCT Images for Phase Recognition in Vitreoretinal Surgery

    Aug 31, 2026Onur Izmitlioglu, Shervin Dehghani, Tarek Ghannoum +2Surgical Video UnderstandingMultimodal Contrastive Learning

    Computer Aided Medical Procedures, Technical University of Munich, Germany · SynthesEyes GmbH, Munich, Germany · Department of Ophthalmology, LMU University Hospital, Munich, Germany

  27. SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

    Aug 31, 2026Chaohui Dang, Zheheng Jiang, James Glasbey +3Surgical Video UnderstandingMultimodal Model Evaluation

    Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK +1

  28. ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation

    Aug 25, 2026Jiaxin Wen, Ming Yin, Lu Liu +1Referring Video Object SegmentationSurgical Video Understanding

    University of Exeter, Exeter, UK

  29. Towards Actionable Surgical Team Dynamics: from Teamwork to Counterfactual Annotations

    Aug 24, 2026Vincenzo Marco De Luca, Antonio Longa, Andrea PasseriniSurgical Video Understanding

    University of Trento Trento, Italy · UiT the Arctic University of Norway Tromsø, Norway

  30. Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition

    Aug 24, 2026Yiyi Zhang, Ying Zheng, Wenxin Fan +5Surgical Video UnderstandingSelf-Training

    Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China · Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China +1