Surgical Video Understanding

Overview

The field studies how to turn surgical video into useful descriptions of what is happening, where, and when, including procedural phases, tool-tissue interactions, anatomy, and surgical skill. Common approaches combine video encoders and temporal models such as temporal convolutional networks or state-space models with segmentation, tracking, and vision-language methods; newer systems also use structured scene representations and evidence-grounded reasoning. Evaluation spans phase and action recognition on datasets such as Cholec80 and AutoLaparo, instrument and interaction benchmarks such as EndoVis, and skill assessment settings such as JIGSAWS, alongside growing long-video question-answering tests. A central challenge is making models reliable across procedures, hospitals, and changing visual conditions while ensuring that their temporal conclusions and explanations rely on clinically relevant evidence.

Latest papers

CardsList

132 papers17 papers in the last four weeks, up 31% on the four weeks before. 0.1% of all new papers.

  1. Toward Cross-Rubric Surgical Skill Assessment via Joint Embedding Prediction

    Nov 11, 2026Jiajun Cheng, Haidar M. Abdul-Muhsin, Juha Choi +1Rubric-Based EvaluationSurgical Video Understanding

  2. Instance-preserving graded semantic reporting for surgical action triplets

    Nov 9, 2026Yanchong Xue, Wanting Zhang, Lingtao Yu +1Surgical Video UnderstandingSelective Prediction

  3. SurgWorld: Learning Surgical Representations Beyond Observation via Latent Prediction

    Nov 8, 2026Jingkun Yue, Pengxiang Wang, Siqi Zhang +7Surgical Video UnderstandingVideo Representation Learning

  4. Remembering to Reason: Structured Memory for Scene and Workflow Understanding in Robot-Assisted Surgery

    Nov 7, 2026Zhiran Zhang, Hu Wang, Yichen Wu +1Surgical Video UnderstandingMemory-Augmented Video Understanding

  5. GRAC: Graph-Based Retrieval-Augmented Captioning for Surgical Scene Description

    Nov 6, 2026Rochak Dhakal, Chen Zhao, Soup Cobb +3Image CaptioningSurgical Video Understanding

  6. SurgTransfer: Multi-Step Latent Action Modeling for Action Transfer in Surgical World Model

    Nov 3, 2026Saurabh Koju, Love Panta, Deepak Thapa +6Latent Action LearningWorld Model Learning

  7. SurgEye: Instruction Tuning-Free Perception-Reasoning Framework for Surgical Scene Understanding

    Nov 2, 2026Qing Xu, Xiechen Shen, Xianjian He +9Surgical Video UnderstandingMedical VLMs

  8. SurgPace: Complexity-Aware Progressive Distillation for Surgical Multimodal Large Language Models

    Nov 2, 2026Zhicheng He, Xiaohan Xing, Hongqiu Wang +7VLM DistillationMultimodal Large Language Models

  9. SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding

    Nov 1, 2026Ruipeng Ma, Qing Xu, Jie Liang +1Streaming Video UnderstandingSurgical Video Understanding

  10. Shifted Window Temporal Transformer for YOLO-Guided Phase Recognition in Laparoscopic Cholecystectomy Surgery

    Oct 26, 2026Sheekar Banerjee, Md. Shahabuddin, Md. Mahbub Hasan Akash +2Surgical Video UnderstandingLaparoscopic Surgery

  11. MIS-VQA: An Annotation-Driven Surgical Vision-Language Dataset for Complex Surgical Understanding

    Oct 25, 2026Mohamed Hamdy, Iheb Zouari, Fatmaelzahraa Ahmed +6Medical Visual Question AnsweringSurgical Video Understanding

  12. Semantic-Coupled Perception Framework for Autonomous Laparoscope Control

    Oct 25, 2026Enduo Zhao, Fangzhou Li, Yujie Zhu +4Surgical RoboticsRobotic Perception

  13. LapaFM: A Laparoscopic Foundation Model via Multi-Stage Continual Learning

    Oct 23, 2026Xiaobo Zhu, Shiwei Wu, Yu An +21Surgical Video UnderstandingMedical Imaging Foundation Models

  14. Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

    Oct 20, 2026Yiqing Shen, Mathias UnberathSurgical Video UnderstandingReasoning Segmentation

  15. SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition

    Oct 18, 2026Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif +1Surgical Video UnderstandingSurgical Robotics

  16. Augmenting Real-time Visualization in Minimally Invasive Surgery with Particle Filtering and Depth-Aware Occlusion Handling

    Oct 13, 2026Anuska Sarkar, Partha Acharya, Vishnu Nandury +2Surgical Instrument TrackingOcclusion Reasoning

  17. Spatiotemporal Recognition of Abdominal Wall Layers During Optical Trocar Entry

    Oct 10, 2026Shreeram Athreya, Andrew Melehy, Carter Prentice +4Medical Image AnalysisSurgical Video Understanding

  18. Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training

    Oct 4, 2026Hanyang Hu, Zekai Liang, Florian Richter +1Self-TrainingSim-to-Real Transfer

  19. Semantic RGB--Depth Based Surgical Skill Assessment in Microscopic Stereo Videos

    Oct 1, 2026Jecia Z. Y. Mao, Sue M. Cho, Francis X. Creighton +3Surgical Video Understanding

  20. Structured Reasoning Agentic Framework for Interpretable Critical View of Safety Assessment

    Sep 25, 2026Qing Xu, Yuxiang Luo, Zhen ChenSurgical Video UnderstandingExplainable Medical Image Analysis

  21. A Surgical Foundation Model Reveals Task-Dependent Label Efficiency

    Sep 25, 2026Florian Philipp Stilz, Lorenzo Arboit, Vinkle Srivastav +6surgical video reasoning benchmarksSurgical Video Understanding

  22. PICO: Projection-Informed Consistency Optimisation for 6DoF Surgical Tool Pose Estimation

    Sep 25, 2026Lucy Fothergill, Pietro Valdastri, Dominic Jones +1Object Pose EstimationSurgical Robotics

  23. CasCVS-Net: A Staged Multi-Task Cascade for Critical View of Safety Assessment

    Sep 23, 2026Bock-Zien Toh, Yuanchuan Ren, Tay Aw Yu +3Multi-Task LearningSurgical Video Understanding

  24. A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery

    Sep 22, 2026Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven +1Video-Language Model EvaluationHierarchical Representation Learning

  25. Video-based Surgical Skill Assessment Using Dynamics-and-Uncertainty-Aware Tree-based Gaussian Process Classifier

    Sep 21, 2026Arefeh Rezaei, Mohammad Javad Ahmadi, Amir Molaei +1Gaussian ProcessesSurgical Video Understanding