cs.CVNov 1, 2026

SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding

Authors: Ruipeng Ma, Qing Xu, Jie Liang, Zhen Chen

Organizations: Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Eastern Institute of Technology, Ningbo · University of Nottingham, UK

Abstract

Surgical video understanding is essential for intraoperative assistance such as workflow monitoring, instrument preparation and procedural documentation. Delivering such assistance in real time requires a model to jointly perceive the ongoing scene, remember past surgical events, and respond before clinically relevant moments pass. Existing approaches provide limited support for jointly retaining procedural evidence and deciding when to respond in streaming surgical video. In this work, we propose SurgStream, a workflow-anchored proactive streaming framework for real-time surgical video understanding. We first introduce Workflow-Anchored Memory (WAM) to record structured instrument-action-target observations and fold them into phase-level summaries, providing compact and queryable surgical memory. Deadline-aware Transition Learning (DTL) is then devised to supervise response states within clinical windows, enabling timely responses before procedural deadlines. We further introduce SurgStream-Bench, comprising 11.7K streaming interactions across various tasks covering real-time perception, backward memory queries, forward proactive QA, proactive alerts and caption memory. Experiments on SurgStream-Bench show gains over state-of-the-art methods, especially in memory and forward proactive QA. The source code will be available.

Figures & tables

Explore similar work

CardsList
  1. SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary

    May 20, 2026Jingyi He, Yue Zhou, Long Bai +3Streaming Video UnderstandingSurgical Video Understanding

  2. Remembering to Reason: Structured Memory for Scene and Workflow Understanding in Robot-Assisted Surgery

    Nov 7, 2026Zhiran Zhang, Hu Wang, Yichen Wu +1Surgical Video UnderstandingMemory-Augmented Video Understanding