cs.CVOct 15, 2026

SurgStreamer: proactive, request-driven monitoring of surgical video streams with structured state and a reusable reasoning layer

Authors: Bowen Xiang, Jon S. Heiselman, Michael I. Miga

Organizations: Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Surgical teams could benefit from assistants that watch the video for requested events, such as "tell me when the clip applier appears during Calot triangle dissection", and answer questions about what has already happened. We formalise this task as request-driven event detection with exact semantics and causal alert decisions, and present SurgStreamer. It separates procedure-specific perception, which maps the 1-FPS stream to a structured state over tools, actions and phases, from procedure-agnostic reasoning, which evaluates a formal monitoring condition over this state and an event memory, either with exact rules or with a small controller that addresses entities through pointers rather than names. We evaluate this monitoring core with gold conditions. On a pre-registered, sealed test set of 12 laparoscopic cholecystectomies, SurgStreamer reached a macro event-F1 of 0.598 over full procedures. On identical 75-s windows with only the window available to every method, it scored 0.677. The tested configurations of the streaming vision-language model OneStreamer-4B, including ones fine-tuned on SurgStreamer's state or on video with an adapted vision encoder, scored at most 0.540 (Holm-adjusted p <= 0.012). At three external centres, re-training only perception raised performance on held-out procedures from 0.239 to 0.470. On a new procedure, pituitary surgery, controller weights trained on cholecystectomy matched an in-domain controller on oracle state (0.947 vs 0.938). On predicted state they missed the pre-registered non-inferiority margin (-0.024 [-0.040, -0.008]) unless the alert thresholds were re-calibrated. The results support separating perception from reasoning and point to perception under distribution shift as the main limitation.

Figures & tables

Figure 1: Request-driven monitoring on a sealed Cholec80 test procedure (75-s window, equal context: every method has only the window). The request is compiled once into an executable monitoring condition. From the 1-FPS frames, perception computes a structured state: predicted phase, hook probability (shaded) and decoded flag (line); grey: reference presence; orange band: matching tolerance around the reference occurrence; dashed line: its confirmation. Bottom: per-frame decisions. SurgStreamer accumulates evidence causally: uncertain at the onset second, a single alert once the onset has persisted K=2K=2 s, then silence. The strongest state-text baseline (C-LLM) and the vision-adapted VLM (V-VA) stay silent over the whole window, as does V-LoRA (not shown). The example was selected for illustration; Fig. 9 shows a failure, and Section 5 the quantitative results.
Figure 2: SurgStreamer overview, shown on a sealed Cholec80 test procedure. A natural-language request is compiled once into an executable monitoring condition. Procedure-specific perception (blue) turns the 1-FPS stream into a structured state over a closed vocabulary of tools, actions and phases (decoded flags as lines, probabilities shaded; four of the seven tools shown). Procedure-agnostic reasoning (orange) addresses the entities of the condition through pointers ( e1e_{1} , φ\varphi ), never by name. Every second, the exact rule controller C-rule, the learned controller C-learn or their per-family hybrid outputs yty_{t} , and an alert is issued causally, once the event is confirmed. Here the earlier irrigator onset at 08:35 ( ×\times ) falls in the clipping phase and is not admissible; the onset at 14:59 ( ▼\blacktriangledown ) is confirmed and reported at 15:00 (dashed line), as in the reference (decisions of C-rule on the decoded state). The event memory logs episodes and phase transitions causally and answers retrospective requests; the count shown, 3, matches the reference. For a new procedure or centre only the blue components are re-trained; the rule semantics and the C-learn weights are reused unchanged, and only the alert thresholds may be re-calibrated.
Figure 3: Event memory on a sealed Cholec80 test procedure (full history). Perception decodes the clip applier (flag, blue; reference, grey) and the phase. The memory logs, causally, one record per episode (onset, last-seen time, predicted phase at onset) and the phase transitions. At 19:09 the question “How many times has the clip applier been introduced so far?” is answered by the running count of the logged episodes (3, as in the reference); the logged onsets coincide with the reference onsets. The same records answer has / count / order requests about any logged entity. Selected for illustration.
Figure 4: Comparison with streaming VLMs under equal context (gold conditions). Macro event-F1 with 95 % procedure-bootstrap intervals on identical 75-s windows. Filled dots: every method has only the window available. Open dots: the state given to state-reading methods was computed with the full procedure history (pre-registered protocol). (a) Cholec80 final test. (b) HeiChole, three external centres, all models trained on Cholec80 / CholecT50 only.
Figure 5: Event-F1 per family on the Cholec80 final-test windows under equal context. The VLMs detect appearances but miss disappearances and phase entries.
Figure 6: A disappearance request on a sealed Cholec80 test window (equal context). The event is the last second the irrigator is visible (“last seen”); it is confirmed only after the instrument has been absent for more than G=2G=2 s, so the 1-s gap in the reference annotation (annotated) is not an offset. C-learn, which answers this family, briefly outputs uncertain around a dip of the irrigator probability without alerting, and alerts once at the confirmation of the real disappearance. C-LLM stays silent, as does V-LoRA (not shown); V-VA alerts 16 s early, while the irrigator is still visible. Selected to illustrate the failure mode quantified in Fig. 5 ; layout as in Fig. 1 .
Figure 7: Transfer with frozen controller weights. (a) New centres (HeiChole 25–33): re-training only the perception doubles performance, with controllers, thresholds and hybrid choices unchanged. (b) New procedure (PitVis sealed test): oracle state (squares) and predicted state (dots); “re-calibrated” keeps the weights frozen and re-calibrates only the alert thresholds. C-rule is exact on oracle state by construction.
Figure 8: The reasoning trained on cholecystectomy, applied unchanged to pituitary surgery (PitVis sealed test). Top: the transfer setting. Perception was re-trained on 15 PitVis procedures; the C-learn weights and alert thresholds are those trained on Cholec80. The controller addresses the ring curette and the tumour-excision phase only through entity pointers; it outputs uncertain at the onset and alerts once at confirmation. Selected for illustration; over the whole test set, the transferred controller scored 0.372 on predicted state (Table 11 ). Layout as in Fig. 1 .
Figure 9: A failure on a sealed Cholec80 test window (equal context). At the window start, perception decodes the scissors as present although the reference marks them absent. When this false detection ends (14:58), the decoded state contains an offset, and C-learn alerts at 14:59, 42 s before the reference disappearance; in once mode, the real disappearance is then not reported. C-LLM stays silent; V-VA alerts 34 s early, and V-LoRA (not shown) also too early. The error originates in perception: on the decoded state, an offset had occurred. Layout as in Fig. 1 .
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Few-shot adaptation with frozen controller weights (exploratory): macro event-F1 versus the number of annotated target videos used to train a light temporal head on the frozen Cholec80-tuned backbone (mean ±\pm s.d. over 5 draws). Solid: test procedures; dashed: development pool. Dash-dotted: full adaptation; dotted: no adaptation.
More in this topic

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

Explore similar work

CardsList
Aug 5, 2026

SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding

Surgical procedures unfold as structured and recurring clinical events, whose real-time understanding via intraoperative surgical videos is critical for intraoperative decision-making and support. However, existing video understanding methods force a trade-off: autoregressive video-language models support comprehensive reasoning but are not practical for time-sensitive clinical applications, whereas contrastive models offer low latency but struggle with complex scene understanding. Recently, generative retrieval has been explored for general-domain video understanding, but transferring it to surgery is not trivial because near-identical visual appearances may indicate semantically distinct events, and the terminology involved is highly surgery-specific. To this end, we propose SurgNarrator, a new generative retrieval framework tailored for surgical video understanding. We construct a well-curated surgery-centric vocabulary from surgical captions to define a clinically meaningful retrieval space. We then adapt the pre-trained Qwen3-VL-Embedding-8B to learn discriminative clinical representations with a temporally-aware contrastive objective. During inference, a hierarchical, procedure-aware retrieval strategy narrows the search space to the relevant procedure type, delivering fast and effective responses. Our method is comprehensively evaluated on twelve benchmarks in a zero-shot setting and achieves consistent performance gains over state-of-the-art baselines, while reducing output-stage latency by more than two orders of magnitude compared with the generative baseline.

Yuqing Feng, Jiawei Ma, Kevin Qinghong Lin +6

Department of Computer Science, University of Liverpool, Liverpool L69 3BX, U.K. · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · Department of Engineering, University of Oxford, Oxford OX1 3PJ, U.K. +2

Surgical Video UnderstandingGenerative RetrievalVisual SearchTemporal Video Understanding
AskPDF
May 20, 2026

SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary

Understanding surgical workflow in real time is fundamental for intelligent surgical embodiment, where AI systems continuously perceive and respond as surgery proceeds. In the operating room, critical decisions depend on subtle, moment-to-moment changes, such as fine instrument movements and evolving tissue states, where even slight perceptual delays can limit assistance or compromise safety. Yet existing methods remain offline or operate at coarse temporal scales, generating descriptions only after processing clips, preventing immediate reaction. We address this by proposing SurgOnAir, a streaming vision-language model that processes frames sequentially without future access and progressively generates narration tokens as visual input arrives. SurgOnAir achieves fine-grained frame-to-token generation, enabling instant responsiveness to evolving surgical dynamics. Built upon our curated hierarchical dataset SurgOnAir-11k spanning action-, step-, and phase-level supervision, the model is trained to produce multi-level textual responses that reflect the inherent hierarchy of surgical procedures. Furthermore, special transition tokens are generated to explicitly mark state changes, allowing SurgOnAir to capture and signal key workflow transitions as they occur. Experiments show that SurgOnAir enables real-time understanding through a single vision-language model that unifies streaming across multiple hierarchies of the surgical workflow, generating superior and hierarchy-aware narrations. The project is publicly available at https://github.com/KarolineZhy/SurgOnAir.

Jingyi He, Yue Zhou, Long Bai +3

Computer Aided Medical Procedures (CAMP), TU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · The Chinese University of Hong Kong, Hong Kong +1

Streaming Video UnderstandingSurgical Video UnderstandingVideo CaptioningMedical VLMs
AskPDF
Mar 6, 2026

SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning

Surgeons don't just see -- they interpret. When an expert observes a surgical scene, they understand not only what instrument is being used, but why it was chosen, what risk it poses, and what comes next. Current surgical AI cannot answer such questions, largely because training data that explicitly encodes surgical reasoning is immensely difficult to annotate at scale. Yet surgical video lectures already contain exactly this -- explanations of intent, rationale, and anticipation, narrated by experts for the purpose of teaching. Though inherently noisy and unstructured, these narrations encode the reasoning that surgical AI currently lacks. We introduce SUREON, a large-scale video QA dataset that systematically harvests this training signal from surgical academic videos. SUREON defines 12 question categories covering safety assessment, decision rationale, and forecasting, and uses a multi-agent pipeline to extract and structure supervision at scale. Across 134.7K clips and 170 procedure types, SUREON yields 206.8k QA pairs and an expert-validated benchmark of 354 examples. To evaluate the extent to which this supervision translates to surgical reasoning ability, we introduce two models: SureonVLM, a vision-language model adapted through supervised fine-tuning, and SureonVLM-R1, a reasoning model trained with Group Relative Policy Optimization. Both models can answer complex questions about surgery and substantially outperform larger general-domain models, exceeding 84% accuracy on the SUREON benchmark while outperforming general-domain models on standard surgical perception tasks. Qualitative analysis of SureonVLM-R1 reveals explicit reasoning behavior, such as inferring operative intent from visual context.

Alejandra Perez, Anita Rau, Lee White +4

Intuitive Surgical Inc., Sunnyvale, United States

Surgical Video UnderstandingMedical VQAMedical VLMsMedical ReasoningVLM Reasoning
AskPDF