cs.CVNov 7, 2026

Remembering to Reason: Structured Memory for Scene and Workflow Understanding in Robot-Assisted Surgery

Authors: Zhiran Zhang, Hu Wang, Yichen Wu, Yutong Xie

Organizations: Zhiran Zhang, Hu Wang, and Yutong Xie are with Mohamed bin Zayed University of Artificial Intelligence. · Yichen Wu is with Imperial College London.

Abstract

Surgical video understanding is an important foundation for intelligent assistance in robot-assisted surgery. Similar surgical views can reflect different actions or procedural stages. Interpreting these views requires access to earlier observations, yet frame-centric prompting leaves this history unavailable. We introduce MemSurg, a framework that remembers surgical observations before reasoning over them. Its structured memory graph stores timestamped entity categories, locations, motion cues, and interaction summaries. Chronological links preserve recent context, while entity-associated links connect recurring observations across the procedure. Task-adaptive retrieval then supplies evidence-grounded prompts for scene and workflow reasoning, supporting both frozen inference and memory-conditioned supervised fine-tuning. Evaluation on robot-assisted prostatectomy and two complementary surgical benchmarks shows a 10.43-percentage-point gain over SurgRAW with GPT-4o. Memory conditioning adds 6.52 points under matched Qwen3-VL-8B fine-tuning, and an inference-time plug-in adds 7.04 points to EndoChat. These results support structured memory as a reusable source of procedural evidence for surgical video understanding. Our code will be publicly released.

Figures & tables

Explore similar work

CardsList
  1. SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding

    Aug 5, 2026Yuqing Feng, Jiawei Ma, Kevin Qinghong Lin +6Surgical Video UnderstandingGenerative Retrieval

  2. SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding

    Nov 1, 2026Ruipeng Ma, Qing Xu, Jie Liang +1Streaming Video UnderstandingSurgical Video Understanding

  3. SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis

    Mar 13, 2025Chang Han Low, Ziyue Wang, Tianyi Zhang +4Surgical Video UnderstandingMultimodal CoT Reasoning