Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining

Keyu Zhou; Peisen Xu; Yahao Wu; Jiming Chen; Gaofeng Li; Shunlei Li

cs.ROFeb 24, 2026

Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining

Authors: Keyu Zhou, Peisen Xu, Yahao Wu, Jiming Chen, Gaofeng Li, Shunlei Li

Organizations: Hangzhou Dianzi University, Hangzhou 310018, China · Zhejiang University, Hangzhou 310058, China

Abstract

Autonomous laparoscopic camera control must maintain a stable and safe surgical view under rapid tool-tissue interactions while remaining interpretable to surgeons. We present a strategy-grounded framework that couples high-level vision-language inference with low-level closed-loop control. Offline, raw surgical videos are parsed into camera-relevant temporal events (e.g., interaction, working-distance deviation, and view-quality degradation) and structured as attributed event graphs. Mining these graphs yields a compact set of reusable camera-handling strategy primitives, which provide structured supervision for learning. Online, a fine-tuned Vision-Language Model (VLM) processes the live laparoscopic view to predict the dominant strategy and discrete image-based motion commands, executed by an IBVS-RCM controller under strict safety constraints; optional speech input enables intuitive human-in-the-loop conditioning. On a surgeon-annotated dataset, event parsing achieves reliable temporal localization (F1-score 0.86), and the mined strategies show strong semantic alignment with expert interpretation (cluster purity 0.81). Extensive ex vivo experiments on silicone phantoms and porcine tissues demonstrate that the proposed system outperforms junior surgeons in standardized camera-handling evaluations, reducing field-of-view centering error by 35.26% and image shaking by 62.33%, while preserving smooth motion and stable working-distance regulation.

Figures & tables

Figure 1: Overview of the proposed strategy-supervised autonomous laparoscopic camera control framework: The bottom block illustrates the offline pipeline, where laparoscopic video demonstrations are parsed into camera-relevant events, organized as attributed event graphs, and clustered to discover dominant camera-handling strategy primitives. These clusters generate strategy labels and corresponding direction supervision signals. The top row shows the online control pipeline: a laparoscopic frame (with optional speech input) is processed by a vision-language backbone with two prediction heads that output a dominant strategy label and discrete 6-DoF motion directions. The predicted directions are executed by an IBVS–RCM controller to maintain stable, safe, and context-aware camera positioning, with surgeon-in-the-loop safety override.

Figure 2: Event parsing from raw laparoscopic video: Three branches detect interaction-driven events, depth-change events, and view-quality constraint events (visibility degradation and lens contamination), whose candidate intervals are fused by temporal event segmentation to form structured events $e_{k}=(\text{type},t_{s},t_{e},x_{k})$ . The bottom timeline illustrates the resulting aligned event segments over time.

Figure 3: Event graph representation and strategy mining: Camera-relevant events are represented as nodes in an attributed graph, with temporal and semantic relations encoded as heterogeneous edges. WSBGC groups events into coherent clusters corresponding to reusable camera-handling strategies.

Cluster	Strategy Description
$C_{1}$	Stable hold with neutral directions.
$C_{2}$	Small translation for tool re-centering.
$C_{3}$	Depth-dominant controlled approach.
$C_{4}$	Depth-dominant controlled withdrawal.
$C_{5}$	Horizontal-dominant motion tracking.
$C_{6}$	Vertical-dominant motion tracking.

Table I: Summary of the 12 mined strategy clusters.

Figure 4: Experimental setup: From left to right, the surgical equipments used in the experiments, including the synthetic abdominal cavity, surgical instruments, and anti-fog devices; an overview of the surgical workspace with the robotic manipulator; and the manipulated objects, consisting of a silicone phantom and porcine tissues used to mimic real organ.

Figure 5: Experimental validation of the proposed autonomous laparoscopic control method in complex surgical tasks: Upper part: Illustration of a multi-site dissection task performed on porcine gastric tissue. The images demonstrate the autonomous following behavior of the laparoscope, as well as its ability to adapt the penetration ratio in response to shifts in the surgical instrument’s workspace. Lower part: Validation during a simulated silicone suturing task. The system exhibits multi-modal interaction capability, allowing the surgeon to refine the laparoscope pose via voice commands without interrupting the manual suturing process. The timestamps shown in the bottom-right corner highlight the continuous operation and efficiency of the proposed robotic assistance system.

Figure 6: Experimental validation of the VLM-based fogging and contamination detection capability: Upper part: System response under mild fogging conditions, where the laparoscope remains stationary, enabling uninterrupted surgical operation. Lower part: Autonomous detection and recovery under severe fogging and contamination. Upon detecting severe fogging, the system automatically initiates an “extraction–watering–wiping” sequence to restore visual clarity. On the other hand, once contamination (e.g., blood or tissue debris) is located on the lens, the system triggers the same cleaning procedure as in the severe fogging scenario to ensure a clear surgical view. The results demonstrate the robustness of the proposed framework in maintaining an optimal surgical field of view (FoV) through autonomous instrument withdrawal and lens cleaning.

Figure 7: Effect of the number of clusters

K

on clustering quality: As

K

increases from 8 to 12, NMI improves and intra-cluster variance decreases, indicating more coherent strategy grouping. For

K\geq 14

, gains saturate and mild over-segmentation effects emerge.

Event	Precision	Recall	F1-score	Mean tIoU
Interaction	0.86	0.81	0.83	0.67
Depth deviation	0.90	0.88	0.89	0.74
Visibility degradation	0.84	0.79	0.81	0.64
Lens contamination	0.93	0.91	0.92	0.80
Average	0.88	0.85	0.86	0.71

Table II: Performance of Event Detection (

\mathrm{tIoU}\geq 0.5

)

Figure 8: 3D spatial trajectories of the autonomous robotic laparoscopic system across diverse surgical tasks and operator expertise levels: The trajectories illustrate the scope’s movement and the control system’s performance during three benchmarking tasks: silicone stitching, intestine stitching, and dissection. Rows represent the performance of senior, junior, and intern surgeons, respectively. In each subplot, the path initiates from the start point (green dot) and concludes at the end point (orange dot). All spatial coordinates are measured in meters (m)

Figure 9: Comparison of Field-of-View Centering and Image Shaking performance between manual camera control and the proposed method: (a): The average pixel errors between the current and the desired instrument tip pixel coordinates under two methods. (b): The average pixel differences of the selected feature point between two adjacent images.

Metric	Method	$1^{\mathrm{st}}$	$2^{\mathrm{nd}}$	$3^{\mathrm{rd}}$	$4^{\mathrm{th}}$	$5^{\mathrm{th}}$	$6^{\mathrm{th}}$	$7^{\mathrm{th}}$	$8^{\mathrm{th}}$	$9^{\mathrm{th}}$
Field-of-View Centering: $e_{cen}$	Human	142.12	130.45	163.61	121.02	121.54	116.17	187.00	109.09	143.25
	Ours	62.47	83.52	104.01	97.35	94.07	99.04	77.59	66.86	114.20
Image Shaking: $S$	Human	1.45	1.68	1.86	1.68	1.07	1.81	1.36	1.63	1.48
	Ours	0.63	0.47	0.77	0.67	0.47	0.61	0.58	0.69	0.38

Table III: Comparison of Field-of-View Centering and Image Shaking performance between manual camera control and the proposed method.

Surgical Robotics

Surgical robotics develops robotic tools and systems for minimally invasive, image-guided, and microsurgical procedures.

See all 187 papers

SurgLAT: Surgical Latent Attention Tracking for Depth-Aware Robotic Laparoscope Control
Aug 8, 2026Rulin Zhou, Qiujie Song, Yujie Ma +10Surgical Video Understanding Vision-Based Robot Control
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China · Shenzhen Loop Area Institute, Shen Zhen 518055, China · College of Mechatronics and Engineering, Shenzhen University, Shenzhen 518060, China +2
Semantic-Coupled Perception Framework for Autonomous Laparoscope Control
Oct 25, 2026Enduo Zhao, Fangzhou Li, Yujie Zhu +4Surgical Robotics Robotic Perception
School of Biomedical Engineering, Tsinghua University, Beijing, China. · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.
Trajectory Divergence Horizon Decision for Reliable Dual-Arm Surgical Subtask Manipulation
Aug 10, 2026Mingwu Su, Guankun Wang, Jinsong Lin +8Robotic Control Bimanual Robotic Manipulation
Department of Electronic Engineering, The Chinese University of Hong Kong (CUHK), Hong Kong, China. · Shenzhen Loop Area Institute, Shenzhen, China. · The Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd., Hong Kong SAR, China.

Aug 8, 2026J/K move · Enter open · S save

SurgLAT: Surgical Latent Attention Tracking for Depth-Aware Robotic Laparoscope Control

Rulin Zhou, Qiujie Song, Yujie Ma, An Wang+9

Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China · Shenzhen Loop Area Institute, Shen Zhen 518055, China · College of Mechatronics and Engineering, Shenzhen University, Shenzhen 518060, China+2

Surgical Video Understanding Vision-Based Robot Control Surgical Robotics Laparoscopic Surgery

Open paper AskPDF arXiv

Autonomous laparoscopic camera control requires continuous understanding of the surgeon's operative intent in dynamic surgical scenes, where the target operative region is not a stable physical object but a latent and temporally evolving attention state. In this work, we present Surgical Latent Attention Tracking (SurgLAT), a causal online framework for latent surgical attention modeling and autonomous laparoscopic view control. SurgLAT uses a frozen DINOv3 encoder and a state-conditioned spatial token mixer to extract operative evidence under a memory-guided spatial prior, while a selective causal latent memory module jointly models short-term motion continuity and long-horizon surgical intent evolution through dynamic retrieval of current, recent, and historical latent states. The learned latent surgical attention state is decoded into a probabilistic attention heatmap and operative region for downstream endoscope guidance. Beyond perception, we further introduce a robotic deployment framework with explicit laparoscopic Remote Center of Motion (RCM) constrained control based on virtual-axis formulation, together with redundancy-aware null-space initialization for stable and smooth manipulator motion. We validate the full system on real laparoscopic surgical videos and a physical robotic laparoscope platform. Experimental results demonstrate robust online operative-region tracking and stable autonomous endoscopy adjustment under occlusion, rapid motion, and target transitions, highlighting the effectiveness of latent surgical intent modeling for surgical autonomy.

Abstract

Surgical Robotics

Explore similar work

SurgLAT: Surgical Latent Attention Tracking for Depth-Aware Robotic Laparoscope Control

Semantic-Coupled Perception Framework for Autonomous Laparoscope Control

Trajectory Divergence Horizon Decision for Reliable Dual-Arm Surgical Subtask Manipulation