Semantic-Coupled Perception Framework for Autonomous Laparoscope Control

Enduo Zhao; Fangzhou Li; Yujie Zhu; Sicheng Lu; Fang Chen; Longfei Ma; Hongen Liao

cs.ROOct 25, 2026

Semantic-Coupled Perception Framework for Autonomous Laparoscope Control

Authors: Enduo Zhao, Fangzhou Li, Yujie Zhu, Sicheng Lu, Fang Chen, Longfei Ma, Hongen Liao

Organizations: School of Biomedical Engineering, Tsinghua University, Beijing, China. · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.

Abstract

In minimally invasive surgery (MIS), maintaining an appropriate field of view (FoV) is essential for surgical safety and efficiency. However, conventional autonomous laparoscope control methods often treat visual tracking and surgical phase recognition independently, limiting robustness under occlusion, entity overlap, and camera motion. This paper proposes a closed-loop autonomous laparoscope control framework that couples real-time surgical phase recognition with camera-motion-compensated visual perception. By exploiting phase-dependent relationships among surgical entities, the framework improves target state estimation under challenging visual conditions, supporting task-aware autonomous FoV control. Experiments on benchmark datasets evaluate the proposed framework in surgical phase recognition and target tracking, including scenarios involving occlusion and entity overlap. Phantom experiments further demonstrate the feasibility of autonomous FoV control, showing improved target visibility and task execution performance compared with fixed-FoV control. These results highlight the potential of incorporating task-level semantics into visual state estimation for autonomous laparoscope guidance.

Figures & tables

Fig. 1: Overview of the proposed semantic-coupled perception and control framework with three modules: Visual Perception, Surgical State Inference, and Autonomous FoV Control. Solid arrows indicate forward information transfer and feedback between the hardware system and the modules, forming a closed perception-control loop. Time-step subscripts are omitted for clarity.

Fig. 2: Illustration of key components of the proposed framework. (a) Camera-motion-aware kinematics on a suturing scene: reference features from non-active entities are used for homography estimation (red arrows), and the task-related velocity $\tilde{\boldsymbol{v}}_{\text{inst2},t}$ (yellow arrow) of the active instrument is derived after compensation. (b) Semantic state maintenance on a peg-transfer scene: occluded green blocks are maintained as non-active static entities and re-associated with their previous states upon reappearance. (c) Spatial coupling on a suturing scene: the occluded needle is localized from the observable instrument 2 and the retained relative position $\boldsymbol{r}_{mg,t-1}$ , with $\boldsymbol{r}_{mg,t}=\boldsymbol{r}_{mg,t-1}$ during occlusion. For clarity, labels of the remaining entities are omitted.

Components				SAR-RARP50 [ 21 ]					Peg Transfer [ 23 ]					JIGSAWS [ 24 ]
Detection	Tracking	Camera Comp.	Semantic Coupling	Tracking			Phase		Tracking			Phase		Tracking			Phase
Detection	Tracking	Camera Comp.	Semantic Coupling	MPE	IDS	ORL	AC	JA	MPE	IDS	ORL	AC	JA	MPE	IDS	ORL	AC	JA
✓	–	–	–	21.7	23.1	13.0	62.7	47.2	52.2	41.4	27.6	71.4	61.0	35.5	26.1	14.1	67.5	50.1
✓	✓	–	–	13.2	11.4	7.6	66.9	54.3	36.9	19.5	12.3	79.5	67.6	21.3	20.5	10.3	72.3	59.4
✓	✓	✓	–	9.4	9.9	6.7	82.3	74.1	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
✓	✓	✓	✓	2.5	4.0	5.2	89.9	79.0	7.1	5.8	5.5	91.7	83.8	4.2	7.0	6.8	90.1	81.2

TABLE I: Ablation study of target tracking and surgical phase recognition.

Dataset	Method	AC (%)	PR (%)	RE (%)	JA (%)
SAR-RARP50 [ 21 ]	NFSM [ 25 ]	82.5	80.1	79.5	71.1
	MS-ASCT [ 26 ]	86.9	84.1	84.0	75.5
	STSANet [ 27 ]	88.4	86.7	86.6	77.2
	LemonFM [ 28 ]	89.1	88.3	88.1	78.7
	Ours	89.9	88.3	88.0	79.0
Peg Transfer [ 23 ]	NFSM [ 25 ]	78.2	77.8	77.7	61.7

TABLE II: Comparison with state-of-the-art phase recognition methods

Fig. 3: Qualitative comparison with SOTA methods on the Peg Transfer dataset. Different colors represent different surgical phases.

Fig. 4: Hardware setup of Experiment III consists of a robotic arm holding a laparoscope, a surgical monitor, and a laparoscopic training box with standard FLS Peg Transfer and Suturing modules.

Fig. 5: Results of Experiment III. (a1)(b1) Spatial heatmaps of active entity positions in Peg Transfer and Suturing under fixed FoV and the proposed control, with the safe region $\Omega_{c}$ (yellow box) and outer triggering region $\Omega_{\text{out}}$ (red box). (a2)(b2) Temporal profiles of horizontal/vertical errors ( $e_{u}$ , $e_{v}$ , normalized), Euclidean pixel error, and cumulative exit count over representative trials. (c) Tracking-based versus proposed control in a needle-passing scene during Suturing, showing the FoV target (red cross) and entity states. (d) Camera path length, control latency, and update frequency in both tasks.

Task	Metric	Fixed	Tracking	Proposed
Peg Transfer	Pixel error	323.6 $\pm$ 21.7	267.3 $\pm$ 19.7	241.9 $\pm$ 13.7
	Time in $\Omega_{c}$	43.0 $\pm$ 6.1	73.0 $\pm$ 7.2	71.4 $\pm$ 4.8
	Exits from $\Omega_{\text{out}}$	12.3 $\pm$ 2.4	14.1 $\pm$ 3.6	7.1 $\pm$ 2.4
	Time	170.9 $\pm$ 24.8	150.1 $\pm$ 22.4	138.1 $\pm$ 14.8
Suturing	Pixel error	202.6 $\pm$ 24.5	161.0 $\pm$ 20.9	155.1 $\pm$ 11.7
	Time in $\Omega_{c}$	59.9 $\pm$ 12.6	78.1 $\pm$ 17.4	79.2 $\pm$ 5.6

TABLE III: Quantitative results of fixed, tracking-based, and semantic-coupled FoV control.

Surgical Robotics

Surgical robotics develops robotic tools and systems for minimally invasive, image-guided, and microsurgical procedures.

See all 187 papers

Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining
Feb 24, 2026Keyu Zhou, Peisen Xu, Yahao Wu +3Surgical Robotics Vision-Based Robot Control
Hangzhou Dianzi University, Hangzhou 310018, China · Zhejiang University, Hangzhou 310058, China
SurgLAT: Surgical Latent Attention Tracking for Depth-Aware Robotic Laparoscope Control
Aug 8, 2026Rulin Zhou, Qiujie Song, Yujie Ma +10Surgical Video Understanding Vision-Based Robot Control
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China · Shenzhen Loop Area Institute, Shen Zhen 518055, China · College of Mechatronics and Engineering, Shenzhen University, Shenzhen 518060, China +2
Autonomous Laparoscope Control through Unified Mechanics-Based Representation of Multimodal Intraoperative Information
May 6, 2026Xiaojian Li, Jin Fang, Yudong Shi +6Vision-Based Robot Control Surgical Robotics
School of Management and the Key Laboratory of Process Optimization and Intelligent Decision-Making (Ministry of Education), Hefei University of Technology, Hefei 230009, China · Department of Thoracic Surgery, Shanghai Pulmonary Hospital, School of Medicine, Tongi University, Shanghai 200433, China · Department of Thoracic Surgery, Changhai Hospital, Shanghai 200433, China

Feb 24, 2026J/K move · Enter open · S save

Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining

Keyu Zhou, Peisen Xu, Yahao Wu, Jiming Chen+2

Hangzhou Dianzi University, Hangzhou 310018, China · Zhejiang University, Hangzhou 310058, China

Surgical Robotics Vision-Based Robot Control VLMs for Robotics

Open paper AskPDF arXiv

Autonomous laparoscopic camera control must maintain a stable and safe surgical view under rapid tool-tissue interactions while remaining interpretable to surgeons. We present a strategy-grounded framework that couples high-level vision-language inference with low-level closed-loop control. Offline, raw surgical videos are parsed into camera-relevant temporal events (e.g., interaction, working-distance deviation, and view-quality degradation) and structured as attributed event graphs. Mining these graphs yields a compact set of reusable camera-handling strategy primitives, which provide structured supervision for learning. Online, a fine-tuned Vision-Language Model (VLM) processes the live laparoscopic view to predict the dominant strategy and discrete image-based motion commands, executed by an IBVS-RCM controller under strict safety constraints; optional speech input enables intuitive human-in-the-loop conditioning. On a surgeon-annotated dataset, event parsing achieves reliable temporal localization (F1-score 0.86), and the mined strategies show strong semantic alignment with expert interpretation (cluster purity 0.81). Extensive ex vivo experiments on silicone phantoms and porcine tissues demonstrate that the proposed system outperforms junior surgeons in standardized camera-handling evaluations, reducing field-of-view centering error by 35.26% and image shaking by 62.33%, while preserving smooth motion and stable working-distance regulation.

Abstract

Surgical Robotics

Explore similar work

Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining

SurgLAT: Surgical Latent Attention Tracking for Depth-Aware Robotic Laparoscope Control

Autonomous Laparoscope Control through Unified Mechanics-Based Representation of Multimodal Intraoperative Information