Language-Conditioned Behavioral Cloning for Autonomous Endoscope Guidance

Ariel Rodriguez; Lorenzo Mazza; Ortrun Hellig; Alejandra Toscano; Rayan Younis; Stefanie Speidel

cs.ROOct 30, 2026

Language-Conditioned Behavioral Cloning for Autonomous Endoscope Guidance

Authors: Ariel Rodriguez, Lorenzo Mazza, Ortrun Hellig, Alejandra Toscano, Rayan Younis, Stefanie Speidel

Organizations: NCT/UCC Dresden, UKDD Dresden, TU Dresden, DKFZ Heidelberg. · BMFTR Research Hub 6G-Life. · The Center for Tactile Internet with Human-in-the-Loop (CeTI), TUD, Dresden, Germany. · Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Dresden, Germany.

Abstract

Automated endoscope guidance reduces the need for continuous manual camera control during minimally invasive surgery. However, the desired camera motion depends on the surgeon's immediate intentions and cannot always be inferred from the visual operative scene alone. We propose to automate the task using language-conditioned behavioral cloning; we define 6 clinically relevant guidance behaviors and collect 235 subtask demonstrations. In preliminary phantom experiments, our policy achieves 100% success on focus, overview, and stop, 80% on search-tip and follow-tip behaviors, and 50% on horizon leveling. These results demonstrate the feasibility of learning heterogeneous surgeon-requested camera behaviors with a single visuomotor policy, while the observed failure modes motivate future work on adding temporal context for more demanding guidance tasks.

Figures & tables

Task	Description
Focus	The instrument operator defines a working region by grasping two tissue segments or pointing with two instruments, with the space between their tips indicating the target. The endoscope moves toward the region while keeping it centered in the image.
Overview	The instrument operator continues the current surgical activity, while the endoscope moves away from the observed working area to provide a wider view and keeps that area centered in the image.
Search tip	The instrument operator indicates a target with one instrument. The endoscope searches for the instrument when it is outside the field of view, follows its visible shaft until the tip is located, and centers the midpoint between the instrument tip and the indicated target.
Follow tip	The instrument operator points toward an anatomical structure and moves an instrument along its contour. The endoscope follows the instrument tip, keeping it near the center of the image while maintaining visibility of the surrounding anatomy.
Level horizon	The endoscope operator rolls the endoscope to correct the camera orientation and restore a level horizon and upright view of the operative field.
Stop	The other operators may continue activity in the phantom, remain inactive, or work outside the field of view, while the endoscope remains stationary and preserves the current view.

TABLE I: Endoscope-guidance tasks and their intended camera behavior.

Fig. 1: Environment for the endoscope-guidance task. The OpenHELP phantom is shown in the center, with the endoscope approaching from above. The setup was used for demonstration collection and policy deployment.

Fig. 2: Representative policy rollouts for five tasks. Each column shows the temporal evolution of one task from top to bottom.

	Focus	Overview	Search tip	Follow tip	Level horizon	Stop
Success rate (%, $\uparrow$ )	100 (10/10)	100 (10/10)	80 (8/10)	80 (8/10)	50 (5/10)	100 (10/10)
Average human completion time (s)	7.38	6.91	8.02	13.06	9.70	–
Average policy completion time (s)	9.36	7.40	15.34	34.71	35.95	–
Completion time ratio ( $\uparrow$ )	0.79	0.93	0.52	0.38	0.27	–

TABLE II: Evaluation of the BC policy on the guidance tasks, with ten rollouts per task. Timing statistics are averaged over successful sub-executions and reference human expert demonstrations. The completion time ratio is the average human completion time divided by the average policy completion time. Timing metrics are not applicable to the Stop task.

Robotic Endoscopy

See all 18 papers

EndoVLA-OOV: Language-Conditioned Out-of-View Target Acquisition in Robotic Endoscopy
Oct 17, 2026Chi Kit Ng, Yidong Zhang, Tianchun Wu +9Robotic Endoscopy Vision-Based Robot Control
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China · Department of Surgery, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong SAR, China +1
Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining
Feb 24, 2026Keyu Zhou, Peisen Xu, Yahao Wu +3Surgical Robotics Vision-Based Robot Control
Hangzhou Dianzi University, Hangzhou 310018, China · Zhejiang University, Hangzhou 310058, China
EBV-CAN: Vision--Language Grounding for Contact-Aided Endoscope Navigation
Oct 22, 2026Chi Kit Ng, Shuchang Liu, Jinsong Lin +4Robotic Endoscopy Endoscopy
The Chinese University of Hong Kong. · National University of Singapore.

Oct 17, 2026J/K move · Enter open · S save

EndoVLA-OOV: Language-Conditioned Out-of-View Target Acquisition in Robotic Endoscopy

Chi Kit Ng, Yidong Zhang, Tianchun Wu, Zhiqing Tang+8

Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China · Department of Surgery, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong SAR, China+1

Robotic Endoscopy Vision-Based Robot Control VLMs for Robotics Surgical Robotics

Open paper AskPDF arXiv

The scan-then-revisit workflow of upper gastrointestinal endoscopy requires bringing a previously observed suspicious region back into view for closer inspection. Acquiring such targets requires translating anatomical and appearance descriptions into camera motions from partial observations of a confined, deformable environment. We present EndoVLA-OOV, a language-conditioned vision-language-action policy for out-of-view target acquisition in robotic endoscopy. The policy adapts a pretrained backbone to continuous two-degree-of-freedom distal bending commands and combines initial, intermediate, and current images to guide search. To support policy learning, we provide an open-source VLA dataset containing 1,050 episodes across six tasks and approximately 262k observation-action pairs. Physical evaluation spans wet-lens and low-light conditions, two unseen stomach phantoms, and cross-embodiment transfer to an unseen ex-vivo environment. In the cross-embodiment evaluation, the full-data policy achieves 93.3% target acquisition success (28/30). In a controlled training-data comparison, expanding the corpus from 312 to 1,050 episodes increases acquisition success from 40.0% to 90.0% under the same model, training, and test configuration. Instruction and observation interventions further support the use of language and temporal context in the evaluated acquisition tasks. These results support language-guided endoscopic revisitation across the evaluated environments and robot embodiments and demonstrate the benefit of expanded task-specific demonstrations.

Abstract

Robotic Endoscopy

Explore similar work

EndoVLA-OOV: Language-Conditioned Out-of-View Target Acquisition in Robotic Endoscopy

Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining

EBV-CAN: Vision--Language Grounding for Contact-Aided Endoscope Navigation