EndoVLA-OOV: Language-Conditioned Out-of-View Target Acquisition in Robotic Endoscopy
Authors: Chi Kit Ng, Yidong Zhang, Tianchun Wu, Zhiqing Tang, Hanlin Zhao, Siu Hing Lui, Tao Yang, Jingsong Lin, Huxin Gao, Yeung Trevor, Raymond Tang, Hongliang Ren
Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China · Department of Surgery, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Medicine and Therapeutics, The Chinese University of Hong Kong, Hong Kong SAR, China
The scan-then-revisit workflow of upper gastrointestinal endoscopy requires bringing a previously observed suspicious region back into view for closer inspection. Acquiring such targets requires translating anatomical and appearance descriptions into camera motions from partial observations of a confined, deformable environment. We present EndoVLA-OOV, a language-conditioned vision-language-action policy for out-of-view target acquisition in robotic endoscopy. The policy adapts a pretrained backbone to continuous two-degree-of-freedom distal bending commands and combines initial, intermediate, and current images to guide search. To support policy learning, we provide an open-source VLA dataset containing 1,050 episodes across six tasks and approximately 262k observation-action pairs. Physical evaluation spans wet-lens and low-light conditions, two unseen stomach phantoms, and cross-embodiment transfer to an unseen ex-vivo environment. In the cross-embodiment evaluation, the full-data policy achieves 93.3% target acquisition success (28/30). In a controlled training-data comparison, expanding the corpus from 312 to 1,050 episodes increases acquisition success from 40.0% to 90.0% under the same model, training, and test configuration. Instruction and observation interventions further support the use of language and temporal context in the evaluated acquisition tasks. These results support language-guided endoscopic revisitation across the evaluated environments and robot embodiments and demonstrate the benefit of expanded task-specific demonstrations.
Figures & tables
Fig. 1: Scan-then-revisit motivation and the language-conditioned distal-control task. The operator specifies target appearance and an anatomical station, and the policy generates bending actions. The schematic illustrates the intended interface; acquisition, centering and language dependence are evaluated separately.
Aspect
Prior EndoVLA [ 21 ]
EndoVLA-OOV
Task
In-view tracking
Initially OOV acquisition; composite FT
Actions
Four directions and stop
Continuous 2D chunks
Data
Approx. 6,000 pairs; 3 tracking tasks
1,050 episodes; 6 tasks; 262k pairs
Evaluation
Tracking protocol
Second robot test; ex-vivo validation
TABLE I: Scope relative to prior EndoVLA.
Fig. 2: Policy interface and adaptation architecture. The initial global, recent-history and current images provide temporal context to the vision-language model (VLM). A state encoder supplies continuous robot-state features directly to the action expert, which predicts two-dimensional bending-action chunks conditioned on the visual and language context.
Fig. 3: Robotic platforms and the collection-to-evaluation workflow. (a) R1 teleoperation in the training phantom produces synchronized endoscopic images, instructions, recorded states and continuous bending commands, stored in LeRobot format for policy adaptation. (b) R1 uses a 6 mm endoscope and R2 a 3.9 mm endoscope. R1-trained policies are deployed on R2 in an unseen ex-vivo environment for zero-shot cross-embodiment evaluation.
Task
Training instruction
A episodes
A pairs
B episodes
B pairs
T0
Tracking white oval suspicious region.
52
13,356
150
36,632
T1
Tracking small round raised nodule.
52
10,667
150
52,843
T2
Tracking the orange protruding lesion.
52
9,386
150
31,564
A (GC)
Search the Greater Curvature for a white oval suspicious region on the mucosal folds.
52
7,364
200
27,999
B (PA)
Search the Pyloric Antrum to find a small round raised nodule.
52
18,377
200
50,536
C (LC)
Search the Lesser Curvature and locate the orange protruding lesion.
52
20,160
200
62,040
TABLE II: Task design and training-data provenance. Ours-A and all VLA baselines are fine-tuned on subset A. B is the full corpus used by Ours-B. Dataset A/B, task A/B/C and organ exA–F are distinct identifiers.
Fig. 4: System-level comparison on R1: (a) finding and (b) composite FT. Each entry is successes among nine trials. HD/HW/LD/LW denote high/low light and dry/wet conditions. Ours-A and all VLA baselines are fine-tuned on subset A. MS and ES denote one medical student and one engineering student, respectively. Supplementary Sec. S8 specifies the R1 finding and FT protocol; Tables S8a and S8b provide full counts and Wilson intervals.
Fig. 5: Environment transfer of Ours-A on R1. Composite FT results use nine trials per unseen-phantom condition and twelve trials per ex-vivo anatomical region. Bars are 95% Wilson intervals. The panel reports FT on R1; the cross-embodiment evaluation on R2 reports Find and Center separately.
Fig. 6: Environment-transfer settings for Ours-A on R1: (a) two unseen stomach phantoms and (b) an ex-vivo preparation. The R2 evaluation uses organs exA–F.
Fig. 7: Selected successful Ours-A FT rollouts on R1 in the two unseen phantoms. Each triplet shows an initial global view, an intermediate view and a final centered view. Panels (a1–a4) and (b1–b4) cover HD/HW/LD/LW on unseen phantoms 1 and 2. The panels illustrate completed search-and-center trajectories; Fig. 5 reports condition-level success counts.
Fig. 8: Selected Ours-B target-acquisition rollouts on R2 in ex-vivo tissue. Rows show GC on exA, PA on exB and LC on exC, from top to bottom. Each sequence progresses from a global view through search to target acquisition. Figure 10 reports Find and Center outcomes across all standard trials.
Fig. 9: Selected Ours-B rollouts on R2 under instruction interventions. Rows show a wrong-region request on exA, unseen correct wording on exB and an empty instruction on exC. The wrong-region rollout bends toward the target-free requested region; the unseen-wording rollout reaches the target, and the empty-instruction rollout continues searching. Figure 10 reports the corresponding ten-trial group outcomes.
Fig. 10: Ours-B on R2: standard exA–C trials, instruction conditions and post-occlusion recovery. Each condition has ten trials; bars are 95% Wilson intervals. Find and Center follow the criteria in Supplementary Sec. S4. Standard exA and the separate full-instruction group each have Center 8/10. Each instruction condition is associated with the organ shown.
Fig. 11: Training-data comparison on R2 using organs exD–F. Ours-A uses 312 episodes and Ours-B uses 1,050 episodes; the other model, training and test conditions are unchanged. Each model has ten trials per organ. Bars are trial-level 95% Wilson intervals; observations are repeated within each organ.
Input (frames)
Find
Center for 2 s
C (1)
6/10 (60; 31.3–83.2)
5/10 (50; 23.7–76.3)
R+C (2)
7/10 (70; 39.7–89.2)
6/10 (60; 31.3–83.2)
G+C (2)
7/10 (70; 39.7–89.2)
7/10 (70; 39.7–89.2)
G+R+C (3)
9/10 (90; 59.6–98.2)
9/10 (90; 59.6–98.2)
TABLE III: Ours-B visual-context ablation. G/R/C: global/recent/current frames; omitted slots are masked. Entries: k/n (%; Wilson 95% interval).
Model
A (Hz)
H
B (Hz)
1/A (ms)
EndoVLA
1.480
1
1.480
675.7
OpenVLA-7B
5.228
1
5.228
191.3
π0
17.69
20
353.8
56.5
π0.5
15.68
20
313.6
63.8
Ours
14.27
20
285.4
70.1
TABLE IV: 300-input timing benchmark. B is the theoretical maximum chunk-equivalent control rate; physical motor dispatch is 20 Hz.
Fig. 12: Selected FT failures on R1 involving (a) a target-like water droplet, (b) a bubble, (c) search in an incorrect local region, and (d) incomplete reacquisition after partial approach. The examples illustrate visual discrimination and search-recovery challenges in the evaluated trajectories.
Figure S1: Robot platforms and ex-vivo evaluation settings. R1 (left) provides demonstrations and baseline/environment-transfer trials. R2 (middle) is used for the exA–exF experiments (right).
Figure S2: YOLO Studio annotation interface. Editable target boxes and an explicit No Detection control support frame-level verification of target visibility and position.
Figure S3: Ours-A action changes under trained-instruction substitution on EA . Each cell is the mean chunk distance in Eq. ( S2 ). Rows identify recorded tasks and columns identify replacement instructions. The diagonal is the unchanged-instruction reference.
Figure S4: Ours-A action sensitivity to alternative instruction strings on EA . Each bar is the mean action-chunk L2 distance from the reference prediction under one shared noise realization.
Figure S5: Motor-direction agreement over 1-, 2- and 20-step prediction prefixes. Each rate averages the per-anchor sign-agreement fraction on eligible demonstrated motor components.
Figure S6: Centering experiment on R1 under (a) high-light dry, (b) high-light wet, (c) low-light dry and (d) low-light wet conditions. Box plots summarize the trial-level ΔPE index. The Ours label denotes Ours-A.
Figure S7: Successful Ours-A finding-and-tracking trajectories on R1 under (a) HD, (b) HW, (c) LD and (d) LW. Image sequences show search and approach; motor-position and pixel-error traces show the progression from finding to tracking.
Endoscopic retrograde cholangiopancreatography (ERCP) demands precise endoscopic navigation and stable biliary cannulation within a narrow monocular field characterized by specular reflections, partial occlusions, and frequent tissue contact. Although recent robotic systems and vision-based assistance techniques improve operator ergonomics and provide perceptual cues, their performance degrades under pronounced anatomical variability and safety-critical visual artifacts, which hinders reliable autonomy in cannulation-grade procedures. Here, we present BiliVLA, a scene-aware Vision-Language-Action (VLA) framework that formulates biliary endoscopic navigation as an instruction-conditioned visuomotor learning problem. Given an endoscopic observation and a stage-specific language instruction, BiliVLA jointly predicts the target category, a grounded bounding box, and a discrete three-degree-of-freedom (3-DoF) motor command for a continuum endoscope. The proposed framework incorporates scene-aware supervision to improve semantic target consistency and safety-aware recovery supervision to induce conservative retreat behaviors under luminal wall contact. A key component of BiliVLA is a two-stage training paradigm that combines grounding-enhanced supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO), thereby improving action reliability and decision consistency during closed-loop navigation. Across three ERCP subtasks, BiliVLA achieves the best overall performance in physical phantom experiments, with a total mIoU of 0.9625, an overall action precision of 91.96%, and an overall success rate (SR) of 84.85%. These results indicate that integrating semantic grounding, scene-aware learning, and reward-guided optimization strengthens perception--action alignment and enables more robust autonomous biliary endoscopic navigation.
Jinsong Lin, Chi Kit Ng, Zhiyong Xiong +8
The Chinese University of Hong Kong, Hong Kong SAR, China. · The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China. · University of Cambridge, Cambridge, United Kingdom. +2
Routine gastrointestinal endoscopy is intrinsically bidirectional: the instrument is advanced to reach target anatomy and later withdrawn or retroflexed for inspection, while an external cue may require earlier reversal. When the requested phase changes before the visual scene does, nearly identical observations can require opposite axial actions. We identify and formalize this ambiguity in bidirectional endoscopic control as intent aliasing. We propose EndoLIFT (Endoscopic Language-Instruction Flow with Trajectory Latents), a vision-language-action policy that combines explicit language-based intent conditioning with a latent-conditioned rectified-flow action expert. The policy receives RGB, a language instruction, and the previous-action state; a 32-D variational trajectory latent stochastically conditions continuous action-chunk generation. Controlled same-observation instruction swaps establish that language selects the axial mode, independently of whether the trajectory latent is present. Relative to the matched model without latent conditioning, EndoLIFT improves navigation-direction accuracy by 11.1 percentage points and reduces wrong-direction advance by 83%. An architecture-controlled 1-bit mode-flag reference exhibits weaker canonical-anchor switching, while EndoLIFT retains 82.8% intent-following accuracy across 44 held-out linguistic variants. In closed-loop evaluation, EndoLIFT improves overall success by 30 percentage points over EndoLIFT w/o VTL on both the seen colon phantom and the unseen lung and stomach phantoms, and completes 10/10 ex-vivo porcine-trachea trials. These results separate language-based intent selection from the trajectory latent's contribution to directional correctness and robust retraction.
Chi Kit Ng, Yidong Zhang, Lui Siu Hing +9
The Chinese University of Hong Kong, Hong Kong SAR, China · The Sixth Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
Vision-language-action (VLA) policies must ground language instructions in the right objects, contact points, and target locations for reliable manipulation. Human gaze provides a natural signal for this grounding, but existing robot datasets rarely pair real human gaze with the multi-camera observations used by modern policies. We introduce GazeAct, a gaze-equipped real-robot manipulation dataset and training framework for multi-view VLA policies. Using screen-based teleoperation, we record gaze directly on the robot camera feed viewed by the operator, yielding 12000 demonstrations across 40 tasks and five domains with 300 Hz gaze synchronized with images, proprioception, and actions. We transfer gaze from the operator-selected view to the remaining synchronized cameras, producing spatially consistent targets across the policy's visual inputs. We then use these gaze-derived targets to regularize language-conditioned visual attention during training while keeping inference gaze-free. Across 18 real-world tasks, our method improves average success from 48.1% to 58.6%, with gains up to 25 percentage points. Ablations show that gaze works best as soft supervision and that real human gaze provides stronger attention targets than generic attention-shaping priors. The gains transfer to LIBERO-Spatial (57.9% to 69.6%) and different VLA backbones.These results show that gaze collected in the robot observation space can provide effective training-time targets for improving visual grounding in VLA manipulation.
Anupam Pani, Wang Yuan, Qifan Wang +1
HKU Musketeers Foundation Institute of Data Science · Department of Electrical and Electronic Engineering, The University of Hong Kong