EndoVLA-OOV: Language-Conditioned Out-of-View Target Acquisition in Robotic Endoscopy
Authors: Chi Kit Ng, Yidong Zhang, Tianchun Wu, Zhiqing Tang, Hanlin Zhao, Siu Hing Lui, Tao Yang, Jingsong Lin, Huxin Gao, Yeung Trevor, Raymond Tang, Hongliang Ren
Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China · Department of Surgery, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Medicine and Therapeutics, The Chinese University of Hong Kong, Hong Kong SAR, China
The scan-then-revisit workflow of upper gastrointestinal endoscopy requires bringing a previously observed suspicious region back into view for closer inspection. Acquiring such targets requires translating anatomical and appearance descriptions into camera motions from partial observations of a confined, deformable environment. We present EndoVLA-OOV, a language-conditioned vision-language-action policy for out-of-view target acquisition in robotic endoscopy. The policy adapts a pretrained backbone to continuous two-degree-of-freedom distal bending commands and combines initial, intermediate, and current images to guide search. To support policy learning, we provide an open-source VLA dataset containing 1,050 episodes across six tasks and approximately 262k observation-action pairs. Physical evaluation spans wet-lens and low-light conditions, two unseen stomach phantoms, and cross-embodiment transfer to an unseen ex-vivo environment. In the cross-embodiment evaluation, the full-data policy achieves 93.3% target acquisition success (28/30). In a controlled training-data comparison, expanding the corpus from 312 to 1,050 episodes increases acquisition success from 40.0% to 90.0% under the same model, training, and test configuration. Instruction and observation interventions further support the use of language and temporal context in the evaluated acquisition tasks. These results support language-guided endoscopic revisitation across the evaluated environments and robot embodiments and demonstrate the benefit of expanded task-specific demonstrations.
Figures & tables
Fig. 1: Scan-then-revisit motivation and the language-conditioned distal-control task. The operator specifies target appearance and an anatomical station, and the policy generates bending actions. The schematic illustrates the intended interface; acquisition, centering and language dependence are evaluated separately.
Aspect
Prior EndoVLA [ 21 ]
EndoVLA-OOV
Task
In-view tracking
Initially OOV acquisition; composite FT
Actions
Four directions and stop
Continuous 2D chunks
Data
Approx. 6,000 pairs; 3 tracking tasks
1,050 episodes; 6 tasks; 262k pairs
Evaluation
Tracking protocol
Second robot test; ex-vivo validation
TABLE I: Scope relative to prior EndoVLA.
Fig. 2: Policy interface and adaptation architecture. The initial global, recent-history and current images provide temporal context to the vision-language model (VLM). A state encoder supplies continuous robot-state features directly to the action expert, which predicts two-dimensional bending-action chunks conditioned on the visual and language context.
Fig. 3: Robotic platforms and the collection-to-evaluation workflow. (a) R1 teleoperation in the training phantom produces synchronized endoscopic images, instructions, recorded states and continuous bending commands, stored in LeRobot format for policy adaptation. (b) R1 uses a 6 mm endoscope and R2 a 3.9 mm endoscope. R1-trained policies are deployed on R2 in an unseen ex-vivo environment for zero-shot cross-embodiment evaluation.
Task
Training instruction
A episodes
A pairs
B episodes
B pairs
T0
Tracking white oval suspicious region.
52
13,356
150
36,632
T1
Tracking small round raised nodule.
52
10,667
150
52,843
T2
Tracking the orange protruding lesion.
52
9,386
150
31,564
A (GC)
Search the Greater Curvature for a white oval suspicious region on the mucosal folds.
52
7,364
200
27,999
B (PA)
Search the Pyloric Antrum to find a small round raised nodule.
52
18,377
200
50,536
C (LC)
Search the Lesser Curvature and locate the orange protruding lesion.
52
20,160
200
62,040
TABLE II: Task design and training-data provenance. Ours-A and all VLA baselines are fine-tuned on subset A. B is the full corpus used by Ours-B. Dataset A/B, task A/B/C and organ exA–F are distinct identifiers.
Fig. 4: System-level comparison on R1: (a) finding and (b) composite FT. Each entry is successes among nine trials. HD/HW/LD/LW denote high/low light and dry/wet conditions. Ours-A and all VLA baselines are fine-tuned on subset A. MS and ES denote one medical student and one engineering student, respectively. Supplementary Sec. S8 specifies the R1 finding and FT protocol; Tables S8a and S8b provide full counts and Wilson intervals.
Fig. 5: Environment transfer of Ours-A on R1. Composite FT results use nine trials per unseen-phantom condition and twelve trials per ex-vivo anatomical region. Bars are 95% Wilson intervals. The panel reports FT on R1; the cross-embodiment evaluation on R2 reports Find and Center separately.
Fig. 6: Environment-transfer settings for Ours-A on R1: (a) two unseen stomach phantoms and (b) an ex-vivo preparation. The R2 evaluation uses organs exA–F.
Fig. 7: Selected successful Ours-A FT rollouts on R1 in the two unseen phantoms. Each triplet shows an initial global view, an intermediate view and a final centered view. Panels (a1–a4) and (b1–b4) cover HD/HW/LD/LW on unseen phantoms 1 and 2. The panels illustrate completed search-and-center trajectories; Fig. 5 reports condition-level success counts.
Fig. 8: Selected Ours-B target-acquisition rollouts on R2 in ex-vivo tissue. Rows show GC on exA, PA on exB and LC on exC, from top to bottom. Each sequence progresses from a global view through search to target acquisition. Figure 10 reports Find and Center outcomes across all standard trials.
Fig. 9: Selected Ours-B rollouts on R2 under instruction interventions. Rows show a wrong-region request on exA, unseen correct wording on exB and an empty instruction on exC. The wrong-region rollout bends toward the target-free requested region; the unseen-wording rollout reaches the target, and the empty-instruction rollout continues searching. Figure 10 reports the corresponding ten-trial group outcomes.
Fig. 10: Ours-B on R2: standard exA–C trials, instruction conditions and post-occlusion recovery. Each condition has ten trials; bars are 95% Wilson intervals. Find and Center follow the criteria in Supplementary Sec. S4. Standard exA and the separate full-instruction group each have Center 8/10. Each instruction condition is associated with the organ shown.
Fig. 11: Training-data comparison on R2 using organs exD–F. Ours-A uses 312 episodes and Ours-B uses 1,050 episodes; the other model, training and test conditions are unchanged. Each model has ten trials per organ. Bars are trial-level 95% Wilson intervals; observations are repeated within each organ.
Input (frames)
Find
Center for 2 s
C (1)
6/10 (60; 31.3–83.2)
5/10 (50; 23.7–76.3)
R+C (2)
7/10 (70; 39.7–89.2)
6/10 (60; 31.3–83.2)
G+C (2)
7/10 (70; 39.7–89.2)
7/10 (70; 39.7–89.2)
G+R+C (3)
9/10 (90; 59.6–98.2)
9/10 (90; 59.6–98.2)
TABLE III: Ours-B visual-context ablation. G/R/C: global/recent/current frames; omitted slots are masked. Entries: k/n (%; Wilson 95% interval).
Model
A (Hz)
H
B (Hz)
1/A (ms)
EndoVLA
1.480
1
1.480
675.7
OpenVLA-7B
5.228
1
5.228
191.3
π0
17.69
20
353.8
56.5
π0.5
15.68
20
313.6
63.8
Ours
14.27
20
285.4
70.1
TABLE IV: 300-input timing benchmark. B is the theoretical maximum chunk-equivalent control rate; physical motor dispatch is 20 Hz.
Fig. 12: Selected FT failures on R1 involving (a) a target-like water droplet, (b) a bubble, (c) search in an incorrect local region, and (d) incomplete reacquisition after partial approach. The examples illustrate visual discrimination and search-recovery challenges in the evaluated trajectories.
Figure S1: Robot platforms and ex-vivo evaluation settings. R1 (left) provides demonstrations and baseline/environment-transfer trials. R2 (middle) is used for the exA–exF experiments (right).
Figure S2: YOLO Studio annotation interface. Editable target boxes and an explicit No Detection control support frame-level verification of target visibility and position.
Figure S3: Ours-A action changes under trained-instruction substitution on EA . Each cell is the mean chunk distance in Eq. ( S2 ). Rows identify recorded tasks and columns identify replacement instructions. The diagonal is the unchanged-instruction reference.
Figure S4: Ours-A action sensitivity to alternative instruction strings on EA . Each bar is the mean action-chunk L2 distance from the reference prediction under one shared noise realization.
Figure S5: Motor-direction agreement over 1-, 2- and 20-step prediction prefixes. Each rate averages the per-anchor sign-agreement fraction on eligible demonstrated motor components.
Figure S6: Centering experiment on R1 under (a) high-light dry, (b) high-light wet, (c) low-light dry and (d) low-light wet conditions. Box plots summarize the trial-level ΔPE index. The Ours label denotes Ours-A.
Figure S7: Successful Ours-A finding-and-tracking trajectories on R1 under (a) HD, (b) HW, (c) LD and (d) LW. Image sequences show search and approach; motor-position and pixel-error traces show the progression from finding to tracking.
The Chinese University of Hong Kong, Hong Kong SAR, China. · The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China. · University of Cambridge, Cambridge, United Kingdom. +2