Routine gastrointestinal endoscopy is intrinsically bidirectional: the instrument is advanced to reach target anatomy and later withdrawn or retroflexed for inspection, while an external cue may require earlier reversal. When the requested phase changes before the visual scene does, nearly identical observations can require opposite axial actions. We identify and formalize this ambiguity in bidirectional endoscopic control as intent aliasing. We propose EndoLIFT (Endoscopic Language-Instruction Flow with Trajectory Latents), a vision-language-action policy that combines explicit language-based intent conditioning with a latent-conditioned rectified-flow action expert. The policy receives RGB, a language instruction, and the previous-action state; a 32-D variational trajectory latent stochastically conditions continuous action-chunk generation. Controlled same-observation instruction swaps establish that language selects the axial mode, independently of whether the trajectory latent is present. Relative to the matched model without latent conditioning, EndoLIFT improves navigation-direction accuracy by 11.1 percentage points and reduces wrong-direction advance by 83%. An architecture-controlled 1-bit mode-flag reference exhibits weaker canonical-anchor switching, while EndoLIFT retains 82.8% intent-following accuracy across 44 held-out linguistic variants. In closed-loop evaluation, EndoLIFT improves overall success by 30 percentage points over EndoLIFT w/o VTL on both the seen colon phantom and the unseen lung and stomach phantoms, and completes 10/10 ex-vivo porcine-trachea trials. These results separate language-based intent selection from the trajectory latent's contribution to directional correctness and robust retraction.
Figures & tables
Fig. 1: Motivation and system concept. (a) Intent aliasing : visually similar endoscopic observations can require opposite axial actions, so pixels alone cannot select the active procedural mode. (b) An external monitor–speech supervisor selects a canonical forward or retraction instruction for one shared VLA policy, which maps the image, instruction, and previous action to three-axis endoscope motion.
Fig. 2: Robotic platform and evaluated domains. (a) The setup combines a motorized feeding unit for longitudinal advance/retraction with two-axis distal bending, matching the policy’s three action channels. (b) Bidirectional trials use the colon phantom as the seen domain and lung and stomach phantoms as unseen domains; stars mark the prescribed start and end regions, with lung trials terminating at any target distal branch endpoint.
Fig. 3: EndoLIFT model architecture. The RGB frame and canonical instruction are encoded by a LoRA-adapted PaliGemma 2 into prefix Pt . The previous executed action st , noisy chunk xτ , flow time τ , and trajectory latent z condition an eight-block Transformer action expert, whose velocity field is integrated for 20 Euler steps to yield a normalized 32×3 action chunk. The dashed upper branch is used only in training: a posterior pools Pt , st , and the target chunk At to parameterize qϕ(z∣Pt,st,At) ; deployment instead samples z∼N(0,I) .
Base-table instruction
Mode
Frames
failed recovery navigation
forward
8,344
normal navigation
forward
6,850
successful recovery navigation
forward
1,278
urgent retraction
retract
28,470
Total
44,942
TABLE I: Instruction composition of the base imitation dataset. The four surface forms correspond to two axial control modes; the canonical deployment instructions are defined in the text.
Fig. 4: Architecture-controlled ModeFlag reference comparison. (a) Mean normalized longitudinal output for EndoLIFT’s single canonical anchor (horizontal) and ModeFlag-LCRF’s literal bit (vertical) on the same 40 frames and noise seeds. (b) Additional held-out colon diagnostic on 4,884 frames from four episodes; bars show 95% bootstrap intervals. Overall accuracy, moving-only accuracy, retraction F1, and wrong-direction advance are reported together because hold steps are prevalent.
Condition interface
IFA ↑
Canonical paired flip ↑
Separation ↑
Literal 1-bit flag (ModeFlag-LCRF)
68.8
37.5
0.0166
Canonical text (EndoLIFT)
85.0
70.0
0.0476
TABLE II: Architecture-controlled ModeFlag reference versus canonical-text intervention on 40 frozen frames. Image, previous action, and per-frame flow-noise seed are identical. IFA and canonical paired flip are percentages; separation is normalized longitudinal action.
Method
Seen canon.
EN para.
ZH para.
Oral
ASR/noise
Distr.
All unseen
Multi-instr. flip
Qwen3-VL + MLP
57.5
60.0
60.6
57.5
59.5
61.3
59.7
20.0
Qwen3-VL + GR00T
74.2
62.9
62.2
66.6
59.5
65.8
63.1
30.0
EndoLIFT w/o VTL
90.8
79.0
79.1
77.8
75.5
80.4
78.2
55.0
Keyword router + canonical policy
85.8
50.0
50.0
50.0
57.0
61.7
53.2
5.0
TF–IDF router + canonical policy
85.8
50.0
50.0
67.5
57.0
50.0
54.8
7.5
ModeFlag-LCRF (literal bit; no text)
68.8
–
–
–
–
–
–
–
TABLE III: Instruction-following accuracy (IFA, %) on the frozen 40-frame pool. “All unseen” aggregates 44 held-out commands. Multi-instruction flip is computed over the multi-string instruction pool used in this table; it is distinct from the two-anchor canonical paired flip in Table II and the test-set instruction-swap flip in Table IV . The ModeFlag row uses literal bits and therefore has no language-category or multi-instruction value (“–” denotes N/A). The EndoLIFT (ours) canonical cell averages three seen canonical strings.
Method
Shift [95% CI]
CSF
Test-swap flip
Qwen3-VL + MLP
1.615 [1.399, 2.683]
0.979
0.376
Qwen3-VL + GR00T
1.587 [1.234, 4.645]
0.613
0.248
EndoLIFT w/o VTL
6.014 [5.762, 7.983]
0.856
0.644
EndoLIFT (ours)
3.484 [3.234, 5.503]
0.681
0.452
TABLE IV: Test-set instruction-swap diagnostic with image and previous-action state fixed. Shift is the axial-output change [95% CI] in raw motor units; CSF is correct-sign fraction and test-swap flip is the test-set instruction-swap flip rate. These sensitivity diagnostics are interpreted separately from task-performance metrics.
Fig. 5: Wrong-direction advance on held-out true-retraction targets under the correct instruction; lower is better. Points indicate the observed rates, and error bars denote 95% confidence intervals. EndoLIFT w/o VTL is the matched ablation without trajectory-latent conditioning.
Method
Nav. direction ↑
Wrong-direction advance ↓
Δ Nav. vs. matched
Error reduction vs. matched
DINOv2 + MLP
0.395 [0.328, 0.580]
0.064 [0.037, 0.557]
–
–
Qwen3-VL + MLP
0.392 [0.324, 0.576]
0.046 [0.033, 0.307]
–
–
Qwen3-VL + GR00T
0.351 [0.288, 0.535]
0.204 [0.181, 0.453]
–
–
EndoLIFT w/o VTL
0.453 [0.412, 0.589]
0.059 [0.036, 0.067]
reference
reference
EndoLIFT (ours)
0.565 [0.534, 0.668]
0.010 [0.005, 0.021]
+11.1 pp
83%
TABLE V: Directional performance on the held-out test set. Navigation direction accuracy and wrong-direction advance are reported with 95% episode-bootstrap confidence intervals. The last two columns report the difference between EndoLIFT (ours) and EndoLIFT w/o VTL.
Fig. 6: Frozen-test action-chunk comparison for (a) Qwen3-VL + MLP, (b) Qwen3-VL + GR00T, (c) EndoLIFT w/o VTL, and (d) EndoLIFT (ours). Within each panel, the top and bottom rows show forward navigation and urgent retraction, respectively; the columns show axial movement, vertical bending, and horizontal bending. Blue solid curves are human targets, and orange dashed curves are open-loop predictions.
Fig. 7: Instruction-stratified validation transition structure for forward navigation (left) and urgent retraction (right), with the common bidirectional task illustrated centrally. Black arrows denote majority transitions, gray arrows all observed transitions, and dashed blue/orange arrows the monitor/speech interfaces.
Method
Nav.
Retr.
Overall
DINOv2 + MLP
0 [0,30.8]
0 [0,30.8]
0 [0,16.8]
Qwen3-VL + GR00T
0 [0,30.8]
0 [0,30.8]
0 [0,16.8]
Qwen3-VL + MLP
40 [12.2,73.8]
0 [0,30.8]
20 [5.7,43.7]
EndoLIFT w/o VTL
100 [69.2,100]
40 [12.2,73.8]
70 [45.7,88.1]
EndoLIFT (ours)
100 [69.2,100]
100 [69.2,100]
100 [83.2,100]
TABLE VI: Closed-loop pilot on the seen colon phantom. Entries are success rate in percent [exact 95% CI]; n=10 per mode and n=20 overall.
Metric
Result
Trial success
10/10; exact 95% CI [69.2, 100]%
Total duration / control steps
162.9 s / 3,257 steps
Human trigger → instruction switch
445±126 ms (350–700)
Instruction switch → first Retract
54±61 ms (23–224); ≈ 1 step at 20 Hz
Retract among post-trigger axial-dominant steps
1,127/1,127 = 100%
Post-trigger bending activity
5.2% (36.4% pre-trigger)
TABLE VII: Quantitative ex-vivo porcine-trachea validation of EndoLIFT over ten human-triggered retraction trials.
Method
Nav.-Lung SR [95% CI]
Nav.-Stomach SR [95% CI]
Ret.-Lung SR [95% CI]
Ret.-Stomach SR [95% CI]
Overall SR [95% CI]
DINOv2 + MLP
0 [0, 30.8]
0 [0, 30.8]
0 [0, 30.8]
0 [0, 30.8]
0 [0, 8.8]
Qwen3-VL + GR00T
0 [0, 30.8]
0 [0, 30.8]
0 [0, 30.8]
0 [0, 30.8]
0 [0, 8.8]
Qwen3-VL + MLP
80 [44.4, 97.5]
50 [18.7, 81.3]
0 [0, 30.8]
0 [0, 30.8]
32.5 [18.6, 49.1]
EndoLIFT w/o VTL
100 [69.2, 100]
100 [69.2, 100]
30 [6.7, 65.2]
50 [18.7, 81.3]
70 [53.5, 83.4]
EndoLIFT (ours)
100 [69.2, 100]
100 [69.2, 100]
100 [69.2, 100]
100 [69.2, 100]
100 [91.2, 100]
TABLE VIII: Closed-loop transfer pilot on the unseen lung and stomach phantoms ( n=10 per condition; n=40 overall). Each cell reports success rate in percent [exact 95% CI].
Fig. 8: Domain-stratified closed-loop action transitions for the unseen lung, seen colon, and unseen stomach phantoms (left to right). Edges aggregate 0.5-s executed-action windows; black arrows show majority transitions, gray arrows all observed transitions, and dashed arrows the monitor/speech interfaces.
Fig. 9: Closed-loop bidirectional behavior and transfer. (a) The robot advances until an external trigger switches the instruction, then retracts from a nearly unchanged observation. (b) Advance and retract sequences in the seen colon and unseen lung and stomach phantoms. (c) Trigger-centered rollouts for EndoLIFT w/o VTL and EndoLIFT; the former becomes stuck after initiating retraction, whereas the latter retracts smoothly. (d) Ex-vivo validation in a porcine trachea, including a representative trigger-centered sequence and the corresponding action-transition graph.
Endoscopic retrograde cholangiopancreatography (ERCP) demands precise endoscopic navigation and stable biliary cannulation within a narrow monocular field characterized by specular reflections, partial occlusions, and frequent tissue contact. Although recent robotic systems and vision-based assistance techniques improve operator ergonomics and provide perceptual cues, their performance degrades under pronounced anatomical variability and safety-critical visual artifacts, which hinders reliable autonomy in cannulation-grade procedures. Here, we present BiliVLA, a scene-aware Vision-Language-Action (VLA) framework that formulates biliary endoscopic navigation as an instruction-conditioned visuomotor learning problem. Given an endoscopic observation and a stage-specific language instruction, BiliVLA jointly predicts the target category, a grounded bounding box, and a discrete three-degree-of-freedom (3-DoF) motor command for a continuum endoscope. The proposed framework incorporates scene-aware supervision to improve semantic target consistency and safety-aware recovery supervision to induce conservative retreat behaviors under luminal wall contact. A key component of BiliVLA is a two-stage training paradigm that combines grounding-enhanced supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO), thereby improving action reliability and decision consistency during closed-loop navigation. Across three ERCP subtasks, BiliVLA achieves the best overall performance in physical phantom experiments, with a total mIoU of 0.9625, an overall action precision of 91.96%, and an overall success rate (SR) of 84.85%. These results indicate that integrating semantic grounding, scene-aware learning, and reward-guided optimization strengthens perception--action alignment and enables more robust autonomous biliary endoscopic navigation.
Jinsong Lin, Chi Kit Ng, Zhiyong Xiong +8
The Chinese University of Hong Kong, Hong Kong SAR, China. · The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China. · University of Cambridge, Cambridge, United Kingdom. +2
Autonomous endoscopic navigation requires the policy model to predict actions from texture-poor monocular observations, make safe control decisions, and retain evidence of lesions after they leave the field of view. Existing vision-language-action (VLA) models primarily rely on visual appearance and short-term context, limiting geometric grounding and episode-level reporting. We introduce StenoVLA-3D, a 3D-aware VLA framework for navigating through stenotic regions. We integrate point-maps into the Cosmos-Reason 2 backbone through learned geometry-gated fusion, and also propose a temporal state branch to model traversal progress. Our reasoning-and-action backbone predicts grounded reasoning with actions, while dedicated heads estimate stenosis shape and generate the final lesion report. We further introduce EndoCausal, an episode-level dataset with lesion annotations, actions, and temporally grounded reasoning. On 40 held-out recorded test episodes, StenoVLA-3D reaches 95.2% semantic accuracy and 83.4% action accuracy. On the physical 3-DoF endoscope, it attains 88.9% and 77.8% task success in esophageal and colonic phantoms (36 trials each), substantially outperforming the evaluated baselines.
We address language-conditioned robotic manipulation using flow-based trajectory generation, which enables training on human and web videos of object manipulation and requires only minimal embodiment-specific data. This task is challenging, as object trajectory generation from pre-manipulation images and natural language instructions requires appropriate instruction-flow alignment. To tackle this challenge, we propose the flow-based Language Instruction-guided open-Loop ACtion generator (LILAC). This flow-based Vision-Language-Action model (VLA) generates object-centric 2D optical flow from an RGB image and a natural language instruction, and converts the flow into a 6-DoF manipulator trajectory. LILAC incorporates two key components: Semantic Alignment Loss, which strengthens language conditioning to generate instruction-aligned optical flow, and Prompt-Conditioned Cross-Modal Adapter, which aligns learned visual prompts with image and text features to provide rich cues for flow generation. Experimentally, our method outperformed existing approaches in generated flow quality across multiple benchmarks. Furthermore, in physical object manipulation experiments using free-form instructions, LILAC demonstrated a superior task success rate compared to existing methods. The project page is available at https://lilac-75srg.kinsta.page/.