cs.ROOct 22, 2026
SaveEBV-CAN: Vision--Language Grounding for Contact-Aided Endoscope Navigation
Organizations: The Chinese University of Hong Kong. · National University of Singapore.
Abstract
Flexible endoscopes exploit structural compliance to traverse narrow lumens, pipes, and cavities, where navigation involves distributed environmental contact. This compliance makes tip motion depend on the evolving support configuration. We present external bird's-eye-view contact-aided navigation (EBV-CAN), a vision-language policy that combines one initial global static bird's-eye image with evolving endoscopic observations and motor state. Three supervised semantic queries locate the tip, next waypoint, and instructed target on the cached global image; their residual representations predict bending and insertion commands. We introduce the Vision-Language Contact-Aided (VLCA) dataset, linking 517 demonstrations and 75k action samples to 958k spatial entries and 93k remaining-route records. On 50 held-out episodes, tip-coordinate and waypoint localization reach 97.5% and 94.8%. Swapping the target instruction under identical observations changes target attention and motor output across 17 dual-target episodes. Across 1,920 physical trials of eight policies trained under a shared protocol, EBV-CAN achieves 183/240 successes (76.25%) under automatic blue-area scoring. Gated recurrent unit behavior cloning (GRU-BC) achieves the lowest matched-task offline mean absolute error (MAE), 0.056, and 1/240 physical successes, establishing distinct rankings for action imitation and task completion. Successful multi-guide executions and representative failures connect spatial progress, steering transitions, and visibility to physical navigation.
Figures & tables
Fig. 1: Contact-aided navigation mechanism. A guide supports the long passive shaft, allowing insertion and distal bending to redirect the endoscope toward a target beyond the illustrated free-space bending envelope. The envelope is a qualitative schematic.
Fig. 2: Physical endoscope platform. A motorized bending knob controls the distal segment, and counter-rotating rollers control insertion. The overhead camera records the global scene; the illuminated tip camera supplies the EV. PCB denotes the printed circuit board.
Fig. 3: Complementary uses of environmental support. (a) Contact-aided navigation redirects the bending section toward the target. (b) Contact-aided insertion constrains shaft progression through a guide. The workspace envelopes and paths illustrate the mechanisms qualitatively.
| Level | Guides | Target / variant | Episodes | Frames |
|---|---|---|---|---|
| L0 | 0 | Star | 81 | 10,080 |
| L1 | 1 | Star | 44 | 7,613 |
| L1.5 | 1 | Star / I | 111 | 15,538 |
| L2 | 2 | Star | 50 | 9,837 |
| L2.5 | 2 | Star / I | 19 | 2,889 |
| L3 | 3 | Star | 15 | 3,295 |
TABLE I: VLCA policy corpus. Guides are those in the annotated plan; frames are retained action samples. I indicates an insertion-dependent task variant.
| Entity | Count | Supervision role |
|---|---|---|
| Guide/object boxes | 770,788 | Scene geometry |
| Target boxes | 93,415 | Goal grounding |
| Robot-tip entries | 93,413 | Localization |
| Canonical reference boxes | 4,041 | Guide identity |
| Ordered episode plans | 518 | Route order |
| Remaining-route records | 93,085 | Next waypoint |
TABLE II: Linked VLCA annotation inventory. Counts refer to stored entities; tip points and tip entries share underlying annotations.
Fig. 4: Implemented EBV-CAN architecture. (a) Initial-EBV patch features are cached, projected, and combined with fixed two-dimensional positional encodings. (b) EV history, the repeated current-state embedding, and current-image CNN features produce ; the text encoder produces . (c) Both condition all three learned queries. Two residual cross-attention blocks read the fixed memory. (d) The action head concatenates with all three final query tokens and predicts six two-motor commands; only the first is executed. The auxiliary coordinate head uses only the tip token. C denotes concatenation; CLS denotes the class token. Q, K, and V denote query, key, and value; WP and Tgt denote waypoint and target. Input thumbnails are representative.
Fig. 5: Dynamic predictions on a static map in held-out episode L3/0004. Columns show three retained observation indices. Top: tip attention and a cyan cross at the annotated tip. Middle: waypoint attention and its green dashed reference box. Bottom: current EV. Every attention background uses the same initial EBV. Each heatmap is normalized by its own peak for visual comparison of spatial location. Maps are model outputs over original recorded images.
Fig. 6: Matched-observation instruction intervention. (a,b) Target attention for star and circle instructions on the same L4/0013 test observation; dashed boxes mark the two references, one template-derived. Heatmaps are peak-normalized separately. (c) Mean first-action difference for each dual-target test episode (L4 in teal, L4.5 in navy). The dashed line is the equally weighted episode mean; is the normalized first-action difference defined in the text. The measured endpoint is motor-output sensitivity to the instruction.
Fig. 7: Complementary evaluation endpoints for the selected models. (a) First-action MAE over all 7,279 test samples; copying the preceding expert action is a teacher-forced reference. (b) Physical completion for the same model order. (c) EBV-CAN success by task level, with nominal 95% Wilson intervals over 30 trials per level.
Fig. 8: Spatial analysis of all 7,279 held-out samples. (a) Empirical cumulative distributions of normalized tip error; the dashed line marks the 0.05 accuracy threshold. (b) Frame-weighted waypoint accuracy for guide versus final-target approach, and for samples near versus away from annotated route changes. Each pair partitions the test set. Near-switch windows comprise five samples before and five starting at each of 46 changes.
Fig. 9: Successful EBV-CAN physical executions in demanding task families. (a) L3.5 multi-guide insertion. (b,c) L4 executions instructed to reach the star and circle, respectively. Each row shows the fixed initial global image followed by four original EV observations. Times are relative to the first recorded frame; EV crops emphasize task geometry while retaining the endpoint region. The final images show approach to the blue endpoint region. These selected examples illustrate progression; Table V reports the complete outcome counts.
Fig. 10: Representative L1 failures from original endoscopic recordings, uniformly cropped to emphasize task geometry. Each pair shows an earlier and later observation; time is relative to the first recorded frame. (a) GRU-BC, view occlusion. (b) ACT-style, late steering. (c) Qwen3-VL with a chunk MLP, reverse progression. (d) Frozen GR00T, insufficient steering. (e) GR00T with LoRA, loss of target view. (f) EBV-CAN, stalled progression. Descriptions follow the recorded case notes; images document the visible outcome. SI Sec. S8 analyzes representative failures from every policy.
| Input condition | Tip acc. (%) | WP acc. (%) | MAE |
|---|---|---|---|
| Recorded inputs | 95.78 | 94.78 | 0.094 |
| Repeat current EV feature | 92.79 | 94.23 | 0.104 |
| Freeze EV at first sample | 22.01 | 43.78 | 0.186 |
| Permute global patches | 29.69 | 3.04 | 0.101 |
TABLE III: Inference-only interventions on all 7,279 test samples. Accuracy is abbreviated acc.; WP denotes waypoint. Weights remain fixed, isolating inference-time input dependence. Bold indicates highest accuracy or lowest MAE.
| Policy | Test MAE | Selected epoch | Trainable (M) | Deployed (M) | |
| Zero-action reference | 1 | 0.480 | — | — | — |
| Copy preceding expert action | 1 | 0.051 | — | — | — |
| GRU-BC | 1 | 0.057 | 12 | 2.78 | 2.8 |
| ACT-style | 16 | 0.074 | 20 | 9.47 | 9.47 |
| Diffusion-MLP | 1 | 0.130 | 2 | 3.25 | 3.25 |
| Qwen3-VL + MLP | 1 | 0.104 | 14 | 9.84 | 4,447.7 |
TABLE IV: Held-out first-action MAE for the selected models over 7,279 samples. All models share a 30-epoch cap and validation-MAE selector. Selected epochs start at one; is the action horizon. Parameter counts are in millions; LoRA deployment uses merged weights. Bold marks the lowest comparable error or parameter count within each block. SI Sec. S7 gives actual training budgets, raw errors, and curves.
| Physical success (automatic blue-area criterion) | ||||||||||
| Policy | Offline MAE | L0 | L1 | L1.5 | L2 | L2.5 | L3 | L3.5 | L4 | All |
| GRU-BC | 0.056 | 1/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 1/240 (0.42%) |
| ACT-style | 0.073 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/240 (0.00%) |
| Diffusion-MLP | 0.129 | 1/30 | 0/30 | 0/30 | 0/30 | 0/30 | 1/30 | 0/30 | 0/30 | 2/240 (0.83%) |
| Qwen3-VL + MLP | 0.102 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/240 (0.00%) |
| Qwen3-VL + chunk MLP | 0.148 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/30 | 0/240 (0.00%) |
TABLE V: Physical success and matched-task offline MAE for the selected checkpoints in SI Sec. S7. MAE uses 6,271 samples across the eight physical levels; Table IV additionally includes L4.5. Every cell includes 30 trials under the automatic blue-area criterion. Bold marks the lowest MAE and highest success count in each column. GR00T denotes the StarVLA/GR00T implementations.
Fig. S1: Original initial EBV examples for all offline task families. Green boxes identify guides appearing in each episode’s plan; red identifies the instructed target. Each level identifies a family of route, insertion, and target-selection requirements.
| Level | Training | Validation | Test | Total |
|---|---|---|---|---|
| L0 | 65 / 8,451 | 8 / 749 | 8 / 880 | 81 / 10,080 |
| L1 | 36 / 6,312 | 4 / 588 | 4 / 713 | 44 / 7,613 |
| L1.5 | 89 / 12,815 | 11 / 1,284 | 11 / 1,439 | 111 / 15,538 |
| L2 | 40 / 7,687 | 5 / 1,066 | 5 / 1,084 | 50 / 9,837 |
| L2.5 | 15 / 2,268 | 2 / 231 | 2 / 390 | 19 / 2,889 |
| L3 | 11 / 2,361 | 2 / 425 | 2 / 509 | 15 / 3,295 |
TABLE S1: Exact policy partition sizes, reported as episodes / action samples.
Fig. S2: VLCA composition. (a) Episode partitions by task family. (b) Retained action samples. (c) Spatial annotation entries across the raw annotation export; tip annotations are counted once, with the coordinate field treated as an alternative representation. (d) Retained episode lengths, showing medians, quartiles, and whiskers based on 1.5 times the interquartile range (IQR), with outliers omitted from the plot.
Fig. S3: The proposed VLCA YOLO-backboned annotation platform and the user interface. The central view links tip, object, target, and waypoint annotations; side panels expose editing, propagation, and JSON export. The interface illustrates the annotation workflow and stored field relationships; VLM denotes vision-language model.
| Subset | Episodes | Frames | Critical frames |
|---|---|---|---|
| SFT training | 300 | 300 | 0 |
| RFT training | 415 | 73,169 | 15,770 |
| Evaluation | 103 | 19,015 | 3,915 |
TABLE S2: Structured-language annotation exports for 518 recordings. Counts apply separately to both plan and wild formats. The SFT subset draws from the 415 training episodes.
| Component | Setting |
|---|---|
| EBV input / patch grid | / |
| EV feature input | |
| EV window / action horizon | 8 / 6 retained samples |
| Current-EV CNN input | |
| Motor state / embedding | 4 / 64 dimensions |
| Navigation GRU | width 256; reset each window |
TABLE S3: EBV-CAN architecture and training settings. Spatial weights refer to tip, waypoint (wp), and target (tgt). The shared training protocol is detailed in Sec. S7.
| Level | Samples | Tip-peak acc. | Waypoint acc. | Norm. MAE | Insertion MAE | Bending MAE |
|---|---|---|---|---|---|---|
| L0 | 880 | 93.18 | 100.00 | 0.077 | 1.605 | 11.158 |
| L1 | 713 | 88.08 | 99.30 | 0.081 | 1.788 | 10.985 |
| L1.5 | 1,439 | 98.33 | 98.75 | 0.084 | 1.596 | 13.370 |
| L2 | 1,084 | 97.32 | 86.44 | 0.075 | 1.443 | 11.739 |
| L2.5 | 390 | 100.00 | 96.92 | 0.090 | 1.815 | 13.391 |
| L3 | 509 | 97.45 | 88.21 | 0.078 | 1.648 | 10.989 |
TABLE S4: Selected-model results by held-out task family. Accuracy (acc.) is in percent; normalized (norm.) MAE averages both channels; raw action errors use controller units.
Fig. S4: Recorded overhead frames from demonstration L3/0004 showing approach and guide-supported shaft redirection. These original images provide qualitative mechanical context. The sequence comes from the demonstration annotation stream; attention plots use the fixed initial image.
| Level | Positive / total | Rate (%) | Interval (%) |
|---|---|---|---|
| L0 | 29/30 | 96.7 | 83.3–99.4 |
| L1 | 29/30 | 96.7 | 83.3–99.4 |
| L1.5 | 24/30 | 80.0 | 62.7–90.5 |
| L2 | 23/30 | 76.7 | 59.1–88.2 |
| L2.5 | 24/30 | 80.0 | 62.7–90.5 |
| L3 | 19/30 | 63.3 | 45.5–77.9 |
TABLE S5: Automatic blue-area physical outcomes for EBV-CAN and nominal Wilson 95% intervals. Bold marks the highest observed rate across levels.
| Policy | Levels with 0/30 | Representative recorded behaviors |
|---|---|---|
| GRU-BC | L1, L1.5, L2, L2.5, L3, L3.5, L4 | Occlusion, incorrect steering, pass-by, target-view loss |
| ACT-style | L0, L1, L1.5, L2, L2.5, L3, L3.5, L4 | Pass-by, incorrect steering, early/late steering |
| Diffusion-MLP | L1, L1.5, L2, L2.5, L3.5, L4 | Incorrect steering, early steering |
| Qwen3-VL + MLP | L0, L1, L1.5, L2, L2.5, L3, L3.5, L4 | Target-view loss, early/incorrect steering |
| Qwen3-VL + chunk MLP | L0, L1, L1.5, L2, L2.5, L3, L3.5, L4 | Reverse progression, target-view loss |
| GR00T, LoRA | L0, L2, L2.5, L3, L3.5 | Incorrect/early/insufficient steering, target-view loss |
TABLE S6: All zero-success cells and relevant L1 case evidence. The listed levels identify observed 0/30 outcomes; behavioral examples originate from L1.
| Level | GRU | ACT | Diffusion | Qwen MLP | Qwen chunk | GR00T F | GR00T L | EBV-CAN | |
|---|---|---|---|---|---|---|---|---|---|
| L0 | 880 | 0.046 | 0.061 | 0.123 | 0.090 | 0.142 | 0.048 | 0.052 | 0.077 |
| L1 | 713 | 0.052 | 0.062 | 0.124 | 0.082 | 0.132 | 0.053 | 0.055 | 0.081 |
| L1.5 | 1439 | 0.049 | 0.066 | 0.122 | 0.104 | 0.140 | 0.056 | 0.054 | 0.084 |
| L2 | 1084 | 0.044 | 0.056 | 0.121 | 0.085 | 0.137 | 0.048 | 0.050 | 0.075 |
| L2.5 | 390 | 0.060 | 0.081 | 0.127 | 0.106 | 0.148 | 0.062 | 0.061 | 0.090 |
| L3 | 509 | 0.046 | 0.069 | 0.118 | 0.082 | 0.129 | 0.051 | 0.048 | 0.078 |
TABLE S7: Selected-checkpoint offline MAE by physical task level. GR00T F/L denote frozen/LoRA variants. Bold marks the lowest MAE per row. Physical outcomes for the same weights appear in main-paper Table V.
| Policy | Epochs trained / selected | Updates | Final train loss | Validation MAE | Test MAE | Insertion MAE | Bending MAE |
|---|---|---|---|---|---|---|---|
| GRU-BC | 17 / 12 | 16,150 | 0.007 | 0.056 | 0.057 | 1.133 | 8.640 |
| ACT-style | 25 / 20 | 23,750 | 0.045 | 0.069 | 0.074 | 1.553 | 10.579 |
| Diffusion-MLP | 7 / 2 | 6,650 | 0.125 | 0.128 | 0.130 | 2.491 | 20.354 |
| Qwen3-VL + MLP | 19 / 14 | 18,050 | 0.018 | 0.100 | 0.104 | 2.555 | 12.082 |
| Qwen3-VL + chunk MLP | 30 / 25 | 28,500 | 0.047 | 0.148 | 0.151 | 3.236 | 21.077 |
| GR00T, frozen | 29 / 24 | 27,550 | 0.025 | 0.058 | 0.063 | 1.355 | 8.623 |
TABLE S8: Training with a common optimization cap and validation selector. Epochs start at one. Validation uses 6,614 rows and test uses 7,279 rows; raw MAEs use controller units. Loss scales are model-specific. Bold marks the lowest validation or test error in its column.
Fig. S5: Optimization trajectories under the shared protocol. Each panel shows the epoch-mean training objective and complete-validation first-action MAE on separately labeled axes. The marked validation point identifies the stored best checkpoint. Each epoch contains 950 optimizer updates and 60,800 optimization-row exposures. Objective scales follow each model’s learning formulation.
Fig. S6: Original EV sequences from selected L1 failures of all eight policies. Frame times are relative to each case’s first recorded timestamp. Identical margins are cropped from each image to emphasize task geometry; the source images remain unchanged. Behavioral labels follow the recorded notes and aligned command evidence.