EndoWorld-JEPA: Motion-Guided Global-Local Latent World Modeling for Endoscopic Video Representation Learning

Jie Wang; Shilong Yang; Tao Zhou; Juan Yu; Rong Qu; Jianfeng Ren; Zhicheng Zhang; Linlin Shen

cs.CVOct 20, 2026

EndoWorld-JEPA: Motion-Guided Global-Local Latent World Modeling for Endoscopic Video Representation Learning

Authors: Jie Wang, Shilong Yang, Tao Zhou, Juan Yu, Rong Qu, Jianfeng Ren, Zhicheng Zhang, Linlin Shen

Organizations: University of Nottingham · Shenzhen University · Tsinghua University · The First Affiliated Hospital of Shenzhen University ·

Abstract

Endoscopic video contains state transitions induced by camera displacement, deformable tissue, and instrument interaction, yet existing pre-training methods treat dynamics implicitly. Inspired by action-conditioned joint-embedding predictive world models and by the observation that endoscopic optical flow contains both shared image-plane displacement and localized residual motion, we reformulate passive endoscopic video pre-training as causal motion-guided latent-state prediction. We present EndoWorld-JEPA, to our knowledge the first latent world-model pre-training framework designed for general-purpose endoscopic video representation learning. In place of recorded actions, calibrated poses, or robot kinematics, the model conditions prediction of the next latent state on motion observed up to the current state. Its Optical Motion-guided module separates dominant displacement from residual motion and converts the resulting evidence into an ordered temporal motion condition and spatial local-motion weights. Complementary global and motion-local predictors use these outputs: the temporal condition modulates both latent transitions, whereas the local weights define an auxiliary motion-local state that extends prediction from scene-level evolution to spatially selective dynamics. Patch-wise motion weighting emphasizes localized change while retaining anatomical context, and past-motion-window dropout randomly withholds the complete motion condition to reduce dependence on unreliable motion. This global--local formulation learns predictable visual--motion dynamics in representation space without reconstructing stochastic appearance artifacts. Evaluation shows that a common pre-trained checkpoint achieves strong transfer across four downstream benchmarks, supporting motion-guided global--local latent prediction as an effective pre-training paradigm for endoscopic video.

Figures & tables

Figure 1: Comparison of three pre-training paradigms and their treatment of local dynamics. (A) Existing endoscopic video pre-training captures temporal context, but transitions remain implicit: invariance may attenuate change, masking models transitions indirectly, and observed motion does not condition prediction. (B) Medical world models rely on domain-specific data and transition conditions unavailable in passive endoscopy. (C) EndoWorld-JEPA couples an Optical Motion-guided (OM) block that derives motion guidance from unlabeled passive video with a causal global–local JEPA, requiring no recorded controls or calibrated poses. Bottom: a schematic comparison illustrates that existing video pre-training models localized dynamics only implicitly within global latent features, whereas the OM block produces an explicit local-motion map whose patch weighting emphasizes motion-relevant regions.

Figure 2: Overview of EndoWorld-JEPA, a causal motion-conditioned JEPA world model. Top: for the implemented $K=5$ horizon, the OM block converts six consecutive observations ending at $x_{s_{t}}$ into five incoming optical flows, motion descriptors, a local-motion weight map, and a motion condition. Middle: the global–local JEPA block uses this five-transition motion history with the current state $x_{s_{t}}$ to predict local and global latent targets at $x_{s_{t}+5}$ . Bottom: the pretrained encoder and OM block are transferred with task-specific decoders for four downstream tasks.

Figure 3: Representative OM responses to a whole-view shift (top), a local-region change (middle), and their combination (bottom). Columns show preceding and current frames, dense flow $\mathbf{u}_{s_{t}}$ , residual magnitude $R_{s_{t}}$ , local map $\mathbf{M}_{t}$ , relative weights $Nw_{t,i}$ , and the weight overlay. Dense-flow hue follows the direction key, while intensity denotes displacement magnitude. Row annotations report $\|\mathbf{d}_{s_{t}}\|_{2}$ and $P_{90}(R_{s_{t}})$ . Residual magnitude and $\mathbf{M}_{t}$ share a within-row display range. Here, $Nw_{t,i}=1$ denotes uniform pooling; weight maps and overlays share the clipped $0.3$ – $7.0$ scale. Display scaling does not affect OM computation.

Method	Video Classification F1 (%)	Polyp Segmentation Dice (%)	Polyp Detection F1 (%)	Surgical Phase Recognition Accuracy (%)
Scratch	$83.5\pm 1.3$	$53.2\pm 3.2$	$73.5\pm 4.3$	$80.3\pm 8.7$
TimeSformer [ 24 ]	$84.2\pm 0.8$	$56.3\pm 1.5$	$75.8\pm 2.1$	$81.2\pm 8.9$
CORP [ 25 ]	$87.1\pm 0.6$	$68.4\pm 1.1$	$78.2\pm 1.4$	$82.6\pm 9.6$
FAME [ 26 ]	$85.4\pm 0.8$	$67.2\pm 1.3$	$76.9\pm 1.2$	$81.9\pm 9.2$
ProViCo [ 27 ]	$86.9\pm 0.5$	$69.0\pm 1.5$	$78.6\pm 1.7$	$82.3\pm 8.5$
VCL [ 28 ]	$87.6\pm 0.6$	$69.1\pm 1.2$	$78.1\pm 1.9$	$82.7\pm 9.4$

Table 1: Comparison across four downstream endoscopic tasks under the corresponding EndoFM-LV evaluation protocols. Values are mean

\pm

standard deviation; best results are shown in bold.

Variant	G	TC	ML	MP	WD	Classification F1 (%)	Segmentation Dice (%)	Detection F1 (%)	Phase Recognition Accuracy (%)
Global prediction baseline	$\checkmark$	–	–	–	–	$91.0\pm 0.9$	$73.7\pm 2.0$	$83.9\pm 1.5$	$83.0\pm 6.8$
$+$ temporal conditioning	$\checkmark$	$\checkmark$	–	–	–	$95.0\pm 0.7$	$79.3\pm 1.5$	$89.0\pm 1.4$	$86.0\pm 6.3$
$+$ temporal conditioning + dropout	$\checkmark$	$\checkmark$	–	–	$\checkmark$	$95.7\pm 0.3$	$80.2\pm 1.2$	$89.8\pm 1.2$	$86.7\pm 6.4$
$+$ motion-local prediction	$\checkmark$	–	$\checkmark$	–	–	$93.4\pm 0.6$	$81.1\pm 1.3$	$89.6\pm 1.3$	$85.0\pm 6.7$
$+$ motion-local + mixed pooling	$\checkmark$	–	$\checkmark$	$\checkmark$	–	$94.2\pm 0.5$	$82.7\pm 1.2$	$90.9\pm 1.1$	$85.7\pm 7.0$
$+$ temporal conditioning + motion-local	$\checkmark$	$\checkmark$	$\checkmark$	–	–	$97.0\pm 0.6$	$84.9\pm 1.2$	$93.4\pm 1.2$	$88.1\pm 7.1$

Table 2: Dependency-aware component ablation and motion-design controls. Values are shown as mean

\pm

standard deviation over three runs using seeds 0, 1, and 2. G: global prediction; TC: OM temporal conditioning; ML: motion-local prediction; MP: motion–uniform mixed pooling within patch-wise motion weighting; WD: past-motion-window dropout.

Video Representation Learning

Video representation learning develops features that capture appearance, motion, and temporal structure for recognition, prediction, and control.

See all 143 papers

EndoWAM: A Grounded World-Action Model for Generalizable Endoscopic Navigation
Aug 2, 2026Jinsong Lin, Zikang Pan, Wanhao Liu +10World Models for Robotics Robotic Endoscopy
CUHK · SLAI · WFU +3
What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction
Jun 5, 2026Jewon Yeom, Hanseul Kim, Jeongjae Park +3Video Representation Learning Predictive Representation Learning
Graduate School of Data Science, Seoul National University
MORE: Learning Motion-Oriented Registers for Driving World Models and Planning
Oct 10, 2026Yingdong Gu, Haoyu Wang, Shaocheng Yan +9World Models for Autonomous Driving Video Representation Learning
Wuhan University · CowaRobot Co. Ltd. · University of Zaragoza

Aug 2, 2026J/K move · Enter open · S save

EndoWAM: A Grounded World-Action Model for Generalizable Endoscopic Navigation

Jinsong Lin, Zikang Pan, Wanhao Liu, Chi Kit Ng+9

CUHK · SLAI · WFU+3

World Models for Robotics Robotic Endoscopy World Action Models Surgical Robotics

Open paper AskPDF arXiv

Autonomous endoscopic navigation can reduce clinicians' operational burden, yet robust control remains challenging due to tissue deformation, transient occlusions, and rapidly changing viewpoints. Existing learning-based policies typically predict actions from current observations without explicitly modeling future dynamics, limiting their robustness and reliability in safety-critical settings. World Action Models (WAMs) offer a promising alternative by coupling predictive visual dynamics with action generation, but extending them to robotic endoscopy remains challenging due to limited training data, restricted viewpoint diversity, deformable anatomy, and high inference latency. We present EndoWAM, which is, to our knowledge, the first WAM for generalizable robotic endoscopic navigation. EndoWAM introduces future grounding, which predicts task-relevant target regions in future observations from intermediate denoising features of a video world model. Specifically, EndoWAM couples a lightweight diffusion transformer for future target-region prediction with a discrete action expert through a shared predictive representation. This design injects target-aware supervision into predictive dynamics modeling, improving robustness to visual degradation and viewpoint changes while enabling real-time control in a single denoising pass. We further introduce EndoMotion, a robotic endoscopic motion dataset spanning three anatomically distinct procedures: ureteroscopy, esophagoscopy, and endoscopic retrograde cholangiopancreatography (ERCP). EndoWAM consistently outperforms all baselines and alternative grounding strategies, while demonstrating strong zero-shot generalization to unseen viewpoints, environments, and targets. These results establish EndoWAM as a predictive, target-grounded framework for accurate, generalizable, and long-horizon navigation in visually constrained endoscopic environments.

Abstract

Video Representation Learning

Explore similar work

EndoWAM: A Grounded World-Action Model for Generalizable Endoscopic Navigation

What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction

MORE: Learning Motion-Oriented Registers for Driving World Models and Planning