Organizations: Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education, Xiangtan University · School of Computer Science, Xiangtan University · Key Laboratory for Artificial Intelligence and International Communication, Hunan Normal University
Endoscopic video analysis is essential for early gastrointestinal screening but remains hindered by limited high-quality annotations. While self-supervised video pre-training shows promise, existing methods developed for natural videos prioritize dense spatio-temporal modeling and exhibit motion bias, overlooking the static, structured semantics critical to clinical decision-making. To address this challenge, we propose Focus-to-Perceive Representation Learning (FPRL), a cognition-inspired hierarchical framework that emulates clinical examination. FPRL first focuses on intra-frame lesion-centric regions to learn static semantics, and then perceives their evolution across frames to model contextual semantics. To achieve this, FPRL employs a hierarchical semantic modeling mechanism that explicitly distinguishes and collaboratively learns both types of semantics. Specifically, it begins by capturing static semantics via teacher-prior adaptive masking (TPAM) combined with multi-view sparse sampling. This approach mitigates redundant temporal dependencies and enables the model to concentrate on lesion-related local semantics. Following this, contextual semantics are derived through cross-view masked feature completion (CVMFC) and attention-guided temporal prediction (AGTP). These processes establish cross-view correspondences and effectively model structured inter-frame evolution, thereby reinforcing temporal semantic continuity while preserving global contextual integrity. Extensive experiments on 11 endoscopic video datasets show that FPRL achieves superior performance across diverse downstream tasks, demonstrating its effectiveness in endoscopic video representation learning. The code is available at https://github.com/MLMIP/FPRL.
Figures & tables
Figure 1: (a) Dense spatio-temporal modeling often captures irrelevant semantics due to motion reliance. (b) Our hierarchical approach isolates static from dynamic context, extracting appearance features required for diagnosis.
Figure 2: The pipeline of the proposed FPRL . For static semantic focus, we employ a teacher-prior adaptive masking strategy coupled with pixel reconstruction. For contextual semantic perception, we use cross-view masked feature completion with latent spatial alignment and an attention-guided temporal prediction module with contrastive learning.
Method
Venue
Year
Pretrain Time(h)
PolypDiag (Classification)
CVC-12k (Segmentation)
KUMC (Detection)
Scratch (Rand.init.)
-
-
N/A
83.5 ± 1.3
53.2 ± 3.2
73.5 ± 4.3
FAME [ 13 ]
CVPR
2022
48.9
85.4 ± 0.8
67.2 ± 1.3
76.9 ± 1.2
ProViCo [ 50 ]
CVPR
2022
71.2
86.9 ± 0.5
69.0 ± 1.5
78.6 ± 1.7
VCL [ 53 ]
ECCV
2022
74.9
87.6 ± 0.6
69.1 ± 1.2
78.1 ± 1.9
ST-Adapter [ 49 ]
NeurIPS
2022
8.1
84.8 ± 0.7
64.3 ± 1.9
74.9 ± 2.9
VideoMAE [ 63 ]
NeurIPS
2022
25.3
91.4 ± 0.8
80.9 ± 1.0
82.8 ± 1.9
Table 1: Comparison with other latest SOTA methods on 3 downstream tasks. We report F1 score (%) for PolypDiag, Dice (%) for CVC-12k, and F1 score (%) for KUMC, respectively.
Figure 3: Qualitative results of segmentation and detection tasks. The segmentation results (green, red, and yellow regions represent the true positive, false positive, and false negative, respectively) on the left are from the CVC-12k dataset, while the detection results on the right are from the KUMC dataset.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Example frames of the 7 pre-training datasets used in this work.
Figure 5: Example frames of the 4 downstream datasets used in this work.
Hyperparameter
Value
Sampling strategies
temporal window length
50
number of view frames
2
crop size
224×224
mask ratio
0.9
Optimizing settings
Appendix
Table 6: Pre-training settings.
Figure 6: Qualitative results for segmentation task on the CVC-12k dataset, where green, red, and yellow regions represent the true positive, false positive, and false negative, respectively.
Figure 7: Qualitative results for detection task on the KUMC dataset.
Figure 8: Illustration of our teacher-prior adaptive masking strategy. We visualize the feature heatmaps from both the teacher and the lightweight network (columns 2 & 3). The important regions (column 4) are then selected through a weighted aggregation of these heatmaps, where visible patches are sampled via a Top-K strategy.
Figure 9: Examples of low-quality sampling frames.
The development of foundation models (FMs) is crucial for advancing endoscopic image analysis. However, existing endoscopy FMs mainly rely on self-supervised learning from uni-modal images or videos, overlooking the rich semantic knowledge contained in clinical reports. Furthermore, effectively leveraging these records is hindered by a fundamental modality gap: structured anatomical descriptions are not naturally mapped to specific frames within the high-redundancy, uncurated visual streams. In this paper, we present EndoVLM, a novel vision-language FM pre-trained on over 348K endoscopic examinations, each pairing a clinical report with its corresponding image collection. An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention, efficiently aggregating semantically salient frames into anatomy-specific visual representations across redundant image-sets. Next, a Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets, bridging the gap from global patient-level matching to fine-grained localized alignment. Finally, a Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation. Extensive experiments across various downstream tasks demonstrate that EndoVLM outperforms existing foundation models and remains competitive with task-specific methods. Remarkably, EndoVLM also exhibits robust zero-shot generalization capabilities, highlighting its potential for broader clinical application.
Zhenyu Yi, Jianwei Xu, Yue Hu +6
1DAMO Academy, Alibaba Group · 3Shanghai Jiao Tong University · 4Hupan Lab +3
Endoscopic video contains state transitions induced by camera displacement, deformable tissue, and instrument interaction, yet existing pre-training methods treat dynamics implicitly. Inspired by action-conditioned joint-embedding predictive world models and by the observation that endoscopic optical flow contains both shared image-plane displacement and localized residual motion, we reformulate passive endoscopic video pre-training as causal motion-guided latent-state prediction. We present EndoWorld-JEPA, to our knowledge the first latent world-model pre-training framework designed for general-purpose endoscopic video representation learning. In place of recorded actions, calibrated poses, or robot kinematics, the model conditions prediction of the next latent state on motion observed up to the current state. Its Optical Motion-guided module separates dominant displacement from residual motion and converts the resulting evidence into an ordered temporal motion condition and spatial local-motion weights. Complementary global and motion-local predictors use these outputs: the temporal condition modulates both latent transitions, whereas the local weights define an auxiliary motion-local state that extends prediction from scene-level evolution to spatially selective dynamics. Patch-wise motion weighting emphasizes localized change while retaining anatomical context, and past-motion-window dropout randomly withholds the complete motion condition to reduce dependence on unreliable motion. This global--local formulation learns predictable visual--motion dynamics in representation space without reconstructing stochastic appearance artifacts. Evaluation shows that a common pre-trained checkpoint achieves strong transfer across four downstream benchmarks, supporting motion-guided global--local latent prediction as an effective pre-training paradigm for endoscopic video.
Jie Wang, Shilong Yang, Tao Zhou +5
University of Nottingham · Shenzhen University · Tsinghua University +2
Surgical video understanding is fundamental to navigation systems. Endoscopic perception is often hindered by a limited field-of-view and frequent instrument occlusions, making spatio-temporal context essential for robust inference. These challenges have motivated video models that aggregate information across frames. However, existing video models typically store past observations implicitly in learned feature representations, often requiring task-specific video training, substantial annotated data, and increased computational cost. We propose Stitch-Inferencer, a real-time, model-agnostic inference framework that replaces implicit feature memory with an explicit image-space panoramic canvas. By stitching valid observations across frames, Stitch-Inferencer preserves previously observed pixels in an online, instrument-free view, expanding the effective field-of-view and providing direct access to regions that are temporarily occluded or absent from the current frame. Downstream segmentation or tracking models are applied to a compact region of interest on the panorama, and their predictions are reprojected to the current frame, enabling existing models to exploit long-range context without retraining. Experiments on anatomy segmentation and point/box tracking demonstrate consistent improvements across diverse baselines while preserving real-time throughput. The stitching module alone runs at over 60 FPS, providing a practical inference-time solution to enhance endoscopic perception in computationally constrained intraoperative environments. Source code will be made publicly available.