cs.CVNov 11, 2026

Feature-Space Latent Forecasting for Endoscopic Surgical Video Understanding

Authors: Songtao Hu, Hang Su

Organizations: Xiamen University, Xiamen, China

Abstract

Surgical workflow assistance is inherently anticipatory: systems must reason about how procedures evolve, not just recognize the current frame. However, many prior approaches perform prediction in pixel space by synthesizing future frames, which primarily focuses on appearance fidelity rather than the semantic representations required for downstream reasoning. To address this limitation, we propose a feature-space semantic forecasting framework that models temporal dynamics directly within the representations of a frozen Vision Foundation Model (VFM). Specifically, we build a hierarchical target representation from multi-layer VFM features, replace future-frame feature tokens with learnable [MASK] tokens, and train a masked transformer to forecast the corresponding future latent features from observed context, thereby formulating procedural evolution as masked future feature prediction. Experiments on public laparoscopic benchmarks demonstrate consistent improvements in phase anticipation, multi-label tool presence prediction, and remaining-time estimation over RGB-based forecasting baselines, while enabling a plug-and-play representation interface for lightweight downstream heads.

Figures & tables

Explore similar work

CardsList
  1. Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models

    Oct 1, 2026Efstathios Karypidis, Spyros Gidaris, Nikos KomodakisVideo PredictionPredictive Representation Learning

  2. SurgWorld: Learning Surgical Representations Beyond Observation via Latent Prediction

    Nov 8, 2026Jingkun Yue, Pengxiang Wang, Siqi Zhang +7Surgical Video UnderstandingVideo Representation Learning

  3. State-Change Learning for Prediction of Future Events in Endoscopic Videos

    Oct 14, 2025Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter +1Event Time PredictionSurgical Video Understanding