Feature-Space Latent Forecasting for Endoscopic Surgical Video Understanding
Organizations: Xiamen University, Xiamen, China
Abstract
Surgical workflow assistance is inherently anticipatory: systems must reason about how procedures evolve, not just recognize the current frame. However, many prior approaches perform prediction in pixel space by synthesizing future frames, which primarily focuses on appearance fidelity rather than the semantic representations required for downstream reasoning. To address this limitation, we propose a feature-space semantic forecasting framework that models temporal dynamics directly within the representations of a frozen Vision Foundation Model (VFM). Specifically, we build a hierarchical target representation from multi-layer VFM features, replace future-frame feature tokens with learnable [MASK] tokens, and train a masked transformer to forecast the corresponding future latent features from observed context, thereby formulating procedural evolution as masked future feature prediction. Experiments on public laparoscopic benchmarks demonstrate consistent improvements in phase anticipation, multi-label tool presence prediction, and remaining-time estimation over RGB-based forecasting baselines, while enabling a plug-and-play representation interface for lightweight downstream heads.
Figures & tables
| Method | Acc. | mF1 | Prep | Calot | C&C | GDis | GPack | Clean | GRet |
|---|---|---|---|---|---|---|---|---|---|
| Baseline (RGB) | 0.5118 | 0.3626 | 0.2633 | 0.5552 | 0.0115 | 0.6562 | 0.2847 | 0.4540 | 0.3132 |
| Oracle (GT feat.) | 0.7006 | 0.6570 | 0.6369 | 0.6022 | 0.3135 | 0.8879 | 0.6800 | 0.8292 | 0.6490 |
| Copy-Last (feat.) | 0.6547 | 0.5799 | 0.5586 | 0.5834 | 0.1812 | 0.8423 | 0.5987 | 0.7618 | 0.5334 |
| Ours (forecast feat.) | 0.6753 | 0.6101 | 0.6016 | 0.6062 | 0.2314 | 0.8617 | 0.6389 | 0.7896 | 0.5414 |
| Method | mF1 | Grasp | Bip | Hook | Scis | Clip | Irrig | Bag |
|---|---|---|---|---|---|---|---|---|
| Baseline (RGB) | 0.4145 | 0.6631 | 0.2974 | 0.8046 | 0.0817 | 0.1789 | 0.3572 | 0.5186 |
| Oracle (GT feat.) | 0.5756 | 0.6858 | 0.7394 | 0.9258 | 0.2555 | 0.5970 | 0.2893 | 0.5362 |
| Copy-Last (feat.) | 0.5267 | 0.6486 | 0.6591 | 0.8709 | 0.2467 | 0.5433 | 0.2447 | 0.4736 |
| Ours (forecast feat.) | 0.5521 | 0.6656 | 0.6862 | 0.9087 | 0.2503 | 0.5635 | 0.2690 | 0.5213 |
| Method | wMAE | inMAE | outMAE | ||||||
|---|---|---|---|---|---|---|---|---|---|
| 2m | 3m | 5m | 2m | 3m | 5m | 2m | 3m | 5m | |
| Bayesian | 0.39 | 0.59 | 0.85 | 0.63 | 0.86 | 1.17 | 0.15 | 0.32 | 0.52 |
| IIA-Net | 0.36 | 0.49 | 0.68 | 0.62 | 0.81 | 1.08 | 0.10 | 0.18 | 0.28 |
| Persistence (Copy-Last) | 0.35 | 0.51 | 0.73 | 0.61 | 0.80 | 1.10 | 0.10 | 0.17 | 0.29 |
| Ours (Forecast) | 0.34 | 0.48 | 0.66 | 0.59 | 0.78 | 1.06 | 0.09 | 0.16 | 0.26 |
| Oracle | 0.33 | 0.47 | 0.64 | 0.58 | 0.76 | 1.04 | 0.09 | 0.15 | 0.24 |
| (A) Current-frame segmentation baselines (frame ) | ||||||||||||
| Networks | Dice Bkg | Dice AW | Dice Liver | Dice GT | Dice Fat | Dice Ins | Dice Gb | Dice Misc. | Dice Mean | IoU Mean | Recall Mean | Precision Mean |
| U-Net | 0.97 | 0.73 | 0.52 | 0.26 | 0.87 | 0.52 | 0.40 | 0.08 | 0.54 | 0.43 | 0.67 | 0.55 |
| U-Net++ | 0.98 | 0.83 | 0.75 | 0.11 | 0.91 | 0.61 | 0.63 | 0.16 | 0.62 | 0.55 | 0.72 | 0.73 |
| DyUNet | 0.98 | 0.84 | 0.82 | 0.13 | 0.89 | 0.57 | 0.57 | 0.00 | 0.60 | 0.52 | 0.70 | 0.63 |
| UNETR | 0.97 | 0.76 | 0.74 | 0.35 | 0.88 | 0.71 | 0.42 | 0.00 | 0.60 | 0.49 | 0.72 | 0.59 |
| DeepLabV3+ | 0.97 | 0.81 | 0.74 | 0.14 | 0.86 | 0.62 | 0.60 | 0.12 | 0.61 | 0.50 | 0.69 | 0.62 |
Surgical Video Understanding
Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.
See all 132 papers