Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning
Authors: Weiliang Huang, Huanrong Liu, Bob Zhang, Qi Dou, Zhen Chen, Yun Gu, Guy Rosman, Qingbiao Li
Organizations: Faculty of Information Science and Computing, University of Macau, Macau, China · University of Macau Advanced Research Institute in Hengqin, Zhuhai, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong, China · Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · School of Medicine, Duke University, Durham, North Carolina, USA · Faculty of Engineering, University of Macau, Macau, China
Reliable surgical planning requires models to anticipate not only how instruments will move, but also how the operative visual state will evolve together with such motion. Existing approaches typically treat future scene generation and instrument trajectory prediction as two separate tasks. Scene-only models cannot directly evaluate the accuracy of future instrument motion at the trajectory level, while trajectory-only models fail to capture the visual consequences of instrument movement, leaving the consistency between predicted trajectories and future scene evolution unaddressed. Jointly forecasting both provides a more complete account of surgical action-scene dynamics by enabling explicit trajectory-level evaluation while simultaneously modeling the corresponding visual evolution. To bridge this gap, we present a preliminary joint visual-trajectory world-action model that simultaneously forecasts future visual states and instrument trajectories from historical surgical observations. Specifically, we encode historical video frames and tool trajectories into latent representations, which are processed by a temporal-spatial encoder and subsequently decoded through separate visual-state and trajectory prediction heads. Based on this preliminary architecture, a chunked autoregressive rollout is repeatedly applied to predict fifteen future steps. The chunked strategy consistently outperforms direct one-shot prediction across all evaluated horizons, improving first-segment PSNR from 18.86 to 23.11 dB and reducing ADE from 45.77 to 22.22 pixels. These results demonstrate the initial feasibility of joint visual-motion forecasting. However, we observe progressive visual degradation and accumulated trajectory errors over longer prediction horizons, which remain important challenges for future surgical world-action modeling.
Figures & tables
Figure 1: Overview of the proposed method. (a) Causal history encoding: five observed RGB frames are processed by the frozen SurgMotion ViT-L encoder to obtain causal token grids. (b) Closed-loop chunked future prediction: the predictor jointly forecasts future token grids and normalized 2-D tool positions in 3-step chunks; each predicted chunk updates the 5-step history for the next rollout, producing a total 15-step forecast.
Visual Metrics
Trajectory Metrics
Setting
Segment
PSNR ↑
SSIM ↑
LPIPS ↓
ADE ↓
FDE ↓
Chunked rollout
t+1:3
23.105
0.7971
0.1783
22.22
31.64
t+4:6
20.395
0.7187
0.2380
52.78
62.64
t+7:9
19.254
0.6799
0.2700
84.13
94.27
t+10:12
18.495
0.6527
0.2906
119.38
133.06
t+13:15
17.982
0.6343
0.3047
158.52
172.12
Table 1: Segment-wise comparison between chunked autoregressive rollout and direct one-shot prediction. Higher values are better for PSNR and SSIM, while lower values are better for LPIPS, ADE, and FDE. Trajectory errors are measured in pixels.
Figure 2: Qualitative comparison of direct one-shot prediction and chunked autoregressive rollout. Columns show the future states at t+3 , t+6 , t+9 , t+12 , and t+15 . Rows show the (a) direct one-shot prediction, (b) chunked autoregressive rollout and (c) ground truth. Instrument trajectories are overlaid on the corresponding visual states. For visualizaton purpose, we cropped the region of interest for comparision.
Reliable surgical planning requires models that move beyond recognizing the current surgical step or imitating expert demonstrations, and instead anticipate how instrument motion reshapes subsequent operative states. Most surgical video understanding methods focus on recognizing phases, actions, or workflow states, while providing limited support for explicitly modeling instrument motion. Conversely, existing tool motion prediction methods can forecast instrument trajectories, but they generally do not capture the coupled evolution of future surgical video states. World models offer a natural framework for jointly modeling visual state transitions and instrument motion dynamics. However, existing surgical world model studies remain largely centered on visual generation quality, relying on generation-oriented metrics such as FVD and CD-FVD. These metrics are poorly aligned with instrument motion planning, as they do not directly measure whether predicted trajectories are geometrically accurate, temporally coherent, or actionable for downstream planning. This limitation is partly structural, since the field lacks public datasets and standardized evaluation protocols that provide the benchmarking infrastructure needed to assess motion-centric capabilities in surgical world models. In this paper, we introduce SurgWMBench, a vision-based benchmark for short-horizon surgical motion planning and dynamics prediction. Given intraoperative image sequences and historical instrument trajectory, SurgWMBench evaluates both near-future instrument motion prediction and stability under continuous rollout or input perturbations.
Huanrong Liu, Weiliang Huang, Bob Zhang +3
University of Macau · Xiamen University · National University of Singapore
Surgical world models must turn prospective robot actions into visually faithful, structurally consistent instrument motion. We introduce SurgCast, which predicts future instrument skeletons from current structure, robot state, and future actions, then uses the forecast to control video generation. Dual-path injection combines skeleton geometry with derived, instrument-specific motion descriptors through geometric residuals and conditioning modulation. Both paths complement native action conditioning and are retained in a four-step causal student. On SutureBot and SRTH-Porcine-Cholei, the teacher and student improve all seven reported quality and instrument-consistency metrics over their respective direct-action baselines; teacher FVD decreases by 37.6% and 67.3%. Ablations show that future actions improve skeleton prediction and that combining both injection paths performs best across all six metrics under predicted skeletons. These results support using predicted articulated structure and its motion semantics as a control interface for surgical video prediction. Project videos and examples are available at https://surgcast.netlify.app/.
Wanhao Liu, Rulin Zhou, Liangjing Shao +9
Shenzhen Loop Area Institute, Shenzhen, China. · Guangdong University of Technology, Guangzhou, China. · The Chinese University of Hong Kong, Hong Kong SAR, China.
Scaling robot policy learning for autonomous surgery is challenging, as expert demonstrations are expensive and in vivo exploration poses substantial safety risks. Surgical world models address this by generating realistic, action-conditioned future frames from an initial observation, but existing methods exhibit two persistent failure modes: spatial interaction incoherence, where visible instrument contact fails to induce spatially consistent tissue deformation, and temporal fidelity collapse, where prediction errors compound across autoregressive rollouts and progressively corrupt visual quality. We present SurgVista, a surgical world model that mitigates both failures through two training recipes. Deformation Consistency Regularization extracts scene-point trajectories from training videos and enforces cross-frame coherence through latent contrastive learning, strengthening physically consistent instrument-tissue dynamics. Drift Adaptation Training mitigates long-horizon drift by perturbing conditioning frames with online prediction residuals and photometric augmentations calibrated to long-horizon drift statistics, sustaining visual fidelity over extended rollouts. To enable rigorous evaluation, we further introduce SurgWorld-Bench, featuring diverse procedure types, long-range rollouts, and decoupled metrics for instrument-motion accuracy and tissue-response fidelity. Extensive experiments show that SurgVista consistently outperforms state-of-the-art methods across visual quality, temporal consistency, and interaction fidelity, with gains widening as the prediction horizon grows.
Wentao Pan, Wuyang Li, Shengyuan Liu +3
The Chinese University of Hong Kong · EPFL · Imperial College London