SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models

Wanhao Liu; Rulin Zhou; Liangjing Shao; Zhaocheng Lin; Dongyue Li; Jinsong Lin; Guoheng Ma; Hangheng Yang

cs.CVOct 11, 2026

SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models

Authors: Wanhao Liu, Rulin Zhou, Liangjing Shao, Zhaocheng Lin, Dongyue Li, Jinsong Lin, Guoheng Ma, Hangheng Yang,

Organizations: Shenzhen Loop Area Institute, Shenzhen, China. · Guangdong University of Technology, Guangzhou, China. · The Chinese University of Hong Kong, Hong Kong SAR, China.

Abstract

Surgical world models must turn prospective robot actions into visually faithful, structurally consistent instrument motion. We introduce SurgCast, which predicts future instrument skeletons from current structure, robot state, and future actions, then uses the forecast to control video generation. Dual-path injection combines skeleton geometry with derived, instrument-specific motion descriptors through geometric residuals and conditioning modulation. Both paths complement native action conditioning and are retained in a four-step causal student. On SutureBot and SRTH-Porcine-Cholei, the teacher and student improve all seven reported quality and instrument-consistency metrics over their respective direct-action baselines; teacher FVD decreases by 37.6% and 67.3%. Ablations show that future actions improve skeleton prediction and that combining both injection paths performs best across all six metrics under predicted skeletons. These results support using predicted articulated structure and its motion semantics as a control interface for surgical video prediction. Project videos and examples are available at https://surgcast.netlify.app/.

Figures & tables

Fig. 2: SurgCast’s action-to-structure-to-video framework. Read (A) $\rightarrow$ (B) for construction of the control interface: (A) forecasts future instrument skeletons from current structure and prospective actions; (B) converts the forecast into geometry maps for residual injection and derived kinematic descriptors for modulation. Both paths condition the teacher (C) and causal student (D), alongside native action inputs. The arrow from (C) to (D) denotes training-time distillation; teacher and student are alternative video generators. Kinematic quantities describe projected keypoint motion, including shaft-direction angular velocity in the image plane.

Method	SutureBot							SRTH-Porcine-Cholei
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	GATC $\uparrow$	TCD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	GATC $\uparrow$	TCD $\downarrow$
Teachers
Cosmos-Predict2.5-2B [ 12 ]	8.1037	0.2249	0.7951	394.0091	6815.0560	0.0036	293.72	10.4087	0.1888	0.7492	458.5920	9802.3112	0.1340	293.7209
Ctrl-World [ 6 ]	21.5191	0.7856	0.2321	49.6254	248.3254	0.3656	71.95	19.3131	0.6584	0.3222	88.1025	750.3728	0.3507	70.3725
IRASim [ 33 ]	17.9001	0.5556	0.4226	135.8555	1226.6812	0.3551	68.04	16.6724	0.3632	0.4786	218.6194	1564.5107	0.4632	81.2678
Cosmos-H-Surgical-Simulator [ 18 ]	22.1164	0.8041	0.1719	31.7472	172.4712	0.6522	47.59	18.6471	0.6387	0.3027	92.6880	698.0070	0.4363	63.5268

TABLE I: Generation quality on SutureBot and SRTH-Porcine-Cholei, with teachers and four-step students grouped separately. SurgCast (Predicted) is the main method and uses forecast future skeletons; SurgCast (Oracle) supplies ground-truth future skeletons as a reference condition. These labels specify the skeleton source, while the teacher and student groups specify the video-generation mode. The causal student uses EMA-600. Green (bold) and yellow mark the best and second-best results per group and dataset. Cosmos-Predict2.5-2B results are from [ 12 ] .

Fig. 3: Teacher outputs on SutureBot (top), SRTH-Porcine-Cholei (middle), and Cao_cautery_combin with zero-shot video-generator transfer (bottom). The figure label ’SurgCast (Ours)’ denotes SurgCast (Predicted), conditioned on predicted future skeletons. White dashed annotations identify geometry drift, interaction misalignment, and structural artifacts in baseline outputs; compare the annotated instrument regions with SurgCast under the corresponding scene and action sequence.

Fig. 4: Video-generator transfer to Cao_cautery_combin without video-model fine-tuning: ranks across seven quality and instrument- consistency metrics. ’SurgCast (Ours)’ labels the SurgCast teacher; the outer ring denotes rank 1. The radial positions encode ranking within each metric, not metric magnitudes.

Method	Geo.	Kin.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	GATC $\uparrow$	TCD $\downarrow$
Action only	—	—	22.1164	0.8041	0.1719	31.7472	0.6522	47.59
Geometry only	$\checkmark$	—	23.0351	0.8337	0.1212	27.8214	0.7815	41.7564
Kinematics only	—	$\checkmark$	23.5359	0.8368	0.1370	28.8493	0.7869	41.3922
Geometry + Kinematics	$\checkmark$	$\checkmark$	24.4687	0.8458	0.1073	27.0716	0.7902	40.37

TABLE II: Predicted-skeleton ablation in the SutureBot teacher. Enabled paths use forecast skeletons; all variants, including action-only, use the same training recipe and retain native action conditioning. Yellow marks the best single-path result; green marks the combined model.

Fig. 5: Segment-wise rollout on 28 SutureBot validation episodes. C1–C6 cover successive 12-frame portions of the 72-frame prediction. Blue squares denote SurgCast (Predicted), conditioned on predicted future skeletons; student curves compare four-step causal generators. Panels (a,b) report frame fidelity; (c,d) report tool-region consistency and centroid distance.

Overall Performance
Method	Input	ADE $\downarrow$	FDE@H12 $\downarrow$	PCK@20px $\uparrow$	Vis. Acc. $\uparrow$
Persistence	$S_{0}$	55.64	80.83	49.04%	92.96%
Predictor w/o future action	$+r_{0}$	29.30	41.33	57.65%	97.33%
Action-to-Skeleton Predictor	$+a_{1:H}$	15.94	20.03	75.08%	97.89%

TABLE III: H12 skeleton prediction: three-domain macro-average in

960\times 540

coordinates. Top: jointly trained predictor without further adaptation. Bottom: domain-adapted predictors over steps 1–4, 5–8, and 9–12 (stride 5). Inputs cumulatively add

r_{0}

and

a_{1:H}

S_{0}

. Yellow marks the best baseline; green marks the action-conditioned predictor.

Fig. 6: Operator-driven SurgCast with a physical 6-DoF surgical master device (left) and VR controllers (right) as sources of prospective action commands. Displays show the generated surgical view; action-dependent outputs are compared in Fig. 7 .

Fig. 7: Action branching from the same initial image and skeleton (left). Distinct 6-DoF commands (insets) produce different predicted skeletons and video rollouts (upper and lower rows).

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 384 papers

Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning
Aug 20, 2026Weiliang Huang, Huanrong Liu, Bob Zhang +5Trajectory Prediction Video World Models
Faculty of Information Science and Computing, University of Macau, Macau, China · University of Macau Advanced Research Institute in Hengqin, Zhuhai, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China +5
SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics
Jun 18, 2026Wentao Pan, Wuyang Li, Shengyuan Liu +3World Models Video Prediction
The Chinese University of Hong Kong · EPFL · Imperial College London
SurgWMBench: A Vision-Based Benchmark for World-Modeling Surgical Instrument Motion Planning
Aug 8, 2026Huanrong Liu, Weiliang Huang, Bob Zhang +3World Models for Robotics World Model-Based Planning
University of Macau · Xiamen University · National University of Singapore

Aug 20, 2026J/K move · Enter open · S save

Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning

Weiliang Huang, Huanrong Liu, Bob Zhang, Qi Dou+4

Faculty of Information Science and Computing, University of Macau, Macau, China · University of Macau Advanced Research Institute in Hengqin, Zhuhai, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China+5

Trajectory Prediction Video World Models Motion Planning

Open paper AskPDF arXiv

Reliable surgical planning requires models to anticipate not only how instruments will move, but also how the operative visual state will evolve together with such motion. Existing approaches typically treat future scene generation and instrument trajectory prediction as two separate tasks. Scene-only models cannot directly evaluate the accuracy of future instrument motion at the trajectory level, while trajectory-only models fail to capture the visual consequences of instrument movement, leaving the consistency between predicted trajectories and future scene evolution unaddressed. Jointly forecasting both provides a more complete account of surgical action-scene dynamics by enabling explicit trajectory-level evaluation while simultaneously modeling the corresponding visual evolution. To bridge this gap, we present a preliminary joint visual-trajectory world-action model that simultaneously forecasts future visual states and instrument trajectories from historical surgical observations. Specifically, we encode historical video frames and tool trajectories into latent representations, which are processed by a temporal-spatial encoder and subsequently decoded through separate visual-state and trajectory prediction heads. Based on this preliminary architecture, a chunked autoregressive rollout is repeatedly applied to predict fifteen future steps. The chunked strategy consistently outperforms direct one-shot prediction across all evaluated horizons, improving first-segment PSNR from 18.86 to 23.11 dB and reducing ADE from 45.77 to 22.22 pixels. These results demonstrate the initial feasibility of joint visual-motion forecasting. However, we observe progressive visual degradation and accumulated trajectory errors over longer prediction horizons, which remain important challenges for future surgical world-action modeling.

Abstract

World Models for Robotics

Explore similar work

Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning

SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics

SurgWMBench: A Vision-Based Benchmark for World-Modeling Surgical Instrument Motion Planning