Multi-modal Representation Fusion for Long-term Microsurgical Anastomosis Trajectory Anticipation

Yuxuan Liu; Yating Luo; Yunfei Luan; Xinyao Zhou; Yuyang Zhuge; Yao Guo; Guang-Zhong Yang

cs.ROOct 10, 2026

Multi-modal Representation Fusion for Long-term Microsurgical Anastomosis Trajectory Anticipation

Authors: Yuxuan Liu, Yating Luo, Yunfei Luan, Xinyao Zhou, Yuyang Zhuge, Yao Guo, Guang-Zhong Yang

Abstract

Microsurgical anastomosis is one of the most challenging and sophisticated procedures for surgical robotics, requiring accurate manipulation of surgical instruments within a limited space to achieve precise operations. Accurate surgical trajectory anticipation is essential for surgical scene understanding, intraoperative planning, robotic manipulation, and reliable behaviour analysis. However, existing approaches mainly focus on features extracted from surgical videos and historical trajectories, while overlooking rich semantic and motion information that can provide more discriminative multi-modal representations for complex surgical instrument-tissue interactions. In this work, we propose a cross-modal feature fusion framework for long-term 2D and 3D microsurgical anastomosis trajectory anticipation. The proposed framework consists of a multi-modal representation extractor and a cross-modal feature fusion network. Semantic recognition of surgical objects and consecutive motion estimation are incorporated as multi-modal visual cues, providing complementary spatial and temporal guidance for instrument-tissue interaction characterization. A coarse-to-fine progressive fusion strategy with a long-short-term memory mechanism is introduced to capture temporal dependencies step by step. The training and evaluation are conducted on our established robotic platform and a comprehensive dataset containing both phantom and ex-vivo surgical scenarios is constructed. Extensive quantitative and qualitative experiments demonstrate that the proposed framework consistently outperforms existing methods and exhibits improved robustness across different surgical scenes, highlighting its potential for both intelligent robot-assisted and autonomous surgical systems.

Figures & tables

Fig. 1: Illustration of the microsurgical anastomosis trajectory anticipation. (a) Existing methods perform this task directly based on either past trajectories or observed surgical videos, while overlooking diverse multi-modal representations for long-term prediction and complex instrument-tissue interactions. (b) Our proposed framework incorporates the fusion of multi-modal representations for long-term 2D and 3D trajectory anticipation on microsurgical anastomosis.

Fig. 2: Overview of our established microsurgical anastomosis robotic platform and collected dataset. (a) Our established robot implements a dual-arm robotic system for anastomosis and a high-resolution microscope to capture surgical videos. (b) Our collected dataset includes surgical data for two different scenarios, i.e., anastomosis on the phantom vessel and ex-vivo vessel.

Fig. 3: Overview of the proposed multi-modal feature representation extraction network. Five semantic segmentations and the motion field are predicted by the multi-modal feature extractor, where the left instrument, right instrument, needle, thread, and operated vessel are selected as the semantic guidance. The network introduces a multi-task learning scheme with a unified feature encoder and several task-specific decoders to extract different feature representations simultaneously.

Fig. 4: Illustration of the proposed cross-modal representation fusion network. The network takes the image feature, text feature, observed trajectory feature, and the combination of semantic and motion guidance as the input, and implements a coarse-to-fine cross-modal feature fusion scheme to generate the fused representation. After that, a 2D trajectory decoder and a 3D trajectory decoder is utilized to anticipate surgical trajectories in the future. During the training stage, a long-short-term fusion block is introduced to progressively capture temporal dependencies, and three constraints, including a L1 distance term, an angular distance term, and a smoothness regularity term, are imposed on the predicted trajectories to improve the accuracy.

Fig. 5: Detailed architecture of the progressive fusion block. Given the extracted semantic feature, motion feature, image feature, text feature, and trajectory feature, the fusion block takes the calculated coarse fused feature maps as input and generates the fine feature maps with respect to different periods in a progressive manner. After that, a learnable gate vector is implemented to alleviate weights for the final output.

Approaches	Datasets	Instruments		Needles		Threads		Vessels
Approaches	Datasets	IOU (%)	Dice (%)	IOU (%)	Dice (%)	IOU (%)	Dice (%)	IOU (%)	Dice (%)
Segformer [ 49 ]	D-I	95.44	97.64	59.72	74.13	52.00	62.01	94.94	97.40
TPSIS [ 50 ]		92.13	95.83	55.74	70.66	47.77	58.54	94.17	96.98
TAPIS [ 25 ]		95.15	97.48	57.70	72.34	55.47	64.75	96.09	98.00
SurgicalSAM [ 51 ]		96.60	98.24	61.13	75.21	50.32	60.33	97.96	98.97
Ours		97.58	98.76	67.70	80.25	57.91	66.32	98.21	99.09

TABLE I: Comparative results of semantic segmentation with existing methods

Approaches	D-I		D-II		Average
Approaches	PSNR (dB)	SSIM (%)	PSNR (dB)	SSIM (%)	PSNR (dB)	SSIM (%)
EndFlow [ 52 ]	32.48	95.34	31.95	93.17	32.38	94.52
RAFT [ 53 ]	33.72	96.17	32.33	93.89	33.20	95.31
GMA [ 54 ]	33.34	96.07	33.39	94.89	33.36	95.63
Ours	35.62	97.09	34.10	96.08	35.05	96.71

TABLE II: Comparative results of motion estimation with existing methods

Fig. 6: Qualitative results of semantic segmentation with different methods on both phantom and ex-vivo datasets. The five classes are visualized with different colors, and the white arrows indicate the regions with significant improvements by our method.

Approaches	Datasets	2D Trajectory				3D Trajectory
Approaches	Datasets	ADE	FDE	FRD	HD	ADE	FDE	FRD	HD
BC [ 55 ]	D-I	14.21	15.71	19.20	14.24	57.00	72.92	87.93	58.33
TTM [ 27 ]		12.32	15.69	18.44	13.13	29.13	47.55	55.03	38.38
LED [ 28 ]		13.83	17.07	24.19	20.82	26.88	44.86	52.25	40.48
USST [ 44 ]		14.80	17.37	20.36	15.21	33.85	55.85	66.95	37.98
SDMamba [ 56 ]		13.50	15.59	18.88	12.01	42.58	58.62	73.96	41.46

TABLE III: Comparative results of 2D and 3D surgical trajectory anticipation with existing methods

Fig. 7: Qualitative results of 2D trajectory anticipation with different methods on both phantom and ex-vivo datasets. The blue line refers to the past observation, the red line refers to the ground truth collected by human demonstration, and the green line refers to the anticipation by different methods. The zoomed images visualized in the right part indicate the region of surgical instrument-tissue interactions.

Fig. 8: Qualitative results of 3D trajectory anticipation with different methods on both phantom and ex-vivo datasets. The blue line refers to the past observation, the red line refers to the ground truth collected by human demonstration, and the green line refers to the anticipation by different methods. The left and right instrument trajectories are visualized separately.

Fig. 9: Additional qualitative results of our proposed method on both the phantom dataset and the ex-vivo dataset. The 2D and 3D trajectory anticipations are visualized simultaneously. The blue, red, and green lines indicate the observation, ground truth, and the anticipation of our method, respectively.

Description	Approaches	2D Trajectory				3D Trajectory
Description	Approaches	ADE	FDE	FRD	HD	ADE	FDE	FRD	HD
Multi-modal Feature Extraction	w/o semantics and motion $\{z^{\mathrm{M}},z^{\mathrm{F}}\}$	13.64	16.16	20.13	14.85	30.71	48.54	58.19	42.55
	w/o detailed semantics $\{\mathrm{M}^{nd},\mathrm{M}^{tr}\}$	11.28	14.60	16.65	12.69	26.92	42.36	49.49	36.08
	w/o operating semantics $\{\mathrm{M}^{l},\mathrm{M}^{r},\mathrm{M}^{tr}\}$	11.55	14.60	16.70	12.68	26.64	42.09	49.18	35.75
	w/o motion $\mathrm{F}$	12.84	15.22	18.78	14.28	27.95	45.08	53.61	39.67
	w/o instruction $\mathrm{\Omega}$	11.50	14.21	16.37	12.29	26.52	42.18	49.33	34.06

TABLE IV: Ablated results of 2D and 3D surgical trajectory anticipation with different network architectures and training schemes

Approaches	2D Trajectory		3D Trajectory
Approaches	ADE	FDE	ADE	FDE
$((z^{\mathrm{I}}\odot z^{\mathrm{\Omega}})\oplus z^{\mathrm{P/J}})\odot(z^{\mathrm{M}}\oplus z^{\mathrm{F}})$	11.20	13.81	25.28	40.83
$((z^{\mathrm{I}}\odot z^{\mathrm{\Omega}})\odot z^{\mathrm{P/J}})\odot(z^{\mathrm{M}}\odot z^{\mathrm{F}})$	11.77	14.44	27.17	43.40
$((z^{\mathrm{I}}\oplus z^{\mathrm{\Omega}})\oplus z^{\mathrm{P/J}})\oplus(z^{\mathrm{M}}\oplus z^{\mathrm{F}})$	11.47	13.90	28.23	44.83
$\{\mathrm{T_{obs}},\mathrm{T_{pred}}\}=\{5s,3s\}$	9.73	12.09	18.76	31.54
$\{\mathrm{T_{obs}},\mathrm{T_{pred}}\}=\{5s,5s\}$	11.20	13.81	25.28	40.83

TABLE V: Ablated results of 2D and 3D surgical trajectory anticipation with different fusion schemes and anticipation periods

Fig. 10: Performance of 2D and 3D trajectory anticipation for different anticipation periods on the phantom case. The left axis refers to the error value of the 2D trajectory, and the right axis refers to the error value of the 3D trajectory. The lower part shows the predicted 2D trajectories with the same observation.

Fig. 11: Qualitative results of our proposed method for different anticipation periods $\mathrm{T_{pred}=\{3s,5s,7s,10s,15s\}}$ . The 2D and 3D trajectory anticipations are visualized simultaneously. The blue, red, and green lines indicate the observation, ground truth, and the anticipation of our method, respectively.

$\alpha_{pos}^{3D}$	2D Trajectory				3D Trajectory
$\alpha_{pos}^{3D}$	ADE	FDE	FRD	HD	ADE	FDE	FRD	HD
15.0	12.15	14.26	17.23	12.78	27.51	44.14	51.74	37.38
10.0	11.39	13.95	16.10	11.56	26.78	42.32	49.36	36.24
5.0	11.20	13.81	15.63	11.28	25.28	40.83	46.74	34.13
1.0	11.80	14.16	16.31	11.63	25.64	41.66	48.09	34.32
0.5	12.39	14.20	17.13	12.26	27.04	42.50	49.80	34.82

TABLE VI: Ablated results of 2D and 3D surgical trajectory anticipation with different coefficients

\alpha_{pos}^{3D}

Fig. 12: Performance of 2D and 3D trajectory anticipation for different choices of the coefficient $\alpha_{pos}^{3D}$ . The left axis refers to the error value of the 2D trajectory, and the right axis refers to the error value of the 3D trajectory.

Fig. 13: Performance of 2D and 3D trajectory anticipation for different surgical actions on the ex-vivo case. The left axis refers to the error value of the 2D trajectory, and the right axis refers to the error value of the 3D trajectory. The lower part shows the representative frames for each action.

Action	I	II	III	IV	V	VI	VII	VIII	IX	X
ADE-2D	13.80	11.18	4.95	5.22	17.36	12.78	13.01	12.87	21.69	19.14
FDE-2D	14.66	10.97	5.14	5.92	19.22	14.85	7.58	15.41	22.63	18.38
ADE-3D	29.57	25.61	15.77	16.74	51.37	36.03	21.04	24.02	53.01	54.28
FDE-3D	40.47	44.60	24.22	29.13	69.17	48.45	21.50	40.87	76.09	63.19

TABLE VII: Results of 2D and 3D surgical trajectory anticipation for different surgical actions

Trajectory Prediction

Trajectory prediction forecasts how people, vehicles, robots, and other dynamic systems will move from observed histories and context.

See all 187 papers

Feb 18, 2026

SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment

Robotic-assisted surgery (RAS) is established in clinical practice, and automated surgical skill assessment utilizing multimodal data offers transformative potential for surgical analytics and education. However, developing effective multimodal methods remains challenging due to the task complexity, limited annotated datasets and insufficient techniques for cross-modal information fusion. Existing state-of-the-art relies exclusively on RGB video and only applies on dry-lab settings, failing to address the significant domain gap between controlled simulation and real clinical cases, where the surgical environment together with camera and tissue motion introduce substantial complexities. This work introduces SurgFusion-Net and Divergence Regulated Attention (DRA), an innovative fusion strategy for multimodal surgical skill assessment. We contribute two first-of-their-kind clinical datasets: the RAH-skill dataset containing 279,691 RGB frames from 37 videos of Robot-assisted Hysterectomy (RAH), and the RARP-skill dataset containing 70,661 RGB frames from 33 videos of Robot-Assisted Radical Prostatectomy (RARP). Both datasets include M-GEARS skill annotations, corresponding optical flow and tool segmentation masks. DRA incorporates adaptive dual attention and diversity-promoting multi-head attention to fuse multimodal information, from three modalities, based on surgical context, enhancing assessment accuracy and reliability. Validated on the JIGSAWS benchmark, RAH-skill, and RARP-skill datasets, our approach outperforms recent baselines with SCC improvements of 0.02 in LOSO, 0.04 in LOUO across JIGSAWS tasks, and 0.0538 and 0.0493 gains on RAH-skill and RARP-skill, respectively.

Runlong He, Freweini M. Tesfai, Matthew W. E. Boal +13

the UCL Hawkes Institute and the Department of Medical Physics & Biomedical Engineering, UCL, UK. · the Griffin Institute, Northwick Park and St Mark’s Hospital, UK. · the UCL Hawkes Institute and the Division of Informatics, Imaging & Data Sciences, The University of Manchester, UK +3

Surgical Video Understanding Multimodal Fusion Surgical Robotics

Ask PDF

Jul 5, 2026

SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy

Surgical automation is being increasingly studied, yet bridging visual scene understanding with autonomous action planning remains a fundamental challenge. While much research effort has been made on scene perception (e.g., tool recognition and scene segmentation), understanding and predicting actionable possibilities for surgical automation is still underexplored. In this paper, we introduce surgical affordance prediction, which identifies actionable regions for fundamental surgical actions from visual data. Specifically, a novel adaptive feature fusion framework is proposed that leverages the complementary strengths of a self-supervised vision transformer encoder for its superior semantic understanding and a large-scale generative model encoder for its spatially-aware capability. Furthermore, we introduce a hierarchical prompt learning mechanism to adapt to varying procedural contexts. Finally, a scene-guided attention decoder is proposed to focus on critical surgical areas while suppressing background distractions. To validate the effectiveness, we established a new dataset, derived from publicly available surgical datasets with affordance annotations for three basic surgical actions: aspiration, clipping, and retraction. Extensive experiments demonstrate that our approach achieves state-of-the-art performance. Moreover, we validate our framework's applicability for downstream automation on a realistic lung and prostate phantom, and results show that the predicted affordance maps successfully enable autonomous surgical actions.

Lei Song, Yonghao Long, Mengya Xu +3

Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Thoracic Surgery, Peking University People’s Hospital, Beijing, China · Thoracic Oncology Institute, Peking University People’s Hospital, Beijing, China +3

Robotic Perception Surgical Robotics

Ask PDF

Feb 27, 2026

Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics

Automated recognition of surgical phases and steps is a fundamental capability for intraoperative decision support, workflow automation, and skill assessment in robotic-assisted surgery. Existing approaches either depend on large-scale annotated surgical datasets or require expensive domain-specific pretraining on thousands of labeled videos, limiting their practical deployability across diverse robotic platforms and clinical environments. In this work, we propose TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free framework for surgical temporal segmentation that requires no task-specific annotations or surgical-domain pretraining. TASOT extends the Action Segmentation Optimal Transport (ASOT) formulation by incorporating temporally aligned textual descriptions generated directly from the input video, fusing visual and semantic cues within a unified unbalanced Gromov-Wasserstein optimal transport objective. Visual representations are extracted using DINOv3, while temporal captions produced by a vision-language model are encoded via CLIP and temporally aligned to individual frames, providing complementary semantic structure to the transport cost. We evaluate TASOT on three public surgical datasets and four benchmark settings spanning laparoscopic and robotic procedures, showing substantial improvements over the strongest zero-shot baselines: +18.9 F1 on Cholec80, +33.7 on AutoLaparo, +23.7 on StrasByPass70, and +4.5 on BernByPass70. These results suggest that fine-grained surgical workflow understanding in robotic settings can be achieved without manual training annotations or surgical-specific pretraining pipelines, offering a promising alternative for real-world robotic surgical systems.

Omar Mohamed, Edoardo Fazzari, Ayah Al-Naji +5

Authos are with the Dept. of Robotics, Mohamed bin Zayed University of AI, Abu Dhabi, UAE.

Temporal Action Segmentation Surgical Video Understanding Surgical Robotics Optimal Transport

Ask PDF

Abstract

Trajectory Prediction

Explore similar work

SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment

SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy

Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics