Authors: Marko Haralović, Zhiqi Miao, Alexander Machiel Bont, Jiapan Guo, Frans van Workum, Estefanía Talavera
Organizations: University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Twente, Enschede, The Netherlands · University of Groningen, Bernoulli Institute, Groningen, The Netherlands · Radboud University Medical Center, Nijmegen, The Netherlands · Department of Surgery, Canisius-Wilhelmina Hospital, Nijmegen, The Netherlands
Surgical performance assessment in minimally invasive surgery largely relies on manual expert review, making it time-consuming, subjective, and difficult to scale. While existing surgical video understanding methods address tasks such as instrument segmentation, surgical phase recognition, and action recognition, they do not explicitly capture fine-grained tissue handling, a key indicator of surgical quality. To address this gap, we introduce tissue tension recognition, a new clinically motivated video understanding task for laparoscopic and robot-assisted rectal cancer surgery. To support this task, we construct SurgTension, the first expert-annotated tissue tension dataset, providing a benchmark for objective tissue tension recognition. We further propose TensionTRAC, a lightweight trajectory-based framework that models tissue tension from sparse point trajectories. Using a compact trajectory encoder, TensionTRAC achieves competitive performance against strong pretrained video backbones.
Figures & tables
Figure 1: Motivation for trajectory-based tissue tension recognition. If we uniformly sample points on a frame and employ a point tracker to track their movement over time (generating trajectories), we can observe that the tension-present clip (a) exhibits spatially structured local tissue deformation, whereas the no-tension clip (b) contains limited or tension-unrelated motion (e.g., instrument movement). This contrast motivates the use of sparse trajectory modeling for tissue tension recognition.
Category
Clinical description
#
%
0 Non-tension ( ✓ )
No visible traction-induced tissue tension or deformation.
874
22.6
1 Low ( ✓ )
Minimal traction or tissue deformation, with no apparent risk of tissue injury.
545
14.1
2 Moderate ( ✓ )
Visible traction and tissue deformation sufficient to facilitate exposure or dissection.
1349
34.9
3 Adequate ( ✓ )
Pronounced but controlled traction and tissue deformation within the clinically acceptable range.
749
19.4
4 Excessive ( ✓ )
Marked or uncontrolled tissue stretching beyond the clinically acceptable range, with increased risk of tissue injury or loss of the dissection plane.
76
2.0
Reverse ( × )
Reduction, release, or reversal of previously applied tissue tension.
208
5.4
Table 1: SurgTension tissue tension dataset categories and category distribution.
Metric
Non-Tension
Tension
Δ
p
Visible pts.
87.11
77.33
-9.78
1.06×10−8
Velocity mean
1.78
2.29
+0.51
1.11×10−5
Velocity max
27.84
30.90
+3.06
7.60×10−3
Accel. mean
0.66
0.69
+0.03
2.40×10−2
Accel. max
15.31
11.76
-3.55
0.633
Dist. change mean
0.23
-0.30
-0.53
0.209
Table 2: Trajectory-derived motion and deformation statistics for Non-Tension and Tension clips. For each metric, Non-Tension and Tension report the average value over clips, and Δ=Tension−Non-Tension indicates the magnitude of the difference. The p value is obtained from a two-sided statistical test comparing the two groups. Significant differences at the nominal level of α=0.05 are bolded. The results show that tension clips exhibit stronger motion and deformation patterns, supporting the use of sparse trajectories for tissue tension recognition.
Figure 2: Overview of TensionTRAC . Frozen DINOv2 and CoTrackerV3 models are used offline for semantic-feature extraction and trajectory generation, respectively. The intra-trajectory and inter-trajectory motion branches, feature fusion, and spatio-temporal transformer form the pretrained trajectory encoder, which is frozen to extract clip-level representations for downstream k -NN and MLP classification on SurgTension.
Method
Trainable Params
Frames
Encoder GFLOPs
Pretraining
Dataset
Scale
SlowFast [ 9 ] R50 4×16
34M
32×1
36
K400 [ 18 ]
240K videos
Video Swin-3D [ 26 ]
88M
32×1
282
IN-21K → K400 [ 18 ]
14M img. + 240K vid.
TimeSformer [ 5 ] DivST
122M
16×1
362
IN-21K → K600 [ 7 ]
14M img. + 392K vid.
SurgMotion [ 48 ] ViT-L
304M
64×1
5,785
SurgMotion-15M [ 48 ]
15M images
TensionTRAC
18M
8×1
35
K400-10K [ 18 , 49 ]
10K videos
Table 3: Comparison of representation-encoder complexity and pretraining scale. Trainable parameters and pretraining refers to the encoders optimized during their representation-learning stages. For TensionTRAC , DINOv2 and CoTrackerV3 remain frozen and performed offline; the reported parameters, GFLOPs, and pretraining scale cover only the trajectory encoder after offline feature and trajectory extraction.
Task
Method
Classifier
Stratified
Video-grouped
Acc.
BalAcc.
M-F1
Acc.
BalAcc.
M-F1
Binary
SlowFast [ 9 ]
KNN
87.6
80.4
82.2
81.1
72.7
73.6
MLP
87.7
84.3
83.7
79.7
75.9
74.3
TimeSformer [ 5 ]
KNN
89.7
83.6
85.2
81.9
72.4
73.9
MLP
88.6
85.1
84.8
83.0
76.2
76.6
Video Swin-3D [ 26 ]
KNN
89.7
83.6
85.3
81.5
73.1
74.0
Table 4: Comparison of KNN ( k=3 ) and MLP classifiers on frozen features for binary tension classification and end-to-end cascade tension recognition. Stratified evaluation uses clip-level stratified splits, whereas video-grouped evaluation assigns all clips from the same surgical video exclusively to the same fold, ensuring strict source-video separation. All metrics are reported in %.
Figure 3: Qualitative results of TensionTRAC for tissue tension recognition. The examples present predictions for binary tension classification (left) and cascade two-stage evaluation (right). Sparse trajectories visualize local tissue and instrument motion in tension-present clips, while failure cases arise from camera, instrument, or other non-tension-related motion, as well as confusion between neighboring tension levels.
Recognizing instruments' interactions with tissues is essential for building context-aware AI assistants in robotic surgery. Vision-language models (VLMs) have opened a new avenue for surgical perception and achieved better generalization on a wide range of tasks compared to conventional task-specific deep learning approaches. However, their performance on instrument--tissue interaction recognition remains limited, largely due to two challenges: (1) many models do not effectively leverage temporal information, and (2) alignment between vision and text often misses fine-grained action details. To address these issues, we propose TrajPred, a framework that encodes instrument trajectories to incorporate temporal motion cues and, conditioned on these trajectories, introduces a predictor module to generate visual semantic embeddings that better capture fine-grained action details. We further incorporate prompt tuning and a verb-rephrasing technique to enable smooth adaptation to the instrument--tissue interaction recognition task. Extensive experiments on the public laparoscopic benchmark, CholecT50, show that our method improves both Average Precision and Top-K accuracy. We also investigate whether visual embeddings of instrument--tissue interaction regions align better with the corresponding text by visualizing the cosine similarity between visual and textual embeddings. The visualization results indicate that the proposed method improves alignment between relevant visual and textual representations.
Robust point tracking in endoscopic videos is essential for computer-assisted intervention and autonomous robotic surgery, enabling continuous registration between intraoperative video and preoperative imaging despite soft tissue deformation. However, supervised tracking methods depend on large annotated datasets, while surgical conditions make reliable trajectory annotation challenging. We propose a self-supervised Track-Any-Point approach that learns from unlabeled surgical videos by establishing global pixel correspondences and inferring point trajectories through contrastive random walks. Trained without annotations, our method achieves performance comparable to existing semi-supervised approaches while implicitly handling tissue deformation. These findings demonstrate the feasibility of self-supervised point tracking in surgical environments and its potential to reduce reliance on annotated data.
Jiaming Zhang, Zijian Wu, Mehran Armand +1
Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas
Surgical action automation has progressed rapidly toward achieving surgeon-like dexterous control, driven primarily by advances in learning from demonstration and vision-language-action models. While these have demonstrated success in table-top experiments, translating them to clinical deployment remains challenging: current methods offer limited predictability on where instruments will interact on tissue surfaces and lack explicit conditioning inputs to enforce tool-action-specific safe interaction regions. Addressing this gap, we introduce AffordTissue, a multimodal framework for predicting tool-action specific tissue affordance regions as dense heatmaps during cholecystectomy. Our approach combines a temporal vision encoder capturing tool motion and tissue dynamics across multiple viewpoints, language conditioning enabling generalization across diverse instrument-action pairs, and a DiT-style decoder for dense affordance prediction. We establish the first tissue affordance benchmark by curating and annotating 15,638 video clips across 103 cholecystectomy procedures, covering six unique tool-action pairs involving four instruments (hook, grasper, scissors, clipper) and their associated tasks: dissection, grasping, clipping, and cutting. Experiments demonstrate substantial improvement over vision-language model baselines (20.6 px ASSD vs. 60.2 px for Molmo-VLM), showing that our task-specific architecture outperforms large-scale foundation models for dense surgical affordance prediction. By predicting tool-action specific tissue affordance regions, AffordTissue provides explicit spatial reasoning for safe surgical automation, potentially unlocking explicit policy guidance toward appropriate tissue regions and early safe stop when instruments deviate outside predicted safe zones.