TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models

Jiajun Cheng; Xiaofan Yu; Subarna Tripathi; Sainan Liu; Shan Lin

cs.CVMar 7, 2026

TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models

Authors: Jiajun Cheng, Xiaofan Yu, Subarna Tripathi, Sainan Liu, Shan Lin

Abstract

Recognizing instruments' interactions with tissues is essential for building context-aware AI assistants in robotic surgery. Vision-language models (VLMs) have opened a new avenue for surgical perception and achieved better generalization on a wide range of tasks compared to conventional task-specific deep learning approaches. However, their performance on instrument--tissue interaction recognition remains limited, largely due to two challenges: (1) many models do not effectively leverage temporal information, and (2) alignment between vision and text often misses fine-grained action details. To address these issues, we propose TrajPred, a framework that encodes instrument trajectories to incorporate temporal motion cues and, conditioned on these trajectories, introduces a predictor module to generate visual semantic embeddings that better capture fine-grained action details. We further incorporate prompt tuning and a verb-rephrasing technique to enable smooth adaptation to the instrument--tissue interaction recognition task. Extensive experiments on the public laparoscopic benchmark, CholecT50, show that our method improves both Average Precision and Top-K accuracy. We also investigate whether visual embeddings of instrument--tissue interaction regions align better with the corresponding text by visualizing the cosine similarity between visual and textual embeddings. The visualization results indicate that the proposed method improves alignment between relevant visual and textual representations.

Figures & tables

Fig. 1: (a) Cosine similarity heatmaps between embeddings of image patches and the text (e.g., “grasper retract gallbladder”). Two representative contrastive learning-based VLMs (CLIP-ViT-L [ 24 ] and SurgVLP [ 38 ] ) exhibit high similarity between background patches and the text, whereas TrajPred (ours) produces embeddings that concentrate on the Instrument-Tissue interaction region. (b) Comparison of the architectures of (i) popular contrastive learning-based surgical VLMs, (ii) VL-JEPA [ 2 ] , and (iii) TrajPred .

Fig. 2: Comparison between frames of the actions (a)“Grasper grasp” and (b)“Hook dissect”.

Fig. 3: Overview of TrajPred pipeline. Video clips are processed by both the video encoder and an object detector. Visual tokens within detector-predicted bounding boxes are mean-pooled and combined with bounding box coordinate embeddings through element-wise addition, forming trajectory tokens that are subsequently encoded via cross-attention. The resulting trajectory tokens are jointly fed with video encoder tokens into the predictor.

Verb	Rephrased Phrase
grasp	holding and gripping
retract	pulling aside
dissect	separating by cutting
coagulate	stopping bleeding by heating
clip	clipping closed
cut	cutting through

TABLE I: Verb rephrasing used for prompt construction.

Setting	Method	AP I	AP V	AP T	AP IVT	TOP@( $K{=}\|GT\|$ )	TOP@5	TOP@10	TOP@20
Standard RDV	CLIP-VIT-L-Patch14-FT	73.63	44.43	31.67	12.62	59.32	80.32	87.04	89.60
	CLIP-VIT-L-Patch14-GOAL-FT	76.04	45.93	32.08	13.15	61.07	80.93	87.99	95.04
	SurgVLP-FT ‡	71.18	47.70	33.35	12.68	59.02	81.98	88.72	94.93
	HecVL-FT ‡	76.07	50.92	32.09	12.67	58.03	78.80	86.59	93.58
	PeskaVL-FT ‡	75.20	48.71	30.29	12.12	57.05	76.76	86.43	93.54
	VL-JEPA (Image)	72.51	49.22	31.58	12.65	60.06	81.58	90.04	95.31

TABLE II: Overall Average Precision (AP) and TOP@K=

|GT|

under RDV [ 22 ] split and unseen-verb settings.

\ddagger

represents surgical pretrained VLMs.

Setting	Verb Rephrase	Traj Tokens	TOP@( $K{=}\|GT\|$ )
Standard RDV	$\times$	$\times$	60.51
	✓	$\times$	60.23
	$\times$	✓	65.37
	✓	✓	65.45
Unseen Verb	$\times$	$\times$	17.89
	✓	$\times$	19.28

TABLE III: Ablation study of TrajPred under the Standard RDV and Unseen-Verb settings. We report TOP@(

K{=}|GT|

Instrument	Verb	Occurrence	VL-JEPA	TrajPred
Instrument	Verb	(%)	(video)	TrajPred
Bipolar	coagulate ‡	71.7	18.0	18.8
	dissect	20.9	0.4	0.9
	grasp	3.1	13.7	14.6
	retract	4.3	2.8	4.2
Grasper	retract ‡	80.8	37.8	40.2

TABLE IV: Average Precision (AP) on triplets containing uncommon instrument–verb pairs.

\ddagger

denotes the common verb associated with the corresponding instrument.

Triplet	HecVL-FT	PeskaVLP-FT	TrajPred
“grasper pack gallbladder”	0.39	0.61	1.02
“grasper retract gut”	4.79	3.82	8.40
“grasper retract liver”	28.81	26.35	38.17
“grasper retract omentum”	16.84	14.26	27.18
“irrigator aspirate fluid”	12.22	14.22	22.49
“irrigator irrigate cystic_pedicle”	0.54	0.92	3.05

TABLE V: Per-class Average Precision (AP) on six selected triplets containing unseen verbs.

Fig. 4: Image–text similarity heatmaps over six consecutive frames, where red indicates higher cosine similarity. Similarity is computed between predictor-generated spatiotemporal tubelet tokens and the embedding of the ground-truth triplet (when it appears in the top-5 predictions). Each heatmap represents a two-frame temporal unit and is overlaid on the second frame. Top-5 triplet scores are shown alongside the heatmaps. Green and orange bounding boxes denote the grasper and bipolar instruments, respectively.

Fig. 5: Comparison of parameter count and inference latency

Explore similar work

Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition
Jul 22, 2026Jiajun Cheng, Sainan Liu, Subarna Tripathi +2Latent Action Learning Surgical Video Understanding
Arizona State University, Tempe, AZ, USA · Intel Corporation, USA · University of California, Merced, CA, USA
AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction
Apr 1, 2026Aiza Maksutova, Lalithkumar Seenivasan, Hao Ding +5Surgical Robotics
Johns Hopkins University, Baltimore MD, USA
SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
May 16, 2025Jiajun Cheng, Xianwu Zhao, Sainan Liu +5VLM Evaluation Surgical Video Understanding
Arizona State University · Intel Labs · University of California, Merced +2

Jul 22, 2026J/K move · Enter open · S save

Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition

Jiajun Cheng, Sainan Liu, Subarna Tripathi, Xiaofan Yu+1

Arizona State University, Tempe, AZ, USA · Intel Corporation, USA · University of California, Merced, CA, USA

Latent Action Learning Surgical Video Understanding Video Representation Learning Medical VLMs

Open paper AskPDF arXiv

Recognizing instrument--tissue interactions is essential for context-aware surgical AI. Vision-language models offer a natural way to inject semantic structure into surgical representations by aligning video features with textual action descriptions. However, pretrained encoders may lack spatial coherence, while global semantic alignment does not ensure precise spatial and temporal representations. By analyzing frame-to-frame feature changes, we find that semantic alignment increases their dimensionality, but larger increases do not necessarily improve recognition; encoders also differ in how strongly dominant changes localize to interaction regions. Motivated by these findings, we introduce \ours{}, which compresses frame-to-frame changes into latent actions and predicts next-frame features during end-to-end video--language alignment. Without additional spatial or motion annotations, \ours{} improves the interaction grounding of leading feature changes and temporal-direction sensitivity in our evaluated settings. We further characterize how action capacity and prediction strength affect recognition across encoders and triplet components. Using image encoders without large-scale video pretraining, \ours{} achieves competitive recognition with faster inference and smaller INT4 accuracy drops than V-JEPA2/2.1.

Abstract

Explore similar work

Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition

AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction

SurgXBench: Explainable Vision-Language Model Benchmark for Surgery