Optimize Surgical Triplet Recognition: A Knowledge-Driven Mixture-of-Experts Solution

Yiyi Zhang; Yuchen Yuan; Ying Zheng; Jialun Pei; Jinpeng Li; Zheng Li; Pheng-Ann Heng

cs.CVAug 24, 2026

Optimize Surgical Triplet Recognition: A Knowledge-Driven Mixture-of-Experts Solution

Authors: Yiyi Zhang, Yuchen Yuan, Ying Zheng, Jialun Pei, Jinpeng Li, Zheng Li, Pheng-Ann Heng

Organizations: Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China · Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China

Abstract

Surgical action triplet recognition constitutes a critical task in context-aware robot-assisted surgery, facilitating automatic surgical action perception by identifying instrument, verb, target, and their association. However, existing works struggle to analyze such complex surgical scenes due to three main issues: (1) component-level optimization conflicts caused by entangled feature spaces, (2) category-level optimization conflicts arising from severe data imbalance, and (3) lack of domain knowledge guidance that limits model interpretability and robustness. To address these challenges, we propose a Mixture-of-Experts-guided Co-Optimization (MoeCo) framework powered by knowledge-driven learning. Within the co-optimization pipeline, to first mitigate component-level conflicts, we introduce a component-tailored adapter that disentangles task-specific features across spatial-temporal regimes, facilitating effective component specialization. Next, we develop a coordinated gradient learning strategy to handle category-level conflicts, which adaptively rebalances positive-negative gradients to enhance the perception of rare categories. Notably, inspired by surgical domain expertise, we introduce a knowledge-driven mixture-of-experts mechanism that dynamically integrates multimodal large language model-guided knowledge via activated experts, thereby enriching the co-optimization pipeline with more expressive and robust representations. Extensive experiments on the public CholecT45 and CholecT50 datasets confirm the effectiveness of the proposed co-optimization pipeline and the superiority of dynamic priors integration via the knowledge-driven mixture-of-experts mechanism.

Figures & tables

Fig. 1 : (a) Hierarchical optimization conflicts in triplet (I/V/T) recognition: We illustrate the conflict using a common head triplet ( $<$ grasper, retract, gallbladder $>$ ) and a rare tail triplet ( $<$ grasper, retract, gut $>$ ). At the Component Level (Left): In the Instrument and Verb layers, the two features are pulled closer because they share the same instrument ( grasper ) and verb ( retract ). However, in the Target and Triplet layer, they must be pushed apart to distinguish gallbladder from gut and the two triplet categories. At the Category Level (Right): This conflict is exacerbated by dataset imbalance. The gradient from the head category (red arrow) dominates the optimization direction, overpowering the tail category (blue arrow). (b) Domain Priors: Case studies show how integrating explicit knowledge helps the reasoning process of valid triplets. For example, knowing a forked tip tool is likely a clipper (not an irrigator ) narrows the search space, guiding the model toward valid triplets despite noisy visual cues.

Fig. 2 : Overview of the proposed Mixture-of-Experts-guided Co-optimization ( MoeCo ) framework. We first employ an MLLM to generate domain knowledge used in constructing Gaussian models for each expert. During training, the original visual embeddings from an input video sequence are enriched by concatenating them with MoE-purified semantics. Multi-head self-attention layers process the enriched embeddings, followed by a component-tailored adapter for task-specific feature modeling via temporal-spatial prompting. Individual component features drive their recognition outputs, while triplet features integrate with them and undergo refinement with a feature pyramid network. Finally, the coordinated gradient learning strategy is employed to decompose and rebalance positive-negative gradients for better perception of rare triplet categories.

Fig. 3 : An illustration of the component-tailored adapter. CTA learns the temporal- and spatial-wise task-specific features in a parameter-efficient way with no need for multiple network architectures for each sub-task.

Fig. 4 : Predicted mean probability and mean gradient curves on the CholecT45 dataset during training. H and T denote head and tail classes, respectively, while

+

and

-

denote positive and negative samples.

H^{+}_{\mathrm{CGL}}

H^{-}_{\mathrm{CGL}}

T^{+}_{\mathrm{CGL}}

, and

T^{-}_{\mathrm{CGL}}

denote the corresponding curves after applying CGL. The gradient magnitude is computed from the predicted probability. Compared with BCE, CGL substantially reduces the discrepancy in optimization behavior between tail and head classes.

Method	Backbone	$\bm{AP_{I}}$	$\bm{AP_{V}}$	$\bm{AP_{T}}$	$\bm{AP_{IV}}$	$\bm{AP_{IT}}$	$\bm{AP_{IVT}}$
TripNet [ 7 ]	ResNet-18	89.9±1.0	59.9±0.9	37.4±1.5	–	–	24.4±4.7
RDV [ 12 ]	ResNet-18	89.3±2.1	62.0±1.3	40.0±1.4	34.0±3.3	30.8±2.1	29.4±2.8
RiT [ 14 ]	ResNet-18	88.6±2.6	64.0±2.5	43.4±1.4	38.3±3.5	36.9±1.0	29.7±2.6
TripDis [ 35 ]	ResNet-50	91.2±1.9	65.3±2.8	43.7±1.6	–	–	33.8±2.5
SelfD [ 36 ]	SwinB×2+SwinL	–	–	–	–	–	38.5±0.0
MT4MTL-KD [ 25 ]	ResNet-18+SwinL	93.9±2.0	73.8±2.0	52.1±5.2	46.5±3.4	46.2±2.3	38.9±1.6

TABLE I : Quantitative comparisons between the proposed method and the SOTA methods using 5-fold cross-validation on the CholecT45 dataset. The mean and standard deviation results of AP are reported. Best and Second Best .

Panel of Figure 6 — Fig. 5 : Box plots with overlaid dot distributions compare the per-category performance of the baseline and MoeCo-T across four task metrics. Each red dot represents a category, while the blue lines indicate the median AP values.

Method	$\bm{AP_{I}}$	$\bm{AP_{V}}$	$\bm{AP_{T}}$	$\bm{AP_{IV}}$	$\bm{AP_{IT}}$	$\bm{AP_{IVT}}$
TCN [ 7 ]	48.9	29.4	21.4	17.7	15.5	12.4
MLT [ 7 ]	84.5	48.4	28.2	26.6	21.2	17.6
Tripnet [ 7 ]	92.1	54.5	33.2	29.7	26.4	20.0
RDV [ 12 ]	92.0	60.7	38.3	39.4	36.9	29.9
MPCNet [ 21 ]	90.9	62.1	41.0	39.0	36.1	30.2
Forest GCN [ 42 ]	93.1	60.1	40.2	36.2	37.5	36.7

TABLE II : Quantitative evaluation on CholecT50 dataset.

Fig. 6 : Performance comparison between BCE and CGL.

KD-MoE	CTA	CGL	$\bm{AP_{I}}$	$\bm{AP_{V}}$	$\bm{AP_{T}}$	$\bm{AP_{IV}}$	$\bm{AP_{IT}}$	$\bm{AP_{IVT}}$
			89.4	67.2	53.2	42.1	45.1	38.3
✓			90.4	69.9	55.4	43.3	47.2	40.3 ( $\uparrow$ 2.0)
	✓		91.2	67.3	57.4	41.6	48.0	40.0 ( $\uparrow$ 1.7)
		✓	91.1	70.8	55.9	43.3	47.5	40.4 ( $\uparrow$ 2.1)
✓	✓		90.7	71.0	57.1	43.5	50.5	41.2 ( $\uparrow$ 2.9)
✓	✓	✓	91.8	72.5	58.0	44.7	50.7	42.3 ( $\uparrow$ 4.0)

TABLE III : Module contribution ablation of the proposed framework (MoeCo-T). The first row represents the results of the baseline method.

Text Input Configuration	$\bm{AP_{I}}$	$\bm{AP_{V}}$	$\bm{AP_{T}}$	$\bm{AP_{IVT}}$
Baseline (Vision-Only)	89.4	67.2	53.2	38.3
Random Feature Initialization	89.1	67.5	54.9	38.7
Random Prior Activation	89.6	68.4	55.9	39.6
w/ KD-MoE (Ours)	90.4	69.9	55.4	40.3
Oracle Priors (Upper Bound)	90.7	69.1	57.3	40.9

TABLE IV : Ablation study isolating the semantic value of the knowledge base and gating mechanism.

Fig. 7 : KD-MoE ablation at different MSA layers.

Fig. 8 : Performance under different $k$ options in KD-MoE.

Loss Function	$\bm{AP_{I}}$	$\bm{AP_{V}}$	$\bm{AP_{T}}$	$\bm{AP_{IV}}$	$\bm{AP_{IT}}$	$\bm{AP_{IVT}}$
BCE	89.4	67.2	53.2	42.1	45.1	38.3
Focal Loss [ 44 ]	90.2	70.5	56.9	43.0	46.9	39.1
CB Loss [ 45 ]	90.1	68.6	55.2	42.5	46.0	38.5
EQ Loss [ 34 ]	90.9	70.2	55.6	42.6	46.3	39.0
CGL (Ours)	91.1	70.8	55.9	43.3	47.5	40.4

TABLE V : Comparison with representative long-tailed loss functions on CholecT45.

Method	Params	FLOPs	Training Time	$\bm{AP_{IVT}}$
Baseline	95.8M	4.57G	19.37s	38.3
+ 4 Task Branches	108.4M	4.68G	19.62s	38.8
+ VPT [ 46 ]	95.8M	4.62G	19.51s	39.4
+ ST-Adapter [ 47 ]	101.0M	4.62G	19.53s	39.5
+ CTA	100.9M	4.63G	19.47s	40.0
+ KD-MoE	110.8M	4.60G	19.40s	40.3

TABLE VI : Comparison of computational efficiency and recognition performance.

Fig. 9 : Representative qualitative examples of attribute activation.

Fig. 10 : Case studies involve the baseline method and our proposed MoeCo-T. Ground truth annotations are displayed on the images. Correct predictions are highlighted in green, and incorrect ones are in red.

Fig. 11 :

AP_{IVT}

under different fusion weight of

\alpha

and

\lambda

Split Criterion	$\bm{AP_{I}}$	$\bm{AP_{V}}$	$\bm{AP_{T}}$	$\bm{AP_{IVT}}$
Ours w/o CGL	90.7	71.0	57.1	41.2
$>5\%/<0.5\%$	91.0	71.8	57.6	41.6
$>8\%/<0.8\%$	91.8	72.5	58.0	42.3
$>10\%/<1.0\%$	91.3	72.5	57.7	42.0

TABLE VII : Sensitivity analysis (

{AP_{IVT}}

) of CGL under different relative head/tail split criteria.

Surgical Robotics

Surgical robotics develops robotic tools and systems for minimally invasive, image-guided, and microsurgical procedures.

See all 189 papers

TRCoRSurg: Temporal-Relational Co-Reasoning for Surgical Video Triplet Recognition
Aug 5, 2026Fang Li, Shihao Zou, Weixin Si +3Surgical Video Understanding Relational Deep Learning
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology
SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition
Aug 3, 2026Saurav Sharma, Lorenzo Arboit, Nabani Banik +11Domain Generalization Relational Reasoning
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France · IHU Strasbourg, France · University of Pavia, Italy +6
Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
May 13, 2026Jincai Huang, Shihao Zou, Yuchen Guo +5Surgical Video Understanding Vision-Language Grounding
Southern University of Science and Technology · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Northwestern University +4

Aug 5, 2026J/K move · Enter open · S save

TRCoRSurg: Temporal-Relational Co-Reasoning for Surgical Video Triplet Recognition

Fang Li, Shihao Zou, Weixin Si, Yang Gao+2

State Key Laboratory of Virtual Reality Technology and Systems, Beihang University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology

Surgical Video Understanding Relational Deep Learning Temporal Attention Temporal Video Understanding

Open paper AskPDF arXiv

Understanding complex surgical scenes requires recognizing multiple interdependent entities, such as instruments, actions, and targets, while maintaining their relational consistency across time. Existing surgical triplet recognition methods struggle to jointly model intra-frame label dependencies and inter-frame temporal semantics in a unified manner. To address these limitations, we propose a unified framework that integrates spatial, relational, and temporal cues for robust surgical triplet recognition. Specifically, class-specific spatial priors are first extracted through a multi-scale encoder. These priors are then refined by a Label Correlation Modeling module with multi-scale class activation map-guided relational extraction (MS-CAMRE), enabling the model to capture both static co-occurrence patterns and dynamic contextual dependencies among triplet components. Furthermore, a Bidirectional Temporal-Relational Fusion Attention (BTRFA) module harmonizes temporal and relational representations to achieve coherent temporal reasoning. We also introduce a new evaluation metric, the Triplet Consistency Error Rate (TCER), which quantitatively measures the model's ability to preserve causal and semantic consistency across triplets. Extensive experiments on the CholecT45 and ProstaTD datasets show that our method achieves state-of-the-art performance, improving AP_IVT by 5.1 percent and 7.8 percent, respectively. Moreover, according to TCER, our approach achieves relative reductions of more than 36 percent and 25 percent on the two datasets, respectively, demonstrating the effectiveness of our framework in temporal-relational co-reasoning.

Abstract

Surgical Robotics

Explore similar work

TRCoRSurg: Temporal-Relational Co-Reasoning for Surgical Video Triplet Recognition

SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition

Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs