Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

Yiqing Shen; Mathias Unberath

cs.CVOct 20, 2026

Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

Authors: Yiqing Shen, Mathias Unberath

Organizations: Department of Computer Science, Johns Hopkins University

Abstract

Conventional surgical segmentation relies on closed-set categories or explicit text queries, which cannot resolve implicit queries that require reasoning about workflows, instrument functions, and temporal relationships. To address these limitations, we propose SurgReasonSeg, a complexity-aware method that brings reasoning segmentation over digital twin (DT) representations to surgical video. Specifically, SurgReasonSeg constructs DT representations from surgical foundation models and trains two low-rank adapters (LoRA) of one large language model with reinforcement learning (RL), each specialized for a different query complexity. A lightweight router, also trained with RL, selects between the two adapters to balance latency against accuracy. SurgReasonSeg also sets the temporal context with an adaptive sliding window conditioned on query complexity and workflow phase, and retrieves relevant historical cases from a surgical context memory by workflow similarity and procedural relevance. We further introduce EndoVis-ReasonSeg, a surgical reasoning segmentation benchmark with 1,460 training and 961 test samples across semantic, spatial, and temporal categories. On EndoVis-ReasonSeg, SurgReasonSeg improves J&F over the strongest baseline, ReSurgSAM2, by 0.178, 0.237, and 0.297 on semantic, spatial, and temporal queries, and it also achieves the best results on the Ref-EndoVis17 and Ref-EndoVis18 referring segmentation benchmarks. A backbone-matched attribution protocol separates the contribution of the upstream segmenter from that of the reasoning modules. Our method runs at 18.8 FPS averaged over the query distribution, enabling real-time reasoning segmentation in surgical videos.

Figures & tables

Component	Origin	Status
DT-as-JSON and index retrieval	[ 22 , 23 , 24 ]	Inherited
Reasoning segmentation on surgical DTs	[ 21 ]	Inherited
Surgical DT from foundation models	[ 4 , 5 ]	Adapted
Two LoRA adapters with an RL router	This paper	New
Adaptive sliding window	This paper	New
Surgical context memory	This paper	New

Table 1: SurgReasonSeg inherits the DT encoding and adds four components. Adapted: a prior component modified for surgery.

Figure 1: Overview of SurgReasonSeg. Given a surgical video and an implicit text query, our method constructs DT representations from surgical foundation models, applies an adaptive sliding window for temporal context selection, and routes each query to the LoRA adapter matched to its complexity, which generates the reasoning chain.

Figure 2: Dataset composition and adaptive window sizes. The left panel shows the number of EndoVis-ReasonSeg samples per reasoning category in the training and test splits, and the right panel shows the distribution of adaptive sliding window sizes across query types, with diamond markers indicating means. Temporal queries during phase transitions receive the largest windows (mean 6.1 frames), while semantic queries remain close to

w_{\text{base}}=2

		Semantic			Spatial			Temporal
Method	Type	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	FPS $\uparrow$
LISA [ 11 ]	Reas.	.556 $\pm$ .031	.568 $\pm$ .024	.562 $\pm$ .025	.201 $\pm$ .042	.213 $\pm$ .035	.207 $\pm$ .036	.238 $\pm$ .039	.251 $\pm$ .033	.245 $\pm$ .034	1.33
VISA [ 31 ]	Reas.	.528 $\pm$ .023	.547 $\pm$ .021	.538 $\pm$ .019	.457 $\pm$ .032	.481 $\pm$ .028	.469 $\pm$ .027	.319 $\pm$ .037	.338 $\pm$ .031	.329 $\pm$ .032	1.87
LLM-Seg [ 28 ]	Reas.	.387 $\pm$ .034	.394 $\pm$ .031	.391 $\pm$ .029	.245 $\pm$ .043	.258 $\pm$ .038	.252 $\pm$ .037	.176 $\pm$ .048	.189 $\pm$ .041	.183 $\pm$ .042	0.29
SegZero [ 16 ]	Reas.	.509 $\pm$ .025	.526 $\pm$ .022	.518 $\pm$ .021	.398 $\pm$ .033	.421 $\pm$ .029	.410 $\pm$ .028	.312 $\pm$ .038	.331 $\pm$ .032	.322 $\pm$ .033	0.11
CoReS [ 2 ]	Reas.	.560 $\pm$ .022	.580 $\pm$ .019	.570 $\pm$ .018	.439 $\pm$ .028	.463 $\pm$ .024	.451 $\pm$ .023	.351 $\pm$ .033	.372 $\pm$ .029	.362 $\pm$ .028	0.27

Table 2: SurgReasonSeg achieves the best

\mathcal{J}

\mathcal{F}

, and

\mathcal{J}\&\mathcal{F}

on all three reasoning categories of EndoVis-ReasonSeg. Reas./Ref.: reasoning/referring methods;

\pm

: standard deviation over three random seeds; the FPS of SurgReasonSeg includes DT construction and is averaged over routing. Bold: best; underline: second best.

Figure 3: Accuracy, latency, and component analysis. The left panel plots the average

\mathcal{J}\&\mathcal{F}

over the three reasoning categories against FPS for all methods, where SurgReasonSeg (star) gives the best balance between reasoning accuracy and speed. The middle panel breaks down the latency of the

L_{\text{simple}}

and

L_{\text{complex}}

inference paths per component on a single RTX 4090 GPU. The right panel shows the drop in

\mathcal{J}\&\mathcal{F}

when the router, ASW, or SCM is removed, and each of the three components has its largest effect on temporal reasoning.

	Ref-EndoVis17 [ 14 ] (tool)			Ref-EndoVis18 [ 14 ] (tool)			Ref-EndoVis18 [ 14 ] (tissue)
Method	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$
LISA [ 11 ]	.388 $\pm$ .032	.420 $\pm$ .029	.355 $\pm$ .034	.293 $\pm$ .041	.284 $\pm$ .043	.303 $\pm$ .037	.237 $\pm$ .044	.244 $\pm$ .042	.229 $\pm$ .045
VISA [ 31 ]	.507 $\pm$ .022	.479 $\pm$ .026	.535 $\pm$ .019	.431 $\pm$ .028	.398 $\pm$ .031	.464 $\pm$ .024	.384 $\pm$ .031	.354 $\pm$ .033	.414 $\pm$ .027
LLM-Seg [ 28 ]	.307 $\pm$ .039	.177 $\pm$ .047	.437 $\pm$ .031	.177 $\pm$ .048	.105 $\pm$ .052	.249 $\pm$ .041	.153 $\pm$ .049	.103 $\pm$ .053	.203 $\pm$ .043
SegZero [ 16 ]	.468 $\pm$ .027	.388 $\pm$ .032	.548 $\pm$ .022	.388 $\pm$ .031	.349 $\pm$ .034	.427 $\pm$ .028	.349 $\pm$ .033	.321 $\pm$ .035	.377 $\pm$ .030
CoReS [ 2 ]	.495 $\pm$ .023	.415 $\pm$ .028	.575 $\pm$ .019	.415 $\pm$ .027	.377 $\pm$ .030	.453 $\pm$ .024	.377 $\pm$ .029	.351 $\pm$ .031	.403 $\pm$ .026

Table 3: SurgReasonSeg achieves the best

\mathcal{J}\&\mathcal{F}

on all three referring segmentation settings.

\pm

: standard deviation over three random seeds; bold: best; underline: second best.

Semantic

Spatial

Temporal

Router

ASW

SCM

\mathcal{J}\uparrow

\mathcal{F}\uparrow

\mathcal{J}\&\mathcal{F}\uparrow

\mathcal{J}\uparrow

\mathcal{F}\uparrow

\mathcal{J}\&\mathcal{F}\uparrow

\mathcal{J}\uparrow

\mathcal{F}\uparrow

\mathcal{J}\&\mathcal{F}\uparrow

FPS

✗

.587

\pm

.031

.601

\pm

.028

.594

\pm

.027

.445

\pm

.038

.468

\pm

.033

.457

\pm

.033

.382

\pm

.041

.405

\pm

.037

.394

\pm

.036

1.5

✓

✗

✓

.718

\pm

.021

.734

\pm

.018

.726

\pm

.017

.651

\pm

.024

.675

\pm

.021

.663

\pm

.020

.578

\pm

.027

.602

\pm

.023

.590

\pm

.022

12.5

✓

✗

✓

.738

\pm

.018

.754

\pm

.015

.746

\pm

.014

.678

\pm

.022

.702

\pm

.019

.690

\pm

.018

.589

\pm

.025

.613

\pm

.022

.601

\pm

.021

17.8

✓

✗

.745

\pm

.015

.761

\pm

.013

.753

\pm

.012

.685

\pm

.020

.710

\pm

.017

.698

\pm

.016

.598

\pm

.023

.623

\pm

.020

.611

\pm

.019

20.0

✓

.762

\pm

.013

.779

\pm

.011

.771

\pm

.010

.704

\pm

.018

.729

\pm

.014

.717

\pm

.013

.641

\pm

.021

.667

\pm

.018

.654

\pm

.017

18.8

Table 4: Removing any component lowers

\mathcal{J}\&\mathcal{F}

, most on temporal queries. Router: routing between two LoRA adapters; ASW: adaptive sliding window; SCM: surgical context memory;

\pm

: standard deviation over three random seeds. Bold: best; underline: second best; FPS is a cost and is not ranked.

Figure 4: Qualitative comparison of reasoning segmentation on EndoVis-ReasonSeg. Each row shows the predictions of one method on five consecutive frames, with the frame index at the top left and the $\mathcal{J}$ score at the top right of each frame. Ground truth is shown in green and predictions in purple, and SurgReasonSeg obtains the highest $\mathcal{J}$ on every frame of the three examples.

Figure 5: Routing analysis. The left panel compares

\mathcal{J}\&\mathcal{F}

across reasoning categories for the single-adapter variants and full routing, with the FPS of each variant in the legend. The right panel shows the percentage of queries routed to

L_{\text{simple}}

and

L_{\text{complex}}

in each category, with the dashed line indicating the overall average.

Setting	Masks	Semantic $\uparrow$	Spatial $\uparrow$	Temporal $\uparrow$
ReSurgSAM2 [ 14 ]	own	.593	.480	.357
ReSurgSAM2 [ 14 ]	DT	.607	.492	.371
LISA [ 11 ]	DT	.572	.224	.258
ReSurgSAM2, fine-tuned	own	.621	.515	.388
Base LLM, zero-shot	DT	.618	.442	.379
SurgReasonSeg (Ours)	DT	.771	.717	.654

Table 5: Matched masks and fine-tuning explain at most 16% of the margin. Masks: the method’s own masks or the DT masks of SurgReasonSeg; values are

\mathcal{J}\&\mathcal{F}

unless noted. Bold: best; underline: second best; the last block is not ranked.

	Semantic	Spatial	Temporal	Latency
$\|\mathcal{M}\|$	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	(ms)
0	.753 $\pm$ .012	.698 $\pm$ .015	.611 $\pm$ .019	0.0
100	.762 $\pm$ .011	.709 $\pm$ .014	.637 $\pm$ .018	0.5
250	.768 $\pm$ .011	.714 $\pm$ .014	.649 $\pm$ .018	0.8
500	.771 $\pm$ .010	.717 $\pm$ .013	.654 $\pm$ .017	1.2
1000	.770 $\pm$ .011	.716 $\pm$ .014	.653 $\pm$ .017	2.1

Table S1 :

\mathcal{J}\&\mathcal{F}

of SurgReasonSeg (Ours) peaks at 500 memory entries.

\pm

: standard deviation over three seeds; latency is a cost and is not ranked.

LLM	Semantic	Spatial	Temporal
backbone	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$	FPS
Qwen3-0.6B	.721 $\pm$ .016	.668 $\pm$ .019	.598 $\pm$ .024	24.7
Qwen3-1.7B	.748 $\pm$ .013	.695 $\pm$ .016	.631 $\pm$ .020	19.5
Qwen3-4B (Ours)	.771 $\pm$ .010	.717 $\pm$ .013	.654 $\pm$ .017	18.8
Qwen3-8B	.774 $\pm$ .010	.720 $\pm$ .013	.659 $\pm$ .017	8.6

Table S2 : Qwen3-8B adds at most 0.005

\mathcal{J}\&\mathcal{F}

over Qwen3-4B at less than half the FPS.

\pm

: standard deviation over three seeds; FPS is end-to-end, a cost, and not ranked.

Adapters	Router	$v_{j}$	Semantic $\uparrow$	Spatial $\uparrow$	Temporal $\uparrow$
SFT	RL	Judge	.742	.688	.615
RL	Rule	Judge	.731	.706	.643
RL	Classifier	Judge	.762	.710	.648
RL	RL	Uniform	.758	.702	.640
RL	RL	Judge	.771	.717	.654
RL	RL	GT IoU	.775	.721	.658

Table S3 : Every design choice of SurgReasonSeg (Ours) outperforms its replacement, and the VLM judge is within 0.004 of the ground-truth IoU bound. Adapters: RL or supervised fine-tuning (SFT); Router: RL, query-category rule, or supervised classifier;

v_{j}

: VLM judge, uniform weighting, or ground-truth IoU; gray: as in the full model. Values are

\mathcal{J}\&\mathcal{F}

; bold: best; underline: second best; the ground-truth IoU row is an upper bound and is not ranked.

Sequence	Source	Semantic	Spatial	Temporal
seq-1	EndoVis17	.728	.663	.591
seq-2	EndoVis17	.743	.682	.613
seq-3	EndoVis17	.763	.707	.642
seq-4	EndoVis18	.776	.723	.661
seq-5	EndoVis18	.789	.740	.680
seq-6	EndoVis18	.795	.747	.689

Table S4 : Every test video stays above the strongest control of the main paper. Per-sequence

\mathcal{J}\&\mathcal{F}

of SurgReasonSeg.

	Semantic	Spatial	Temporal	All
Templates	20	15	12	47
Held-out templates	4	3	3	10
Test samples, held out	85	68	52	205
$\mathcal{J}\&\mathcal{F}$ , held out	.746	.692	.628	.698

Table S5 : Ten of the 47 templates are held out and cover 205 test samples. The last row is the template-disjoint model; All is weighted by sample count.

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

Sep 18, 2026

SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment

Surgical instrument segmentation (SIS) plays a critical role in robotic assistance and surgical workflow analysis. However, most existing SIS methods formulate segmentation as a category-driven localization problem, limiting their ability to capture procedural context and task-dependent semantics in surgical workflows. We introduce Reasoning-Aware Surgical Instrument Segmentation (RA-SIS), a task formulation that frames segmentation as query-conditioned inference under surgical context. To benchmark this setting, we construct SurgRS, a surgical reasoning segmentation dataset consisting of 41,000 image-text pairs, which aligns instance-level masks with structured query-answer supervision to enable semantic grounding at the pixel level. Based on SurgRS, we propose Surgical Instrument Reasoning and Segmentation Assistant (SIRA), a multimodal framework that disentangles target-level and query-level semantics and integrates them with visual features through query-anchored dual alignment. By aligning query semantics with spatial features and segmentation prompts, SIRA enhances semantic-visual consistency in mask prediction. Extensive experiments on SurgRS demonstrate improvements over existing reasoning-aware baselines. Code is available at https://github.com/linxir226/SIRA.

Zhibo Zhang, Qijie Wang, Zengqiang Yan

School of Electronic Information and Communications, Huazhong University of Science and Technology

Medical Image Segmentation Reasoning Segmentation Instance Segmentation Text-Guided Image Segmentation Cross-Modal Alignment

Ask PDF

Jun 15, 2026

Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA

Surgical video question answering requires multi-step reasoning across semantic, spatial, and temporal dimensions. Existing methods architecturally compress videos into discrete token representations and couple visual perception with reasoning. This approach fragments continuous spatial-temporal relationships and has been shown to restrict multi-step reasoning capabilities. We introduce a reinforcement learning (RL) framework that trains large language models (LLMs) to decouple perception from reasoning by operating over digital twin representations constructed from surgical foundation models. Additionally, we introduce hierarchical representations across frame, temporal window, and procedure levels with probabilistic uncertainty estimates. Finally, we propose a novel reward that combines format validation with accuracy assessment through clinical plausibility evaluation and uncertainty-aware calibration for training. To demonstrate the capabilities of this approach, we introduce REAL-Colon-Reason, a colonoscopic benchmark with 2000 question-answer pairs across three complexity levels. We achieve state-of-the-art performance on REAL-Colon-Reason and two existing surgical VideoQA benchmarks REAL-Colon-VQA and EndoVis18-VQA.

Yiqing Shen, Han Zhang, Mathias Unberath

Johns Hopkins University, Baltimore, MD, USA

RL for Language Model Reasoning Medical VQA Video Reasoning Digital Twins

Ask PDF

Aug 25, 2026

ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation

Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression. Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking. Although effective, their performance is highly sensitive to the quality of the initial grounded mask: once an incorrect anchor is selected, subsequent tracking tends to propagate the error. This issue is especially challenging in surgical videos due to visually similar instruments, occlusion, and complex tissue-tool interactions. To address this issue, we propose ReGround-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation. It first predicts a text-conditioned spatial reliability map from the referring expression and current-frame visual features. The map is then reused in two complementary branches: a Gated Side Adapter enhances expression-relevant visual regions before text-to-vision fusion, while a Reliability-Weighted Vision-to-Text Attention module suppresses off-target visual evidence during prompt-token aggregation. Experiments on Ref-EndoVis17 and Ref-EndoVis18 show consistent improvements over state-of-the-art methods across three evaluation splits with negligible speed reduction. Code is publicly available at https://github.com/JiaxinWen1/ReGround-Surg.

Jiaxin Wen, Ming Yin, Lu Liu +1

University of Exeter, Exeter, UK

Referring Video Object Segmentation Surgical Video Understanding Visual Grounding

Ask PDF

Abstract

Surgical Video Understanding

Explore similar work

SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment

Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA

ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation