SurgQ-VLM: Question-Conditioned Dual-Stream Compression and Answer-Type-Aware Reinforcement for Surgical Video Question Answering

Dhruv Makwana; Ananya Mantravadi

cs.CVNov 1, 2026

SurgQ-VLM: Question-Conditioned Dual-Stream Compression and Answer-Type-Aware Reinforcement for Surgical Video Question Answering

Authors: Dhruv Makwana, Ananya Mantravadi

Organizations: Centific Global Solutions

Abstract

Reasoning about surgical video -- which phase is underway, which instruments act, and when events occur -- underpins operative documentation, workflow analysis, and skill assessment. Yet surgical visual question answering is still largely built on single frames and closed labels that cannot express a duration, an ordering, or a count, and the video models that lift the frame limit typically compress every clip the same way regardless of the question, discarding the temporal evidence the answer needs. Worse, the answers are structurally heterogeneous, spanning five incompatible structures -- binary, classification, multi-label, counting, and sentence-temporal -- that no single training signal scores uniformly well. We propose SurgQ-VLM, a two-stage model for surgical video question answering. Stage 1 trains a question-conditioned dual-stream (slow/fast) compressor for a LoRA-adapted Qwen2.5-VL-7B: cross-attention conditions both streams on the question, and each clip's visual tokens shrink 49x from the raw ViT patches (11x vs. post-merge backbone tokens). Stage 2 post-trains with GRPO under an answer-type-aware reward that scores each question on the metric matching its answer structure (micro-F1 for label sets, normalised error for counts, a content-timing blend for temporal windows). On a sealed 21,840-question test split over Cholec80 and CholecT50, SurgQ-VLM reaches 0.779 keyword accuracy and a 10x BLEU-4 gain over zero-shot; GRPO adds 2.5 pp K-Acc (2.0 pp attributable to the answer-type-aware reward), concentrated in the structurally hard types, with answer format unchanged (ROUGE-L). Code is available at https://github.com/DhruvMakwana/SurgQ-VLM.

Figures & tables

Figure 1: The same 30-second laparoscopic clip requires different temporal evidence for different questions. Prior methods ( a ) compress without question context, producing incomplete or mistimed answers. SurgQ-VLM ( b ) conditions both streams on the question via four independent GuideInjectors, then refines with GRPO’s answer-type-aware reward.

Figure 2: SurgQ-VLM pipeline. Dual-stream sampling (slow: 0.5 FPS/15f; fast: 1.0 FPS/30f) is encoded by a frozen Qwen2.5-VL ViT, then compressed by four question-conditioned GuideInjectors into 671 tokens (405 slow-local $+$ 250 fast-local $+$ 16 global) before a LoRA-adapted LLM. Blue: frozen; green/orange: slow/fast streams; purple: global queries; gold: GuideInjectors. Bottom panel: two-stage training.

Answer structure	Q-types	Test QA	Metric
Binary	Q02, Q06, Q07, Q13, Q17	7,120	Bin. accuracy
Classification	Q01, Q10, Q14, Q15	3,522	K-Acc
Multi-label	Q05, Q11	2,446	Micro-F1
Counting	Q12	542	Exact-match
S-temporal	Q03, Q04, Q08, Q09, Q16, Q18	8,210	K-Acc, T-Acc ±2
Total	Q01–Q18 (18 types)	21,840	—

Table 1: Benchmark distribution across the five answer structures. Q-type inverse-frequency weighting compensates for the imbalance between binary and counting questions.

Method	Backbone	Type	#Fr.	#Tok.	K-Acc	Primary	BLEU-4	ROUGE-L	METEOR	CIDEr	T-Acc ±2
Zero-shot
Qwen2.5-VL (backbone)	7B	ZS	45	7,605	0.220	0.293	0.057	0.255	0.193	0.260	0.510
uAI-NEXUS MedVLM [ 34 ]	7B	ZS	8	1,352	0.398	0.376	0.038	0.214	0.183	0.125	0.338
Fine-tuned on our SFT split
LLaVA-OneVision-7B [ 15 ]	Qwen2	+LoRA	8	1,568	0.653	0.621	0.481	0.759	0.703	n/a	n/a
SurgVidLM [ 37 ]	Qwen2-VL	+LoRA	30	540	0.740	0.710	0.554	0.751	0.710	n/a	n/a

Table 2: Results on the sealed 21,840-QA test split; our final model in bold.

\dagger

Backbone-only row also serves as the architecture ablation. n/a : not logged for baseline/ablation runs; --- : not applicable.

Answer structure	Metric	SFT	GRPO	$\Delta$
Binary †	B-Acc	0.727	0.727	$0.000$
Classification	K-Acc	0.818	0.818	$0.000$
Multi-label	F1	0.766	0.778	$+$ 0.012
Counting	Exact	0.328	0.354	$+$ 0.026
Sent.-temporal	K-Acc	0.690	0.741	$+$ 0.051
Primary (wtd. avg.)	—	0.722	0.743	$+$ 0.021

Table 3: GRPO gains by answer structure (SFT

\,{\to}\,

SFT+GRPO, sealed test). Bold : improvement

>

1 pp;

\dagger

: Q13 regression (

-3.9

pp) discussed in text. Counting has

n{=}542

; its

\Delta

is within sampling error. Sentence-temporal scored by K-Acc; its saturating T-Acc

{}_{\pm 2\text{s}}

: Tab. 2 , supp. §A.7.

Reward design
Variant	Removed	K-Acc	Binary	Classif.	Multi-lbl	Count	S-Temp	ROUGE-L
Full SurgQ-VLM (SFT+GRPO)	—	0.779	0.799	0.818	0.820	0.645	0.741	0.803
$-$ Answer-type dispatch ( $R_{1}$ uniform) $\star$	per-structure reward	0.759	0.799	0.819	0.784	0.627	0.699	0.803
Architecture
$-$ Compressor (backbone-only)	dual-stream compressor	0.746	0.785	0.808	0.742	0.611	0.656	0.707
$-$ Question conditioning	GuideInjectors	0.673	0.756	0.793	0.755	0.576	0.529	0.733

Table 4: Core ablations on the 7B SurgQ-VLM, grouped by the component removed. Every removal lowers the headline K-Acc below the full model (0.779); the

-

Compressor and

-

GRPO rows are SFT-only (SFT reference: 0.754). Group columns report per-group K-Acc (structure metrics: Tab. 3 ).

\star

central contribution. The “

-

Question conditioning” row is a dual-stream unconditioned baseline (SlowFast-LLaVA-style [ 40 ] ) at matched 671-token budget.

Figure 3: Qualitative comparison (sealed test set). Two frames, GT, ZS, SFT, and GRPO per row. Rows 1–3: GRPO recovers on Q03, Q12, and Q05 where ZS and SFT fail. Row 4: all models err on Q13 (failure case).

Question group	Q	$-$ Slow	$-$ Fast	$-$ Global
Phase	Q03	$+0.3$	$\mathbf{-22}$	$-0.7$
Instrument	Q07	$-0.5$	$\mathbf{-2.6}$	$+0.3$
Triplet/action	Q12	$\mathbf{-2.1}$	$-0.7$	$-0.6$
Binary action	Q13	$-1.7$	$-0.6$	$\mathbf{-2.7}$
Temporal	Q16	$-2.3$	$\mathbf{-12.7}$	$+0.3$

Table 5: Stream specialisation: K-Acc change (pp) when each token group is zeroed at inference (7B+GRPO model). Most-critical component per row in bold . Most diagnostic Q-type shown per group; full breakdown in supplementary.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Q	Name	Group	Answer struct.	Annotation source	Temporal	Coverage
Q01	Current phase	Phase	Classification	phase majority vote	No	Both
Q02	Phase change	Phase	Binary	phase change flag	No	Both
Q03	Phase transition time	Phase	Sent.-temporal	phase segment boundary second	Yes	Both
Q04	Phase sequence	Phase	Sent.-temporal	phase segments, ordered	Yes	Both
Q05	Active instrument list	Instrument	Multi-label	instrument union across clip	No	Both
Q06	Instrument presence	Instrument	Binary	per-instrument membership	No	Both

Appendix

Table A1 : All 18 question types. Coverage : Both = Cholec80 + CholecT50; T50 = CholecT50 only. Temporal : question or answer references a specific second value.

Group	Q	Example question	Ground-truth answer
Phase	Q01	Which phase of the cholecystectomy procedure is shown here?	The clip primarily shows calot triangle dissection.
	Q02	Does the surgical phase change at any point during this clip?	No, the surgical phase remains constant throughout this clip.
	Q03	How many seconds into the clip does the surgical phase change?	The phase changes from gallbladder packaging to cleaning coagulation at second 18 of the clip.
	Q04	Which phases appear in this clip and in what order?	gallbladder packaging from second 0 to second 17, then cleaning coagulation from second 18 to the end of the clip.
Instrument	Q05	What instruments are active at any point in this clip?	hook, grasper
	Q06	Is the grasper used at any point in this clip?	Yes, the grasper is used in this clip.

Appendix

Table A2 : One representative example per Q-type (Q01–Q18), verbatim from the sealed test split. Q03/Q04 are from the same clip. Q10–Q15 and Q18 require CholecT50 triplet annotations. Q07 gold answers embed a second value but the Q-type is scored by binary accuracy; Q08 is scored by T-Acc

{}_{\pm 2\text{s}}

. Q09 answers are instrument lists with no second embedded, so their T-Acc is vacuously 1 ( Appendix A.3 ).

Q	Name (group)	SFT	GRPO	$\Delta$ (pp)
Q01	Current phase (cls)	0.8163	0.8186	$+0.23$
Q02	Phase change (bin)	0.6694	0.6612	$-0.82$
Q03	Phase transition time (s-tmp)	0.8793	0.8793	$0.00$
Q04	Phase sequence (s-tmp)	0.9253	0.9253	$0.00$
Q05	Active instrument list (mul)	0.7738	0.7896	$+1.58$
Q06	Instrument presence (bin)	0.8623	0.8623	$0.00$

Appendix

Table A3 : Per-Q-type primary metric, SFT vs. GRPO (sealed test split,

n{=}21{,}840

). Metric per group: binary accuracy (bin), classification K-Acc (cls), multi-label F1 (mul), counting exact-match (cnt), sentence-temporal T-Acc

{}_{\pm 2\text{s}}

(s-tmp).

Variant	K-Acc	Binary	Classif.	Multi-lbl	Counting	S-Temp
Full $R=R_{1}+0.3\,R_{2}+0.05\,R_{3}-0.3\,R_{\text{len}}$	0.779	0.799	0.818	0.820	0.645	0.741
$R_{1}$ only (strip $R_{2}$ , $R_{3}$ , $R_{\text{len}}$ )	0.760	0.799	0.820	0.788	0.630	0.700
$R_{1}+R_{2}$ (strip $R_{3}$ , $R_{\text{len}}$ )	0.761	0.799	0.820	0.786	0.633	0.703
$R_{1}+R_{3}$ (strip $R_{2}$ , $R_{\text{len}}$ )	0.760	0.799	0.819	0.788	0.629	0.701
No length penalty ( $R_{\text{len}}=0$ )	0.760	0.798	0.820	0.787	0.636	0.701
No Q-type IFW in reward (uniform weighting)	0.765	0.799	0.820	0.794	0.628	0.712

Appendix

Table A4: Full reward decomposition on the sealed test split (

n{=}21{,}840

). Each row removes or replaces one component from the full GRPO reward; all other hyperparameters unchanged. Group columns report per-group K-Acc.

Figure A1 : GRPO training dynamics (seed 42, 3,256 steps). Left: primary reward

R_{1}

(20-step rolling average) rises then plateaus at

{\approx}0.79

. Right: KL divergence from the reference policy stays bounded (peak 0.73 nats) with no collapse. Both curves confirm stable optimisation without reward hacking.

Q	Full	$-$ slow	$-$ fast	$-$ global	$\Delta$ slow	$\Delta$ fast	$\Delta$ global
Q01	0.8201	0.8199	0.7932	0.8194	$-0.0002$	$-0.0269$	$-0.0007$
Q02	0.6694	0.6557	0.5082	0.6585	$-0.0137$	$\mathbf{-0.1612}$	$-0.0109$
Q03	0.7680	0.7706	0.5485	0.7611	$+0.0026$	$\mathbf{-0.2195}$	$-0.0069$
Q04	0.8285	0.8293	0.7350	0.8289	$+0.0008$	$\mathbf{-0.0935}$	$+0.0004$
Q05	0.8276	0.8236	0.8201	0.8275	$-0.0040$	$-0.0075$	$-0.0001$
Q06	1.0000	1.0000	1.0000	1.0000	$0.0000$	$0.0000$	$0.0000$

Appendix

Table A5: Stream zeroing, per-Q-type K-Acc (7B + GRPO, sealed test split). Each

\Delta

column is that condition minus Full. Largest-magnitude

\Delta

per row in bold .

Model	$\pm 1$ s	$\pm 2$ s	$\pm 3$ s
7B SFT	0.860	0.874	0.884
7B + GRPO	0.876	0.888	0.899
$\Delta$	$+0.016$	$+0.014$	$+0.015$

Appendix

Table A6: Temporal accuracy at

\pm 1

\pm 2

\pm 3

s on sentence-temporal Q-types (

n{=}8{,}210

sentence-temporal test QA pairs).

Metric	Dispatch (ours)	$+$ per-group norm.
K-Acc	0.779	0.779
Primary	0.743	0.743
ROUGE-L	0.803	0.811
Per-group K-Acc $\Delta$ (norm. $-$ dispatch)
Binary	—	$-0.3$
Classification	—	$-0.2$

Appendix

Table A7: Answer-type dispatch (our full model) vs. MedGRPO-style per-group reward normalisation [ 34 ] , both warm-started from the SFT checkpoint with a matched one-epoch budget (sealed test split,

n{=}21{,}840

Figure A2 : Additional qualitative examples across six Q-types (sealed test set). Rows 1–4 cover binary and classification Q-types (Q01, Q06, Q10, Q14); rows 5–6 cover sentence-temporal Q-types (Q16, Q08). Each row shows two representative frames, the question, ground truth, and predictions from zero-shot (ZS), SFT, and SFT $+$ GRPO. ZS fails on all six through systematic errors — wrong phase names, wrong polarity, wrong instrument or action, vague temporal descriptions with no named content. For the binary and classification rows, SFT already matches the ground truth and GRPO adds no further gain, consistent with the near-zero deltas in Tab. 3 of the main paper. For the sentence-temporal rows, SFT learns the surface format but fills it with wrong phase names, wrong instruments, or wrong durations; GRPO recovers the fully correct structured answer in both cases.

Figure A3 : Frame-count robustness. (a) Fast stream (slow=15 fixed): Q03 drops 7.5 pp at fast=5; K-Acc flat above 10 frames. (b) Slow stream (fast=30 fixed): K-Acc varies by only 0.77 pp across 3–15 frames.

Surgical VQA

See all 8 papers

May 18, 2026

SurgLQA: Scalable Long-Horizon Surgical Video Question Answering

Surgical Video Question Answering (VideoQA) provides a promising paradigm for dynamic intraoperative interpretation, enabling real-time decision support and context-aware retrieval in clinical environments. Nevertheless, existing approaches are predominantly restricted to images or short clips, limiting their ability to model long-range procedural dynamics and causal dependencies across extended surgical workflows. To address this challenge, we propose SurgLQA, a unified long-horizon VideoQA framework for scalable surgical reasoning. This framework incorporates Faithful Temporal Consolidation (FTC), which leverages intrinsic temporal cues to construct compact long-range representations while preserving fine-grained temporal fidelity. Further, we develop Temporally-Grounded Multi-Policy Scaling (TMS), an adaptive test-time inference paradigm that strategically adjusts policy-level reasoning capacity within temporally grounded contexts. To facilitate systematic evaluation, we restructured a long-duration colonoscopy VideoQA benchmark, Colon-LQA, and conducted extensive experiments on Colon-LQA and REAL-Colon-VQA. Experimental results demonstrate that our approach achieves consistent performance gains in long-range reasoning with temporally grounded inference. Code link: https://github.com/RascalGdd/SurgLQA.

Diandian Guo, Xikai Yang, Ruiyang Li +2

The Chinese University of Hong Kong

Temporal Video Grounding Test-Time Scaling Medical Visual Question Answering Surgical Video Understanding Long-Video QA

Ask PDF

Sep 8, 2026

STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs

Despite recent advances in surgical vision-language models (VLMs), temporal reasoning remains limited because existing supervision is largely frame-centric. Frame-level scene graphs (SGs) have proven effective in providing structured representations of surgical environments but do not explicitly model the dynamics of surgical workflows. To explicitly model how surgical states evolve across time, we introduce a multi-level structured temporal supervision methodology that augments frame-level surgical SGs with object-level continuity, event-level interaction continuity, and procedure-level connectivity. We then execute temporal queries over the resulting spatio-temporal scene graphs (STSGs) to generate evidence-grounded question-answer pairs, which together form the STSG-VQA benchmark. Each question is linked to the temporal interval and STSG evidence used to derive its reference answer, enabling traceable verification. The benchmark contains 18,458 question-answer pairs across seven temporal categories. Fine-tuning Qwen3-VL-4B and Hulu-Med-4B with STSG-derived supervision improves question-level micro accuracy by 24.39 and 19.56 percentage points over their zero-shot baselines and by 16.50 and 14.25 points over static scene-graph supervision, respectively. These gains span all temporal categories, indicating that STSG-derived supervision helps surgical VLMs reason over temporally grounded interactions rather than isolated frames. The code and dataset will be made publicly available upon acceptance.

Jing Li, Duygu Sarikaya

School of Computer Science, University of Leeds, UK

Medical Visual Question Answering Question Answering Surgical Video Understanding Temporal QA Temporal Video Understanding

Ask PDF

Mar 31, 2026

SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomy

Surgical procedures are inherently complex and risky, requiring extensive expertise and constant focus to navigate evolving intraoperative scenes. Computer-assisted systems such as surgical visual question answering (VQA) offer promises for education and intraoperative support. Current surgical VQA research largely focuses on static frame analysis, overlooking rich temporal semantics. Surgical video question answering is further challenged by low visual contrast, its highly knowledge-driven nature, diverse analytical needs spanning scattered temporal windows, and the hierarchy from basic perception to high-level intraoperative assessment. To address these challenges, we propose SurgTEMP, a multimodal LLM framework featuring (i) a query-guided token selection module that builds hierarchical visual memory (spatial and temporal memory banks) and (ii) a Surgical Competency Progression (SCP) training scheme. Together, they enable effective modeling of variable-length surgical videos while preserving procedure-relevant cues and temporal coherence, and better support diverse downstream assessment tasks. To support model development, we introduce CholeVidQA-32K, a surgical video question answering dataset comprising 32K open-ended QA pairs and 3,855 video segments (approximately 128 h total) from laparoscopic cholecystectomy. The dataset is organized into a three-level hierarchy -- Perception, Assessment, and Reasoning -- spanning 11 tasks from instrument/action/anatomy perception to Critical View of Safety (CVS), intraoperative difficulty, skill proficiency, and adverse event assessment. In comprehensive evaluations against state-of-the-art open-source multimodal and video LLMs (fine-tuned and zero-shot), SurgTEMP achieves substantial performance improvements, advancing the state of video-based surgical VQA. The project page is available at: https://camma-public.github.io/SurgTEMP/

Shi Li, Vinkle Srivastav, Shih-Min Yin +7

University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France · IHU Strasbourg, Strasbourg, France · Department of General Surgery, Kaohsiung Chang Gung Memorial Hospital, Chang Gung University College of Medicine, Kaohsiung, Taiwan +1

Medical Visual Question Answering Surgical Video Understanding Surgical VQA Medical VLMs Temporal Video Understanding

Ask PDF

Abstract

Surgical VQA

Explore similar work

SurgLQA: Scalable Long-Horizon Surgical Video Question Answering

STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs

SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomy