TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering
Authors: Luca Carlini, Chiara Lena, Cesare Hassan, Danail Stoyanov, Elena De Momi, Sophia Bano, Mobarak I. Hoque
Organizations: Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano, Italy · IRCCS Humanitas Research Hospital, Italy · UCL Hawkes Institute and Department of Computer Science, University College London, UK · Division of Informatics, Imaging and Data Science, University of Manchester, UK
Surgical Video Question Answering (VideoQA) requires accurate temporal grounding while remaining robust to natural variation in how clinicians phrase questions, where linguistic bias can arise. Standard Parameter Efficient Fine Tuning (PEFT) methods adapt pretrained projections without explicitly modeling frame-to-frame interactions within the adaptation pathway, limiting their ability to exploit sparse temporal evidence. We introduce TemporalDoRA, a video-specific PEFT formulation that extends Weight-Decomposed Low-Rank Adaptation by (i) inserting lightweight temporal Multi-Head Attention (MHA) inside the low-rank bottleneck of the vision encoder and (ii) selectively applying weight decomposition only to the trainable low-rank branch rather than the full adapted weight. This design enables temporally-aware updates while preserving a frozen backbone and stable scaling. By mixing information across frames within the adaptation subspace, TemporalDoRA steers updates toward temporally consistent visual cues and improves robustness with minimal parameter overhead. To benchmark this setting, we present REAL-Colon-VQA, a colonoscopy VideoQA dataset with 6,424 clip--question pairs, including paired rephrased Out-of-Template questions to evaluate sensitivity to linguistic variation. TemporalDoRA improves Out-of-Template performance, and ablation studies confirm that temporal mixing inside the low-rank branch is the primary driver of these gains. We also validate on EndoVis18-VQA adapted to short clips and observe consistent improvements on the Out-of-Template split. Code and dataset available at~Anonymous GitHub.
Figures & tables
Figure 1: TemporalDoRA overview. (A) VideoQA pipeline. (B) TemporalDoRA: down-project to rank- r , apply temporal MHA over T , then selectively DoRA up-project and add to frozen W0 .
Figure 2: REAL-Colon-VQA question distribution and an example instance.
Data
Model
PEFT
In-Template
Out-of-Template
B4
RL
MET
Acc
B4
RL
MET
Acc
REAL-Colon-VQA
SurgViVQA [ 5 ]
0.649
0.812
0.815
0.472
0.142
0.375
0.379
0.199
Qwen3-VL-2B
Zero-shot
0.196
0.514
0.538
0.457
0.112
0.433
0.459
0.470
LoRA [ 7 ]
0.742
0.815
0.809
0.636
0.435
0.623
0.622
0.607
DoRA [ 12 ]
0.730
0.836
0.826
0.680
0.246
0.611
0.617
0.669
VeRA [ 10 ]
0.249
0.524
0.539
0.501
0.138
0.436
0.463
0.504
Table 1: PEFT comparison on REAL-Colon-VQA and EndoVis18-VQA for In-Template and Out-of-Template splits using Qwen3-VL-2B [ 1 ] , InternVL3-1B [ 25 ] , and SurgViVQA [ 5 ] . Bold indicates best and underlined second best within each model block and split.
REAL-Colon-VQA (Out-of-Template, Qwen3-VL-2B)
Q1: When motion occurs, does the scope pull backward?
GT: No, the scope is advancing.
Pred (Zero-Shot): No, the scope is advancing forward.
Pred (TemporalDoRA): No, the scope is advancing.
EndoVis18-VQA (Out-of-Template, Qwen3-VL-2B)
Q2: What specific abdominal organ is currently undergoing surgical intervention during the robotic-assisted procedure?
Table 2: Out-of-Template examples with Qwen3-VL-2B.
Temporal operator
In-Template
Out-of-Template
B4
RL
MET
Acc
B4
RL
MET
Acc
3D Convolution
0.618
0.781
0.761
0.647
0.312
0.606
0.616
0.609
LSTM
0.717
0.828
0.818
0.669
0.375
0.653
0.648
0.612
Mamba
0.646
0.765
0.737
0.658
0.336
0.593
0.592
0.651
Self-Attention
0.554
0.706
0.680
0.620
0.367
0.628
0.623
0.610
MHA (Ours)
0.703
0.823
0.808
0.678
0.484
0.731
0.716
0.646
Table 3: REAL-Colon-VQA (Qwen3-VL-2B). Ablation of the temporal operator in the low-rank bottleneck.
PEFT
In-Template
Out-of-Template
B4
RL
MET
Acc
B4
RL
MET
Acc
LoRA
0.742
0.814
0.809
0.636
0.435
0.623
0.622
0.607
LoRA+MHA
0.638
0.801
0.791
0.689
0.305
0.627
0.642
0.661
DoRA
0.730
0.836
0.826
0.680
0.246
0.611
0.617
0.669
DoRA+MHA
0.662
0.750
0.741
0.601
0.510
0.649
0.647
0.597
TemporalDoRA (Ours)
0.703
0.823
0.808
0.678
0.484
0.731
0.716
0.646
Table 4: REAL-Colon-VQA (Qwen3-VL-2B). Effect of inserting MHA in the low-rank bottleneck of LoRA and DoRA compared to TemporalDoRA.
Surgical Video Question Answering (VideoQA) provides a promising paradigm for dynamic intraoperative interpretation, enabling real-time decision support and context-aware retrieval in clinical environments. Nevertheless, existing approaches are predominantly restricted to images or short clips, limiting their ability to model long-range procedural dynamics and causal dependencies across extended surgical workflows. To address this challenge, we propose SurgLQA, a unified long-horizon VideoQA framework for scalable surgical reasoning. This framework incorporates Faithful Temporal Consolidation (FTC), which leverages intrinsic temporal cues to construct compact long-range representations while preserving fine-grained temporal fidelity. Further, we develop Temporally-Grounded Multi-Policy Scaling (TMS), an adaptive test-time inference paradigm that strategically adjusts policy-level reasoning capacity within temporally grounded contexts. To facilitate systematic evaluation, we restructured a long-duration colonoscopy VideoQA benchmark, Colon-LQA, and conducted extensive experiments on Colon-LQA and REAL-Colon-VQA. Experimental results demonstrate that our approach achieves consistent performance gains in long-range reasoning with temporally grounded inference. Code link: https://github.com/RascalGdd/SurgLQA.
Parameter-efficient fine-tuning (PEFT) and probing enable adaptation of foundation models using only a small number of trainable parameters, making it attractive for video understanding where annotation and computation are expensive. However, video PEFT has focused on adapting image-pretrained models, while standard PEFT methods can also be applied to video representations. These settings are rarely compared and both confine temporal reasoning to a single component of the model, leaving open how temporal context should be distributed across backbone, PEFT and probe. In this work we provide a systematic study of model adaptation strategies for video understanding. We evaluate methods across appearance-focused, motion-focused and spatially dense settings, with a particular focus on scenarios with limited data where parameter-efficiency is most beneficial. Our results provide new insights into PEFT and probing across settings and demonstrate the importance of temporal context allocation for effective video adaptation
Long Video Question Answering (LVQA) requires identifying sparse, query-relevant evidence within hours-long untrimmed videos. Existing approaches either process videos densely with large vision-language models (VLMs), incurring prohibitive computational cost, or rely on sparse caption-based reasoning, which often misses temporally localized and motion-centric evidence. We introduce TimeProVe, a cost-efficient hybrid framework for temporally grounded reasoning in long videos. TimeProVe first employs lightweight modules to generate action-grounded answer--evidence hypotheses and subsequently invokes an expensive VLM only for targeted verification. The core of our framework lies in the Action-based Candidate Evidence (ACE) module, which converts temporally localized actions into query-conditioned candidate answers and supporting evidence windows through lightweight LLM reasoning. We further introduce OpenTSUBench (OTB), an open-ended benchmark designed to evaluate temporally grounded reasoning in real-world Activities of Daily Living (ADL) scenarios. Experiments show that TimeProVe outperforms the strongest baseline on OTB by 7.3%, while reducing VLM calls by 75% and inference cost by 93%. Furthermore, without explicit temporal grounding training, TimeProVe achieves competitive performance on Charades-STA, and reaches state-of-the-art results when enhanced with grounding VLMs.