MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning
Authors: Yingying Fan, Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Yuxuan Zhang, Liang Chen, Jiahao Xie, Jiangtang Wang, Shuai Shao, Anchao Yang, Yutong Bai, Yan Wang
Organizations: School of Automation and Intelligence, Beijing Jiaotong University, Beijing 100044, China · Department of Electrical and Computer Engineering, University of Maryland, College Park, MD 20742, USA · University of British Columbia · Suzhou Institute for Advanced Research, University of Science and Technology of China · Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University
Understanding tens-of-minutes surgical videos requires long-horizon temporal reasoning, answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time. Existing approaches handle this poorly: a one-shot vision-language model (VLM) compresses the whole procedure to fit its context window and loses the detail a "before" or "after" question depends on, while video agents that train the model where to look are data-hungry and transfer poorly to out-of-domain surgery. We build an agent harness that separates reasoning from perception and improves by evolving context rather than optimizing weights. A text-only orchestrator plans which evidence to gather and issues an auditable sequence of tool calls, while frozen vision-language sub-agents execute each call over the pixels, viewing, cropping, inspecting frames, and retrieving external knowledge. We further propose a gradient-free, reward-gated Heuristic Skill Distillation loop that mines the agent's own low-scoring traces and keeps a candidate skill only when it raises a validation reward, yielding reusable retrieval skills, notably directed re-look. Growing an external skill library rather than tuning weights, the loop adapts from only about 100 labeled examples, far fewer than supervised or reinforcement fine-tuning requires. To evaluate this agent, we introduce MedClawBench, a de-leaked, doctor-grounded benchmark of 1,123 questions over self-built long neurosurgery recordings and a held-out public lecture-video test split. Across both datasets and all four evaluation dimensions, our agent consistently outperforms one-shot VLMs and general video-agent frameworks, with the largest gains on the long, out-of-domain neurosurgery videos. Project page: https://fyycs.github.io/medclaw/.
Figures & tables
Fig. 1: Long-horizon temporal reasoning on a long surgery video, where an “after” question is answered by evidence that lies outside the window depicting the named event. (a) A one-shot VLM reads the whole procedure in a single pass, losing the detail the question needs. (b) SFT-based agents learn where to look through supervised or reinforcement fine-tuning, which needs massive data and transfers poorly out of domain. (c) MedClaw keeps every model frozen and instead distills reusable skills through Heuristic Skill Distillation, collecting failure traces, diagnosing the gap, and retaining a skill only when it passes a reward gate, adapting from only about one hundred examples while keeping every decision explicit and auditable.
Fig. 2: Overview of MedClaw. Left: a real trace on a temporal “before” question, where the orchestrator loads the relevant skill and issues an auditable sequence of tool calls that frozen vision sub-agents execute on the pixels, viewing the full video to localize a window, cropping it, and inspecting a frame, before grounding its answer. Top right: the orchestrator is backed by an external knowledge base queried via RAG and by a library of tunable skills derived from past failures. Bottom right: these skills come from our Heuristic Skill Distillation loop, which collects low-scoring traces on a mining set, diagnoses each gap as operational, unreachable, or already correct, and retains a candidate only if it raises the four-dimensional reward on a validation set.
Dataset
Video len.
≥20 min
Temp.
De-leak
Neuro.
EndoVis18-VQA [ 65 ]
per-frame
–
✗
✓
✗
PitVQA [ 44 ]
per-frame
–
✗
✓
✓
SurgViVQA [ 49 ]
8 frames
–
short
✓
✗
SVU-31K (test) [ 5 ]
13.9 min
28%
partial
✗
✗
MedClawBench (ours)
18.4 min
41%
✓
✓
✓
neurosurgery split
28.5 min
71%
✓
✓
✓
TABLE I: Comparison with surgical-video QA datasets. Video len. : median source-video duration. ≥20 min : share of videos at least twenty minutes long. Temp. : long-horizon temporal reasoning. De-leak : not answerable without watching the video (by explicit filtering or by the nature of the source). Neuro. : includes neurosurgery videos. Durations and percentages are rounded.
Fig. 3: MedClawBench statistics. (a) Neurosurgery question types ( 259 ): the inner ring is the Visual Temporal (VTR, 109 ) vs. Visual Perception (VPR, 150 ) split and the outer ring the ten fine-grained types. (b) Source-video duration for MedClawBench (Ours) against SVU-31K, as the percentage of videos per bin; our filter drops all sub-ten-minute videos and shifts mass toward the long ( ≥20 min, shaded) tail. (c) Surgical-type distribution over both splits (# questions, # videos), with the neurosurgery half split into MVD, tumour resection and epileptogenic-lesion resection (HPB* stands for Hepatobiliary & Pancreas).
Cat.
Method
Neurosurgery ( 259 )
SVU ( 864 )
VTR ( 109 )
VPR ( 150 )
VTR ( 163 )
VPR ( 701 )
CI
DO
CU
TU
CI
DO
CU
TU
CI
DO
CU
TU
CI
DO
CU
TU
VLM
Qwen2.5-VL-7B
1.62
1.31
2.28
2.01
1.80
1.61
2.53
2.90
1.64
1.54
2.31
2.01
2.25
1.85
2.84
2.73
InternVL3-8B
1.35
1.05
1.78
1.65
1.65
1.43
2.29
2.60
1.51
1.18
1.98
1.75
1.81
1.16
2.11
2.09
VideoLLaMA3-7B
1.49
1.09
1.96
1.74
1.73
1.39
2.28
2.50
1.60
1.29
2.17
1.79
1.87
1.34
2.26
2.28
Reasoning
Video-R1-7B
1.61
1.17
2.24
2.28
1.77
1.55
2.55
3.03
1.67
1.31
2.33
2.02
2.10
1.50
2.68
3.04
TABLE II: Main results on MedClawBench, broken down by reasoning type ( VTR , VPR ) with the four scoring dimensions each 1 – 5 : CI (correctness), DO (detail), CU (context), TU (temporal). Best in bold , second best underlined . Shaded cells mark the one-shot backbone used for the Δ row. MedClaw uses the flash visual backbone on neurosurgery and the cheaper lite backbone on SVU, and the Δ row compares against the corresponding one-shot baseline in each split.
Setting
Video-R1-7B
Qwen2.5-VL-7B
ReWatch-R1-7B
one-shot
2.06
2.04
1.54
+ harness
2.60
2.58
2.56
Gain
+0.55
+0.54
+1.02
TABLE III: Backbone-lift ablation on the neurosurgery split ( 259 questions): one-shot vs. the same backbone wrapped as the sub-agent in our frozen-weight harness. Scores are the average over the four dimensions. Best per column in bold .
Configuration
CI
DO
CU
TU
Avg
Δ
MedClaw (full)
2.54
2.52
3.30
3.22
2.90
—
w/o coarse-to-fine
2.36
2.36
3.19
3.03
2.74
−0.16
w/o directed re-look
2.19
2.17
2.93
2.76
2.51
−0.38
w/o HSD
2.37
2.39
3.17
3.14
2.77
−0.13
TABLE IV: Component ablation on the neurosurgery split ( 259 questions): from the full MedClaw harness, we remove or alter one learned component at a time. Scores are the four dimensions and their average; Δ is the change in average from the full model.
Orchestrator
CI
DO
CU
TU
Avg
Δ
Opus-4.8
2.54
2.52
3.30
3.22
2.90
—
DeepSeek-V3
2.07
2.36
2.94
2.89
2.57
−0.33
DeepSeek-R1
2.22
2.21
3.04
2.75
2.56
−0.34
TABLE V: Orchestrator ablation on the neurosurgery split ( 259 questions): the visual sub-agent and all tools are held fixed; only the text-only orchestrator changes. Scores are the four dimensions and their average; Δ is the change from the Opus-4.8 orchestrator.
Dimension
Neurosurgery
SVU
κ (GPT-5.5)
κ (GPT-5.5)
CI (correctness)
0.71
0.87
DO (detail)
0.66
0.86
CU (context)
0.72
0.85
TU (temporal)
0.74
0.86
Overall
0.72
0.86
TABLE VI: Judge reliability: quadratic weighted Cohen’s κ between the GPT-5.5 judge and expert ratings, per dimension, on each split.