Organizations: Department of Artificial Intelligence, Asian Institute of Gastroenterology, India · Department of Gastroenterology, Asian Institute of Gastroenterology, India · University of Vermont Medical Center, USA
Endoscopic diagnosis unfolds over native video, where a single positive finding during traversal of the gastrointestinal tract sets off a multi-minute chain of reasoning over location, morphology, severity and intervention. Existing endoscopy multimodal benchmarks test only static images or single-disease videos, which miss the temporal and clinical reasoning that real-world reporting demands. We introduce EndoSCOP-V, a video-grounded multi-turn evaluation framework for structured endoscopy reporting and clinical reasoning. It contains 1,307 real-world endoscopy and colonoscopy videos (63.2 h; 97% 1080p) averaging about 174 s. Its 20,274 question-answer turns are instantiated deterministically from clinician-filled structured endoscopy-reporting EHRs, so no per-case annotation is synthetic or LLM-rephrased. They come from 1,033 clinician-reviewed templates, about 18 per disease, that span a clinically aligned 9-task x 30-sub-task reporting taxonomy over 54 World Endoscopy Organization (WEO) MST 3.0 diseases. 22% of procedures carry co-existing pathologies, up to six per case. A Markovian context-pruned oracle-forcing protocol evaluates the dialogue turn by turn, injecting the gold answer before each subsequent turn so that per-turn reasoning is isolated from cascading dialogue errors. A panel of 15 board-certified endoscopists sets a 78% strict-accuracy baseline on 1,500 questions. Across 14 open-weight LMMs, the best 27B-class model reaches 59.8% strict accuracy, falling to 53.4% once format-trivial disease-rejection items are excluded. On a between-subjects spot-check, the best of 3 frontier proprietary models trails the clinician panel by 23 pp on strict NDR accuracy. Sub-task analysis reveals consistent gaps in quantitative measurement, multi-variable grading and intervention reasoning on long-horizon video. Data: Hugging Face (EndoSCOP-V). Code: GitHub (abhi25t/EndoSCOP-V).
Figures & tables
Figure 1: Endoscopy reporting is inherently multi-turn. A single finding cascades through a chain of follow-up questions (for example, Disease? → Ulcer, Organ? → Duodenum, Sub-location? → Duodenal Bulb, Wall? → Posterior, Grade? → IIB) that mirrors how an endoscopist composes a structured EHR report. EndoSCOP-V evaluates models on this multi-turn dialogue rather than on isolated single-shot recognition. Illustration generated with Gemini 3.1 Pro.
Benchmark
Modality / Clip length
Diseases / Categories
Multi-turn
Multi-disease
PanEndoX Johri et al. [2025]
Image
111 GI conditions
N
N
EndoBench Liu et al. [2025]
Image
4 modalities
N
N
ColonVQA / Colon-X Ji et al. [2025]
Image
76 findings (Colon only)
N
N
GI-Bench Zhu et al. [2026]
Image
20 lesions (EGD+Colon)
N
N
SurgViVQA Drago et al. [2025]
60 videos (8-frame)
Polyps only; 20 Q-cats
N
N
Kvasir-VQA-x1 Gautam et al. [2025]
Image (perturbed)
5 categories (EGD+Colon)
N
N
Table 1: Compact comparison of EndoSCOP-V with peer endoscopy benchmarks. Appendix D (Table 16 ) gives the full version with an eval-items column.
Task
Abbr.
Sub-tasks
Disease Identification
DI
DI-M, DI-C, DI-R, DI-N
Anatomical Localization
AL
AL-O, AL-P, AL-W
Morphological Assessment
MA
MA-T, MA-S, MA-C, MA-M, MA-SZ
Pathological Feature Assessment
PF
PF-D, PF-M, PF-U, PF-S, PF-B
Quantification
QN
QN-L, QN-S
Grading & Scoring
GS
GS-MC, GS-SS, GS-EC, GS-AG
Table 2: 9-task × 30-sub-task reporting taxonomy.
Rank
Model
Strict Overall
Strict NDR
Partial Overall
Partial NDR
EGD
Colonoscopy
1
Qwen3.6-27B
59.8
53.4
62.6
57.0
60.5
57.2
2
Qwen3.5-9B
58.4
51.6
62.0
56.5
60.3
52.8
3
Qwen3.5-27B
58.2
52.2
61.2
56.0
58.3
56.2
4
Qwen3-VL-32B
56.4
48.8
60.3
53.3
56.5
53.9
5
Hulu-Med-32B
55.0
46.9
58.8
51.2
55.9
52.5
6
Hulu-Med-14B
54.6
47.0
56.9
49.2
55.5
51.4
Table 3: Accuracy (%). Best per column in bold .
Model
DI
AL
MA
PF
QN
GS
CF
CR
AR
n
6,585
1,390
3,387
3,503
609
1,176
1,701
1,026
897
Qwen3.6-27B
76.0
47.2
48.3
51.6
64.7
40.3
61.3
40.7
79.5
Qwen3.5-9B
75.4
40.1
45.1
52.6
52.7
40.9
59.0
43.4
77.1
Qwen3.5-27B
76.5
46.6
43.8
46.9
58.6
46.0
59.1
37.5
80.7
Qwen3-VL-32B
74.9
41.8
42.5
44.2
58.0
46.5
57.4
35.1
75.5
Hulu-Med-32B
66.8
39.6
45.0
52.6
54.4
42.9
60.5
38.7
72.8
Table 4: Strict accuracy (%) by clinical task across the 14 open-weight models, where n is the total number of questions per task across the 1,307 cases.
Video-exclusive sub-task
n
Keyframe (oracle)
25 frames
50 frames
Full video
Video − keyframe
AR-V
41
40.5
50.7
51.2
54.3
+13.8
AR-B
856
72.5
78.6
80.2
79.1
+6.5
CF-B
718
82.0
77.9
77.2
76.2
− 5.8
CF-S
641
50.6
49.6
49.0
48.5
− 2.1
Table 5: Strict accuracy (%) on the four video-exclusive sub-tasks, mean over five models, equal weight per model. Paired McNemar tests (video vs. keyframe, pooled over model × question outcomes) give p<0.001 in all four sub-tasks.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Sub-task
Abbr.
Description
Video advantage
MCQ Identification
DI-M
Identify disease from 4 options
Multi-angle, temporal context
Confirmation
DI-C
Confirm suspected disease
Multi-angle, temporal context
Disease-Rejection
DI-R
Reject a clinically plausible look-alike not present in the case
Multi-angle, temporal context
Normal Assessment
DI-N
Identify absence of pathology at a location
Full mucosal sweep of region
Organ
AL-O
Upper-GI organ identification
Navigational context
Part / Segment
AL-P
Sub-organ segment localization
Scope entry into segment visible
Appendix
Table 6: Full task taxonomy (9 tasks × 30 sub-tasks). Bold rows are video-exclusive sub-tasks.
Sub-task
N
% all
% task
EGD
Colon
Binary
Single-MCQ
Multi-MCQ
AL-O
348
1.72%
25.04%
348
0
46
297
5
AL-P
713
3.52%
51.29%
679
34
49
503
161
AL-W
329
1.62%
23.67%
302
27
32
235
62
AR-B
856
4.22%
95.43%
720
136
788
68
0
AR-V
41
0.20%
4.57%
18
23
41
0
0
CF-A
342
1.69%
20.11%
256
86
121
16
205
Appendix
Table 7: Per-sub-task question counts.
Disease
Cases
Standalone
Multi
EGD
Colon
Questions
Binary
Single-MCQ
Multi-MCQ
EGD Q
Colon Q
Gastric Polyp
52
20
32
52
0
669
203
383
83
669
0
Reflux Esophagitis/GERD
52
25
27
52
0
401
98
202
101
401
0
Duodenal Ulcer
49
21
28
49
0
593
138
249
206
593
0
Erosion
47
13
34
47
0
350
204
77
69
350
0
Diverticulum
43
36
7
37
6
668
270
256
142
606
62
Esophageal Varices
42
0
42
42
0
371
176
167
28
371
0
Appendix
Table 8: Per-disease coverage (Convention A). Sorted by total cases descending. Rows for non-disease findings (Hills hiatus, Normal, Ampulla, Mucosa) appear in italics, and their cases are also counted under the strict diseases they co-occur with, which is why per-disease cases sum to more than 1,307. The Disease-Rejection (DI-R) row aggregates over all 1,297 cases that have ≥1 DI-R question.
#
Entity
Binary
Single-MCQ
Multi-MCQ
Total templates
1
Ampulla of Vater / Papilla
14
2
2
18
2
Angioectasia
18
14
2
34
3
Barrett’s esophagus
9
6
1
16
4
Candidiasis
4
3
0
7
5
Caustic Injury
13
7
1
21
6
Coeliac Disease
19
1
3
23
Appendix
Table 9: Per-entity template counts. Alphabetical by entity name. Format columns are template counts, and the question-level format mix appears in Table 7 . Italicized rows are the four non-disease findings.
Quantity
Value
Cases (videos)
1,307
EGD / colonoscopy split
1,004 / 303 (76.8% / 23.2%)
Total video duration
63.2 hours
Mean clip length
∼ 174 s
1080p / 720p
1,269 / 38 (97.1% / 2.9%)
Diseases (+ non-disease findings)
54 ( + 4)
Appendix
Table 10: EndoSCOP-V dataset summary.
Sub-task
n
Qwen3.6-27B
Qwen3.5-9B
Qwen3.5-27B
Qwen3-VL-32B
Hulu-Med-32B
MedGemma-27B
MedGemma-1.5-4B
ColonR1
AL-O (Organ)
348
54.9
45.4
52.0
47.4
42.5
37.4
31.9
35.6
AL-P (Part / Segment)
713
50.2
44.0
46.4
47.4
37.3
42.2
41.4
25.9
AL-W (Wall)
329
31.9
25.5
39.5
21.3
39.8
43.5
33.7
20.5
AR-B (Biopsy)
856
82.2
81.6
84.0
80.5
74.9
72.0
62.2
66.7
AR-V † (Visualization)
41
48.8
43.9
43.9
34.1
36.6
65.9
68.3
34.8
CF-A (Associated Findings)
342
39.5
42.7
44.7
44.4
45.6
46.2
25.7
22.5
Appendix
Table 11: Per-sub-task strict accuracy (%) on Set A, where n is the number of questions per sub-task in the 20,274-question full-cohort denominator. ColonR1 is evaluated on its 9,897-question eligibility-filtered subset, so its sub-task-level denominators are smaller than the cohort n shown in the table.
Disease
n
Qwen3.6-27B
Qwen3.5-9B
Qwen3.5-27B
Qwen3-VL-32B
Hulu-Med-32B
MedGemma-27B
MedGemma-1.5-4B
ColonR1
Angioectasia
405
60.2
60.5
65.4
51.1
43.2
52.3
33.8
40.8
Barrett’s esophagus
398
68.8
49.7
54.0
50.5
47.2
45.0
43.0
—
Candidiasis
161
60.9
47.2
55.3
57.8
45.3
34.2
31.1
—
Caustic Injury †
174
42.0
54.6
43.1
53.4
46.0
39.7
37.4
—
Coeliac Disease
172
44.2
50.0
29.7
20.9
51.7
30.2
16.9
—
Crohn’s Disease
459
36.6
44.0
40.5
44.0
42.0
46.4
36.6
31.9
Appendix
Table 12: Per-disease strict accuracy (%) on Set A, where n is the number of questions per disease. Italicized rows are the four non-disease findings (Ampulla of Vater / Papilla, Hills hiatus, Mucosa, Normal). ColonR1 cells show “—” for the 32 of 58 entities outside its eligibility subset. A question in a multi-disease case is attributed to each disease present, so per-disease n ’s may sum above the 20,274 per-question total. A disease occurring in two organs is counted twice. DI-R items are routed to a separate case-level row, so per-disease rows exclude DI-R scores.
Task
Questions
Labelled
α
Fleiss’ Kappa
Raw
AR Action Recognition
92
92
0.91
0.91
0.95
DI Disease Identification
629
629
0.88
0.88
0.92
QN Quantification
76
76
0.78
0.78
0.82
CF Complication Recognition
169
159
0.78
0.78
0.81
PF Pathological Feature Assessment
383
354
0.71
0.71
0.75
MA Morphological Assessment
397
371
0.67
0.67
0.71
Appendix
Table 13: Reliability per task (five raters, 2,148 questions). Questions counts all questions of the task; Labelled counts those with a ≥ 3/5 consensus. Krippendorff’s α treats unanswered items as missing; raw = mean pairwise agreement.
Sub-task
Questions
α
Fleiss’ Kappa
Raw
DI-M MCQ Identification
166
0.83
0.83
0.88
DI-C Confirmation †
166
0.20
0.20
0.88
DI-R Disease-Rejection †
297
0.23
0.23
0.96
AL-O Organ
29
0.91
0.91
0.93
AL-P Part / Segment
76
0.64
0.64
0.71
AL-W Wall
35
0.51
0.50
0.61
Appendix
Table 14: Reliability per sub-task (five raters; sub-tasks with at least 10 questions in the subset). AR-V (8 questions) is pooled into AR in Table 13 ; DI-N does not occur in the subset.
Model
Questions
Recorded label
Consensus
Δ
Qwen3.6-27B
2,044
62.9
61.4
− 1.4
Qwen3.5-9B
2,044
59.9
59.5
− 0.3
Qwen3-VL-32B
2,044
59.4
58.3
− 1.1
Qwen3-VL-8B
2,044
57.3
55.9
− 1.5
Qwen3.5-27B
2,044
55.9
55.4
− 0.5
Hulu-Med-14B
2,044
56.0
55.1
− 0.9
Appendix
Table 15: Strict accuracy (%) on the 2,044 consensus-labelled questions, scored against the recorded EHR label and against the consensus; Δ = consensus − recorded (pp). Sorted by consensus accuracy. Values differ from Table 3 because the subset is not the full benchmark; the quantity compared is the within-model Δ . ColonR1 is eligible for 79 of the 100 cases (1,668 questions), is scored on that subset and is not rank-comparable with the other 13 models.
Benchmark
Eval items
Modality
Procedures
Categories
Tasks
Multi-turn
Multi-disease
EHR-GT
EndoBench Liu et al. [2025]
6,832 VQA
Image
EGD + Colon + Capsule + Surgical
4 scenarios
12 + 12 sub
N
N
N
GI-Bench Zhu et al. [2026]
300-Q Human-AI subset †
Image
EGD + Colon
20 lesions
5 stages
N
N
N
ColonVQA Ji et al. [2025]
4,568 VQA
Image
Colon only
76 findings
16 of 18
N
N
N
ColonGPT Ji et al. [2026]
450K dialogues ‡
Image
Colon only
62 sub-categories
4 (CLS/DET/SEG/VQA)
N
N
N
Colon-Bench Hamdi et al. [2026]
528 videos / 464K frames
Video
Colon only
14 lesions
3 (VQA/CLS/OV-VOS)
N
N
N
PanEndoX Johri et al. [2025]
420K imgs / 30 datasets
Image
EGD + Colon + Capsule + Small-bowel
111 GI conditions
10 (CLS / grading / landmark)
N
N
N
Appendix
Table 16: EndoSCOP-V vs. peer multimodal endoscopy benchmarks (Y = Yes, N = No). This is the full version of the table summarised in main-text Table 1 .
Vision-language models remain underused in colonoscopy despite the rich expert descriptions recorded in routine reports. These reports document lesion appearance, size and location but summarise entire procedures rather than caption individual frames, leaving clinical findings only weakly linked to the corresponding images. Here we develop EndoCLIP, a colonoscopy vision-language foundation model trained on 125,756 lesion-level image-text pairs progressively recovered from 280,476 routine colonoscopy records. Across lesion-level image-text retrieval, structured report generation and six multi-centre clinical classification tasks, EndoCLIP outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings. On benign-versus-malignant classification, its linear probe approaches the performance of expert readers in a blinded study involving 12 endoscopists. These results suggest that recovering finding-to-frame correspondence can transform routine documentation into scalable supervision, enabling clinical targets to be specified in language rather than separately annotated for each task.
Jia Yu, Yan Zhu, Yili He +12
Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China. · Shanghai Collaborative Innovation Center of Endoscopy, Shanghai, China. · Shanghai Institute for Advanced Study of Zhejiang University, Shanghai, China. +5
Capsule endoscopy (CE) enables non-invasive gastrointestinal screening, but current CE research remains largely limited to frame-level classification and detection, leaving video-level analysis underexplored. To bridge this gap, we introduce and formally define a new task, diagnosis-driven CE video summarization, which requires extracting key evidence frames that covers clinically meaningful findings and making accurate diagnoses from those evidence frames. This setting is challenging because diagnostically relevant events are extremely sparse and can be overwhelmed by tens of thousands of redundant normal frames, while individual observations are often ambiguous due to motion blur, debris, specular highlights, and rapid viewpoint changes. To facilitate research in this direction, we introduce VideoCAP, the first CE dataset with diagnosis-driven annotations derived from real clinical reports. VideoCAP comprises 240 full-length videos and provides realistic supervision for both key evidence frame extraction and diagnosis. To address this task, we further propose DiCE, a clinician-inspired framework that mirrors the standard CE reading workflow. DiCE first performs efficient candidate screening over the raw video, then uses a Context Weaver to organize candidates into coherent diagnostic contexts that preserve distinct lesion events, and an Evidence Converger to aggregate multi-frame evidence within each context into robust clip-level judgments. Experiments show that DiCE consistently outperforms state-of-the-art methods, producing concise and clinically reliable diagnostic summaries. These results highlight diagnosis-driven contextual reasoning as a promising paradigm for ultra-long CE video summarization.
Bowen Liu, Li Yang, Shanshan Song +6
1The Hong Kong University of Science and Technology · 2Renji Hospital, Shanghai Jiao Tong University School of Medicine
Endoscopic diagnosis is an iterative process in which clinicians acquire, compare, and verify local visual evidence before reaching a conclusion. Current AI systems do not adequately support this process because fine-grained evidence acquisition and multi-step reasoning remain weakly coupled, complicating reconciliation of image-derived findings with their textual interpretations. This gives rise to two failure modes, hallucinated evidence and uncorrected error accumulation, that undermine diagnostic reliability. We propose EndoCogniAgent, a closed-loop agentic framework that formulates endoscopic diagnosis as a controlled state update process for integrating complementary visual and textual evidence. At each reasoning round, a central planner selects an evidence acquisition action, specialized expert tools extract spatial and semantic observations as structured textual evidence, and a self-consistency validation mechanism examines this evidence along two dimensions, knowledge consistency against the input image and temporal consistency with prior validated findings, before updating the diagnostic state. Validated observations are admitted into the evolving state to condition subsequent planning, while insufficiently supported or conflicting findings are retained with corrective feedback that redirects the planner toward additional verification. We further introduce EndoAgentBench, a workflow-oriented benchmark comprising 6,132 question-answer pairs from 11 endoscopic datasets, to evaluate diagnostic agents across a comprehensive diagnostic chain, from fine-grained visual perception to high-level diagnostic reasoning. EndoCogniAgent achieves 85.23% overall accuracy on perception tasks and 71.13% clinical acceptance rate on reasoning tasks. Blinded clinician evaluation further shows consistent improvements in diagnostic response quality over the evaluated baselines.
Yi Tang, Kai-Ni Wang, Liang-Peng Pu +3
School of Biological Science and Medical Engineering, Southeast University, Nanjing 211189, China · Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University, Nanjing 211189, China · State Key Laboratory of Digital Medical Engineering, Southeast University, Nanjing 211189, China +2