DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis

Jiahui Li; Yutong Guo; Nan Yang; Wenzhan Song; Jin Lu; Fei Dou

cs.CVSep 26, 2026

DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis

Authors: Jiahui Li, Yutong Guo, Nan Yang, Wenzhan Song, Jin Lu, Fei Dou

Organizations: University of Georgia, Athens, USA · Beijing Luhe Hospital, Beijing, China

Abstract

Diagnosing a patient from video requires more than recognizing the sign: a vision-language model must turn what it sees into hypotheses, questions and tests. DynamicDx evaluates each step in 71 neurological consultations across 11 sign categories, linking authentic patient videos to confirmed diagnoses and fixed charts built from the same case reports, so that every model queries the same evidence. Across five such models, video improves accuracy by 9.9-22.5 percentage points over blind input, but neither recognition alone nor temporal order explains the gain: the cause is usually missing from the model's video-only differential diagnosis even when the sign is recognized, and shuffling the frames produces no reliable accuracy loss. Instead, a trajectory replay traces most of the gain to the investigation results the video prompts. Evidence acquisition is the bottleneck: supplying the decisive investigations raises accuracy to 73.2-93.0%. Two interventions act on it. A post-trained 4B video describer improves sign descriptions, especially from a short, densely sampled segment, and source-clean literature retrieval expands initial hypotheses; both bring the tests a model orders closer to those the treating clinicians documented and, through them, raise accuracy. For video-based diagnosis, seeing better helps when it leads to asking better.

Figures & tables

Figure 1: Video helps diagnosis through the evidence it prompts, not through its dynamics. (a) Diagnostic accuracy on the 71 cases as the input grows (Table 2 ); black, mean over the five models. (b) Accuracy gained at a replayed diagnosis turn from each component, mean over the five models with

95\%

bootstrap intervals (Table 11 ).

Benchmark	Source	Cases	Video	Sign	Diff.	Hx	Tests	Dx
MedVidQA	Instructional video	899	✓	–	–	–	–	–
MediQ	Simulated cases	1,413	–	–	–	✓	–	✓
AgentClinic	Simulated cases/records	120 + 415	–	–	–	✓	✓	✓
Multimodal AMIE	Actors + artifacts	105	–	–	–	✓	–	✓
AI co-clinician	Actors	20	✓	–	–	✓	–	✓
DynamicDx (ours)	Patient clips + records	71	✓	✓	✓	✓	✓	✓

Table 1: Benchmark comparison (✓: included; Diff.: differential; Hx: history; Dx: diagnosis). Cases: evaluation cases or scenarios (MedVidQA: videos; AgentClinic: multimodal + text).

Figure 2: Overview of DynamicDx . Stage 1 evaluates reactive perception (sign recognition and hypotheses from video); Stage 2 evaluates proactive evidence acquisition (history and investigations) and diagnosis from the case chart.

Figure 3: Teacher-guided temporal-window sampling. Clinical context supervises the offline teacher only; benchmark phenomenology supplies the recognition target. The student predicts at most one window from 32 survey frames and describes $K$ window frames.

Figure 4: Retrieval pipeline from predicted phenomenology to candidate causes.

Figure 5: Stage 1 performance on 71 clips. (a) Phenomenology recognition. (b) True aetiology in the differential. Model curves vary the sampled-frame budget; dashed lines show one neurologist’s clip-only reference (9.9% recognition; 2.8% aetiological coverage).

		GPT-5.6-luna	Gemma-4-31B	MiMo-v2.5	MiniMax-M3	Qwen3.8-flash	Clinician
Diagnosis Accuracy (%)	Blind	23.9 [14.1,33.8]	23.9 [14.1,33.8]	8.5 [2.8,15.5]	9.9 [4.2,16.9]	2.8 [0.0,7.0]	—
	Single Frame	36.6 [25.4,47.9]	28.2 [18.3,39.4]	21.1 [12.7,31.0]	15.5 [7.0,23.9]	19.7 [11.3,29.6]	—
	Shuffled	45.1 [33.8,56.3]	28.2 [18.3,39.4]	22.5 [12.7,32.4]	15.5 [7.0,23.9]	12.7 [5.6,21.1]	—
	Video	46.5 [35.2,57.7]	33.8 [22.5,45.1]	22.5 [12.7,32.4]	25.4 [15.5,35.2]	22.5 [12.7,32.4]	35.2 [23.9,46.5]
	Multi-Turn	52.1 [40.8,63.4]	26.8 [16.9,36.6]	19.7 [11.3,29.6]	33.8 [22.5,45.1]	23.9 [14.1,33.8]	—
	Own Words	47.9 [36.6,59.2]	33.8 [22.5,45.1]	22.5 [12.7,32.4]	22.5 [14.1,32.4]	21.1 [12.7,31.0]	—

Table 2: Consultation outcomes (%;

95\%

bootstrap intervals in brackets). Seven primary conditions are defined in Section 3.4 ; multi-turn uses up to ten adaptive rounds. The clinician completed only the video condition (—: not run).

Figure 6: Frame-order effects across clinical categories. Five systems pooled over

71

clips; clip counts in parentheses. (a) Accuracy for one frame,

32

shuffled frames, and

32

ordered frames. (b) Ordered-minus-shuffled accuracy with bootstrap intervals (

10{,}000

clip resamples).

$K$	Window accuracy	Random accuracy	No-window accuracy	$\tau$ (W / R / N)
8	29.1 [20.7, 38.0]	29.6 [21.6, 38.5]	26.8 [19.2, 34.3]	17.3 / 16.4 / 19.0
16	30.0 [21.1, 39.4]	28.6 [20.2, 37.6]	27.2 [19.2, 35.7]	18.6 / 17.9 / 18.3
32	33.8 [24.4, 43.7]	30.5 [21.6, 39.9]	27.2 [19.2, 35.7]	19.4 / 20.9 / 18.9

Table 3: Matched LoRA sampling outcomes (Section 3.5.1 ; 71 cases; three-run per-clip means). Accuracy and source-workup coverage

\tau

are percentages; brackets give 95% clip-level bootstrap intervals (Appendix D ). No window: sign-only adapter on

32{+}K

uniform frames (Table 24 ). W/R/N : window, random and no-window arms.

Figure 7: Retrieval acts on evidence acquisition. (a) Diagnostic accuracy and (b) source-workup coverage on the 71 cases, with

95\%

bootstrap intervals. Own candidates supplies the model’s Stage 1 differential alone; retrieval conditions append retrieved causes (Section 3.5.2 ): Original keeps all records, Source-clean removes the source article and near duplicates, and Strict no-answer also removes records naming the diagnosis.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Who is in the benchmark. a , Age of the $67$ patients whose age the source states, stacked by sex; four further patients are described only as adults and one has no stated sex. The dashed line marks $18$ years. b , Region of the first author’s affiliation, by case ( $23$ countries). c , Publication year of the $66$ source articles. d , Cases per category; hatched segments are the eight cases drawn from the three articles that contribute more than one patient.

period	$n$	video	video $-$ blind	oracle $-$ video	reference $-$ video	video $-$ single	video $-$ shuffled	clean $-$ video
All	71	30.1	$+16.3$ $[+9.3,+23.7]$	$+51.8$ $[+43.4,+60.0]$	$+15.8$ $[+8.7,+22.8]$	$+5.9$ $[-0.3,+12.4]$	$+5.4$ $[-0.3,+11.3]$	$+9.6$ $[+2.5,+16.6]$
$\leq$ 2022	17	49.4	$+32.9$ $[+14.1,+49.4]$	$+38.8$ $[+23.5,+55.3]$	$+12.9$ $[-1.2,+27.1]$	$+16.5$ $[+3.5,+31.8]$	$+7.1$ $[-5.9,+21.2]$	$+2.4$ $[-15.3,+21.2]$
2023–24	17	22.4	$+12.9$ $[+2.4,+23.6]$	$+60.0$ $[+43.5,+76.5]$	$+3.5$ $[-8.2,+16.5]$	$+3.5$ $[-7.1,+14.1]$	$+12.9$ $[+5.9,+21.2]$	$+7.1$ $[-5.9,+21.2]$
2025–26	37	24.9	$+10.3$ $[+1.6,+18.9]$	$+54.1$ $[+42.2,+65.4]$	$+22.7$ $[+13.5,+32.4]$	$+2.2$ $[-5.9,+10.8]$	$+1.1$ $[-7.0,+9.2]$	$+14.1$ $[+5.4,+22.7]$

Appendix

Table 4: Main contrasts by publication period of the source article. Stage-2 diagnosis accuracy (%) averaged over the five systems; differences in percentage points with

95\%

paired case-level bootstrap intervals. Clean: source-clean retrieval.

	GPT-5.6-luna	Gemma-4-31B	MiMo-v2.5	MiniMax-M3	Qwen3.8-flash	Clinician
95% CI by article	[34.8, 58.1]	[22.9, 45.3]	[13.0, 33.3]	[15.7, 35.7]	[13.6, 32.4]	[24.2, 46.5]
width ratio	1.04	1.00	1.03	1.01	0.95	0.99

Appendix

Table 5: Article-clustered bootstrap intervals for diagnosis accuracy in the video condition,

10{,}000

resamples. Clustered resampling draws the

66

source articles with replacement, keeping all their cases; the width ratio divides the clustered interval’s width by the case-level one in Table 2 .

subgroup	$n$	GPT-5.6- luna	Gemma-4- 31B	MiMo- v2.5	MiniMax- M3	Qwen3.8- flash	Clinician
male	38	47.4	34.2	23.7	28.9	18.4	39.5
female	32	43.8	34.4	21.9	21.9	25.0	28.1
age $<$ 18	4	50.0	50.0	25.0	50.0	25.0	50.0
age 18–59	38	42.1	34.2	21.1	18.4	23.7	28.9
age $\geq$ 60	25	44.0	28.0	28.0	28.0	16.0	36.0
East Asia	21	42.9	33.3	19.0	28.6	9.5	19.0

Appendix

Table 6: Diagnosis accuracy (%) in the video condition by subgroup. The denominator is the number of cases in the subgroup; a case without a usable answer counts as wrong. Strata with

n<10

are shown for completeness only.

Source	PMCID	Licence	Country	Patient(s)
Abbassi O et al., Cureus, 2024	PMC11283634	CC BY	Morocco	chorea, 66F
Antova I et al., Cureus, 2025	PMC12596229	CC BY	Bulgaria	chorea, 62M
Bartl M et al., Neurological research and practice, 2020	PMC7713151	CC BY	Germany	functional, 48F
Batot C et al., Tremor and other hyperkinetic movements, 2022	PMC9122005	CC BY	France	chorea, 90M
Calvo PA et al., Cureus, 2024	PMC11586875	CC BY	Portugal	chorea, 84F
Chen W et al., Frontiers in neurology, 2022	PMC9815763	CC BY	China	chorea, 44F

Appendix

Table 7: Source articles. Licence, country of the first author’s affiliation, and the category, age and sex of each patient filmed ( – : not stated in the source).

	sign recognition (%)		diagnosis accuracy (%)
	dec. / undec.	gap [95% CI]	dec. / undec.	gap [95% CI]
GPT-5.6-luna	30.6 / 22.7	$+7.9$ $[-14.4,+29.2]$	46.9 / 45.5	$+1.5$ $[-23.7,+26.9]$
Gemma-4-31B	14.3 / 22.7	$-8.4$ $[-28.7,+11.3]$	36.7 / 27.3	$+9.5$ $[-13.7,+31.7]$
MiMo-v2.5	26.5 / 22.7	$+3.8$ $[-18.5,+24.0]$	24.5 / 18.2	$+6.3$ $[-14.8,+26.1]$
MiniMax-M3	10.2 / 18.2	$-8.0$ $[-27.7,+9.7]$	30.6 / 13.6	$+17.0$ $[-2.8,+35.8]$
Qwen3.8-flash	26.5 / 27.3	$-0.7$ $[-23.5,+21.1]$	22.4 / 22.7	$-0.3$ $[-22.1,+19.9]$

Appendix

Table 8: Results by clinician-judged decidability. The neurologist judged

22

clips unidentifiable from the video alone, before seeing case material. Recognition is the Stage-1 sign grade at

32

frames ( correct ); accuracy is Stage-2 video-condition accuracy. Cells give decidable (

n{=}49

) / undecidable (

n{=}22

); the gap is their difference with a

10{,}000

-resample bootstrap interval.

	video	shuffled	single	video $-$ shuffled	video $-$ single	shuffled $-$ single
GPT-5.6-luna	46.9	51.0	44.9	$-4.1$ $[-16.3,+6.1]$	$+2.0$ $[-12.2,+16.3]$	$+6.1$ $[-8.2,+20.4]$
Gemma-4-31B	36.7	28.6	28.6	$+8.2$ $[-4.1,+20.4]$	$+8.2$ $[-6.1,+22.4]$	$+0.0$ $[-14.3,+14.3]$
MiMo-v2.5	24.5	26.5	22.4	$-2.0$ $[-18.4,+16.3]$	$+2.0$ $[-14.3,+18.4]$	$+4.1$ $[-10.2,+18.4]$
MiniMax-M3	30.6	14.3	18.4	$+16.3$ $[+2.0,+30.6]$	$+12.2$ $[-2.0,+28.6]$	$-4.1$ $[-16.3,+8.2]$
Qwen3.8-flash	22.4	14.3	22.4	$+8.2$ $[-2.0,+20.4]$	$+0.0$ $[-14.3,+14.3]$	$-8.2$ $[-22.4,+6.1]$
Mean of five	32.2	26.9	27.3	$+5.3$ $[-1.6,+12.2]$	$+4.9$ $[-3.3,+13.5]$	$-0.4$ $[-7.8,+6.9]$

Appendix

Table 9: Visual conditions on the

49

clinician-decidable clips. Stage-2 diagnosis accuracy (%) with

32

ordered frames (video),

32

shuffled frames, and a single frame, and paired differences with

95\%

case-level bootstrap intervals. The last row averages the five systems, resampling cases jointly.

system	gain	more decisive	same decisive	fewer decisive	cases with a decisive entry
GPT-5.6-luna	$+22.5$	24: $+16.9$	39: $+9.9$	8: $-4.2$	34 $\rightarrow$ 45
Gemma-4-31B	$+9.9$	18: $+11.3$	40: $+4.2$	13: $-5.6$	34 $\rightarrow$ 38
MiMo-v2.5	$+14.1$	27: $+14.1$	33: $+2.8$	11: $-2.8$	16 $\rightarrow$ 33
MiniMax-M3	$+15.5$	29: $+16.9$	29: $+2.8$	13: $-4.2$	26 $\rightarrow$ 42
Qwen3.8-flash	$+19.7$	28: $+19.7$	40: $+1.4$	3: $-1.4$	5 $\rightarrow$ 30
Pooled	$+16.3$	126: $+15.8$	181: $+4.2$	48: $-3.7$	115 $\rightarrow$ 188

Appendix

Table 10: Where the video gain arises. For each system the

71

cases are split by whether the video consultation obtains more, the same, or fewer distinct decisive entries than the blind consultation. Each cell gives the number of cases and that group’s contribution to the video

-

blind accuracy gain (percentage points of the

71

-case rate); contributions sum to the gain. The last column counts consultations with at least one decisive entry, blind

\rightarrow

video.

	incorrect	of which (%)
system	(of 71)	decisive entry	no decisive entry,	no results
		returned	some results returned	returned
GPT-5.6-luna	38	37	34	29
Gemma-4-31B	47	34	60	6
MiMo-v2.5	55	35	45	20
MiniMax-M3	53	49	45	6

Appendix

Table 12: Evidence returned in incorrect video-condition consultations. Consultations not graded Accurate , split by the recorded investigation results into three exclusive categories.

source of the entry	entries	decisive	specific values
reported in the source article	$426$	$256$	$394$
derived, category menu (annotators)	$2{,}473$	$44$	$808$
derived, generic panel (DeepSeek-V4.1-Flash)	$3{,}499$	$1$	$278$
total	$6{,}398$	$301$	$1{,}480$

Appendix

Table 13: Chart composition over the

71

cases. Neutral: normal, negative, not performed, not recorded or not tried; specific: any other value.

	original	neutral	neutral $-$ original
Blind	$14.4$	$9.3$	$-5.1$ $[-8.2,-2.3]$
Video	$29.3$	$19.4$	$-9.9$ $[-13.5,-6.2]$
Oracle evidence	$81.4$	$80.0$	$-1.4$ $[-5.4,+2.5]$
Source-clean retrieval	$39.7$	$30.4$	$-9.3$ $[-13.5,-5.1]$
Video $-$ blind	$+14.9$	$+10.1$	$-4.8$ $[-9.3,-0.3]$
Retrieval $-$ video	$+10.4$	$+11.0$	$+0.6$ $[-4.8,+5.9]$

Appendix

Table 14: Neutral-value sensitivity. Accuracy (%), mean of five systems, with the diagnosis turn re-run on fixed trajectories under the original chart and with every derived specific value neutralised. Differences are pooled over systems with

95\%

paired case-level bootstrap intervals; all rows use the

71

cases, with an empty reply counted as incorrect.

system		blind	single frame	shuffled	video	multi-turn	reference
GPT-5.6-luna	questions	29.8	14.9	17.6	16.2	25.1	18.5
GPT-5.6-luna	unknown	92.1	86.5	84.8	85.7	84.8	83.3
Gemma-4-31B	questions	4.7	9.0	8.0	7.9	23.5	6.8
Gemma-4-31B	unknown	82.5	81.1	77.7	77.5	84.4	66.6
MiMo-v2.5	questions	12.4	8.2	7.1	8.2	19.8	9.3
MiMo-v2.5	unknown	89.3	88.7	84.2	84.2	86.4	70.5

Appendix

Table 15: Availability of requested evidence by condition. Per-case means over the usable consultations of the

71

clips;

\tau

as in Table 2 . History reports questions and unknown answers; investigations report orders, unavailable returns, distinct entries released, reported or derived (Table 13 ), and

\tau

. Multi-turn uses up to ten rounds; reference uses audited descriptions.

cases	$n$	video	video $-$ blind	oracle $-$ video	reference $-$ video	video $-$ single	video $-$ shuffled
All cases	71	30.1	$+16.3$ $[+9.3,+23.7]$	$+51.8$ $[+43.4,+60.0]$	$+15.8$ $[+8.7,+22.8]$	$+5.9$ $[-0.3,+12.4]$	$+5.4$ $[-0.3,+11.3]$
Above-median answerability	35	30.3	$+21.7$ $[+13.1,+30.9]$	$+48.6$ $[+36.6,+61.1]$	$+13.1$ $[+2.9,+23.4]$	$+3.4$ $[-5.1,+12.0]$	$+3.4$ $[-5.1,+12.0]$
At or below median	36	30.0	$+11.1$ $[+0.0,+22.2]$	$+55.0$ $[+43.9,+66.1]$	$+18.3$ $[+8.9,+27.8]$	$+8.3$ $[+0.0,+17.2]$	$+7.2$ $[-0.6,+14.4]$
Top third	24	31.7	$+22.5$ $[+11.7,+34.2]$	$+49.2$ $[+33.3,+65.0]$	$+15.8$ $[+3.3,+28.3]$	$+0.0$ $[-10.0,+10.0]$	$+3.3$ $[-7.5,+15.0]$

Appendix

Table 16: Main contrasts by record answerability. Answerability is the share of a case’s questions, pooled over five systems and six conditions, that its record settles (median

14.7\%

). Stage-2 accuracy (%) averaged over systems; differences in points with

95\%

paired case-level bootstrap intervals.

system	blind	video	multi-turn	video $-$ blind	multi-turn $-$ video
GPT-5.6-luna	19.0	34.8	43.6	$+15.9$ $[+8.9,+23.2]$	$+8.8$ $[+1.4,+16.3]$
Gemma-4-31B	7.9	19.5	31.6	$+11.6$ $[+6.0,+17.4]$	$+12.1$ $[+4.7,+19.3]$
MiMo-v2.5	14.7	23.8	38.8	$+9.1$ $[+1.6,+16.7]$	$+15.0$ $[+6.3,+23.8]$
MiniMax-M3	11.9	21.2	37.6	$+9.3$ $[+4.2,+14.7]$	$+16.4$ $[+9.4,+23.2]$
Qwen3.8-flash	7.4	14.8	20.9	$+7.4$ $[+2.5,+12.4]$	$+6.0$ $[+0.3,+11.8]$

Appendix

Table 17: Acquisition of documented history : share (%) of the findings each record documents as present that the consultation’s questions would have revealed, pooled over

71

cases, with

95\%

case-level bootstrap intervals; differences are paired over cases.

category	$n$	example
Decisive finding returned under a second entry for the same test	15	PSP: hummingbird sign under “MRI brain with gadolinium”
Different but reasonable diagnostic route	3	mucormycosis: sinonasal–orbital disease on MRI rather than CT
Decisive order lost to a matching miss	1	“MRI Brain” not matched to the pontine-infarct MRI
No supporting result; diagnosis from video and history	3	post-hypoxic myoclonus without the cardiac-arrest history

Appendix

Table 18: Accurate video-condition consultations with name-only

\tau=0

(

22

107

), classified by a verifier review of the orders, returned results and source report.

Arm	Orders	Unav. (%)	Entries	Accurate (%)	$\Delta$ (pp)	$\tau$ (%)
Released (free)	8.5	35.0	16.7	46.5 [35.2, 57.7]	—	28.9 [22.1, 35.9]
Budget, 10 tests	10	25.1	10.0	43.7 [32.4, 54.9]	$-2.8$ [ $-12.7$ , 7.0]	25.9 [20.1, 31.9]
Budget, single entry	10	29.2	6.5	36.6 [25.4, 47.9]	$-9.9$ [ $-19.7$ , 0.0]	20.9 [16.2, 26.0]
Checklist, 10	10	3.8	12.4	42.3 [31.0, 53.5]	$-4.2$ [ $-14.1$ , 5.6]	22.9 [18.0, 28.0]
Checklist, 15	15	8.5	20.8	40.8 [29.6, 52.1]	$-5.6$ [ $-15.5$ , 4.2]	31.9 [26.4, 37.4]
Atomic, named	10	14.1	8.6	32.4 [21.1, 43.7]	$-14.1$ [ $-23.9$ , $-4.2$ ]	14.6 [10.3, 19.2]

Appendix

Table 19: Order-selection controls on GPT-5.6-luna across

71

clips. Orders denotes mean orders per clip; unav. is the share of orders returning nothing. Entries denotes mean distinct chart entries released per clip, deduplicated across orders within each case (entries divided by orders gives the entries released per order). Brackets show

95\%

case-level bootstrap intervals.

\Delta

is the paired difference in accuracy relative to the released free-text arm, which is the video arm of Table 2 .

\tau

as in Section 3.3 . The two atomic checklists hold the ten-item checklist to one chart entry per order: named resolves each item to a single named chart entry of the case, matched submits the checklist as written and lets the matcher release at most one entry per order. Budget, single entry releases the budget arm’s own orders under the matched rule. Blinded ratings of these lists are in Table 20 .

Strategy	Overall score (1–5)	Orders judged unnecessary	$\tau$ (%)
Random, 10	1.35 / 1.85	7.7 / 5.8	7.3
Checklist, 10	2.82 / 2.67	2.6 / 2.9	22.9
Atomic, named	2.80 / 2.79	2.6 / 3.1	14.6
Atomic, matched	2.76 / 2.77	2.8 / 3.1	15.9
Model-selected, 10	2.82 / 2.69	3.4 / 3.6	25.9

Appendix

Table 20: Blinded rating of the ten-item order lists on all

71

clips. Paired values report Rater 1 / Rater 2; scores and unnecessary-order counts are means;

\tau

is computed on the same clips.

$K$	$\Delta$ accuracy (pp)	$\Delta\tau$ (pp)
8	$-0.5$ $[-7.5,+6.6]$	$+0.9$ $[-2.2,+4.0]$
16	$+1.4$ $[-7.0,+9.4]$	$+0.7$ $[-1.9,+3.3]$
32	$+3.3$ $[-4.2,+10.8]$	$-1.6$ $[-5.0,+1.8]$

Appendix

Table 21: Paired window-minus-random differences (percentage points; 95% paired case-level bootstrap intervals, 71 clips). W: window arm; R: random window of the same length. Both arms share the adapter and the

32

-frame survey, as in Table 3 .

sentence	body part	character	side	condition	judged closer (clips)
no adapter	27/68 (40%)	7/68 (10%)	3/57 (5%)	13/34 (38%)	14
window-trained adapter	45/68 (66%)	37/68 (54%)	5/57 (9%)	6/34 (18%)	48

Appendix

Table 22: Blinded judgement of the student’s sign sentences ,

K{=}32

with the

32

-frame survey. For each clip the judge saw the reference phenomenology and the two sentences and marked each component independently; unstated components are not scored, so denominators differ.

$K$	acc. [95% CI]	cov.	acc. [95% CI]	cov.	$\Delta$ acc. [95% CI]	$\Delta$ cov. [95% CI]
	untrained		adapter		adapter $-$ untrained
8	23.9 [16.4, 31.9]	13.4	29.1 [20.7, 38.0]	17.3	$+5.2$ $[-3.8,+14.1]$	$\mathbf{+3.9}$ $[+0.1,+7.9]$
16	22.1 [14.6, 30.5]	13.4	30.0 [21.1, 39.4]	18.6	$+8.0$ $[-1.4,+17.4]$	$\mathbf{+5.2}$ $[+1.5,+9.2]$
32	27.2 [18.3, 36.6]	14.3	33.8 [24.4, 43.7]	19.4	$+6.6$ $[-1.9,+15.5]$	$\mathbf{+5.1}$ $[+0.6,+9.8]$

Appendix

Table 23: Untrained Qwen3.5-4B versus the window-trained student on the same survey

+

window input (

32

-frame survey, same day, denominator

71

, three runs per clip, clip-level bootstrap). Coverage is source-workup coverage (

\tau

); differences are paired over clips.

$K$	arm	acc. [95% CI]	cov.	$\Delta$ acc. [95% CI]	$\Delta$ cov. [95% CI]
8	no window	26.8 [19.2, 34.3]	19.0	–	–
	window	29.1 [20.7, 38.0]	17.3	$-2.3$ $[-10.8,+5.6]$	$+1.8$ $[-1.0,+4.6]$
	random	29.6 [21.6, 38.5]	16.4	$-2.8$ $[-10.8,+4.7]$	$+2.7$ $[-0.3,+5.8]$
	untrained	23.9 [16.4, 31.9]	13.4	$+2.8$ $[-5.2,+10.3]$	$\mathbf{+5.6}$ $[+2.1,+9.5]$
16	no window	27.2 [19.2, 35.7]	18.3	–	–
	window	30.0 [21.1, 39.4]	18.6	$-2.8$ $[-11.7,+6.1]$	$-0.3$ $[-3.1,+2.4]$

Appendix

Table 24: Sign-only adapter trained without windows versus the window-trained adapter at matched frame budget (denominator

71

, three runs per clip, clip-level paired bootstrap). No window: adapter trained and evaluated on

32{+}K

evenly spaced frames. Window / random: the window-trained adapter on the

32

-frame survey plus

K

frames from the student’s predicted window or from a random window of the same length. Untrained: Qwen3.5-4B on the survey-plus-window input. Coverage is source-workup coverage (

\tau

); differences are no window minus the listed arm.

Figure 9: One clip at three frame budgets. Uniformly sampled frames from the same $17.8$ -second clip at $K{=}1$ , $4$ and $8$ . Frames share a common scale; each row shows the full sampling budget.

$K$	Phenomenology recognition	Differential diagnosis	Sign grade
$1$	“lying on their side … a prominent dark/purplish area with swelling around the ear/temple and upper cheek”	acute trauma to the head; periorbital or auricular haematoma	wrong
$2$	“repeatedly bend forward and hold/rub the upper abdomen or lower chest”	upper abdominal pain; gastritis, ulcer or biliary disease	wrong
$4$	“markedly bent forward/stooped … intermittent, somewhat irregular/flinging or jerky movement of the opposite arm”	camptocormia from Parkinson disease or another parkinsonian syndrome	partial
$8$	“the movements look irregular, flowing, and somewhat twisting rather than rhythmic”	hyperkinetic movement disorder; chorea or choreoathetosis, possibly hemichorea	correct
$32$	“irregular, seemingly involuntary movements … jerky and variable, with prominent upper-limb and whole-body involvement”	functional movement disorder; “the marked variability … support[s] this over a fixed neurologic syndrome”	correct
$128$	“recurrent, forceful, involuntary movements of the trunk and upper limbs … episodic and somewhat irregular”	functional movement disorder, particularly functional truncal jerks or dystonia	partial

Appendix

Table 25: Responses at six frame budgets. Phenomenology recognition and differential diagnosis, quoted from the responses. Sign grade is the grader’s rating of phenomenology recognition.

Figure 10: Recognition and aetiological coverage by category. Phenomenology recognition and aetiological coverage pooled over five models and eight frame budgets and ordered by their difference. Facial palsy is excluded because it contains only one clip.

Figure 11: Interaction counts under retrieval conditions (71 cases): ( a ) questions asked and ( b ) investigations ordered per case. Colours denote no retrieval (grey), Own candidates (orange), Original (light blue), Source-clean (blue), and Strict no-answer (dark blue). Bars show case-level means with 95% bootstrap intervals.

Figure 12: The retrieved list covers causes the model misses. Coverage of reference causes by the model’s initial differential and the Source-clean retrieved list. Stacked segments distinguish causes covered only by the model, by both lists, or only by retrieval; the total is their union.

System	Condition	$\Delta$ accuracy [95% CI]	$\Delta\tau$ [95% CI]
GPT-5.6-luna	Source-clean	$+2.8$ $[-9.9,+15.5]$	$+8.3$ $[+0.9,+15.6]$
	Strict no-answer	$+1.4$ $[-11.3,+14.1]$	$+5.3$ $[-3.0,+13.4]$
Gemma-4-31B	Source-clean	$-2.8$ $[-14.1,+8.5]$	$+0.7$ $[-5.7,+6.5]$
	Strict no-answer	$-4.2$ $[-15.5,+7.0]$	$-2.0$ $[-8.3,+3.9]$
MiMo-v2.5	Source-clean	$+21.1$ $[+8.5,+33.8]$	$+13.3$ $[+5.0,+21.3]$
	Strict no-answer	$+15.5$ $[+2.8,+28.2]$	$+11.0$ $[+4.7,+17.3]$

Appendix

Table 26: Retrieval conditions minus the video condition on the same

71

clips (percentage points,

95\%

case-level bootstrap intervals). Accuracy and coverage

\tau

as in Table 2 .

	candidate coverage (%)			accuracy (%)
system	original	source-clean	strict	original	source-clean	strict
GPT-5.6-luna	70.4	63.4	52.1	50.7	49.3	47.9
Gemma-4-31B	70.4	64.8	52.1	29.6	31.0	29.6
MiMo-v2.5	64.8	62.0	47.9	36.6	43.7	38.0
MiniMax-M3	69.0	63.4	47.9	45.1	40.8	49.3
Qwen3.8-flash	67.6	64.8	52.1	33.8	33.8	33.8

Appendix

Table 27: Candidate coverage and accuracy under the three filtering levels (

71

cases). Coverage is the share of cases whose supplied list (the model’s own Stage 1 differential followed by the retrieved causes) contains the true cause, graded with the Appendix H rank probe; accuracy is the Accurate rate of the consultation.

system	arm	acc. (%)	$\tau$ (%)	orders	quality	unnecessary (%)
GPT-5.6-luna	video	46.5	28.9	8.5	3.38 / 3.10	22 / 25
	own	52.1	37.9	7.5	3.56 / 3.20	19 / 24
	clean	49.3	37.2	9.1	3.35 / 3.22	24 / 25
Gemma-4-31B	video	33.8	20.3	3.5	2.67 / 2.46	24 / 31
	own	18.3	12.0	3.0	2.43 / 2.46	32 / 38
	clean	31.0	20.9	3.4	2.76 / 2.52	20 / 36

Appendix

Table 28: Retrieval under the full set of measures (

71

cases). Accuracy and

\tau

as in Table 2 ; orders is the mean per case. Quality (

1

–

5

) and unnecessary share (unnecessary orders over orders placed) are blinded ratings of each order list, reported as Rater 1 / Rater 2. Own: the model’s own Stage 1 differential without retrieval; clean: the same list followed by source-clean retrieved causes.

system	video	own	mismatched	clean	clean $-$ mismatched	mismatched $-$ own
GPT-5.6-luna	46.5 / 28.9	52.1 / 37.9	53.5 / 37.9	49.3 / 37.2	$-4.2$ $[-15.5,+7.0]$	$+1.4$ $[-7.0,+9.9]$
Gemma-4-31B	33.8 / 20.3	18.3 / 12.0	22.5 / 17.3	31.0 / 20.9	$+8.5$ $[-2.8,+19.7]$	$+4.2$ $[-5.6,+14.1]$
MiMo-v2.5	22.5 / 16.3	28.2 / 15.3	22.5 / 16.6	43.7 / 29.6	$+21.1$ $[+9.9,+32.4]$	$-5.6$ $[-16.9,+5.6]$
MiniMax-M3	25.4 / 19.3	18.3 / 17.9	29.6 / 22.9	40.8 / 27.2	$+11.3$ $[-1.4,+23.9]$	$+11.3$ $[0.0,+22.5]$
Qwen3.8-flash	22.5 / 16.3	18.3 / 12.6	31.0 / 15.9	33.8 / 25.2	$+2.8$ $[-7.0,+12.7]$	$+12.7$ $[+2.8,+22.5]$

Appendix

Table 29: Length- and form-matched control for retrieval (

71

cases). Cells give accuracy /

\tau

(%). Own: the model’s own Stage 1 differential; mismatched: the same list followed by source-clean causes retrieved for an unrelated case, truncated to the length of the source-clean list; clean: source-clean retrieval. Differences are in accuracy percentage points with

95\%

paired case-level bootstrap intervals. The paired

\tau

difference, clean

-

mismatched, is

-0.7

[-7.4,+6.2]

for GPT-5.6-luna,

+3.7

[-3.9,+11.0]

for Gemma-4-31B,

+13.0

[+6.7,+19.6]

for MiMo-v2.5,

+4.3

[-1.6,+10.2]

for MiniMax-M3 and

+9.3

[+4.3,+14.5]

for Qwen3.8-flash.

Figure 13: Consultation outcomes across history-corruption levels : ( a ) diagnostic accuracy, ( b ) investigation count, and ( c ) source-workup coverage. The video and history questions are held fixed; investigation orders and final diagnoses are generated using the modified responses. Shaded bands are marginal 95% bootstrap intervals over clips.

	Question	Basis	Uncorr.	Corr.
1 ∗	Are you experiencing involuntary shaking, jerking, or flapping movements?	S1	yes	no
2 ∗	Did these movements begin suddenly?	—	unknown	yes
3 ∗	Are the movements present when your hands are at rest?	S2	no	yes
4	Do the movements worsen when you hold your arms out or use your hands?	S2	yes	yes
5	Do both hands move in the same way?	—	unknown	unknown
6 ∗	Have you noticed weakness or clumsiness on one side of your body?	—	unknown	yes

Appendix

Table 30: History questions and source-aligned responses in the worked example (a 34-year-old man, the sole patient of the source case report). Basis keys into Table 31 , which quotes the supporting source evidence. A dash means no passage is quoted here, not verified absence. Question 8 is supported by S6. Starred rows are corrupted at the nominal

40\%

rate.

Key	Chart entry	Sentence in the source	Loc.
S1	hand tremor when holding a posture: yes ; the tremor started about six months ago: yes	“[He] came to our hospital with a complaint of tremors of the hands of 6 months duration.” — “The tremor was limited to the hands.”	Case pres.
S2	tremor at rest: no	“We noted a postural tremor that attenuated when the patient assumed an anatomic position.”	Case pres.
S3	reduced sensation in the feet: yes	“His sensory examination revealed a mildly decreased vibratory sensation; otherwise, he had no sensory level.”	Clin. findings
S4	known liver disease: no ; known kidney disease: no ; diabetes: no	“[T]he patient had no history of diabetes, hypertension, psychiatric, cardiac, renal, liver diseases, or cancers.”	Case pres.
S5	illicit drug use: no	“He had no history of illicit drug use and denied constitutional symptoms.”	Case pres.
S6	difficulty walking: yes	Walking difficulty and bilateral distal weakness are documented.	Patient info.

Appendix

Table 31: Patient-level provenance for every non- unknown response in the worked example. Sentences quote the single-patient source report ( Mohammed et al., 2024 ) ( PMC11600610 , Journal of Medical Case Reports , CC BY-NC-ND 4.0); Loc. gives its section.

Stage	Outcome	$n$	Agreement	$\kappa_{w}$
1	Phenomenology recognition	$96$	$89.6\%$	$0.903$
	Aetiology	$96$	$96.9\%$	$0.921$
2	Final diagnosis	$71$	$91.5\%$	$0.947$

Appendix

Table 32: Agreement between the model grader and the independent human reviewer ; Stage 2 evaluation is restricted to the video condition (GPT-5.6-luna, all

71

consultations).

Model / Human	Accurate	Partial	Not accurate
Accurate	$32$	$1$	$0$
Partial	$1$	$10$	$2$
Not accurate	$0$	$2$	$23$

Appendix

Table 33: Paired Stage 2 grades in the video condition (

71

consultations). Rows represent the model grader and columns the human reviewer.

system	grader	video	source-clean	$\Delta$
GPT-5.6-luna ( $71$ )	human	46.5	53.5	$+7.0$
	automatic	46.5	49.3	$+2.8$
MiMo-v2.5 ( $71$ )	human	26.8	47.9	$+21.1$
	automatic	22.5	43.7	$+21.1$
MiniMax-M3 ( $71$ )	human	31.0	45.1	$+14.1$
	automatic	25.4	40.8	$+15.5$

Appendix

Table 34: Paired re-grading of video-condition and source-clean final diagnoses. Accurate rates in percent;

\Delta

is source-clean minus video (pp); paired clip counts follow each system.

Medical Diagnosis

Medical diagnosis uses clinical data, tests, and images to identify diseases, assess risk, and guide care.

See all 262 papers

May 22, 2026

DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs

Medical diagnosis is not a single prediction from a fully specified vignette. It is a sequential workup: clinicians decide what evidence to obtain, revise a differential diagnosis, and stop when the diagnosis is sufficiently supported. Most medical AI benchmarks instead reveal the relevant context upfront and score only the final answer, making unsupported correct guesses, premature closure, inefficient workups, and poor uncertainty updating invisible. We introduce DDX-TRACE, a physician-adjudicated benchmark for multimodal neuroradiology that evaluates diagnostic trajectories under hidden evidence over 211 challenging cases. Each case begins with limited clinical history; models request imaging studies in free form, receive matched image bundles when available, update a probabilistic differential diagnosis after each turn, and stop with a localized final diagnosis. Evaluating state-of-the-art VLMs, we find that final diagnosis scores can substantially misrepresent workup quality: models may guess plausible diagnoses without essential evidence, request useful studies but misinterpret raw images, or acquire evidence inefficiently while updating uncertainty poorly. Controlled evidence variants isolate bottlenecks in planning, visual evidence extraction, and downstream differential reasoning. DDX-TRACE shifts medical AI evaluation from final answers to evidence-supported diagnostic trajectories.

Jiazhen Pan, Weixiang Shen, Jun Li +7

Technical University of Munich (TUM) · TUM University Hospital · Munich Center for Machine Learning (MCML) +3

Medical Imaging Radiology VLMs Healthcare Clinical Reasoning Multimodal Model Evaluation Differential Diagnosis+1

Ask PDF

Jul 14, 2026

Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

Benchmark accuracy in video large language models (LLMs) is often treated as evidence of visual understanding. We audit this assumption across twenty models spanning 2-78B parameters and ten architecture families. We introduce the Visual Dependency Gap (VDG), the difference in per-question correctness between original-video and black-screen conditions. Paired McNemar tests on MVBench show that accuracy and visual dependency are separable: models differ on original video (p = 0.0003) but not on black screens (p = 0.53). Across models, task-type rankings are stable: Attribute Perception is strongly visual, whereas Temporal Reasoning approaches the language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original video reveals that frame diversity supplies most of the visual benefit, while temporal order contributes near-zero accuracy across sixteen open-weight models. An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. The diagnostic also generalizes to four API-accessed models, whose VDG values range from 0.025 to 0.315. These results motivate VDG as a standard audit for whether video benchmarks measure visually grounded capability. Code is available at https://github.com/JaeLee18/accuracy-without-grounding.

Jae Joong Lee

Department of Computer Science Purdue University West Lafayette, Indiana, USA

VLM Evaluation Video-Language Model Evaluation Video-Language Models Temporal Video Understanding

Ask PDF

Aug 26, 2026

Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos

While LVLMs rapidly improve, long-video question answering still remains challenging: relevant evidence is sparse, and question-relevant context often fails to provide cues that discriminate the correct answer from plausible alternatives. Diagnostic analysis on a manually annotated subset of MMR-V shows that prior agentic systems substantially improve cue retrieval over direct VLM inference yet fail to achieve a corresponding gain in answer accuracy, indicating that the bottleneck lies in option-discriminative evidence rather than topical relevance alone. We propose PACE (Progressive Acquisition of Critical Evidence), a factor-guided framework for long-video evidence acquisition. PACE proceeds in two stages: it first indexes clip-level descriptions guided by question-derived factors without observing the candidate answers; it then uses the candidate answers to derive contrastive cues and queries the index for verification. On MMR-V with the open-source Qwen3-VL backbone, PACE achieves 42.6% accuracy, outperforming direct inference and prior agentic baselines including Deep Video Discovery (DVD). On the same diagnostic subset, PACE recovers 66.9% of the annotated cues, providing empirical evidence that its gains are associated with improved evidence recovery rather than stronger answer-side priors alone. Consistent gains over DVD on LVBench, Video-MME, EgoSchema, and LongVideoBench suggest that option-aware evidence acquisition transfers beyond MMR-V. Code is available at https://github.com/HKUST-KnowComp/PACE.

Baixuan Xu, Yinyui Xu, Tianshi Zheng +9

Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China

Long-Video QA Long-Video Understanding Evidence Retrieval Memory-Augmented Video Understanding

Ask PDF

Abstract

Medical Diagnosis

Explore similar work

DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs

Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos