Agentic video understanding requires models to actively interact with video environments to gather evidence and form judgments. Evaluating this capability therefore requires assessing whether models can translate opportunities for interaction into better understanding. We introduce VideoACE, a benchmark for assessing video understanding under conditions that support sustained interaction. We focus on high-frame-rate action videos, where rapid, subtle changes make targeted observation especially valuable. The benchmark comprises 1,000 multiple-choice questions spanning six categories of fine-grained action understanding, accompanied by temporal evidence annotations and user hints screened for answer leakage. Its evaluation environment combines video inspection tools with a user simulator, allowing models to adjust temporal coverage, sampling density, and spatial detail, and to revisit evidence and revise their judgments under simulated user feedback. We evaluate single-pass observation, autonomous tool use, and tool use combined with user feedback to examine how effectively models exploit these opportunities. Evaluation of five models reveals distinct patterns in how they acquire evidence and respond to opportunities for further interaction. These findings establish VideoACE as a testbed for examining agentic video understanding beyond performance under a fixed observation setting.
Figures & tables
Figure 1: Targeted video inspection for understanding a fast action.
Type
What must be resolved
n
Binding
Which person acts
75
Interaction
Who acts on whom
165
Order
Which action comes first
227
Counting
How often an action repeats
84
Spatial
Where and in which direction
132
Transient
Which brief motion occurs
317
Table 1: Six action reasoning skills. Counts refer to questions; each question is assigned one primary type.
Figure 2: The action tasks used to evaluate interaction. Left: source clip categories. Center: question counts by estimated temporal density. Right: annotated evidence-window lengths, with a median of 0.90 s.
Figure 3: Question construction and the autonomous observation path. (a) Dense video and captions support question generation; a weaker caption supplies candidate mistakes. Option refinement controls surface form and balances answer positions. (b) Starting from a sparse overview, a model requests further observations and accumulates evidence before answering. The User Simulator ( section 4.3 ) extends this interaction through feedback.
Model
Single-pass (%)
Tool use (%)
Gain
Gemini 4 Argon
80.8
88.1
+7.3
Gemini 3.8 Flash
75.4
86.4
+11.0
Gemini 3.6 Flash
65.6
79.8
+14.2
Qwen3.8-Flash
63.5
76.2
+12.7
Qwen3.7-Flash
56.3
59.3
+3.0
Table 2: Autonomous tool use on VideoACE. Each model answers all 1,000 questions. Gain is the accuracy difference in percentage points. The shared interface provides the same operations, but models differ in the benefit they obtain.
Model
R1
Deferred
Rounds 1 / 2 / 3
Single ($)
Agentic ($)
Gemini 4 Argon
79.9
0
1 / 266 / 33
12.81
72.39
Gemini 3.8 Flash
71.8
18
1 / 152 / 147
5.76
26.96
Gemini 3.6 Flash
64.1
0
1 / 279 / 20
3.02
16.19
Qwen3.8-Flash
60.1
23
0 / 101 / 199
2.96
11.59
Qwen3.7-Flash
53.8
59
167 / 124 / 9
1.68
3.48
Table 3: Round usage and recorded cost. R1 is first-round agentic accuracy (%); deferred answers count as incorrect. Round counts sum to 300 groups. Costs (USD) are logged estimates for completed records.
Figure 4: Accuracy on the temporal-density strata. Each segment connects single-pass accuracy (open circle) to autonomous tool-use accuracy (filled circle).
Figure 5: Autonomous tool-use gains by action task, in percentage points. Each cell uses all questions of that type; sample sizes appear in table 1 . Color is centered at zero and numerical values retain the sign.
Figure 6: Selected successes and regressions. Four selected questions with real source frames and predictions from all five models. The first two are fixed by every model, the third by four models, and the fourth is broken by all five. English text paraphrases the Chinese questions and selected options. Frames are taken from the annotated evidence. Boxed predictions match the annotated key.
Figure 7: Performance and use of interaction opportunities. (a) Accuracy through the combined tool-and-feedback episode, always over 1,000 questions; early-ending episodes carry their final answer forward. (b) Inner-round usage in autonomous tool sessions. (c) Outer-turn usage with feedback. Each distribution covers 300 groups; segment labels are rounded percentages, and narrow segments are unlabeled. Inner rounds and outer turns are distinct units.
Model
First (%)
Final (%)
Gain
Gemini 4 Argon
84.8
87.1
+2.3
Gemini 3.8 Flash
80.3
83.5
+3.2
Gemini 3.6 Flash
73.0
78.0
+5.0
Qwen3.8-Flash
68.4
73.2
+4.8
Qwen3.7-Flash
51.5
60.3
+8.8
Table 4: Tool use with user feedback. First and final refer to outer turns, each containing an inner tool session. All models answer the full 1,000 questions.
Measure
vanilla
VideoACE
Median evidence duration (s)
3.00
0.90
24 fps annotations
34
307
Transient-detail questions
50
317
Strictly longest key (%)
32.6→6.3
42.7→8.5
Mean relative length spread
.203→.037
.205→.038
Final keys per letter
250
250
Table 5: Task-design and option-quality statistics for two independently constructed question sets. Each has 1,000 items; source-video overlap is 100/100. Arrows compare options before and after refinement within each set.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Type
n
Gemini 4 Argon
Gemini 3.8 Flash
Gemini 3.6 Flash
Qwen3.8-Flash
Qwen3.7-Flash
binding
75
80.0 → 88.0
78.7 → 92.0
61.3 → 84.0
52.0 → 78.7
44.0 → 57.3
interaction
165
86.7 → 90.3
81.2 → 87.9
71.5 → 78.2
66.7 → 75.8
53.9 → 59.4
order
227
84.1 → 91.6
80.2 → 91.2
65.2 → 86.8
65.2 → 78.0
63.9 → 68.7
counting
84
42.9 → 69.0
42.9 → 76.2
36.9 → 75.0
45.2 → 60.7
42.9 → 46.4
spatial
132
84.1 → 90.2
68.9 → 84.1
65.9 → 73.5
60.6 → 78.0
57.6 → 58.3
transient
317
84.2 → 88.6
79.5 → 84.5
71.3 → 78.5
69.4 → 77.9
58.0 → 56.8
Appendix
Table A1: Accuracy (%), Single → video harness, by question type on fineaction.
fmin
n
Gemini 4 Argon
Gemini 3.8 Flash
Gemini 3.6 Flash
Qwen3.8-Flash
Qwen3.7-Flash
8
693
82.4 → 89.0
76.8 → 86.6
68.0 → 80.1
65.2 → 78.1
57.7 → 59.0
24
307
77.2 → 86.0
72.3 → 86.0
60.3 → 79.2
59.6 → 72.0
53.1 → 59.9
Appendix
Table A2: Accuracy (%), Single → video harness, by model-annotated minimum frame rate. These labels are not exposed to the evaluated model.
Table 15
Figure A1 : Paired answer changes and inspection depth. Left: fixes and breaks relative to the independent single-pass run; numbers are question counts. Right: executed answer rounds across 300 groups per model.