This paper presents a multi-agent perception-action exploration alliance, dubbed A4VL, for efficient long-video reasoning. A4VL operates in a multi-round perception-action exploration loop with a selection of VLM agents. In each round, the team of agents performs video question-answer (VideoQA) via perception exploration followed by action exploration. During perception exploration, each agent learns to extract query-specific perception clue(s) from a few sampled frames and performs clue-based alignment to find the video block(s) that are most relevant to the query-specific event. During action exploration, A4VL performs video reasoning in three steps: (1) each agent produces its initial answer with rational, (2) all agents collaboratively scores one another through cross-reviews and relevance ranking, and (3) based on whether a satisfactory consensus is reached, the decision is made either to start a new round of perception-action deliberation by pruning (e.g., filtering out the lowest performing agent) and re-staging (e.g., new-clue and matching block based perception-action exploration), or to conclude by producing its final answer. The integration of the multi-agent alliance through multi-round perception-action exploration, coupled with event-driven partitioning and cue-guided block alignment, enables A4VL to effectively scale to real world long videos while preserving high quality video reasoning. Evaluation Results on five popular VideoQA benchmarks show that A4VL outperforms 18 existing representative VLMs and 11 recent methods optimized for long-video reasoning, while achieving significantly lower inference latency. Our code is released at https://github.com/git-disl/A4VL.
Figures & tables
Figure 1: Responses from A4VL and three of the strongest closed-sourced models.
Figure 2: Overview of A4VL architecture.
Figure 3: A4VL agent teaming workflow.
Figure 4: Example of A4VL on EgoSchema. Here all agents have wrong answers at the beginning, but later one of them turns to the correct answer and other agents are pruned.
Model
NeXT-QA
EgoSchema
LongVideoBench
MLVU
Video-MME
Short
Medium
Long
Average
GPT-4o* [ 25 ]
-
72.2
66.7
54.9
80.0/82.8
70.3/76.6
65.3/72.1
71.9/77.2
Gemini 1.5Pro* [ 37 ]
-
71.1
64.0
-
81.7/84.5
74.3/81.0
67.4/77.4
75.0/81.3
LLaVA-Video-7B-Qwen2 [ 55 ]
81.2
61.8
61.1
51.7
72.4/75.6
58.1/63.3
50.7/60.4
60.4/66.4
VideoLlama3-7B [ 49 ]
67.4
52.6
52.5
43.9
80.1/80.2
63.7/69.6
54.9/61.0
66.2/70.3
InternVL3.5-8B [ 41 ]
76.2
62.0
53.6
47.1
69.6/74.0
56.3/63.9
47.7/61.9
57.9/66.6
Table 1: A4VL compared with other methods on five benchmarks. For Video-MME, the result is in format w. subtitle/wo. subtitle. For Video-MME, the improvement row shows the improvement without subtitle, since A4VL is mainly designed to optimize visual reasoning. * near the model names mean we take the results from the model paper or benchmark leaderboard.
Benchmark
Models
NeXT-QA
Intern-78B
Intern-38B
QwenVL-72B
EgoSchema
Intern-78B
Intern-38B
QwenVL-72B
LongVideoBench
Intern-78B
Intern-38B
QwenVL-72B
MLVU
Intern-78B
Intern-38B
LLaVA-72B
Video-MME
Intern-78B
LLaVA-72B
QwenVL-72B
Table 2: Models used for each benchmark. Intern-78B refers to InternVL3-78B [ 41 ] , Intern-38B refers to InternVL3.5-38B [ 41 ] , QwenVL-72B refers to QwenVL-2.5-72B [ 38 ] , and LLaVA-72B refers to LLaVA-Video-72B-Qwen2 [ 55 ] .
Model
NeXT-QA
EgoSchema
MLVU
GPT-4o
23s
54s
127s
InternVL3-78B
15s
50s
204s
VideoAgent
20s
83s
175s
TraveLER
101s
94s
450s
A4VL
18s
37s
74s
Table 3: Average inference time per sample on four representative methods and A4VL.
Figure 5: Relationship between accuracy and maximum number of rounds.
#Rounds
1
2
3
NeXT-QA
1072
230
191
EgoSchema
303
101
96
LongVideoBench
566
341
430
MLVU
183
116
203
Video-MME
1289/1667
557/421
854/612
Table 4: Number of actual running rounds in A4VL. The data of Video-MME is shown in w/o subtitle / w. subtitle format.
Method
RRSampling
RESampling
ERSampling
Accuracy (%)
80.2
82.2
79.6
Time/Sample
35s
37s
37s
Table 5: Accuracy and latency of different sampling strategies on EgoSchema. R means random sampling and E means sampling from event-based blocks. The first character represents the sampling strategy for N1 perception frames, and the second character represents the sampling strategy for N2 action frames.
Method
MConsens
FConsens
Accuracy (%)
81.4
82.2
Time/Sample
26s
37s
Table 6: Results of different consensus conditions on EgoSchema. M means majority and F means full.
Method
NoPruneSum
NoPruneMaj
A4VL
Accuracy (%)
80.8
79.4
82.2
Time/Sample
60s
60s
37s
Table 7: Results of different pruning strategies on EgoSchema.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm 1 A4VL Algorithm.
Figure 6: Visualization of event-based video partitioning.
Figure 7: Example where the majority is wrong in the first round, but finally A4VL produces the correct answer. The example is taken from NeXT-QA.
Figure 8: Failure case where all agents achieve a wrong consensus in the first round. The example is taken from NeXT-QA.
Figure 9: Failure case where the majority is correct in the first round, but finally the final answer gets wrong. This example is taken from Video-MME (w/o sub).