Video question answering (VideoQA) is a challenging task that requires integrating spatial, temporal, and semantic information to capture the complex dynamics of video sequences. Although recent advances have introduced various approaches for video understanding, most existing methods still rely on locating relevant frames to answer questions rather than reasoning through the evolving storyline as humans do. Humans naturally interpret videos through coherent storylines, an ability that is crucial for making robust and contextually grounded predictions. To address this gap, we propose SVAgent, a storyline-guided cross-modal multi-agent framework for VideoQA. The storyline agent progressively constructs a narrative representation based on frames suggested by a refinement suggestion agent that analyzes historical failures. In addition, cross-modal decision agents independently predict answers from visual and textual modalities under the guidance of the evolving storyline. Their outputs are then evaluated by a meta-agent to align cross-modal predictions and enhance reasoning robustness and answer consistency. Experimental results demonstrate that SVAgent achieves superior performance and interpretability by emulating human-like storyline reasoning in video understanding.
Figures & tables
Figure 2: Overview of SVAgent , which consists of six interacting agents: (1) a Storyline Agent that summarizes sampled frames into a query-oriented narrative, (2) a Hypothesis Agent that proposes and evaluates hypotheses while selecting evidence frames via Determinantal Point Processes (DPPs), (3) two Cross-Modal Decision Agents that respectively conduct modality-specific reasoning to produce decisions, (4) a Meta-Decision Agent that verifies the consistency of cross-modal decisions, and (5) a Suggestion Agent that uses error history to propose informative frames for iterative refinement. The ✓ indicates successful cross-modal decision confirmation, while the × denotes inconsistency detection that triggers the refinement feedback loop.
Model
Size
# Frames
LongVideoBench
MLVU
LVBench
VideoMME (w/o vs. w/ sub.)
val (w/o sub.)
test
test
short
medium
long
overall
Proprietary Video MLLMs
Gemini 1.5 Pro [ 10 ]
-
1fps
64.0
64.0
33.1
81.7 / 84.5
74.3 / 81.0
67.4 / 77.4
75.0 / 81.3
GPT-4o [ 28 ]
-
1fps
66.7
64.6
48.9
80.0 / 82.8
70.3 / 76.6
65.3 / 72.1
71.9 / 77.2
Open-source Video MLLMs
LLaVA-Video [ 59 ]
72B
32 / 64 ∗
61.9
72.9
38.7
80.7 / 81.8
68.7 / 73.8
62.1 / 72.2
70.3 / 75.9
Table 1: Comparison on Long Video Understanding Benchmarks.
Table 2: Ablation Studies on Qwen2.5VL-7B.
Figure 3: Parameter and Latency Analysis on Video-MME.
Retrieval Model
LongVideoBench
MLVU
LVBench
VideoMME
val (w/o sub.)
test
test
long
overall
CLIP [ 30 ]
60.2
61.9
40.4
54.2 / 56.6
60.7 / 65.2
LongCLIP [ 53 ]
59.9
63.1
39.8
52.5 / 55.1
60.4 / 64.9
Siglip
60.7
62.7
40.6
53.3 / 57.3
61.2 / 66.9
Table 3: Ablation study on the retrieval model in DPP.
Figure 4: Comparison of SVAgent and uniform sampling under different frame budgets on Video-MME.
Figure 5: Case study showing how SVAgent maintains coherent temporal reasoning and consistent cross-modal verification. By combining storyline-guided abstraction with complementary text and vision decisions, the system retrieves stable evidence across long sequences and converges on reliable explanations for complex video events.
Method
Mean (%)
StdDev
t -test p
Wilcoxon p
Raw
54.68
0.89
7.09×10−10
9.77×10−4
SVAgent
62.76
0.31
Table 4: Statistical comparison on the MLVU benchmark. StdDev denotes standard deviation. We report one-sided paired t -test and Wilcoxon signed-rank test p -values ( α=5% ). Both tests show that SVAgent significantly outperforms Raw .
School of Computer Science, Wuhan University, Wuhan, 430072, Hubei, China. · 2National Engineering Research Center for Multimedia Software, China. · 3Hubei Key Laboratory of Multimedia and Network Communication Engineering, China.