Organizations: Institute of Automation, Chinese Academy of Sciences · School of Future Technology, University of Chinese Academy of Sciences · Kuaishou Technology
Despite rapid developments and widespread applications of MLLM agents, they still struggle with long-form video understanding (LVU) tasks, which are characterized by high information density and extended temporal spans. Recent research on LVU agents demonstrates that simple task decomposition and collaboration mechanisms are insufficient for long-chain reasoning tasks. Moreover, directly reducing the time context through embedding-based retrieval may lose key information of complex problems. In this paper, we propose Symphony, a multi-agent system, to alleviate these limitations. By emulating human cognition patterns, Symphony decomposes LVU into fine-grained subtasks and incorporates a deep reasoning collaboration mechanism enhanced by reflection, effectively improving the reasoning capability. Additionally, Symphony provides a VLM-based grounding approach to analyze LVU tasks and assess the relevance of video segments, which significantly enhances the ability to locate complex problems with implicit intentions and large temporal spans. Experimental results show that Symphony achieves state-of-the-art performance on LVBench, LongVideoBench, VideoMME, and MLVU, with a 5.0% improvement over the prior state-of-the-art method on LVBench. Code is available at https://github.com/Haiyang0226/Symphony.
Figures & tables
Figure 1: (a) Single-agent approaches face limitations in reasoning capacity when handling long-video understanding tasks requiring multi-step reasoning. (b) Our proposed multi-agent system achieves enhanced reasoning capabilities through task decomposition and collaboration along functional dimensions.
Figure 2: The reflection-enhanced dynamic reasoning framework in Symphony. The planning agent formulates a task plan and dynamically invokes other agents to execute subtasks. Upon obtaining an initial solution, the Reflection agent evaluates the reasoning chain τ , producing a critique C that guides a subsequent round of reasoning exploration.
Figure 3: The CLIP-based method utilizes the original query for retrieval, thereby failing to capture abstract concepts and actions within temporal sequences. Our grounding agent analyzes the query, expands and refines the relevant concepts, and utilizes VLM to evaluate the similarity between the enhanced query and each segment, achieving more comprehensive grounding results.
Score
Relevance Criteria
Key Basis in Reasoning Process
4
Core elements visible; sufficient for answer
Key verification steps and preliminary conclusion
3
Partial evidence present; incomplete information
Visual cues already been identified
2
No explicit clues; indirect association through multi-hop reasoning
Approach for conceptual decomposition, semantic expansion, or associative analysis
1
No observable or inferable relevance to the query
N/A
Table 1: Segment relevance scoring criteria and reasoning outputs in grounding agent.
Methods
LVBench
LongVideoBench (Val)
Video MME Long
MLVU
Commercial VLMs
Gemini-1.5-Pro [ 43 ]
33.1
64.0
67.4
-
GPT-4o [ 44 ]
48.9
66.7
65.3
54.9
OpenAI o3 [ 45 ]
57.1
67.5
64.7
-
Open-Source VLMs
Seed 1.6 VL ∗ [ 46 ]
58.1
66.1
68.4
65.3
Table 2: Comparison on long video understanding benchmarks. Our method achieves SOTA performance across multiple benchmarks. The best results are bolded . Categories are color-coded for clarity.
Figure 4: Experimental results of different agent-based methods.
Sampling Method
LVBench EU Subset
FPS
Clip Interval (s)
Score (%)
Time (s)
0.5
30
70.3
79.0
0.5
120
64.7
50.6
1
60
70.9
74.8
0.25
60
62.1
54.8
0.5
60
68.6
69.4
Table 6: Ablation Study on Sampling Methods in the Grounding Agent.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S5 : Analysis of the reasoning trajectories generated by our proposed Symphony and the single-agent DVD.
Method
Seed 1.6VL
Qwen2.5VL -72B
Qwen2.5VL -7B
GPT 4o
VideoTree
33.7
32.0
29.4
32.8
VideoAgent
37.6
34.6
30.5
32.7
VDR
56.1
52.3
49.4
50.8
VideoRAG
59.2
57.7
50.2
52.3
Ours
71.8
68.2
65.1
67.1
Appendix
Table 8: Performance on LVBench with different base models.