Organizations: Shanghai Jiao Tong University · Fudan University · Shanghai AI Laboratory · Hong Kong University of Science and Technology · MemTensor (Shanghai) Technology Co., Ltd.
Vision agent memory has shown remarkable effectiveness in streaming video understanding. However, storing such memory for videos incurs substantial memory overhead, leading to high costs in both storage and computation. To address this issue, we propose StreamMeCo, an efficient Stream Agent Memory Compression framework. Specifically, based on the connectivity of the memory graph, StreamMeCo introduces edge-free minmax sampling for the isolated nodes and an edge-aware weight pruning for connected nodes, evicting the redundant memory nodes while maintaining the accuracy. In addition, we introduce a time-decay memory retrieval mechanism to further eliminate the performance degradation caused by memory compression. Extensive experiments on three challenging benchmark datasets (M3-Bench-robot, M3-Bench-web and Video-MME-Long) demonstrate that under 70% memory graph compression, StreamMeCo achieves a 1.87* speedup in memory retrieval while delivering an average accuracy improvement of 1.0%. Our code is available at https://github.com/Celina-love-sweet/StreamMeCo.
Figures & tables
Figure 1: An example from M3-Bench-robot. (a) Composition of the memory graph. (b) Analysis of memory retrieval time with respect to the number of text nodes, indicating that the increment of memory nodes makes the retrieval time not acceptable.
Figure 2: The overview of M3-Agent. (a) The memory graph of M3-Agent. (b) The memory storage and retrieval pipeline of M3-Agent, illustrating two different retrieval strategies.
Figure 3: The overview of StreamMeCo. Efficient memory graph compression is achieved via the EMsampling and EWpruning modules, while the TMR mechanism prioritizes retrieving memories from more recent segments, enabling fast and accurate memory retrieval.
Dataset
M3-Bench-robot
M3-Bench-web
Video- MME-Long
Avg.
Model
ME
MH
CM
PU
GK
All
ME
MH
CM
PU
GK
All
Closed-source MLLMs
Gemini-1.5-Pro
6.5
7.5
8.0
9.7
7.6
8.0
18.0
17.9
23.8
23.1
28.7
23.2
38.0
23.1
GPT-4o
9.3
9.0
8.4
10.2
7.3
8.5
21.3
21.9
30.9
27.1
39.6
28.7
38.8
25.3
Gemini-Agent
15.8
17.1
15.3
20.0
15.5
16.9
29.3
20.9
33.8
34.6
45.0
34.1
55.1
35.4
Gemini-GPT4o-Hybrid
21.3
25.5
22.7
28.8
23.1
24.0
35.9
26.2
37.6
43.8
52.2
41.2
56.5
40.6
Table 1: Performance comparison of different models on M3-Bench-robot, M3-Bench-web and Video-MME-Long. The best three results for the streaming video models are highlighted in bold black font .
Components
M3-Bench-Robot
M3-Bench-Web
A
B
C
Accuracy
Accuracy
–
–
–
30.3 (100.0%)
47.9 (100.0%)
28.5 (94.1%)
44.6 (93.1%)
✓
30.3 (100.0%)
45.7 (95.4%)
✓
✓
30.1 (99.3%)
46.1 (96.2%)
✓
✓
30.2 (99.7%)
46.1 (96.2%)
Table 2: Ablation study of the compression components. The first row reports the accuracy of the M3-Agent without compression. A : EMsampling; B : entity importance; C : embedding similarity.
Figure 4: Impact of the TMR Mechanism on M3-Bench-robot and M3-Bench-web at different compression ratios.
Figure 5: Time Efficiency Analysis on M3-Bench-Robot, M3-Bench-Web and Video-MME-Long. SS-Time: Time to compute and sort the similarity between the query embedding and each text node’s content embedding; Avg-Retrieval: Average number of memory graph queries per question; TR-Time: Average total time spent querying the memory graph for each question after obtaining the model’s query embedding.
Figure 6: Impact of cluster ratio a and balance boefficient b on M3-Bench-robot and M3-Bench-web.
Decay Coefficient
Exponential
Linear
0.1
34.6
33.5
0.5
32.1
33.7
1.0
32.9
32.4
2.0
32.0
33.5
Table 3: Comparison between diffenent decay methods on M3-Bench-Robot.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Avg. Duration(s)
#Videos
#QA Pairs
Avg. Text Nodes
#Source
M3-Bench-robot
2039.9
100
1276
2151.98
Real-world filming
M3-Bench-web
1630.7
920
3214
1067.75
YouTube
Video-MME-Long
2466.7
300
900
2499.27
YouTube
Appendix
Table 4: The statistical information of M3-Bench-robot, M3-Bench-web and Video-MME-Long.
Figure 7: Numbers of different types of nodes and their memory proportion in the memory graphs of the three benchmark datasets.
Figure 8: Impact of memory decay coefficient λ on M3-Bench-robot.
Figure 9: Impact of the TMR Mechanism on Video-MME-Long at different compression ratios.
Components
M3-Bench-Robot
M3-Bench-Web
A
B
C
Accuracy
Accuracy
–
–
–
30.3 (100.0%)
47.9 (100.0%)
26.6 (87.8%)
41.8 (87.3%)
✓
27.0 (89.1%)
42.6 (88.9%)
✓
✓
28.7 (94.7%)
43.9 (91.6%)
✓
✓
28.4 (93.7%)
42.3 (88.3%)
Appendix
Table 5: Ablation study of the compression components. The first row reports the accuracy of the M3-Agent without compression. A : EMsampling; B : entity importance; C : embedding similarity.
Model / Dataset
Office (95)
Meeting Room (68)
Total (163)
Mem0 (graph)
42.1
45.6
43.6
+Random
33.7
36.8
35.0
+Clustering
35.8
39.7
38.0
+DART
34.7
42.6
38.0
+TimeChat-Memory
37.9
39.7
38.7
+MemoryLLM
34.7
38.2
36.2
Appendix
Table 6: Mem0 (graph): Performance comparison of different compression methods on the Office and Meeting Room subsets of M3-Bench-robot (compression ratio = 30%).