Retrieving relevant observations from long multi-modal web interaction histories is challenging because relevance depends on the evolving task state, modality (screenshots, HTML text, structured signals), and temporal distance. Prior approaches typically rely on static similarity thresholds or fixed-capacity buffers, which fail to adapt relevance to the current task context. We propose ACGM, a learned graph-memory retriever that constructs task-adaptive relevance graphs over agent histories using policy-gradient optimization from downstream task success. ACGM captures heterogeneous temporal dynamics with modality-specific decay (visual decays 4.3× faster than text: λv=0.47 vs.\ λx=0.11) and learns sparse connectivity (3.2 edges/node), enabling efficient O(logT) retrieval. Across WebShop, VisualWebArena, and Mind2Web, ACGM improves retrieval quality to 82.7 nDCG@10 (+9.3 over GPT-4o, p<0.001) and 89.2% Precision@10 (+7.7), outperforming 19 strong dense, re-ranking, multi-modal, and graph-based baselines. Code to reproduce our results is available at{\color{blue}Saman Forouzandeh}.
Figures & tables
WebShop (1,180 tasks)
VisualWebArena (910 tasks)
Mind2Web (2,350 tasks)
Method
nDCG
MAP
MRR
Rec
Prec
nDCG
MAP
MRR
Rec
Prec
nDCG
MAP
MRR
Rec
Prec
ColBERT-v2 ( Santhanam et al., 2022 )
64.7
54.9
70
75.2
71.3
60.8
51.2
66
73.1
68.4
62.4
53.6
68
74.8
70.1
ANCE ( Xiong et al., 2021 )
66.3
56.8
72
76.9
73.2
62.4
53.1
68
74.8
70.1
64.1
55.4
70
76.3
71.8
Contriever ( Izacard et al., 2021 )
58.9
49.3
65
70.5
66.4
55.2
46.1
61
68.7
63.8
57.6
48.7
63
70.2
65.3
E5-Large
67.8
58.3
73
78.1
74.6
63.7
54.6
69
76.2
71.9
65.8
57.1
71
77.8
73.4
MonoT5 ( Nogueira et al., 2020 )
68.4
59.1
74
78.3
75.6
64.7
55.8
71
76.5
72.9
66.9
58.4
73
78.9
74.8
Table 1. Retrieval performance across three benchmarks. Best , Second , Third . * denotes p<0.001 vs. second-best (Bonferroni-corrected).
Figure 1. (a) ACGM maintains near-constant sparsity across trajectory lengths, reducing edge evaluations. (b) ACGM achieves higher retrieval accuracy with substantially less memory than dense multi-modal retrievers.
Figure 2. Retrieval analysis, ablations, and decay sensitivity. (a) PR curves (AUC: ACGM 0.863 vs. GPT-4o 0.815) show consistently higher precision. (b) Retrieval quality strongly correlates with task success ( ρ=0.93 , R2=0.991 ). (c) Ablations across graph topology (RQ3), decay (RQ2), modality (Mod), and retrieval (RQ1) identify learned edges, no-decay, and dense graphs as most critical ( Δ>10 ). (d) Decay sensitivity shows > 80 nDCG@10 within ± 0.10 of optimum, indicating robustness.
Figure 3. Performance and efficiency across trajectory lengths. (a) ACGM maintains high retrieval quality (nDCG@10 > 79%) up to T=100 , with the gap over baselines widening as history grows. (b) Sub-linear O(logT) latency (14.7 ms at T=100 ) vs. linear scaling for dense retrieval methods. (c) Learned modality-specific decay rates ( λv=0.47 , λk=0.23 , λx=0.11 ) capture heterogeneous temporal dynamics, outperforming uniform decay by 8.7 Prec@10 points.
Long-term memory lets LLM agents recall past interactions and remain consistent across sessions, but memory systems are hard to compare because they often vary in representation, indexing, retrieval, and evaluation. We present a controlled evaluation framework based on shared 5W-style conversational memories. Localized graph configurations traverse a common base graph; AdaptiveGraph adds chronological edges and Personalized PageRank diffusion. We also evaluate BM25 over the same extracted notes and OpenClaw as a raw-input external reference. Retrieval rankings vary across memory settings. On LongMemEval-S, AdaptiveGraph is the strongest graph configuration at 0.844 MRR, but BM25 reaches 0.867 and OpenClaw 0.880. On ATANT Core, localized graph traversal outperforms diffusion and BM25, whereas BM25 leads the stress rounds. Reducing LongMemEval-S within the tested range does not reproduce the ATANT diffusion penalty, but the smallest tested store remains larger than ATANT Core, so store size cannot be ruled out. The penalty also persists under a permissive content-match criterion. Vocabulary normalization and extraction quality substantially affect graph retrieval, and missing extraction tags are common among top-five misses. Retrieval strategies should therefore be evaluated jointly with the memory setting and against strong lexical baselines.
Yu-Shu Chen, Yu-Jung Liang, Pengtao Xie
Department of Electrical and Computer Engineering University of California, San Diego La Jolla, CA 92093, USA
Long-term multimodal memory must support not only retrieving relevant information but also computing over observations accumulated across interactions. Existing systems largely emphasize retrieval memory, organizing interaction histories through summaries and indexes to return query-relevant information at multiple granularities, from high-level abstractions to underlying records. In this paper, we formulate analytic memory as a complementary abstraction that organizes recurring multimodal observations into queryable structures supporting filtering, aggregation, ranking, and temporal comparison. We present AdaMM, a framework that jointly supports retrieval and analytic memory. Rather than relying on application-defined schemas, AdaMM extracts provenance-linked attribute-value observations from dialogue, images, and contextual metadata, discovers recurring field structures, and materializes them for analytical access. At inference time, a memory-aware planner decomposes queries into retrieval and analytic operations and routes each operation to the appropriate tools. Experiments on two long-term multimodal memory benchmarks, MemEye and MemGallery, show that AdaMM improves performance by up to 11.3% and 6.9%, respectively.
Although existing multi-agent Retrieval-Augmented Generation (RAG) systems have demonstrated promise on complex multimodal reasoning tasks, they remain fundamentally limited in reasoning depth and memory structure, suffering from inadequate retrieval and state blindness when answering knowledge-intensive questions. To address these limitations, we propose GraMRAG, a graph memory-guided multi-agent RAG framework that integrates a dynamic multimodal memory graph to enable stable, multi-step multimodal reasoning. We introduce a vision-text bridged reasoning paradigm that unifies multi-scale entity cropping with a ReAct-style visual toolchain, enhancing the long-horizon cross-modal reasoning capability. We further construct a multimodal memory graph that formalizes agent reasoning as a dynamic directed acyclic graph (DAG), explicitly modeling action-observation dependencies to mitigate state blindness and suppress redundant retrieval. Moreover, we propose Topology-Aware Policy Optimization (TAPO) that leverages graph topology for critical path identification and targeted node pruning, enabling fine-grained credit assignment across multi-step reasoning trajectories. Extensive experiments on challenging multimodal benchmarks demonstrate that our approach consistently outperforms existing baselines and achieves state-of-the-art performance on complex long-horizon reasoning tasks.