PACE-Mem: Prefix-Aware Complementary Evidence for Multimodal Agents
Authors: Shidu Ren, Rong-cheng Tu, Hang Zhou, Xiao Luo
Organizations: Shidu Ren is with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 3G4, Canada · Rong-cheng Tu is with the College of Computing and Data Science, Nanyang Technological University, Singapore 639798 · Hang Zhou is with the Department of Statistics and Operations Research and the School of Data and Information Science, The University of North Carolina at Chapel Hill, Chapel Hill, NC 27599 USA · Xiao Luo is with the Department of Statistics, University of Wisconsin–Madison, Madison, WI 53706 USA
As multimodal agents accumulate experience, their long-term memories contain many similar observations of recurring scenes, actions, and entities. Answer quality depends on the evidence retrieved from these memories into the model's context. However, query-relevant memories may contain overlapping information, and a fixed retrieval depth may return too little or too much evidence for a given question. To address these challenges, we propose PACE-Mem, Prefix-Aware Complementary Evidence, which constructs a complementary evidence sequence and determines its length from the same selection trajectory. For structured memories, query-guided pooling combines records within each clip to obtain relevance scores and representations. Using these relevance scores and representations, maximal marginal relevance (MMR) selection balances query relevance against overlap with previously selected evidence. From the resulting score trajectory, a prefix-length selection algorithm determines retrieval depth by comparing length-normalized adjacent declines, accounting for both score changes and the amount of evidence accumulated. PACE-Mem is a training-free, plug-in retrieval method for clip-based agent memory systems, leaving memory construction and answer generation unchanged. Across three long-video benchmarks and two agent frameworks, PACE-Mem improves question-answering accuracy by an average of 1.6% over Adaptive-k, the strongest baseline at comparable mean retrieval depths. Further analysis reveals that jointly adding and removing evidence is associated with larger net accuracy gains than either change alone.
Figures & tables
Fig. 1: Evidence selection in long-video memory, illustrated with VideoMME-Long frames. Top: retrieved views repeat nearby activity around a pole-vault attempt while omitting the view that reveals contact with the bar. Middle: a short prefix misses the check handoff. Bottom: a long prefix includes unrelated artwork after the key view. PACE-Mem chooses prefix length from the scores obtained while selecting complementary clips.
Fig. 2: Retrieval with PACE-Mem. The host encodes video into textual memory. For structured memories, similar records within each clip are grouped before query-guided pooling produces relevance scores and evidence representations. Consistent blue shades track clip identities across the relevance ranking and MMR sequence. The largest raw decline occurs at k=2 , while the length-normalized criterion selects K∗=4 . The answer model receives the selected prefix in chronological order. The question, answer, and memory excerpts are adapted from VideoMME-Long question 769-2; clip IDs and scores are schematic.
TABLE I: Question-answering accuracy and mean retrieval depth in two agent frameworks. Clips reports the mean evidence count per nonempty ordinary retrieval call. Results are compared within each host block, with SiLVR Web evaluated on a fixed subset of 3,171 questions
Variant
MMR
Depth selection
Acc. (%)
Clips
PACE-Mem
✓
✓
59.0
7.2
w/o MMR
–
✓
58.1
7.4
w/o depth selection
✓
–
57.6
7.0
w/o both
–
–
56.9
7.0
TABLE II: Component ablations in SiLVR on VideoMME-Long. Checks indicate enabled components. Removing MMR uses relevance order, and removing depth selection fixes K=7 . Clips is the mean returned depth
Variant
Robot
Web
VideoMME-Long
Avg. Acc.
w/o depth selection
41.1
56.7
57.6
51.8
w/o both
40.8
55.9
56.9
51.2
TABLE III: Complementary evidence selection ablation in SiLVR at fixed depth K=7 . Both variants disable depth selection, and w/o both also removes MMR. Entries are accuracies (%)
Benchmark
PACE-Mem
w/o pooling
Acc. (%)
Clips
Acc. (%)
Clips
M3Bench Robot
40.3
3.9
38.7
3.9
M3Bench Web
57.6
4.1
56.7
4.1
VideoMME-Long
58.7
4.3
57.6
4.3
TABLE IV: Query-guided pooling ablation in M3-Agent. The w/o pooling variant uses each clip’s strongest query-matched node for relevance and representation. MMR and depth selection are retained
Figure 7
Fig. 5: VideoMME-Long cases with M3-Agent under identical single search requests. Top left: Clip 58 recovers both parents’ results at the same five-clip depth. Top right: retaining the explicit budget fact allows a correct answer with three clips instead of five. Bottom: omitting Clip 39 removes the target-final result, and PACE-Mem predicts the winner of another race. Frames and brief descriptions illustrate text-memory records. Rows show returned clip IDs and shortened predictions, with supporting evidence in green and replaced or omitted clips in peach.
Under a limited context budget, a long-term agent must recall not merely memories that are individually relevant, but evidence that is collectively sufficient for the current query. Conventional memory access retrieves, scores, and truncates memories, often spending multiple context slots on redundant facts while leaving complementary evidence behind. We cast the read phase of long-term memory as query-conditioned evidence composition: each memory is valued by what it adds beyond the evidence already selected. We instantiate this operation with MemPursuit, a training-free mechanism that uses the residual of a sparse convex query reconstruction as a representation-space proxy for what the current memory set is missing. By reusing the query and candidate embeddings already available from retrieval, MemPursuit progressively composes nested evidence contexts without invoking a language model or cross-encoder to score each query--memory pair. The effectiveness of our method is verified on 3 popular benchmarks in comparison to strong baselines. In fixed-budget LoCoMo experiments, MemPursuit improves end-to-end accuracy by 7.2 percentage points at Top-10 and 5.5 percentage points at Top-20, with consistent gains across multiple memory frameworks, while achieving a 32.9× speedup in median reranking latency over fully batched BGE-v2-m3. Our implementation is available at https://github.com/FengQi-HITSZ/MemPursuit.
Qi Feng, Jicong Fan
School of Data Science, The Chinese University of Hong Kong, Shenzhen
Organizing long-term memory for multimodal agents remains challenging because existing methods either suffer from expensive question-agnostic offline summaries or naive embedding similarity matching that introduces incomplete and redundant context. To address these issues, we propose GraphMemix, a combinatorial-optimization graph memory framework that models memory organization as query-aware evidence-forest construction. Specifically, our method consists of three key components:(1) candidate graph construction, which expands multi-view seed memories through schema and semantic relations to acquire query-aware original context; (2) evidence utility and activation costs, which decouples direct memory support from anchor-conditioned relation verification to suppress redundant or conflicting information; and (3) forest optimization, which jointly selects a forest-format memory context under a maximum evidence budget and its reliable relational structure. By organizing memory into a query-relevant subgraph, the method avoids substantial lifecycle cost and recovers low-similarity complementary evidence. Experimental results across four long-term multimodal memory benchmarks demonstrate significant improvements with different foundation models and establish a new Pareto frontier between accuracy and lifecycle cost.
Geng Li, Yuhao Wang, Dong Li +2
Wangxuan Institute of Computer Technology, Peking University · MemoraX AI
Long-term memory is essential for LVLM agents to maintain consistency and integrate information across extended multimodal interactions. Existing agent memory systems, however, often reduce visual experiences into textual summaries or rely on static retrieve-then-reason pipelines, which are inefficient at query time and brittle when questions require image-text binding, temporal updates, or visual details. We propose Prospective Multimodal Memory Compilation, a framework that shifts part of the memory reasoning process from query time to memory consolidation time. Given accumulated multimodal interactions, a Questioner predicts future question candidates, a Planner compiles question-conditioned multimodal memory programs, and a Doubter verifies whether the planned evidence path can support the predicted answer. The verified question-program pairs form a structured question bank for efficient query-time routing and evidence retrieval. Experiments on multimodal long-term memory benchmarks show that our method improves answer quality and visual evidence recall while reducing query-time token and latency costs. Extensive ablations analyze the effects of self-feedback, dynamic planning, raw-image access, and question bank coverage.
Jingyu Sun, Yan Lin, Yuyang Xue +10
The University of Manchester · The University of Newcastle · The University of Edinburgh +5