PACE-Mem: Prefix-Aware Complementary Evidence for Multimodal Agents
Authors: Shidu Ren, Rong-cheng Tu, Hang Zhou, Xiao Luo
Organizations: Shidu Ren is with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 3G4, Canada · Rong-cheng Tu is with the College of Computing and Data Science, Nanyang Technological University, Singapore 639798 · Hang Zhou is with the Department of Statistics and Operations Research and the School of Data and Information Science, The University of North Carolina at Chapel Hill, Chapel Hill, NC 27599 USA · Xiao Luo is with the Department of Statistics, University of Wisconsin–Madison, Madison, WI 53706 USA
As multimodal agents accumulate experience, their long-term memories contain many similar observations of recurring scenes, actions, and entities. Answer quality depends on the evidence retrieved from these memories into the model's context. However, query-relevant memories may contain overlapping information, and a fixed retrieval depth may return too little or too much evidence for a given question. To address these challenges, we propose PACE-Mem, Prefix-Aware Complementary Evidence, which constructs a complementary evidence sequence and determines its length from the same selection trajectory. For structured memories, query-guided pooling combines records within each clip to obtain relevance scores and representations. Using these relevance scores and representations, maximal marginal relevance (MMR) selection balances query relevance against overlap with previously selected evidence. From the resulting score trajectory, a prefix-length selection algorithm determines retrieval depth by comparing length-normalized adjacent declines, accounting for both score changes and the amount of evidence accumulated. PACE-Mem is a training-free, plug-in retrieval method for clip-based agent memory systems, leaving memory construction and answer generation unchanged. Across three long-video benchmarks and two agent frameworks, PACE-Mem improves question-answering accuracy by an average of 1.6% over Adaptive-k, the strongest baseline at comparable mean retrieval depths. Further analysis reveals that jointly adding and removing evidence is associated with larger net accuracy gains than either change alone.
Figures & tables
Fig. 1: Evidence selection in long-video memory, illustrated with VideoMME-Long frames. Top: retrieved views repeat nearby activity around a pole-vault attempt while omitting the view that reveals contact with the bar. Middle: a short prefix misses the check handoff. Bottom: a long prefix includes unrelated artwork after the key view. PACE-Mem chooses prefix length from the scores obtained while selecting complementary clips.
Fig. 2: Retrieval with PACE-Mem. The host encodes video into textual memory. For structured memories, similar records within each clip are grouped before query-guided pooling produces relevance scores and evidence representations. Consistent blue shades track clip identities across the relevance ranking and MMR sequence. The largest raw decline occurs at k=2 , while the length-normalized criterion selects K∗=4 . The answer model receives the selected prefix in chronological order. The question, answer, and memory excerpts are adapted from VideoMME-Long question 769-2; clip IDs and scores are schematic.
TABLE I: Question-answering accuracy and mean retrieval depth in two agent frameworks. Clips reports the mean evidence count per nonempty ordinary retrieval call. Results are compared within each host block, with SiLVR Web evaluated on a fixed subset of 3,171 questions
Variant
MMR
Depth selection
Acc. (%)
Clips
PACE-Mem
✓
✓
59.0
7.2
w/o MMR
–
✓
58.1
7.4
w/o depth selection
✓
–
57.6
7.0
w/o both
–
–
56.9
7.0
TABLE II: Component ablations in SiLVR on VideoMME-Long. Checks indicate enabled components. Removing MMR uses relevance order, and removing depth selection fixes K=7 . Clips is the mean returned depth
Variant
Robot
Web
VideoMME-Long
Avg. Acc.
w/o depth selection
41.1
56.7
57.6
51.8
w/o both
40.8
55.9
56.9
51.2
TABLE III: Complementary evidence selection ablation in SiLVR at fixed depth K=7 . Both variants disable depth selection, and w/o both also removes MMR. Entries are accuracies (%)
Benchmark
PACE-Mem
w/o pooling
Acc. (%)
Clips
Acc. (%)
Clips
M3Bench Robot
40.3
3.9
38.7
3.9
M3Bench Web
57.6
4.1
56.7
4.1
VideoMME-Long
58.7
4.3
57.6
4.3
TABLE IV: Query-guided pooling ablation in M3-Agent. The w/o pooling variant uses each clip’s strongest query-matched node for relevance and representation. MMR and depth selection are retained
Figure 7
Fig. 5: VideoMME-Long cases with M3-Agent under identical single search requests. Top left: Clip 58 recovers both parents’ results at the same five-clip depth. Top right: retaining the explicit budget fact allows a correct answer with three clips instead of five. Bottom: omitting Clip 39 removes the target-final result, and PACE-Mem predicts the winner of another race. Frames and brief descriptions illustrate text-memory records. Rows show returned clip IDs and shortened predictions, with supporting evidence in green and replaced or omitted clips in peach.