MMA: Multimodal Memory Agent

Yihao Lu; Wanru Cheng; Zeyu Zhang; Hao Tang

cs.CVFeb 18, 2026

MMA: Multimodal Memory Agent

Authors: Yihao Lu, Wanru Cheng, Zeyu Zhang, Hao Tang

Organizations: School of Computer Science, Peking University

Abstract

Long-horizon multimodal agents depend on external memory; however, similarity-based retrieval often surfaces stale, low-credibility, or conflicting items, which can trigger overconfident errors. We propose Multimodal Memory Agent (MMA), which assigns each retrieved memory item a dynamic reliability score by combining source credibility, temporal decay, and conflict-aware network consensus, and uses this signal to reweight evidence and abstain when support is insufficient. We also introduce MMA-Bench, a programmatically generated benchmark for belief dynamics with controlled speaker reliability and structured text-vision contradictions. Using this framework, we uncover the "Visual Placebo Effect", revealing how RAG-based agents inherit latent visual biases from foundation models. On FEVER, MMA matches baseline accuracy while reducing variance by 35.2% and improving selective utility; on LoCoMo, a safety-oriented configuration improves actionable accuracy and reduces wrong answers; on MMA-Bench, MMA reaches 41.18% Type-B accuracy in Vision mode, while the baseline collapses to 0.0% under the same protocol. Code: https://github.com/AIGeeksGroup/MMA.

Figures & tables

Figure 1: Retrieval Trap Case Study. MIRIX fails by retrieving the high-similarity but irrelevant Memory B. MMA correctly identifies the credible Memory A using multi-dimensional reliability signals.

Benchmark	Setting	Modality	Temp. structure	Paired T–V evidence	Src prior	Epistemic scoring
LongBench ( Bai et al., 2024 )	static LC	Text	static	✗	✗	accuracy
RULER ( Hsieh et al., 2024 )	synth LC	Text	static	✗	✗	accuracy
LoCoMo ( Maharana et al., 2024 )	LT dialog	Text	multi-session / months	✗	✗	accuracy
FEVER ( Thorne et al., 2018 )	verif.	Text	static	✗	✗	accuracy (NEI)
MMA-Bench (Ours)	LT dialog	Multi	10 / $\sim$ 6mo	✓	✓	CoRe

Table 1: Comparison of Benchmarks Related to Long-horizon Evidence Use. MMA-Bench complements prior suites by explicitly controlling source reliability priors and pairing multimodal evidence to enable a controlled diagnosis of belief dynamics and epistemic behavior under conflict.

Figure 2: MMA Framework. The Confidence Module reweights retrieval via source reliability, temporal decay, and network consensus to modulate reasoning and abstention.

Figure 3: MMA-Bench evaluation framework. The benchmark integrates cross-modal consistency analysis, risk-aware betting, and a $2\times 2$ logic matrix for trust conflicts. Performance is assessed through fundamental QA and a 3-step belief probe.

Type	Conflict	Configuration	Target Capability
A	Standard	Visuals support reliable User A.	Baseline consistency.
B	Inversion	Visuals support unreliable User B.	Overcoming authority bias.
C	Ambiguity	Visuals are vague.	Rejecting over-interpretation.
D	Unknowable	No valid evidence.	Absolute abstention.

Table 2: Logic Matrix for MMA-Bench. Categorization of multimodal trust conflicts based on source reliability and visual evidence.

Method	Performance Metrics		Prudence Metrics		Stability (Std)
Method	Raw Acc.	Selective ( $\alpha=0.2$ )	Abstain Rate	Abstain Prec.	Stability (Std)
MIRIX (Baseline) ( Wang and Chen, 2025 )	59.87%	0.6468	44.2%	45.6%	$\pm 2.50\%$
MMA (Ours)	59.93%	0.6484	45.3%	45.8%	$\pm 1.62\%$

Table 3: Main Results on FEVER. MMA matches baseline accuracy while significantly reducing performance variance (

\pm

1.62% vs.

\pm

2.50%) across seeds.

Method	Mode	Overall Metrics			Scenario-Specific Analysis
Method	Mode	Core Acc.	Verdict Acc.	CoRe Score	Type B Acc.	Type D Score
MIRIX (Baseline) ( Wang and Chen, 2025 )	Text	30.94%	47.78%	0.37	0.00%	1.00
MIRIX (Baseline) ( Wang and Chen, 2025 )	Vision	32.67%	46.67%	0.35	0.00%	1.00
MMA (Ours)	Text	13.15%	56.67%	0.28	23.53%	0.69
MMA (Ours)	Vision	13.55%	42.22%	-0.16	41.18%	-0.38

Table 4: MMA-Bench Main Results. Comparison across logic types (Type D uses risk-adjusted CoRe scoring). MMA restores agency in Type B conflict and mitigates the visual placebo effect in Type D scenarios.

Figure 4: Detailed Dynamics Analysis. (a) Step-wise belief revision; (b) Risk-adjusted scores highlighting visual noise sensitivity; (c) Gap analysis between retrieval accuracy and calibration.

Figure 5: Evolutionary Logic Spectrum. Tracing performance from Foundation Model to MMA. MMA restores agency (Activation) and buffers inherited visual bias (Placebo Effect).

Model	Deterministic (Type A, Vis)		Indeterminate (Type D, Vis)
Model	Verdict Acc.	Status	CoRe Score	Interpretation
MMA (Full)	$\mathbf{50.0\%}$	Robust	$\mathbf{-0.38}$	Buffered
tc (w/o Source)	$0.0\%$	Paralyzed	$1.00^{\dagger}$	Artifact of Default
st (w/o Consen.)	$36.7\%$	Unstable	$-0.69$	Hallucinated
cs (w/o Time)	$0.0\%$	Degraded	$1.00^{\dagger}$	Artifact of Default

Table 5: Ablation results on MMA-Bench (Vision Mode). † Perfect scores in Type D coincide with 0% accuracy in Type A, indicating system paralysis rather than genuine calibration.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Selective Prediction Analysis on FEVER. MMA consistently outperforms the Baseline under abstention-based risk control, achieving higher utility and lower risk across evaluation settings.

Mode	Components	Raw Acc. (%)	Act. Acc. (%)	Correct Abstain	Wrong Abstain
MMA (Ours)	$S+T+C_{\text{con}}$	$59.93\pm 1.62$	$71.61\pm\mathbf{0.43}$	$103.7\pm 4.7$	$122.6\pm 11.2$
tc (w/o Source)	$T+C_{\text{con}}$	$\mathbf{60.47\pm 1.33}$	$71.61\pm 2.54$	$102.3\pm 12.6$	$117.7\pm 20.6$
cs (w/o Time)	$S+C_{\text{con}}$	$59.00\pm 2.27$	$68.96\pm 2.35$	$95.0\pm 13.2$	$\mathbf{114.0\pm 32.1}$
st (w/o Consen.)	$S+T$	$58.93\pm 3.48$	$\mathbf{72.05\pm 2.34}$	$\mathbf{105.7\pm 16.1}$	$131.0\pm 36.4$

Appendix

Table 6: Ablation results on FEVER (

N=500,\text{Seeds}=3

). Act. Acc. (Actionable Accuracy) denotes the precision of non-abstained responses (Mean

\pm

Std). Notably, while MMA shares a similar mean accuracy with other variants, it achieves significantly lower variance (

\sigma=0.43

), demonstrating superior stability compared to tc (

\pm 2.54

) and st (

\pm 2.34

Panel of Figure 7 — Figure 7: Ablation Study Results on FEVER. We compare the Full Model (MMA) against variants without Consensus (‘st’), without Time (‘cs’), and without Source (‘tc’). (a) Shows that removing Consensus drastically increases variance. (b) Shows that MMA maintains high utility under strict prudence requirements (high $\alpha$ ). (c) Visualizes the trade-off between sensitivity and conservativeness.

Method	Reasoning Categories (LLM Score)				Overall Metrics		Reliability	Utility
Method	Single-Hop	Multi-Hop	Open-Domain	Temporal	Accuracy	Wrong Ans.	Act. Acc.	( $\lambda=1,r=0.2$ )
MIRIX (Baseline)	80.14	76.01	67.71	78.00	77.37	317	78.96%	880.0
MMA (Ours)	73.76	62.31	59.38	77.05	72.31	335	76.80%	793.6
Variant ‘st’	79.08	67.91	61.46	79.55	75.94	298	79.64%	883.6

Appendix

Table 7: Main Results on LoCoMo. Breakdowns of LLM Scores across reasoning dimensions (

N=1542

). While the Baseline excels in raw accuracy, our ‘st’ variant achieves the highest Actionable Accuracy (79.64%) and Utility , demonstrating superior reliability in safety-critical retrieval tasks.

Panel of Figure 8 — Figure 8: Quantitative Analysis on LoCoMo. While MMA focuses on prudence, its ‘st’ configuration demonstrates robust utility advantages over the Baseline in high-stakes settings.

Mode	Components	Utility	Wrong Ans.	Abstain Count	Act. Acc.
MMA (Full)	$S+T+C_{\text{con}}$	488.0	335	98	76.80%
st (w/o Consen.)	$S+T$	609.0	298	78	79.64%
cs (w/o Time)	$S+C_{\text{con}}$	480.5	335	113	76.56%
tc (w/o Source)	$T+C_{\text{con}}$	471.5	344	77	76.52%

Appendix

Table 8: Ablation results on LoCoMo (

N=1542

). Utility is computed with

\lambda=2.0,r=0.5

. Wrong Ans. denotes hallucinations (Lower is Better). The ‘st’ variant achieves the best safety profile.

Panel of Figure 9 — Figure 9: Ablation Sensitivity on LoCoMo. Removing the Consensus module (‘st’) actually improves robustness in this specific domain, while removing Source (‘tc’) or Time (‘cs’) degrades performance.

Model	Mode	Overall Metrics			Scenario-Specific Analysis
Model	Mode	Core Acc.	Verdict Acc.	CoRe Score	Type B Acc.	Type D Score
GPT-4.1-mini	Text (Oracle)	85.26%	77.78%	0.59	76.47%	0.85
GPT-4.1-mini	Vision (Raw)	80.74%	73.33%	0.51	64.71%	0.23
Qwen-3-VL-Plus	Text (Oracle)	88.05%	65.56%	0.32	88.24%	-0.69
Qwen-3-VL-Plus	Vision (Raw)	88.98%	63.33%	0.28	82.35%	-0.69

Appendix

Table 9: Cognitive dynamics of foundation models on MMA-Bench. Core Acc. measures basic reading comprehension. CoRe Score (Risk-Adjusted) reflects epistemic calibration. Type B Acc. indicates success in Reliability Inversion (overcoming authority bias). Type D Score reflects prudence in unknowable scenarios. Note the significant drop in Type D score for GPT-4.1-mini when switching to Vision mode, illustrating the Visual Placebo Effect .

Panel of Figure 10 — Figure 10: Cognitive Dynamics of Foundation Models on MMA-Bench. We compare GPT-4.1-mini and Qwen-3-VL-Plus across Text (Oracle) and Vision (Raw) modes. (a) Reveals how visual modalities can act as distractors in noise scenarios. (b) Highlights the disconnect between reading comprehension (Core Acc) and epistemic prudence (CoRe Score).

Panel of Figure 11 — Figure 11: Mechanism Ablation on MMA-Bench (Vision Mode). We isolate the failure modes: (a) Accuracy metrics reveal that Source ( $S$ ) and Time ( $T$ ) are prerequisites for agency, as their absence leads to paralysis (0% accuracy in known facts); (b) CoRe Scores demonstrate that Consensus ( $C_{\text{con}}$ ) is essential to buffer against the Visual Placebo Effect in indeterminate queries.

Explore similar work

May 28, 2026

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

Multimodal large language models are increasingly deployed as long-horizon agents, where memory must do more than recall: it must track an evolving world, revise what has gone stale, and surface the right evidence at decision time. Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions, leaving us unable to localize failures to writing, maintenance, retrieval, or use. The rise of agent harnesses that author their own memory sharpens this gap, since we have no principled way to compare hand-designed pipelines with self-managing alternatives. To close these gaps, we formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle, and instantiate it in WorldMemArena: 400 multi-session multimodal tasks spanning Lifelong Evolution (evolving personal and task states) and Agentic Execution (memory from real observations, actions, and feedback), annotated with gold memory points, updates, distractors, and evidence chains for stage-level diagnosis. This enables the first head-to-head comparison of long-context, manually designed (RAG and external memory systems), and harness-based memory agents. Results show that: (1) better memory writing and storage do not guarantee better performance; (2) multimodal memory still struggles to fully use visual evidence; (3) systems are unstable across domains and degrade on realistic agentic trajectories; and (4) harness memory is more flexible but remains costly and less reliable.

Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14

University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3

Multimodal Memory LLM Agent Memory AI Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks Memory Agent Benchmarks

Ask PDF

May 15, 2026

SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory

Existing benchmarks for multimodal memory reasoning largely evaluate systems within pre-assembled contexts, but under-evaluate whether agents can use evidence distributed across independently originated sources. We argue that source-distributed memory composition is an important and under-examined bottleneck in multimodal agent memory, especially when relevant evidence is fragmented across heterogeneous artifacts such as conversations, profiles, screenshots, tables, images, and documents. To address this gap, we introduce Source-distributed Multimodal Memory Benchmark(SMMBench), which measures whether agents can retrieve, align, and compose multimodal evidence scattered across multiple sources rather than reason within a single curated context. SMMBench evaluates four core capabilities: (1) cross-source multimodal reasoning; (2) conflict resolution; (3) preference reasoning; (4) memory-grounded action prediction. The benchmark contains 1877 samples grounded in 264 sources. Experiments on representative memory-style and retrieval-based baselines show that current systems still struggle on these capabilities, positioning source-distributed multimodal memory as an important and still under-evaluated challenge for multimodal agents. Our data are available at https://huggingface.co/datasets/HuacanChai/SMMBench.

Huacan Chai, Yukai Wang, Yingxuan Yang +7

Shanghai Jiao Tong University, China · OPPO, China

Multimodal Memory Multimodal Reasoning Multimodal Agents Agent Benchmarks

Ask PDF

Jun 25, 2026

DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection

Agent benchmarks for measuring memory largely study textual cases, in which information is deliberately extracted from the environment, written down, and then later retrieved. In other words, they assess what agents elected to record, not what they happened to see. We introduce DMV-Bench (code: https://github.com/yyyujintang/DMV-Bench), the first interactive benchmark for visual memory in multimodal agents, to study this often-neglected property. DMV-Bench is built on (1) a controlled home-furnishing e-commerce environment, supported by a catalog of 1,000 product variants, and (2) a text-leakage contract which ensures that the primary discriminative signal of each task is solely in the pixels. In DMV-Bench, agents undergo chains of autonomous shopping sessions in which every visited product image carries a unique, pre-rendered incidental cue that the agent is later asked to recall. We show that conventional solutions struggle with this task. Inspired by dual-coding theory, we propose a memory architecture that uses parallel visual and verbal codes, which we call DualMem. On DMV-Bench, DualMem outperforms a caption-only baseline and three recent multimodal agent-memory systems across multi-session chain lengths on multiple models. These gains persist even adjusting for memory-bank size and encoding-position bias. Further experiments also reveal an asymmetric division of labor between the two codes; a weighted coding scheme is often strongest. We view this as a step towards memory systems that preserve a richer record of agents' observations.

Yujin Tang, Chenming Shang, Ruize Xu +1

Dartmouth College

Multimodal Memory Long-Horizon Agent Evaluation Memory Agent Benchmarks Multimodal Agents

Ask PDF