MMA: Multimodal Memory Agent

Yihao Lu; Wanru Cheng; Zeyu Zhang; Hao Tang

cs.CVFeb 18, 2026

MMA: Multimodal Memory Agent

Authors: Yihao Lu, Wanru Cheng, Zeyu Zhang, Hao Tang

Organizations: School of Computer Science, Peking University

Abstract

Long-horizon multimodal agents depend on external memory; however, similarity-based retrieval often surfaces stale, low-credibility, or conflicting items, which can trigger overconfident errors. We propose Multimodal Memory Agent (MMA), which assigns each retrieved memory item a dynamic reliability score by combining source credibility, temporal decay, and conflict-aware network consensus, and uses this signal to reweight evidence and abstain when support is insufficient. We also introduce MMA-Bench, a programmatically generated benchmark for belief dynamics with controlled speaker reliability and structured text-vision contradictions. Using this framework, we uncover the "Visual Placebo Effect", revealing how RAG-based agents inherit latent visual biases from foundation models. On FEVER, MMA matches baseline accuracy while reducing variance by 35.2% and improving selective utility; on LoCoMo, a safety-oriented configuration improves actionable accuracy and reduces wrong answers; on MMA-Bench, MMA reaches 41.18% Type-B accuracy in Vision mode, while the baseline collapses to 0.0% under the same protocol. Code: https://github.com/AIGeeksGroup/MMA.

Figures & tables

Figure 1: Retrieval Trap Case Study. MIRIX fails by retrieving the high-similarity but irrelevant Memory B. MMA correctly identifies the credible Memory A using multi-dimensional reliability signals.

Benchmark	Setting	Modality	Temp. structure	Paired T–V evidence	Src prior	Epistemic scoring
LongBench ( Bai et al., 2024 )	static LC	Text	static	✗	✗	accuracy
RULER ( Hsieh et al., 2024 )	synth LC	Text	static	✗	✗	accuracy
LoCoMo ( Maharana et al., 2024 )	LT dialog	Text	multi-session / months	✗	✗	accuracy
FEVER ( Thorne et al., 2018 )	verif.	Text	static	✗	✗	accuracy (NEI)
MMA-Bench (Ours)	LT dialog	Multi	10 / $\sim$ 6mo	✓	✓	CoRe

Table 1: Comparison of Benchmarks Related to Long-horizon Evidence Use. MMA-Bench complements prior suites by explicitly controlling source reliability priors and pairing multimodal evidence to enable a controlled diagnosis of belief dynamics and epistemic behavior under conflict.

Figure 2: MMA Framework. The Confidence Module reweights retrieval via source reliability, temporal decay, and network consensus to modulate reasoning and abstention.

Figure 3: MMA-Bench evaluation framework. The benchmark integrates cross-modal consistency analysis, risk-aware betting, and a $2\times 2$ logic matrix for trust conflicts. Performance is assessed through fundamental QA and a 3-step belief probe.

Type	Conflict	Configuration	Target Capability
A	Standard	Visuals support reliable User A.	Baseline consistency.
B	Inversion	Visuals support unreliable User B.	Overcoming authority bias.
C	Ambiguity	Visuals are vague.	Rejecting over-interpretation.
D	Unknowable	No valid evidence.	Absolute abstention.

Table 2: Logic Matrix for MMA-Bench. Categorization of multimodal trust conflicts based on source reliability and visual evidence.

Method	Performance Metrics		Prudence Metrics		Stability (Std)
Method	Raw Acc.	Selective ( $\alpha=0.2$ )	Abstain Rate	Abstain Prec.	Stability (Std)
MIRIX (Baseline) ( Wang and Chen, 2025 )	59.87%	0.6468	44.2%	45.6%	$\pm 2.50\%$
MMA (Ours)	59.93%	0.6484	45.3%	45.8%	$\pm 1.62\%$

Table 3: Main Results on FEVER. MMA matches baseline accuracy while significantly reducing performance variance (

\pm

1.62% vs.

\pm

2.50%) across seeds.

Method	Mode	Overall Metrics			Scenario-Specific Analysis
Method	Mode	Core Acc.	Verdict Acc.	CoRe Score	Type B Acc.	Type D Score
MIRIX (Baseline) ( Wang and Chen, 2025 )	Text	30.94%	47.78%	0.37	0.00%	1.00
MIRIX (Baseline) ( Wang and Chen, 2025 )	Vision	32.67%	46.67%	0.35	0.00%	1.00
MMA (Ours)	Text	13.15%	56.67%	0.28	23.53%	0.69
MMA (Ours)	Vision	13.55%	42.22%	-0.16	41.18%	-0.38

Table 4: MMA-Bench Main Results. Comparison across logic types (Type D uses risk-adjusted CoRe scoring). MMA restores agency in Type B conflict and mitigates the visual placebo effect in Type D scenarios.

Figure 4: Detailed Dynamics Analysis. (a) Step-wise belief revision; (b) Risk-adjusted scores highlighting visual noise sensitivity; (c) Gap analysis between retrieval accuracy and calibration.

Figure 5: Evolutionary Logic Spectrum. Tracing performance from Foundation Model to MMA. MMA restores agency (Activation) and buffers inherited visual bias (Placebo Effect).

Model	Deterministic (Type A, Vis)		Indeterminate (Type D, Vis)
Model	Verdict Acc.	Status	CoRe Score	Interpretation
MMA (Full)	$\mathbf{50.0\%}$	Robust	$\mathbf{-0.38}$	Buffered
tc (w/o Source)	$0.0\%$	Paralyzed	$1.00^{\dagger}$	Artifact of Default
st (w/o Consen.)	$36.7\%$	Unstable	$-0.69$	Hallucinated
cs (w/o Time)	$0.0\%$	Degraded	$1.00^{\dagger}$	Artifact of Default

Table 5: Ablation results on MMA-Bench (Vision Mode). † Perfect scores in Type D coincide with 0% accuracy in Type A, indicating system paralysis rather than genuine calibration.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Selective Prediction Analysis on FEVER. MMA consistently outperforms the Baseline under abstention-based risk control, achieving higher utility and lower risk across evaluation settings.

Mode	Components	Raw Acc. (%)	Act. Acc. (%)	Correct Abstain	Wrong Abstain
MMA (Ours)	$S+T+C_{\text{con}}$	$59.93\pm 1.62$	$71.61\pm\mathbf{0.43}$	$103.7\pm 4.7$	$122.6\pm 11.2$
tc (w/o Source)	$T+C_{\text{con}}$	$\mathbf{60.47\pm 1.33}$	$71.61\pm 2.54$	$102.3\pm 12.6$	$117.7\pm 20.6$
cs (w/o Time)	$S+C_{\text{con}}$	$59.00\pm 2.27$	$68.96\pm 2.35$	$95.0\pm 13.2$	$\mathbf{114.0\pm 32.1}$
st (w/o Consen.)	$S+T$	$58.93\pm 3.48$	$\mathbf{72.05\pm 2.34}$	$\mathbf{105.7\pm 16.1}$	$131.0\pm 36.4$

Appendix

Table 6: Ablation results on FEVER (

N=500,\text{Seeds}=3

). Act. Acc. (Actionable Accuracy) denotes the precision of non-abstained responses (Mean

\pm

Std). Notably, while MMA shares a similar mean accuracy with other variants, it achieves significantly lower variance (

\sigma=0.43

), demonstrating superior stability compared to tc (

\pm 2.54

) and st (

\pm 2.34

Panel of Figure 7 — Figure 7: Ablation Study Results on FEVER. We compare the Full Model (MMA) against variants without Consensus (‘st’), without Time (‘cs’), and without Source (‘tc’). (a) Shows that removing Consensus drastically increases variance. (b) Shows that MMA maintains high utility under strict prudence requirements (high $\alpha$ ). (c) Visualizes the trade-off between sensitivity and conservativeness.

Method	Reasoning Categories (LLM Score)				Overall Metrics		Reliability	Utility
Method	Single-Hop	Multi-Hop	Open-Domain	Temporal	Accuracy	Wrong Ans.	Act. Acc.	( $\lambda=1,r=0.2$ )
MIRIX (Baseline)	80.14	76.01	67.71	78.00	77.37	317	78.96%	880.0
MMA (Ours)	73.76	62.31	59.38	77.05	72.31	335	76.80%	793.6
Variant ‘st’	79.08	67.91	61.46	79.55	75.94	298	79.64%	883.6

Appendix

Table 7: Main Results on LoCoMo. Breakdowns of LLM Scores across reasoning dimensions (

N=1542

). While the Baseline excels in raw accuracy, our ‘st’ variant achieves the highest Actionable Accuracy (79.64%) and Utility , demonstrating superior reliability in safety-critical retrieval tasks.

Panel of Figure 8 — Figure 8: Quantitative Analysis on LoCoMo. While MMA focuses on prudence, its ‘st’ configuration demonstrates robust utility advantages over the Baseline in high-stakes settings.

Mode	Components	Utility	Wrong Ans.	Abstain Count	Act. Acc.
MMA (Full)	$S+T+C_{\text{con}}$	488.0	335	98	76.80%
st (w/o Consen.)	$S+T$	609.0	298	78	79.64%
cs (w/o Time)	$S+C_{\text{con}}$	480.5	335	113	76.56%
tc (w/o Source)	$T+C_{\text{con}}$	471.5	344	77	76.52%

Appendix

Table 8: Ablation results on LoCoMo (

N=1542

). Utility is computed with

\lambda=2.0,r=0.5

. Wrong Ans. denotes hallucinations (Lower is Better). The ‘st’ variant achieves the best safety profile.

Panel of Figure 9 — Figure 9: Ablation Sensitivity on LoCoMo. Removing the Consensus module (‘st’) actually improves robustness in this specific domain, while removing Source (‘tc’) or Time (‘cs’) degrades performance.

Model	Mode	Overall Metrics			Scenario-Specific Analysis
Model	Mode	Core Acc.	Verdict Acc.	CoRe Score	Type B Acc.	Type D Score
GPT-4.1-mini	Text (Oracle)	85.26%	77.78%	0.59	76.47%	0.85
GPT-4.1-mini	Vision (Raw)	80.74%	73.33%	0.51	64.71%	0.23
Qwen-3-VL-Plus	Text (Oracle)	88.05%	65.56%	0.32	88.24%	-0.69
Qwen-3-VL-Plus	Vision (Raw)	88.98%	63.33%	0.28	82.35%	-0.69

Appendix

Table 9: Cognitive dynamics of foundation models on MMA-Bench. Core Acc. measures basic reading comprehension. CoRe Score (Risk-Adjusted) reflects epistemic calibration. Type B Acc. indicates success in Reliability Inversion (overcoming authority bias). Type D Score reflects prudence in unknowable scenarios. Note the significant drop in Type D score for GPT-4.1-mini when switching to Vision mode, illustrating the Visual Placebo Effect .

Panel of Figure 10 — Figure 10: Cognitive Dynamics of Foundation Models on MMA-Bench. We compare GPT-4.1-mini and Qwen-3-VL-Plus across Text (Oracle) and Vision (Raw) modes. (a) Reveals how visual modalities can act as distractors in noise scenarios. (b) Highlights the disconnect between reading comprehension (Core Acc) and epistemic prudence (CoRe Score).

Panel of Figure 11 — Figure 11: Mechanism Ablation on MMA-Bench (Vision Mode). We isolate the failure modes: (a) Accuracy metrics reveal that Source ( $S$ ) and Time ( $T$ ) are prerequisites for agency, as their absence leads to paralysis (0% accuracy in known facts); (b) CoRe Scores demonstrate that Consensus ( $C_{\text{con}}$ ) is essential to buffer against the Visual Placebo Effect in indeterminate queries.

Explore similar work

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Memory LLM Agent Memory
University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3
SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory
May 15, 2026Huacan Chai, Yukai Wang, Yingxuan Yang +7Multimodal Memory Multimodal Reasoning
Shanghai Jiao Tong University, China · OPPO, China
DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection
Jun 25, 2026Yujin Tang, Chenming Shang, Ruize Xu +1Multimodal Memory Long-Horizon Agent Evaluation
Dartmouth College

May 28, 2026J/K move · Enter open · S save

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu, Yepeng Liu+13

University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University+3

Multimodal Memory LLM Agent Memory AI Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks Memory Agent Benchmarks

Open paper AskPDF arXiv

Multimodal large language models are increasingly deployed as long-horizon agents, where memory must do more than recall: it must track an evolving world, revise what has gone stale, and surface the right evidence at decision time. Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions, leaving us unable to localize failures to writing, maintenance, retrieval, or use. The rise of agent harnesses that author their own memory sharpens this gap, since we have no principled way to compare hand-designed pipelines with self-managing alternatives. To close these gaps, we formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle, and instantiate it in WorldMemArena: 400 multi-session multimodal tasks spanning Lifelong Evolution (evolving personal and task states) and Agentic Execution (memory from real observations, actions, and feedback), annotated with gold memory points, updates, distractors, and evidence chains for stage-level diagnosis. This enables the first head-to-head comparison of long-context, manually designed (RAG and external memory systems), and harness-based memory agents. Results show that: (1) better memory writing and storage do not guarantee better performance; (2) multimodal memory still struggles to fully use visual evidence; (3) systems are unstable across domains and degrade on realistic agentic trajectories; and (4) harness memory is more flexible but remains costly and less reliable.

Abstract

Explore similar work

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory

DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection