M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

Zhengjun Huang; Wenxuan Liu; Zhoujin Tian; Wei Chen; Junle Chen; Yuqian Wu; Fangyuan Zhang; Qintian Guo

cs.CLJun 5, 2026

M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

Authors: Zhengjun Huang, Wenxuan Liu, Zhoujin Tian, Wei Chen, Junle Chen, Yuqian Wu, Fangyuan Zhang, Qintian Guo, +1 more

Organizations: The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology · Tencent Hy · Harbin Institute of Technology (Shenzhen) · The Hong Kong University of Science and Technology (Guangzhou) · Beijing Institute of Technology (Zhuhai)

Abstract

Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over authentic multimodal file interaction nor the interpretation of concealed user information. We therefore introduce M $^3$ Exam, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference. Benchmarking MLLMs and memory systems reveals persistent gaps in cross-modal grounding, cross session reasoning, and the efficiency cost of accumulating multimodal context. We further propose M $^3$ Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand, improving accuracy by 13% while cutting index-construction time and retrieved tokens by over 70%.

Explore similar work

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Memory Multimodal Agents
SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory
May 15, 2026Huacan Chai, Yukai Wang, Yingxuan Yang +7Multimodal Memory Multimodal Reasoning Benchmarks
H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions
Jun 8, 2026Shiping Zhu, Yibo Yang, Zhengyang Wang +3Multimodal Memory Agent Memory

May 28, 2026 · cs.CVJ/K move · Enter open · S save

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu, Yepeng Liu+13

University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University · J.P. Morgan Chase · Stanford University

Multimodal large language models are increasingly deployed as long-horizon agents, where memory must do more than recall: it must track an evolving world, revise what has gone stale, and surface the right evidence at decision time. Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions, leaving us unable to localize failures to writing, maintenance, retrieval, or use. The rise of agent harnesses that author their own memory sharpens this gap, since we have no principled way to compare hand-designed pipelines with self-managing alternatives. To close these gaps, we formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle, and instantiate it in WorldMemArena: 400 multi-session multimodal tasks spanning Lifelong Evolution (evolving personal and task states) and Agentic Execution (memory from real observations, actions, and feedback), annotated with gold memory points, updates, distractors, and evidence chains for stage-level diagnosis. This enables the first head-to-head comparison of long-context, manually designed (RAG and external memory systems), and harness-based memory agents. Results show that: (1) better memory writing and storage do not guarantee better performance; (2) multimodal memory still struggles to fully use visual evidence; (3) systems are unstable across domains and degrade on realistic agentic trajectories; and (4) harness memory is more flexible but remains costly and less reliable.

Multimodal Memory Multimodal Agents Agent Memory Agentic Benchmarks Arena

Open paper Ask about it PDF

Abstract

Explore similar work

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory

H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions