CUE-Mem: Benchmarking Long-Term User Memory via Implicit Cues in Multimodal Conversations

Yulin Hu; Yanyan Zhao; Zimo Long; Xing Fu; Mengtong Ji; Weixiang Zhao; Yutai Hou; Qianchao Wang

cs.AISep 26, 2026

CUE-Mem: Benchmarking Long-Term User Memory via Implicit Cues in Multimodal Conversations

Authors: Yulin Hu, Yanyan Zhao, Zimo Long, Xing Fu, Mengtong Ji, Weixiang Zhao, Yutai Hou, Qianchao Wang,

Organizations: Harbin Institute of Technology · Huawei Technologies Co., Ltd.

Abstract

Long-term memory is essential for multimodal agents that interact with users across sustained conversations. However, user memories are not always explicitly stated: they may also be implied by recurring background objects in images, ambient sounds in audio, or other peripheral multimodal cues. Existing benchmarks largely focus on text-only memory or explicit multimodal evidence, leaving implicit multimodal cues underexplored. We introduce CUE-Mem, a text-image-audio benchmark for evaluating long-term user memory from implicit cues. CUE-Mem contains 2,674 questions across explicit and implicit evidence settings and covers four tasks: Entity Recall, Long Pattern, Personalized Recommendation, and Answer Refusal. Across textualized memory systems, implicit performance remains far below oracle evidence, locating the main bottleneck in preserving and retrieving subtle cues rather than question answerability. Increasing caption detail recovers more of this evidence, but brings uneven gains and rapidly growing token costs, motivating native multimodal access. Yet native access does not uniformly resolve the bottleneck: evidence use depends strongly on the backbone, while multimodal indexing introduces substantial retrieval noise. CUE-Mem provides a testbed for memory systems that selectively retain, retrieve, and use subtle multimodal evidence.

Figures & tables

Figure 1: Explicit and implicit evidence settings in CUE-Mem . Explicit memories are supported by directly stated or foregrounded information, whereas implicit memories rely on recurring background cues across images and audio. Profile boxes denote target memories.

Benchmark	Mod.	A.Turn	A.Sess.	A.MM	Imp.
DuLeMon	T	8.16	–	–	No
DialogBench	T	7.48	–	–	No
MemoryBank	T	3.77	10	–	No
LOCCO	T	4.77	10	–	No
LongMemEval	T	5.19	35	–	No
LoCoMo	T+I	10.81	13.6	3.35	No

Table 1: Comparison with representative conversational and multimodal memory benchmarks. T, I, and A denote text, image, and audio. A.Turn denotes turns per session; when session boundaries are unavailable, we use the reported dialogue or conversational-source unit. A.Sess. and A.MM denote sessions and multimodal items per history. MemLens statistics are averaged over its four context-length settings. Imp. distinguishes implicit multimodal cues from semantic (Sem.) inference.

Figure 2: Construction pipeline of CUE-Mem . Starting from predefined user profiles, we derive explicit and implicit evidence plans, distribute them across multi-session scripts, and realize them as text-image-audio conversations. Automatic and human checks are applied to ensure synthesis quality, long-term consistency, and the validity of implicit cues.

Figure 3: Task examples from CUE-Mem , covering Entity Recall, Long Pattern, Personalized Recommendation, and Answer Refusal. The first three illustrate memory evaluation from implicit multimodal cues, while Answer Refusal tests abstention under insufficient evidence.

Qwen3.6-35B-A3B
Method	Entity Recall		Long Pattern		Personalized Rec.		Answer Refusal		Memory Avg.
Method	Explicit	Implicit	Explicit	Implicit	Explicit	Implicit	Explicit	Implicit	Explicit	Implicit	EI-Gap
Human (Full)	90.2	84.7	84.6	82.1	74.9	81.5	–	–	83.2	82.8	0.4
Human (Oracle)	93.6	88.7	92.4	94.1	86.3	76.8	–	–	90.8	86.5	4.3
No Context	31.7	23.8	27.7	22.6	30.5	30.8	74.9	70.9	30.0	25.7	4.2
Oracle Evidence ⋆	87.3	93.8	85.0	81.2	82.6	67.3	–	–	85.0	80.8	4.2

Table 2: Main results of textualized memory systems on CUE-Mem. Human (Full) reads the full textualized history, whereas Human (Oracle) receives only oracle text. Oracle Evidence ⋆ denotes the source text prompts used to synthesize multimodal clues, which provide clean and accurate textual statements of the target information. Memory Avg. averages Entity Recall, Long Pattern, and Personalized Recommendation. Best non-oracle implicit accuracies within each backbone are bold.

Figure 4: Overview of CUE-Mem . Left: QA counts and clue-modality proportions by task and evidence setting (T: text, I: image, A: audio). Right: corpus-scale statistics and average clues per question. Answer Refusal has no supporting clues.

Figure 5: Effects and costs of cue-aware textualization. (a–b) Accuracy changes from Brief captions and ASR, averaged across memory systems; solid/dashed lines denote explicit/implicit results. (c) NaiveRAG accuracy versus retrieved tokens per QA; faint/large markers denote users/means, and ellipses show user variation. ER, LP, PR, and AR denote the tasks.

Qwen3-Omni-30B-A3B-Instruct
Index	Use	Entity Recall		Long Pattern		Personalized Rec.		Answer Refusal		Memory Avg.
Index	Use	Explicit	Implicit	Explicit	Implicit	Explicit	Implicit	Explicit	Implicit	Explicit	Implicit	EI-Gap
Human (Full)		82.4	45.1	88.1	60.3	70.2	57.1	–	–	80.2	54.2	26.0
Human (Oracle) ⋆		94.2	86.1	97.1	95.3	91.4	82.2	–	–	94.2	87.9	6.3
Oracle ⋆	Text	90.7	94.8	79.4	81.9	76.2	67.9	–	–	82.1	81.5	0.6
Oracle ⋆	Multimodal	91.9	84.4	87.8	53.5	81.2	59.6	–	–	87.0	65.8	21.1

Table 3: Results of the RAG-based multimodal memory variant. Human (Full) reads the complete native multimodal history; Human (Oracle) receives only oracle evidence. Oracle ⋆ uses annotated clues from the original multimodal history, bypassing retrieval. Memory Avg. averages the first three tasks; best non-oracle model scores are bold.

Figure 6: Supporting-evidence retrieval on explicit and implicit samples. MM-Index yields modest recall gains but lower precision, particularly for implicit evidence; F1 summarizes the resulting trade-off.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Explicit evidence	Implicit evidence	Construction-time validity test
Text	The user directly states the target fact, preference, relationship, habit, or routine.	Not used as a target-bearing source. Text may establish the foreground conversation, but may neither name nor paraphrase the implicit target.	The target and close paraphrases must occur in an explicit script and must be absent from every implicit dialogue turn and assistant reply.
Image	The target person, pet, place, or object is the centered or otherwise dominant subject and may be discussed in the dialogue.	A recurring target object is visible only at an edge, corner, or background region, possibly partially occluded; it is not discussed.	The target is recognizable at inspection, yet peripheral to the foreground subject and semantically disjoint from the dialogue.
Audio	The target information is conveyed by the user’s speech; the waveform is treated as another carrier of directly stated content.	A recognizable environmental sound is mixed behind the user’s speech and is absent from the speech transcript.	The ambient event is audible without masking speech, is compatible with the scene, and cannot be recovered from ASR alone.

Appendix

Table 4: Operational evidence taxonomy. Foreground/background status and whether the target is stated jointly determine explicitness; modality alone does not.

Family	Representative retained event types
Animals/nature	dog, cat, birds, rooster, waves, rain, wind
Domestic	vacuum, doorbell, clock, alarm, microwave, boiling
Food/drink	chopping, frying, cutlery, pouring, chewing, coffee grinder
Work/learning	typing, mouse click, printer, writing, chalk, telephone
Transport/travel	bicycle bell, subway, engine, horn, aircraft, suitcase
Sports/leisure	running, basketball, tennis, swimming, camera shutter

Appendix

Table 5: Semantic families in the retained ambient-event vocabulary.

Check	Acceptance criterion	Typical rejection
Schema and synthesis	All required fields and referenced assets exist; dialogue and media are natural, readable/audible, and mutually coherent.	Malformed record, corrupt media, unnatural script, modality mismatch.
Cross-session visual identity	SigLIP 2 similarity between each recurring entity and its profile reference passes the fixed identity threshold; incompatible references are rejected.	Changed face, pet markings, object color/material, or swapped identity.
Cross-session speaker identity	ERes2NetV2 embeddings are used to compare every synthesized utterance with the user’s fixed voice anchor and enforce cross-session speaker consistency.	Speaker drift, wrong user voice, unstable timbre.
Implicit visual validity	Dialogue–target semantic overlap remains below the leakage threshold, and the implicit target’s occupied fraction of the full image remains below the salience threshold while remaining recognizable.	Target named/paraphrased, centered, dominant, fully exposed, or absent.
Implicit audio validity	The canonical ambient event is audible, scene-compatible, and absent from the speech transcript; speech is not masked.	ASR leakage, inaudible/wrong ambience, clipping, masked speech.
QA and clue validity	One unambiguous answer is supported by all retained clue IDs; distractors are plausible but unsupported, and evidence type and modality labels agree with the source profile.	Wrong label, invalid clue ID, ambiguous option, insufficient support, or explicitness mismatch.

Appendix

Table 6: Automatic and human-auditable quality criteria.

Stage	Rejected at stage	Remaining
Generated candidates	—	100.00%
Automatic checks	42.80% of input	57.20%
Human review	22.10% of input	44.56%

Appendix

Table 7: Filtering flow. “Input” in the human-review row denotes samples remaining after automatic checks; hence 44.56% of the initial candidates survive both stages. Per-reason counts were not preserved in the transferred snapshot and will be added from the original annotation export.

Task	Target and answerability	Correct option	Distractor/refusal constraint
Entity Recall	A named relationship or pet attribute, or a recurrent user-associated item attribute, grounded in at least one retained clue.	The profile-consistent identity, relationship, appearance, occupation, age, personality, or object attribute.	Three alternatives match the queried attribute and granularity. They may change one or more entity attributes but cannot also be true of the target.
Long Pattern	A profile preference linked to its tagged events; exactly one question is generated for each preference.	The activity, taste, routine, trigger, or owner-level tendency expressed by that preference.	Three plausible patterns in the same life domain. They cannot be eliminated solely by option length or type, and the stem cannot quote the target preference.
Personalized Recommendation	A preference-grounded recommendation scenario; the recommended item itself must not already be mentioned in the history.	A new item or activity that matches the core style, need, setting, or value of the user’s established preference.	At least two hard distractors remain in the same product/activity class but differ on a preference-critical dimension; no “all/none/uncertain” options are allowed.
Answer Refusal	A natural-looking stem formed by changing exactly one premise slot of a real memory. The changed premise must be unsupported.	Exactly one option is “The history does not specify” (the fixed Chinese refusal string used throughout the benchmark JSON).	A clue-supported but question-incompatible answer is retained as the lure, with two additional same-domain distractors. The lure’s source clues are annotations of the trap, not evidence that the corrupted premise is true.

Appendix

Table 8: Task-specific construction and answerability criteria.

Panel of Figure 7 — Figure 7: Complete explicit Entity Recall example.

Panel of Figure 8 — Figure 8: Complete implicit Entity Recall example.

Panel of Figure 9 — Figure 9: Complete explicit Long Pattern example.

Panel of Figure 10 — Figure 10: Complete audio-supported implicit Long Pattern example.

Panel of Figure 11 — Figure 11: Complete explicit Personalized Recommendation example.

Panel of Figure 12 — Figure 12: Complete image-supported implicit Personalized Recommendation example.

Role	Reported model
RQ1/RQ2 answer backbone	Qwen3.6-35B-A3B
RQ1 answer backbone	GPT-5.4-mini
RQ2 image captioner	GPT-5.4
RQ2 speech/background models	Qwen3-ASR-1.7B; Gemini 3.1 Pro
RQ3 native answer backbones	Qwen3-Omni-30B-A3B-Instruct; MiniCPM-o-4.5
RQ3 embedding	Gemini Embedding 2

Appendix

Table 9: Model roles and reported model names.

Panel of Figure 15 — Figure 15: Human-baseline evaluation interface and an example submitted result. The interface presents the multimodal dialogue history and four-option QA side by side, while the completion dialog records coverage and accuracy for a profile–variant assignment.

Paper name	Implementation	Recall	Storage/update policy and system-specific values
Full Memory	FUMemory	Ordered history, 4,096-word budget	Linear, round-level storage; no consolidation or learned update.
NaiveRAG	LTMemory	MiniLM cosine, top–10	Linear, round-level observations; embeddings added at storage time.
Generative Agents	GAMemory	Semantic, recency, and importance fusion; top–10	Exponential recency decay 0.995; importance scored 1–10 by the answer backbone; reflection threshold 0.3, three salient questions and three insights, reflection top–10.
Reflexion	RFMemory	Full-history recall	Full-memory store plus a trial reflector that updates a concise high-level insight from the previous insight and the new trial.
MemGPT	MGMemory	Semantic recall for active/archive stores, top–10	Recursive LLM summary on a 4,096-word flush boundary; warning threshold 0.7; the checked-in benchmark sets skip_trigger=True .
A-Mem	AMemMemory	MiniLM cosine, top–10	Each round/caption is an A-Mem note; robust agentic-memory evolution uses threshold 100 and interval 15; memory-side LLM is patched to the current answer backbone; cached notes are keyed by data/config hashes.

Appendix

Table 10: Checked-in RQ1 memory-system configurations. LTMemory is the implementation of the NaiveRAG row in the paper.

Study	Backbone	Setting	Explicit [95% CI]	Implicit [95% CI]	EI-Gap [95% CI]	Gap run SD
RQ1	Qwen3.6	Base	66.5 [61.6, 71.2]	31.7 [27.3, 36.1]	34.8 [28.2, 41.2]	4.13
RQ2 image	Qwen3.6	Brief	72.4 [65.6, 79.0]	24.2 [17.8, 30.9]	48.2 [38.6, 57.4]	0.17
		Detailed	80.3 [73.9, 86.8]	44.4 [37.3, 51.9]	35.9 [26.1, 45.6]	1.86
RQ2 audio	Qwen3.6	ASR	75.4 [67.9, 82.5]	41.6 [27.5, 53.6]	33.8 [19.3, 49.5]	2.04
		Split	74.4 [66.8, 81.5]	52.6 [41.5, 62.8]	21.8 [9.1, 35.1]	0.89
RQ3	Qwen3-Omni	TT	72.7 [67.1, 78.2]	42.5 [36.5, 49.0]	30.2 [21.6, 38.6]	0.75

Appendix

Table 11: Three-run Memory Avg. accuracy and EI-Gap on the fixed five-profile audit subset. Values are percentages or percentage points. In RQ3, the first/second letter denotes Text (T) or Multimodal (M) index/use.

Study	Backbone	Contrast and estimand	Effect [95% CI]	Run SD	$\boldsymbol{p}$	$\boldsymbol{p_{\mathrm{Holm}}}$
RQ1	Qwen3.6	Base: Explicit $-$ Implicit	34.8 [28.1, 41.3]	4.13	0.0002	0.0026
RQ2 image	Qwen3.6	Detailed $-$ Brief, implicit accuracy	20.2 [14.7, 25.8]	0.93	0.0002	0.0026
		Detailed $-$ Brief, EI-Gap change	$-$ 12.3 [ $-$ 18.6, $-$ 6.0]	2.02	0.0006	0.0060
RQ2 audio	Qwen3.6	Split $-$ ASR, implicit accuracy	11.0 [3.7, 18.3]	1.98	0.0036	0.0306
		Split $-$ ASR, EI-Gap change	$-$ 12.0 [ $-$ 19.6, $-$ 4.0]	1.53	0.0034	0.0306
RQ3	Qwen3-Omni	MM-Use $-$ Text-Use, Text-Index	4.1 [ $-$ 1.0, 9.5]	0.99	0.1204	0.7223

Appendix

Table 12: Prespecified significance contrasts on the fixed five-profile audit. RQ1 compares independently resampled explicit and implicit sets; all RQ2 and RQ3 rows are paired by question. Profiles are not resampled. Effects and run SDs are percentage points. Numerical unadjusted and Holm-adjusted

p

-values are shown without significance symbols.

		Long Pattern						Entity Recall						Personalized Recommendation						Answer Refusal
Backbone	Memory	explicit			implicit			explicit			implicit			explicit			implicit			explicit			implicit
		br.	med.	det.	br.	med.	det.	br.	med.	det.	br.	med.	det.	br.	med.	det.	br.	med.	det.	br.	med.	det.	br.	med.	det.
Qwen3.6- 35B-A3B	Full Memory	77.6	90.6	87.1	36.6	44.3	47.3	77.3	85.8	88.7	16.2	39.3	65.2	71.6	70.2	76.6	40.6	47.4	48.4	18.2	18.2	16.7	73.3	60.0	44.0
	NaiveRAG	80.0	87.1	81.2	29.8	31.3	39.7	65.3	77.5	81.5	9.2	15.9	41.5	67.0	73.6	75.5	44.4	43.6	54.4	25.8	27.3	27.3	81.3	80.0	77.3
	A-Mem	75.3	84.7	83.5	29.8	35.9	41.2	58.9	72.7	78.6	8.7	16.7	44.5	64.8	73.6	74.3	39.1	45.9	52.9	66.4	66.1	65.0	97.4	97.8	97.4
	MemoryOS	27.1	31.8	40.0	13.0	17.6	22.9	9.6	10.1	14.1	3.2	4.0	8.5	50.0	53.1	56.1	39.8	42.9	46.9	92.7	93.2	93.6	99.3	99.7	99.7

Appendix

Table 13: Full image-caption-quality results for Qwen3.6-35B-A3B (accuracy, %). Brief (br.), medium (med.), and detailed (det.) denote the three caption granularities.

		Long Pattern						Entity Recall			Personalized Recommendation						Answer Refusal
Backbone	Memory	explicit			implicit			explicit			explicit			implicit			explicit			implicit
		ASR	hint	split	ASR	hint	split	ASR	hint	split	ASR	hint	split	ASR	hint	split	weak	normal	strong	weak	normal	strong
Qwen3.6- 35B-A3B	Full Memory	81.9	81.9	81.9	41.3	42.8	61.6	88.5	90.8	88.5	77.7	75.7	74.8	47.4	49.6	54.6	47.7	46.6	46.2	93.5	94.2	91.6
	NaiveRAG	81.0	80.2	80.2	32.6	33.3	51.4	88.5	85.1	82.8	68.0	71.8	73.8	47.4	46.7	50.2	61.6	60.2	62.0	96.8	97.1	94.5
	A-Mem	78.4	82.8	79.3	32.6	33.3	51.4	83.9	83.9	82.8	71.8	69.9	69.9	48.9	49.6	52.4	64.5	62.7	63.8	94.8	97.4	92.9
	MemoryOS	34.5	36.2	37.1	15.9	14.5	21.0	16.1	16.1	17.2	49.5	52.4	53.4	40.9	42.3	45.8	90.3	89.6	88.5	99.4	99.4	98.4

Appendix

Table 14: Full audio-caption-quality results for Qwen3.6-35B-A3B (accuracy, %), restricted to questions whose memory clue contains audio. Entity Recall has no implicit subset under this intervention.

Caption	Text	Image cap.	Audio cap.	Total	Text	Image cap.	Audio cap.	Total	Text QA	Image QA
	Memory input per profile				Retrieved input per QA				Question + options
Brief	22,866	1,326	8,880	33,195	$842{\pm}131$	$108{\pm}115$	$144{\pm}160$	$1{,}096{\pm}161$	$110{\pm}28$	$182{\pm}24$
Medium	22,866	4,789	8,880	36,658	$840{\pm}132$	$453{\pm}472$	$145{\pm}156$	$1{,}439{\pm}491$	$110{\pm}28$	$667{\pm}96$
Detailed	22,866	35,966	8,880	67,834	$836{\pm}133$	$4{,}249{\pm}4{,}476$	$143{\pm}149$	$5{,}230{\pm}4{,}475$	$110{\pm}28$	$3{,}585{\pm}2{,}866$

Appendix

Table 15: Token statistics for the image-caption intervention. Memory-input entries are means over five sampled profiles; retrieved-input and question entries are mean

\pm

standard deviation over questions. The fixed text and audio columns are unchanged by image-caption granularity.

		Recall@ $k$					Precision@ $k$
Index	Split	@1	@3	@5	@10	@25	@1	@3	@5	@10	@25
Text Index	Overall	4.5	11.6	17.2	27.4	44.6	43.3	39.7	37.3	32.4	23.9
	Explicit	6.6	17.1	25.2	39.2	61.7	60.8	55.8	52.5	45.3	33.0
	Implicit	1.8	4.7	7.1	12.5	23.1	21.3	19.6	18.2	16.2	12.4
Multimodal Index	Overall	6.2	14.6	20.1	28.6	42.6	42.4	36.0	31.0	23.1	15.4
	Explicit	8.0	20.0	27.1	37.3	52.9	62.5	54.9	47.1	34.7	22.8

Appendix

Table 16: Evidence-retrieval results (%) for text and multimodal indices, reported overall and by evidence explicitness.

Evidence	Targets	2	3	4	5	6	7	Mean	Median
		Occurrence count
Explicit	202	76	36	51	29	8	2	3.32	3
Implicit	152	0	82	50	16	3	1	3.62	3

Appendix

Table 17: Cue recurrence and temporal coverage. The upper panel counts user-specific preference targets in the 671-group construction manifest. The lower panel measures days between realized occurrences in the 648 generated events; it includes targets with at least two realized occurrences. “Minimum gap” is computed within each target and then summarized across targets.

Panel of Figure 16 — Figure 16: Retrieval failure case R1. The side-by-side lists are the stored top-five turn IDs from the Qwen3-Omni RQ3 runs.

Panel of Figure 17 — Figure 17: Retrieval failure case R2. Multimodal retrieval restores the peripheral bookshelf evidence needed to distinguish two plausible venues.

Panel of Figure 18 — Figure 18: Image-textualization failure. A single retained edge detail changes an underdetermined entity match into a supported answer.

Figure 19: Audio-textualization failure. The same Qwen3.6-35B-A3B and memory system change answer when the non-speech channel is retained.

Multimodal Memory

Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.

See all 71 papers

Jun 5, 2026

M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over authentic multimodal file interaction nor the interpretation of concealed user information. We therefore introduce M

^3

Exam, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference. Benchmarking MLLMs and memory systems reveals persistent gaps in cross-modal grounding, cross session reasoning, and the efficiency cost of accumulating multimodal context. We further propose M

^3

Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand, improving accuracy by 13% while cutting index-construction time and retrieved tokens by over 70%.

Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6

The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4

Multimodal Understanding Multimodal Grounding Efficient Multimodal Inference Multimodal Memory AI Agent Benchmarks Memory Agent Benchmarks+3

Ask PDF

Oct 21, 2026

MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents

Omni-modal assistants must remember not only what users say, but also what they show and convey. Existing long-term memory benchmarks are mostly text-and-image or report a single end-to-end score, so they cannot locate whether evidence is lost when it is represented, retrieved, delivered, or read. We introduce MOSAIC, a benchmark of 923 questions over persona-grounded histories of 11-24 sessions spanning text, image, video, and audio. MOSAIC constructs targeted spatial, temporal, and prosodic signals rather than decorative media, with the strongest causal validation for spatial information and selector-level validation for annotated prosody, and pairs them with a matched diagnostic protocol that freezes the question, reader, prompt, and budget while intervening on representation, retrieval, and delivery. Matched decomposition shows that representation loss is modality-dependent, residual reading dominates even under gold evidence, and abstention remains weak on these controls: strong answerable scores coexist with failures to refuse. MOSAIC thus turns omni-modal memory evaluation from a leaderboard into a diagnosis of write, retrieve, deliver, and read failures.

Hong Ting Tsang, Boyi Zhang, Haoyu Huang +7

The Hong Kong University of Science and Technology · University of Edinburgh

Multimodal Memory Long-Term Conversational Memory Multimodal Model Evaluation Multimodal Reasoning

Ask PDF

Aug 31, 2026

UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

Long-term memory is increasingly important for conversational agents, yet existing benchmarks primarily measure memory through pointwise factual recall: whether a system can recover isolated facts or event-level details from prior interactions. Real-world memory use, however, often requires a more demanding capability: integrating distributed, implicit, and noisy evidence across extended interaction histories into coherent, task-oriented outputs. We call this capability memory utilization. Here, we introduce UtilMem, a diagnostic benchmark comprising 1,717 instances across five domains, designed to evaluate four underexplored aspects of memory utilization: reasoning over dense histories, identifying implicitly relevant memories, synthesizing distributed evidence into summaries, analyses, or plans, and resisting interference from semantically similar distractors. Evaluating a diverse set of retrieval-based and memory-augmented systems, we find that strong performance on conventional factual-memory benchmarks does not reliably translate into effective memory utilization. Moreover, retrieval alone is insufficient: even when relevant evidence is successfully recovered, systems frequently fail to integrate information across sessions or to distinguish useful evidence from plausible distractors. These findings expose a substantial gap between accessing stored information and using it effectively, and suggest that progress in long-term conversational memory will require architectures that explicitly support evidence integration and robustness to retrieval interference. Code is available at https://github.com/peijunallin/UtilMem.

Peijun Qing, Fobo Shi, Soroush Vosoughi

Dartmouth College · Wuhan University

LLM Agent Memory Memory Agent Benchmarks Long-Term Conversational Memory

Ask PDF

Abstract

Multimodal Memory

Explore similar work

M3^33Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents

UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions