Omni-modal assistants must remember not only what users say, but also what they show and convey. Existing long-term memory benchmarks are mostly text-and-image or report a single end-to-end score, so they cannot locate whether evidence is lost when it is represented, retrieved, delivered, or read. We introduce MOSAIC, a benchmark of 923 questions over persona-grounded histories of 11-24 sessions spanning text, image, video, and audio. MOSAIC constructs targeted spatial, temporal, and prosodic signals rather than decorative media, with the strongest causal validation for spatial information and selector-level validation for annotated prosody, and pairs them with a matched diagnostic protocol that freezes the question, reader, prompt, and budget while intervening on representation, retrieval, and delivery. Matched decomposition shows that representation loss is modality-dependent, residual reading dominates even under gold evidence, and abstention remains weak on these controls: strong answerable scores coexist with failures to refuse. MOSAIC thus turns omni-modal memory evaluation from a leaderboard into a diagnosis of write, retrieve, deliver, and read failures.
Figures & tables
Figure 1: Example MOSAIC haystack: evidence spans four modalities and months of sessions. Answering the final query requires chaining a January text specification with later audio and video evidence.
Figure 2: Overview of MOSAIC. Top: what the benchmark tests—evidence scattered across months of text, image, video, and audio sessions must be chained across modalities. Construction targets native signals rather than decorative media: spatial layout and background detail in images; action order, repetition count, and state change in video; vocal affect in audio (causal validation is strongest for spatial detail; § 6.2 ). Bottom: where systems lose that signal—matched interventions on write (representation compression), retrieve (missed facts), deliver (retrieved but dropped under media caps), and read (misread complete evidence).
Put the target behind a more salient foreground subject.
328
284
Image
Fine-grained attribute
Require exact text, brand, shade, label, or a small count.
388
333
Image
Spatial configuration
Make relative position, rather than object presence, discriminative.
194
185
Video
Action ordering
Show two distinct completed actions in strict sequence without cuts.
67
64
Video
Repetition count
Repeat one action under a fixed camera, without a counter or overlay.
64
56
Video
State transition
Expose the start, change pattern, and end state.
64
64
Table 2: The seven modality challenges and their overlapping coverage among the 784 answerable items. Counts are required-fact occurrences and distinct items, not independent subsets.
Stage
Condition
Score
Img
Vid
Aud
M-hop
Chain
Diagnostic role
—
Gold symbolic facts
0.846
0.847
0.900
0.821
0.853
1.000
Bounds residual read error
write
Gold native evidence
0.783
0.785
0.729
0.826
0.783
1.000
Native perception cost
write
Gold generic text
0.690
0.693
0.625
0.718
0.682
1.000
Textualization loss
write
Gold structured text
0.704
0.705
0.652
0.742
0.702
1.000
Avoidable share of it
retrieve
Retrieved native, all delivered
0.708
0.709
0.631
0.741
0.696
0.675
Retrieval loss
deliver
Retrieved native, capped
0.670
0.671
0.576
0.696
0.652
0.491
Packing loss
Table 3: Matched write – retrieve – deliver – read decomposition on the 784 answerable items with the fixed Gemini 3.5 reader (Figure 3 ). Image / Video / Audio restrict to the 780/184/424 items whose chain needs that modality; Multi-hop to the 668 needing ≥ 2 facts; Chain is complete-chain delivery into the request. Paired CIs (and W/T/L for write contrasts) are in Appendix K .
Figure 3: Decomposing the loss on the 784 answerable items under one fixed reader, prompt, packer, and judge (Table 3 ). (a) Descending the native pipeline: each arrow is a paired drop associated with the intervened stage under the matched protocol; the shaded band is residual reading error under gold symbolic facts. (b) Write-time representation only: generic captions/ASR cost 9.3 points; a query-blind structured writer recovers a statistically insignificant 1.4.
Figure 4: Delivery-budget sweep over a single frozen top-50 ranked manifest. Retrieval quality is constant (Recall@50 =0.871 ); only the packer’s allowance changes. One-media-per-modality packing collapses complete-chain delivery from 0.675 to 0.184.
Figure 5: Within-item paired effects on the governing challenge stratum (item-bootstrap 95% intervals). Filled markers exclude zero. Spatial structure is the signal for which no tested text surrogate is adequate.
Backend / representation
Chain R@ k
Score
Flat RAG, caption text
0.698
0.522
Flat RAG, omni text
0.705
0.500
Flat RAG, native multimodal
0.466
0.641
A-MEM, caption text
0.787
0.684
A-MEM, omni text
0.702
0.664
MemoryOS, caption text
n/o
0.268
Table 4: Track A under the fixed Gemini 3.5 reader on 784 answerable QIDs. Chain R@ k denotes complete-chain delivery; MemoryOS coverage is unobservable. Paired effects: Appendix K.5 .
Figure 6: Answering versus abstention for all twelve Track B systems. Marker area is proportional to the false-refusal rate. The shaded upper region is unoccupied. Full scores: Appendix K.6 , Table 28 .
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Hops
Sess.
Reasoning
Info. Extraction
1–2
1
Direct recall
Msr-Selection
2–3
≥ 2
Cross-sess. select
Msr-Bridge
3
≥ 3
Sequential chain
Knowledge Update
2
≥ 2
Old → new
Temporal Reasoning
2–3
≥ 2
Plan → disrupt
Answer Refusal
1–2
1
Abstention
Appendix
Table 5: The six MOSAIC question types. Hops = required facts in the reasoning chain; Sess. = evidence sessions that host those facts.
Type
Definition
Example
Information Extraction
Single-session factual recall. Disambiguates among ≥2 same-category items via a discriminator (tone, sequence, count, temporal hint). 1–2 hops.
“In the photo of my fondue set with the colour-coded forks, what brand is stamped on it?”
MSR Selection
Cross-session entity binding: entities of the same category appear in different sessions; a discriminator selects the target. 2–3 hops, ≥2 sessions.
“What background detail is visible in the photo of the tomato seedlings I sounded happy about?”
MSR Bridge
Sequential chain: entity e0 in s0 identifies e1 in s1 , which identifies e2 in s2 ; the answer is an attribute of e2 . 3 hops, ≥3 sessions.
“What is the packing method from the carver in the video where I first hollowed then inlaid, in the town on the sign in my chess set photo?”
Knowledge Update
Evolving attribute tracking (old → new state) across sessions. 2 hops, ≥2 sessions.
“In the two photos, what brand did my chore coat change to?” (Carhartt → Dickies)
Temporal Reasoning
Time-order reasoning with a causal disruption: an old plan is altered; the answer depends on the new outcome. 2–3 hops, ≥2 sessions.
“After I sounded unhappy about the tour cancellation, what was the shade name of the replacement tile I photographed?”
Answer Refusal
Evidence-stripped unanswerable calibration check. The question appears answerable on its surface, but the required evidence has been deliberately removed. 1–2 hops, 1 session.
“In the photo of my Kierkegaard book on the bench by the foggy harbour, what stand-up comedy detail is in the background?” (Expected: “I don’t know.”)
Appendix
Table 6: The six MOSAIC question types with definitions and concrete examples.
Figure 7: The MOSAIC generation pipeline. Stage 1 samples a persona and designs a multi-hop reasoning chain; Stage 2 audits it via a per-fact deletion test against twelve failure modes; Stage 3 realises it as a timestamped multi-session conversation with persona-conditioned filler; Stage 4 synthesises the media assets, followed by structural and LLM-based quality filtering.
Type
Chain template
Info. Extraction
Nar opt → Sel → Ans
Msr-Selection
Nar opt → Sel → Ans (cross-sess.)
Msr-Bridge
Nar opt → Sel → Sel → Ans
Knowledge Update
Nar opt → Sel old → Ans new
Temporal Reasoning
Nar opt → Sel plan → Sel disrupt → Ans out.
Answer Refusal
inherits IE, evidence stripped
Appendix
Table 7: Chain templates. Nar = Narrower , Sel = Selector , Ans = Answer . “opt” denotes an optional narrowing step.
Failure Mode
Definition
Lexical-Leak
The question text paraphrases or contains the answer value.
Bridge-Integrity
Adjacent facts in the chain do not share a concrete referent.
Shortcut
The answer can be derived from text-only evidence, bypassing the intended modality gap.
Entity-Mismatch
The entity described in the Answer fact does not match the entity selected by prior facts.
Gap-Anchoring
The question text does not reference the gap-bearing detail, making it unclear which evidence to retrieve.
Redundant-Gap
Two facts test the same modality gap, reducing diversity.
Appendix
Table 8: The twelve named failure modes checked during Stage 2 validation.
Figure 8: The three system architectures evaluated on MOSAIC: Caption - Text (VL captioner + Whisper ASR, text index), Omni - Text (single omni captioner preserving prosody, text index), and Native - MM (multimodal embeddings over raw artefacts, omni - modal reasoner).
Score
Label
Definition
1.0
Perfect
Semantically identical to expected or alias. Only formatting/capitalization differs.
0.7
Mostly Correct
Right answer but extra context or slightly imprecise wording.
0.5
Partial
Gets category or one part right, misses specifics.
0.3
Attempted
Core fact wrong but shows effort.
0.0
Wrong
Completely incorrect, refused on answerable question, empty, or error.
Appendix
Table 9: LLM-as-Judge scoring rubric.
Tier
Image
Video
Audio
Total
Required
911
195
476
1,582
Evidence (non-req)
861
366
1,127
2,354
Filler
2,525
0
4,294
6,819
Total
4,297
561
5,907
10,765
Appendix
Table 10: Multimodal artefacts in MOSAIC.
Metric
Value
Total items
923
Personas
70
Items per persona (mean)
13.2
Answerable items
784
Answer Refusal items
139
Sessions per item (mean)
17.5
Appendix
Table 11: MOSAIC dataset statistics.
Question Type
Count
%
Knowledge Update
193
20.9%
Temporal Reasoning
189
20.5%
Information Extraction
178
19.3%
Answer Refusal
139
15.1%
Msr Bridge
112
12.1%
Msr Selection
112
12.1%
Appendix
Table 12: Question type distribution.
Modality
All Facts
Required Facts
Text
914
278
Image
1,788
911
Audio
1,304
476
Video
567
195
Total
4,573
1,860
Appendix
Table 13: Evidence modality distribution. 3,659/4,573 (80.0%) of all facts are non-text.
Gap Strategy
Count
Image — Background Detail
328
Image — Fine Grained Attribute
388
Image — Spatial Configuration
194
Video — Action Ordering
67
Video — Action Repetition Count
64
Video — State Transition
64
Appendix
Table 14: Gap strategy distribution on required facts. Named challenge tags cover 1,555/1,582 required non-text facts; the 27 untagged residuals are 26 required audio facts without a tone field and 1 image fact without a visual strategy (release audit of stage2_human_filtered ).
Benchmark
Items
Tok.
Avg. Sess.
Non-text
MOSAIC (Ours)
923
∼ 269K/pers.
17.5/item
80.0%
LongMemEval
500
∼ 158K/pers.
∼ 50
0%
LoCoMo
1,986
∼ 17K/conv.
27.2
images
Appendix
Table 15: Haystack scale comparison with prior conversational memory benchmarks. LoCoMo figures follow Maharana et al. (ACL 2024): 1,986 QA pairs, mean 16.6K tokens and 27.2 sessions per conversation. MOSAIC session counts are per item; token budgets for MOSAIC/LongMemEval are per persona.
Stage
Input
Output
Net
Initial generation
—
1,050
—
(1) Structural filter (9 checks)
1,050
1,019
−31
(2) LLM quality judge (4 checks)
1,019
1,004
−15
(3) Omni probe selection
1,004
885
−119
(4) Second quality intersection
885
879
−6
(5) Human-rescued probe FNs
879
923
+44
Appendix
Table 16: Sample-count lineage. The omni probe checks 864 answerable candidates; all 140 answer-refusal candidates bypass that probe and enter the next quality intersection.
Check
Pass
Rate
Atomicity (one question, one answer)
1,012/1,019
99.3%
Grounding (answer in evidence)
1,014/1,019
99.5%
Multimodal required (needs non-text)
1,015/1,019
99.6%
Gap load-bearing (gaps gate answer)
1,007/1,019
98.8%
Overall (all 4 checks)
1,004/1,019
98.5%
Appendix
Table 17: LLM quality judge per-check pass rates on the 1,019 structurally-filtered items (conjunctive filter; 15 items fail at least one check).
Metric
Value
Answerable items checked
864
Omni probe YES (answerable)
745
Omni probe NO (unanswerable)
119
Raw pass rate
86.2%
Answer Refusal candidates bypassed
140
Probe-YES retained after quality intersection
740
Appendix
Table 18: Omni-probe and final sample-count reconciliation. The 91.3% statistic is computed before the second quality intersection and is therefore not a final-set count; the figure quoted in § 4 is the 90.7% retention rate. Neither is a human answerability ceiling, since annotators adjudicated only a 100-item sample while the probe adjudicated all candidates.
Contrast
n
Mean Δ
MDE
Ordered − shuffled (ordering)
64
−0.009
8.9
Event − frames (repetition)
56
+0.016
10.8
Event − frames (transition)
64
+0.027
11.2
Appendix
Table 19: Paired video MDEs (in judge-score points) at 80% power. n counts unique QIDs, the power-analysis unit used in the main text; intervention-case n values are 67/64/64.
Stratum / sample
Q1
Q2
Q3
Audit probe-YES ( n=30 , 3 ann.)
97.8%
100%
94.4%
Audit probe-NO ( n=70 , 3 ann.)
92.9%
55.7%
52.9%
Follow-up YES ( n=100 , 2 ann.)
98.5%
97.0%
97.0%
Combined probe-YES ( n=130 )
98.3%
97.7%
96.4%
Appendix
Table 20: Annotator-mean Yes rates by probe stratum (Q1 well-posed, Q2 gold correct, Q3 evidence sufficient). Combined YES merges the audit YES subset with the disjoint probe-YES follow-up (item score = mean of annotator Yes/No).
Estimand ( N=784 answerable)
Est.
95% CI
Q1 well-posed
98.4%
[97.3,99.5]
Q2 gold answer correct
97.8%
[96.3,99.4]
Q3 evidence sufficient
96.6%
[95.1,98.2]
Appendix
Table 21: Stratified estimates on the 784 answerable MOSAIC-Eval items. Probe-YES stratum ( N=740 ) uses the combined n=130 sample; the 44 rescued items enter at rate 1.0 (human-verified at rescue). CIs use item-score variance with finite-population correction on the YES stratum. The 139 refusal items are excluded (different label).
Question
Pooled % (3 evaluators)
Q1. Well-posed? — Yes
94.3%
Q1. Well-posed? — Partially
3.0%
Q1. Well-posed? — No
2.7%
Q2. Gold answer correct? — Yes
69.0%
Q2. Gold answer correct? — No
31.0%
Q3. Evidence sufficient? — Yes
65.3%
Appendix
Table 22: Pooled responses on the stratified 100-item audit (70 probe-NO / 30 probe-YES). Not a release estimate; see Tables 20 – 21 .
Question
E1
E2
E3
Q1. Well-posed? — Yes
94
99
90
Q1. Well-posed? — Partially
4
0
5
Q1. Well-posed? — No
2
1
5
Q2. Gold correct? — Yes
65
66
76
Q2. Gold correct? — No
35
34
24
Q3. Evidence sufficient? — Yes
68
52
76
Appendix
Table 23: Per-evaluator responses on the stratified 100-item audit.
Failure Mode (Q4)
Count
%
Evidence item invalid
126
60.0%
Model overlooked gap-bearing detail
34
16.2%
Model refused incorrectly
31
14.8%
Model attended to wrong modality
16
7.6%
Judge format mismatch
3
1.4%
Appendix
Table 24: Failure mode distribution on Judge=NO items (pooled across 3 evaluators).
Pair
κ
Band
ρ
Qwen vs. GLM-5.2
0.962
0.880
0.954
Qwen vs. Grok-4.5
0.966
0.898
0.957
GLM-5.2 vs. Grok-4.5
0.971
0.892
0.963
Mean score (12 runs)
Qwen (primary)
0.536
GLM-5.2
0.544
Appendix
Table 25: Three-judge panel agreement, pooled over all twelve Track B runs (quadratic-weighted κ over the five rubric bands). The three judges differ by at most 0.010 in mean score and agree on the induced system ranking, which justifies reporting every result under the single primary judge.
Condition
Gemini
MiMo
Gold symbolic facts
0.878
0.870
Gold native evidence
0.822
0.818
Gold generic text
0.709
0.739
Gold structured text
0.738
0.707
Retrieved, production-budgeted
0.694
0.669
Retrieved, all delivered †
0.713
0.157
Appendix
Table 26: Mean judge score on the 120-item pilot ( n=120 per cell). † MiMo all-delivered is provider-limited (91/120 timeouts) and is reported only to document the failure mode.
Mod.
Contrast
N
Δ [95% CI]
Img
Raw − OCR (fine/bg)
716
+0.185 [0.153, 0.216]
Img
Raw − OCR (spatial)
194
+0.320 [0.248, 0.392]
Img
Spatial − object-only
194
+0.154 [0.095, 0.215]
Vid
Ordered − shuffled
67
−0.015 [ −0.078 , 0.046]
Vid
Event − frames (rep.)
64
+0.014 [ −0.052 , 0.080]
Vid
Event − frames (trans.)
64
+0.027 [ −0.050 , 0.106]
Appendix
Table 27: Within-item modality interventions (governing stratum). N is intervention cases except correct − swapped, which uses unique QIDs (primary analysis unit). W/T/L: fine/bg 222/434/60; spatial raw − OCR 92/89/13; spatial desc. − object 58/117/19; ordered − shuffled 5/56/6; repetition 6/54/4; transition 11/44/9; raw − ASR 36/387/27; prosody-tag 34/394/22; correct − swapped 46/330/23.
Table 29: Per-question-type diagnostic ladder on the 784 answerable items (IE n=178 , MSR-S 112, MSR-B 112, KU 193, TR 189); refusal metrics are reported separately in Table 28 .
Metric
@3
@5
@10
@20
@50
Original-question ranking (retrieved conditions)
Required-fact recall
0.600
0.700
0.786
0.828
0.871
Complete-chain recall
0.282
0.388
0.515
0.583
0.676
Production fixed-rewrite ranking (end-to-end)
Required-fact recall
0.574
0.677
0.759
0.806
0.863
Complete-chain recall
0.265
0.371
0.490
0.556
0.661
Appendix
Table 30: Required-fact and complete-chain recall over ranked units, on the 784 answerable items.
Drop reason
Image
Video
Audio
All
Media-count cap
5,559
593
3,186
9,338
Context/token cap
0
0
0
0
Serialization/API error
0
0
0
0
Unsupported format
0
0
0
0
Total dropped artefacts
5,559
593
3,186
9,338
Appendix
Table 31: Retrieved-but-dropped artefacts by modality and recorded drop reason, under production delivery caps (media-count cap applied per modality). The end-to-end condition shows the same single-cause pattern (6,184 image, 587 video, 2,533 audio; 9,304 total).
Challenge (required)
Score
N
Image — Background detail
0.670
284
Image — Fine-grained attribute
0.801
333
Image — Spatial configuration
0.714
185
Video — Action ordering
0.773
64
Video — Action repetition count
0.666
56
Video — State transition
0.697
64
Appendix
Table 32: Per-challenge judge score for the best end-to-end system (ReAct over the native MiMo-V2.5 stack). An item may contribute to multiple rows, so the strata overlap and do not sum to 784.
Family
Mean score
Range
Native-MM ReAct
0.703
0.674–0.732
Native-MM RAG
0.651
0.614–0.687
A-MEM (text architectures)
0.617
0.611–0.623
Text-only ReAct
0.417
0.415–0.419
Text-only RAG
0.383
0.375–0.390
MemoryOS (text architectures)
0.249
0.243–0.255
Appendix
Table 33: Method family ranking by mean answerable judge score across the twelve Track B runs (two configurations per family).
Figure 9: Mean judge score on the 784 answerable items for every Track B system, with persona-clustered bootstrap 95% intervals. Colour encodes the architecture family. The native-multimodal configurations occupy the top of the ranking, and the two MemoryOS configurations the bottom.
Figure 10: Judge score by system × question type, with AR the correct-refusal rate on the 139 refusal items. Knowledge Update is the hardest answerable type for essentially every system. The AR column runs opposite to every answerable column, which is the indiscriminate-abstention pattern quantified in § 6.4 .
Figure 11: Judge score by question type: six panels, each ranking all twelve systems for that type. Colour encodes the architecture family. The answerable panels share a near-identical ordering, while the AR panel reverses it.
Figure 12: Birding-log photo from item mm_msr_selection_ac58aedd_002 . The answer-critical detail (receipt price $189.99 under the black 10x42 pair) is only readable in the image; the chat text never states the price.
Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.
Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over authentic multimodal file interaction nor the interpretation of concealed user information. We therefore introduce M3Exam, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference. Benchmarking MLLMs and memory systems reveals persistent gaps in cross-modal grounding, cross session reasoning, and the efficiency cost of accumulating multimodal context. We further propose M3Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand, improving accuracy by 13% while cutting index-construction time and retrieved tokens by over 70%.
Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6
The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4
Long-term memory is essential for multimodal agents that interact with users across sustained conversations. However, user memories are not always explicitly stated: they may also be implied by recurring background objects in images, ambient sounds in audio, or other peripheral multimodal cues. Existing benchmarks largely focus on text-only memory or explicit multimodal evidence, leaving implicit multimodal cues underexplored. We introduce CUE-Mem, a text-image-audio benchmark for evaluating long-term user memory from implicit cues. CUE-Mem contains 2,674 questions across explicit and implicit evidence settings and covers four tasks: Entity Recall, Long Pattern, Personalized Recommendation, and Answer Refusal. Across textualized memory systems, implicit performance remains far below oracle evidence, locating the main bottleneck in preserving and retrieving subtle cues rather than question answerability. Increasing caption detail recovers more of this evidence, but brings uneven gains and rapidly growing token costs, motivating native multimodal access. Yet native access does not uniformly resolve the bottleneck: evidence use depends strongly on the backbone, while multimodal indexing introduces substantial retrieval noise. CUE-Mem provides a testbed for memory systems that selectively retain, retrieve, and use subtle multimodal evidence.
Yulin Hu, Yanyan Zhao, Zimo Long +6
Harbin Institute of Technology · Huawei Technologies Co., Ltd.
Personalized AI assistants must recall and reason over long-term user memory, which naturally spans multiple modalities and sources such as images, videos, and emails. However, existing Long-term Memory benchmarks focus primarily on dialogue history, failing to capture realistic personalized references grounded in lived experience. We introduce ATM-Bench, the first benchmark for multimodal, multi-source personalized referential Memory QA. ATM-Bench contains approximately four years of privacy-preserving personal memory data and human-annotated question-answer pairs with ground-truth memory evidence, including queries that require resolving personal references, multi-evidence reasoning from multi-source and handling conflicting evidence. We propose Schema-Guided Memory (SGM) to structurally represent memory items originated from different sources. In experiments, we implement 5 state-of-the-art memory systems along with a standard RAG baseline and evaluate variants with different memory ingestion, retrieval, and answer generation techniques. We find poor performance (under 20% accuracy) on the ATM-Bench-Hard set, and that SGM improves performance over Descriptive Memory commonly adopted in prior works. Code available at: https://github.com/JingbiaoMei/ATM-Bench
Jingbiao Mei, Jinghong Chen, Guangyu Yang +3
Department of Engineering, University of Cambridge, United Kingdom · Department of Physics, University of Cambridge, United Kingdom · Independent Researcher