cs.CLOct 21, 2026

MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents

Authors: Hong Ting Tsang, Boyi Zhang, Haoyu Huang, Jiaxin Bai, Yufei Li, Yisen Gao, Zhongwei Xie, Hongyu Luo, 

Organizations: The Hong Kong University of Science and Technology · University of Edinburgh

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Omni-modal assistants must remember not only what users say, but also what they show and convey. Existing long-term memory benchmarks are mostly text-and-image or report a single end-to-end score, so they cannot locate whether evidence is lost when it is represented, retrieved, delivered, or read. We introduce MOSAIC, a benchmark of 923 questions over persona-grounded histories of 11-24 sessions spanning text, image, video, and audio. MOSAIC constructs targeted spatial, temporal, and prosodic signals rather than decorative media, with the strongest causal validation for spatial information and selector-level validation for annotated prosody, and pairs them with a matched diagnostic protocol that freezes the question, reader, prompt, and budget while intervening on representation, retrieval, and delivery. Matched decomposition shows that representation loss is modality-dependent, residual reading dominates even under gold evidence, and abstention remains weak on these controls: strong answerable scores coexist with failures to refuse. MOSAIC thus turns omni-modal memory evaluation from a leaderboard into a diagnosis of write, retrieve, deliver, and read failures.

Figures & tables

Figure 1: Example MOSAIC haystack: evidence spans four modalities and months of sessions. Answering the final query requires chaining a January text specification with later audio and video evidence.
Figure 2: Overview of MOSAIC. Top: what the benchmark tests—evidence scattered across months of text, image, video, and audio sessions must be chained across modalities. Construction targets native signals rather than decorative media: spatial layout and background detail in images; action order, repetition count, and state change in video; vocal affect in audio (causal validation is strongest for spatial detail; § 6.2 ). Bottom: where systems lose that signal—matched interventions on write (representation compression), retrieve (missed facts), deliver (retrieved but dropped under media caps), and read (misread complete evidence).
  ModalitiesReasoning  
BenchmarkMulti-Sess.TIVAMulti-HopCross-Mod.#ItemsAvg. Sess.
MemoryBank ( Zhong et al., 2023 )✓✓✗✗✗✗✗1943.8
DialogBench ( Ou et al., 2024 )✓✓✗✗✗✗✗9,7897.5
LongMemEval ( Wu et al., 2025 )✓✓✗✗✗✓✗5005.2
MemAgentBench ( Yu et al., 2025 )✓✓✗✗✗✓✗1,2009.6
LoCoMo ( Maharana et al., 2024 )✓✓✓✗✗✗✗1,98627.2
Table 1: Comparison with prior memory benchmarks. Cross-Mod. : answering requires chaining across modality boundaries. † Text–image co-reference only. ‡ Multi-source personal items (not dialogue sessions); avg. 1.6 evidence items/query. § 110 dialogues, avg. 9 Q&A turns. ¶ Scales with 32K–256K context variants. Artefact counts: Table 10 .
Mod.ChallengeControlled realizationFactsItems
ImageBackground detailPut the target behind a more salient foreground subject.328284
ImageFine-grained attributeRequire exact text, brand, shade, label, or a small count.388333
ImageSpatial configurationMake relative position, rather than object presence, discriminative.194185
VideoAction orderingShow two distinct completed actions in strict sequence without cuts.6764
VideoRepetition countRepeat one action under a fixed camera, without a counter or overlay.6456
VideoState transitionExpose the start, change pattern, and end state.6464
Table 2: The seven modality challenges and their overlapping coverage among the 784 answerable items. Counts are required-fact occurrences and distinct items, not independent subsets.
StageConditionScoreImgVidAudM-hopChainDiagnostic role
—Gold symbolic facts0.8460.8470.9000.8210.8531.000Bounds residual read error
writeGold native evidence0.7830.7850.7290.8260.7831.000Native perception cost
writeGold generic text0.6900.6930.6250.7180.6821.000Textualization loss
writeGold structured text0.7040.7050.6520.7420.7021.000Avoidable share of it
retrieveRetrieved native, all delivered0.7080.7090.6310.7410.6960.675Retrieval loss
deliverRetrieved native, capped0.6700.6710.5760.6960.6520.491Packing loss
Table 3: Matched write – retrieve – deliver – read decomposition on the 784 answerable items with the fixed Gemini 3.5 reader (Figure 3 ). Image / Video / Audio restrict to the 780/184/424 items whose chain needs that modality; Multi-hop to the 668 needing ≥\geq 2 facts; Chain is complete-chain delivery into the request. Paired CIs (and W/T/L for write contrasts) are in Appendix K .
Figure 3: Decomposing the loss on the 784 answerable items under one fixed reader, prompt, packer, and judge (Table 3 ). (a) Descending the native pipeline: each arrow is a paired drop associated with the intervened stage under the matched protocol; the shaded band is residual reading error under gold symbolic facts. (b) Write-time representation only: generic captions/ASR cost 9.3 points; a query-blind structured writer recovers a statistically insignificant 1.4.
Figure 4: Delivery-budget sweep over a single frozen top-50 ranked manifest. Retrieval quality is constant (Recall@50 =0.871=0.871 ); only the packer’s allowance changes. One-media-per-modality packing collapses complete-chain delivery from 0.675 to 0.184.
Figure 5: Within-item paired effects on the governing challenge stratum (item-bootstrap 95% intervals). Filled markers exclude zero. Spatial structure is the signal for which no tested text surrogate is adequate.
Backend / representationChain R@ kkScore
Flat RAG, caption text0.6980.522
Flat RAG, omni text0.7050.500
Flat RAG, native multimodal0.4660.641
A-MEM, caption text0.7870.684
A-MEM, omni text0.7020.664
MemoryOS, caption textn/o0.268
Table 4: Track A under the fixed Gemini 3.5 reader on 784 answerable QIDs. Chain R@ kk denotes complete-chain delivery; MemoryOS coverage is unobservable. Paired effects: Appendix K.5 .
Figure 6: Answering versus abstention for all twelve Track B systems. Marker area is proportional to the false-refusal rate. The shaded upper region is unoccupied. Full scores: Appendix K.6 , Table 28 .
Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

TypeHopsSess.Reasoning
Info. Extraction1–21Direct recall
Msr-Selection2–3≥\geq 2Cross-sess. select
Msr-Bridge3≥\geq 3Sequential chain
Knowledge Update2≥\geq 2Old →\rightarrow new
Temporal Reasoning2–3≥\geq 2Plan →\rightarrow disrupt
Answer Refusal1–21Abstention
Appendix
Table 5: The six MOSAIC question types. Hops = required facts in the reasoning chain; Sess. = evidence sessions that host those facts.
TypeDefinitionExample
Information ExtractionSingle-session factual recall. Disambiguates among ≥2\geq 2 same-category items via a discriminator (tone, sequence, count, temporal hint). 1–2 hops.“In the photo of my fondue set with the colour-coded forks, what brand is stamped on it?”
MSR SelectionCross-session entity binding: entities of the same category appear in different sessions; a discriminator selects the target. 2–3 hops, ≥2\geq 2 sessions.“What background detail is visible in the photo of the tomato seedlings I sounded happy about?”
MSR BridgeSequential chain: entity e0e_{0} in s0s_{0} identifies e1e_{1} in s1s_{1} , which identifies e2e_{2} in s2s_{2} ; the answer is an attribute of e2e_{2} . 3 hops, ≥3\geq 3 sessions.“What is the packing method from the carver in the video where I first hollowed then inlaid, in the town on the sign in my chess set photo?”
Knowledge UpdateEvolving attribute tracking (old →\rightarrow new state) across sessions. 2 hops, ≥2\geq 2 sessions.“In the two photos, what brand did my chore coat change to?” (Carhartt →\rightarrow Dickies)
Temporal ReasoningTime-order reasoning with a causal disruption: an old plan is altered; the answer depends on the new outcome. 2–3 hops, ≥2\geq 2 sessions.“After I sounded unhappy about the tour cancellation, what was the shade name of the replacement tile I photographed?”
Answer RefusalEvidence-stripped unanswerable calibration check. The question appears answerable on its surface, but the required evidence has been deliberately removed. 1–2 hops, 1 session.“In the photo of my Kierkegaard book on the bench by the foggy harbour, what stand-up comedy detail is in the background?” (Expected: “I don’t know.”)
Appendix
Table 6: The six MOSAIC question types with definitions and concrete examples.
Figure 7: The MOSAIC generation pipeline. Stage 1 samples a persona and designs a multi-hop reasoning chain; Stage 2 audits it via a per-fact deletion test against twelve failure modes; Stage 3 realises it as a timestamped multi-session conversation with persona-conditioned filler; Stage 4 synthesises the media assets, followed by structural and LLM-based quality filtering.
TypeChain template
Info. ExtractionNar opt →\rightarrow Sel →\rightarrow Ans
Msr-SelectionNar opt →\rightarrow Sel →\rightarrow Ans (cross-sess.)
Msr-BridgeNar opt →\rightarrow Sel →\rightarrow Sel →\rightarrow Ans
Knowledge UpdateNar opt →\rightarrow Sel old →\rightarrow Ans new
Temporal ReasoningNar opt →\rightarrow Sel plan →\rightarrow Sel disrupt →\rightarrow Ans out.
Answer Refusalinherits IE, evidence stripped
Appendix
Table 7: Chain templates. Nar == Narrower , Sel == Selector , Ans == Answer . “opt” denotes an optional narrowing step.
Failure ModeDefinition
Lexical-LeakThe question text paraphrases or contains the answer value.
Bridge-IntegrityAdjacent facts in the chain do not share a concrete referent.
ShortcutThe answer can be derived from text-only evidence, bypassing the intended modality gap.
Entity-MismatchThe entity described in the Answer fact does not match the entity selected by prior facts.
Gap-AnchoringThe question text does not reference the gap-bearing detail, making it unclear which evidence to retrieve.
Redundant-GapTwo facts test the same modality gap, reducing diversity.
Appendix
Table 8: The twelve named failure modes checked during Stage 2 validation.
Figure 8: The three system architectures evaluated on MOSAIC: Caption - Text (VL captioner + Whisper ASR, text index), Omni - Text (single omni captioner preserving prosody, text index), and Native - MM (multimodal embeddings over raw artefacts, omni - modal reasoner).
ScoreLabelDefinition
1.0PerfectSemantically identical to expected or alias. Only formatting/capitalization differs.
0.7Mostly CorrectRight answer but extra context or slightly imprecise wording.
0.5PartialGets category or one part right, misses specifics.
0.3AttemptedCore fact wrong but shows effort.
0.0WrongCompletely incorrect, refused on answerable question, empty, or error.
Appendix
Table 9: LLM-as-Judge scoring rubric.
TierImageVideoAudioTotal
Required9111954761,582
Evidence (non-req)8613661,1272,354
Filler2,52504,2946,819
Total4,2975615,90710,765
Appendix
Table 10: Multimodal artefacts in MOSAIC.
MetricValue
Total items923
Personas70
Items per persona (mean)13.2
Answerable items784
Answer Refusal items139
Sessions per item (mean)17.5
Appendix
Table 11: MOSAIC dataset statistics.
Question TypeCount%
Knowledge Update19320.9%
Temporal Reasoning18920.5%
Information Extraction17819.3%
Answer Refusal13915.1%
Msr Bridge11212.1%
Msr Selection11212.1%
Appendix
Table 12: Question type distribution.
ModalityAll FactsRequired Facts
Text914278
Image1,788911
Audio1,304476
Video567195
Total4,5731,860
Appendix
Table 13: Evidence modality distribution. 3,659/4,573 (80.0%) of all facts are non-text.
Gap StrategyCount
Image — Background Detail328
Image — Fine Grained Attribute388
Image — Spatial Configuration194
Video — Action Ordering67
Video — Action Repetition Count64
Video — State Transition64
Appendix
Table 14: Gap strategy distribution on required facts. Named challenge tags cover 1,555/1,582 required non-text facts; the 27 untagged residuals are 26 required audio facts without a tone field and 1 image fact without a visual strategy (release audit of stage2_human_filtered ).
BenchmarkItemsTok.Avg. Sess.Non-text
MOSAIC (Ours)923∼\sim 269K/pers.17.5/item80.0%
LongMemEval500∼\sim 158K/pers.∼\sim 500%
LoCoMo1,986∼\sim 17K/conv.27.2images
Appendix
Table 15: Haystack scale comparison with prior conversational memory benchmarks. LoCoMo figures follow Maharana et al. (ACL 2024): 1,986 QA pairs, mean 16.6K tokens and 27.2 sessions per conversation. MOSAIC session counts are per item; token budgets for MOSAIC/LongMemEval are per persona.
StageInputOutputNet
Initial generation—1,050—
(1) Structural filter (9 checks)1,0501,019−31-31
(2) LLM quality judge (4 checks)1,0191,004−15-15
(3) Omni probe selection1,004885−119-119
(4) Second quality intersection885879−6-6
(5) Human-rescued probe FNs879923+44+44
Appendix
Table 16: Sample-count lineage. The omni probe checks 864 answerable candidates; all 140 answer-refusal candidates bypass that probe and enter the next quality intersection.
CheckPassRate
Atomicity (one question, one answer)1,012/1,01999.3%
Grounding (answer in evidence)1,014/1,01999.5%
Multimodal required (needs non-text)1,015/1,01999.6%
Gap load-bearing (gaps gate answer)1,007/1,01998.8%
Overall (all 4 checks)1,004/1,01998.5%
Appendix
Table 17: LLM quality judge per-check pass rates on the 1,019 structurally-filtered items (conjunctive filter; 15 items fail at least one check).
MetricValue
Answerable items checked864
Omni probe YES (answerable)745
Omni probe NO (unanswerable)119
Raw pass rate86.2%
Answer Refusal candidates bypassed140
Probe-YES retained after quality intersection740
Appendix
Table 18: Omni-probe and final sample-count reconciliation. The 91.3% statistic is computed before the second quality intersection and is therefore not a final-set count; the figure quoted in § 4 is the 90.7% retention rate. Neither is a human answerability ceiling, since annotators adjudicated only a 100-item sample while the probe adjudicated all candidates.
ContrastnnMean Δ\DeltaMDE
Ordered −- shuffled (ordering)64−0.009-0.0098.9
Event −- frames (repetition)56+0.016+0.01610.8
Event −- frames (transition)64+0.027+0.02711.2
Appendix
Table 19: Paired video MDEs (in judge-score points) at 80% power. nn counts unique QIDs, the power-analysis unit used in the main text; intervention-case nn values are 67/64/64.
Stratum / sampleQ1Q2Q3
Audit probe-YES ( n=30n{=}30 , 3 ann.)97.8%100%94.4%
Audit probe-NO ( n=70n{=}70 , 3 ann.)92.9%55.7%52.9%
Follow-up YES ( n=100n{=}100 , 2 ann.)98.5%97.0%97.0%
Combined probe-YES ( n=130n{=}130 )98.3%97.7%96.4%
Appendix
Table 20: Annotator-mean Yes rates by probe stratum (Q1 well-posed, Q2 gold correct, Q3 evidence sufficient). Combined YES merges the audit YES subset with the disjoint probe-YES follow-up (item score == mean of annotator Yes/No).
Estimand ( N=784N{=}784 answerable)Est.95% CI
Q1 well-posed98.4%[97.3, 99.5][97.3,\,99.5]
Q2 gold answer correct97.8%[96.3, 99.4][96.3,\,99.4]
Q3 evidence sufficient96.6%[95.1, 98.2][95.1,\,98.2]
Appendix
Table 21: Stratified estimates on the 784 answerable MOSAIC-Eval items. Probe-YES stratum ( N=740N{=}740 ) uses the combined n=130n{=}130 sample; the 44 rescued items enter at rate 1.0 (human-verified at rescue). CIs use item-score variance with finite-population correction on the YES stratum. The 139 refusal items are excluded (different label).
QuestionPooled % (3 evaluators)
Q1. Well-posed? — Yes94.3%
Q1. Well-posed? — Partially3.0%
Q1. Well-posed? — No2.7%
Q2. Gold answer correct? — Yes69.0%
Q2. Gold answer correct? — No31.0%
Q3. Evidence sufficient? — Yes65.3%
Appendix
Table 22: Pooled responses on the stratified 100-item audit (70 probe-NO / 30 probe-YES). Not a release estimate; see Tables 20 – 21 .
QuestionE1E2E3
Q1. Well-posed? — Yes949990
Q1. Well-posed? — Partially405
Q1. Well-posed? — No215
Q2. Gold correct? — Yes656676
Q2. Gold correct? — No353424
Q3. Evidence sufficient? — Yes685276
Appendix
Table 23: Per-evaluator responses on the stratified 100-item audit.
Failure Mode (Q4)Count%
Evidence item invalid12660.0%
Model overlooked gap-bearing detail3416.2%
Model refused incorrectly3114.8%
Model attended to wrong modality167.6%
Judge format mismatch31.4%
Appendix
Table 24: Failure mode distribution on Judge=NO items (pooled across 3 evaluators).
Pairκ\kappaBandρ\rho
Qwen vs. GLM-5.20.9620.8800.954
Qwen vs. Grok-4.50.9660.8980.957
GLM-5.2 vs. Grok-4.50.9710.8920.963
Mean score (12 runs)
Qwen (primary)0.536
GLM-5.20.544
Appendix
Table 25: Three-judge panel agreement, pooled over all twelve Track B runs (quadratic-weighted κ\kappa over the five rubric bands). The three judges differ by at most 0.010 in mean score and agree on the induced system ranking, which justifies reporting every result under the single primary judge.
ConditionGeminiMiMo
Gold symbolic facts0.8780.870
Gold native evidence0.8220.818
Gold generic text0.7090.739
Gold structured text0.7380.707
Retrieved, production-budgeted0.6940.669
Retrieved, all delivered †0.7130.157
Appendix
Table 26: Mean judge score on the 120-item pilot ( n=120n{=}120 per cell). † MiMo all-delivered is provider-limited (91/120 timeouts) and is reported only to document the failure mode.
Mod.ContrastNNΔ\Delta [95% CI]
ImgRaw −- OCR (fine/bg)716+0.185+0.185 [0.153, 0.216]
ImgRaw −- OCR (spatial)194+0.320+0.320 [0.248, 0.392]
ImgSpatial −- object-only194+0.154+0.154 [0.095, 0.215]
VidOrdered −- shuffled67−0.015-0.015 [ −0.078-0.078 , 0.046]
VidEvent −- frames (rep.)64+0.014+0.014 [ −0.052-0.052 , 0.080]
VidEvent −- frames (trans.)64+0.027+0.027 [ −0.050-0.050 , 0.106]
Appendix
Table 27: Within-item modality interventions (governing stratum). NN is intervention cases except correct −- swapped, which uses unique QIDs (primary analysis unit). W/T/L: fine/bg 222/434/60; spatial raw −- OCR 92/89/13; spatial desc. −- object 58/117/19; ordered −- shuffled 5/56/6; repetition 6/54/4; transition 11/44/9; raw −- ASR 36/387/27; prosody-tag 34/394/22; correct −- swapped 46/330/23.
Back-endRepr.Ans.C.refF.refLat.Rdr
Naive RAGCaption0.3900.8350.4315.8DS
Naive RAGOmni0.3750.8710.4216.2DS
Naive RAGNative-MM0.6140.7340.14329.6MiMo
Naive RAGNative-MM0.6870.4960.02231.2Gem.
ReActCaption0.4150.8200.40418.6DS
ReActOmni0.4190.7990.39017.5DS
Appendix
Table 28: Track B on 923 items (vs. Figure 6 ). Ans./C.ref/F.ref/Lat. = answerable, correct/false refusal, latency (s). Rdr: DS=DeepSeek-v4-Flash, MiMo=MiMo-V2.5, Gem.=Gemini 3.5.
ConditionAllIEMSR-SMSR-BKUTR95% CI
Gold symbolic facts0.8460.7300.7410.9240.8980.916[0.822, 0.869]
Gold native evidence0.7830.7990.7770.7900.6680.885[0.759, 0.806]
Gold generic text0.6900.6880.6600.6650.5900.826[0.662, 0.718]
Gold structured text0.7040.6880.6750.7370.6110.811[0.677, 0.731]
Retrieved native, all0.7080.7740.6830.5890.5960.843[0.679, 0.734]
Retrieved native, capped0.6700.7340.7120.5620.5690.752[0.642, 0.698]
Appendix
Table 29: Per-question-type diagnostic ladder on the 784 answerable items (IE n=178n{=}178 , MSR-S 112, MSR-B 112, KU 193, TR 189); refusal metrics are reported separately in Table 28 .
Metric@3@5@10@20@50
Original-question ranking (retrieved conditions)
Required-fact recall0.6000.7000.7860.8280.871
Complete-chain recall0.2820.3880.5150.5830.676
Production fixed-rewrite ranking (end-to-end)
Required-fact recall0.5740.6770.7590.8060.863
Complete-chain recall0.2650.3710.4900.5560.661
Appendix
Table 30: Required-fact and complete-chain recall over ranked units, on the 784 answerable items.
Drop reasonImageVideoAudioAll
Media-count cap5,5595933,1869,338
Context/token cap0000
Serialization/API error0000
Unsupported format0000
Total dropped artefacts5,5595933,1869,338
Appendix
Table 31: Retrieved-but-dropped artefacts by modality and recorded drop reason, under production delivery caps (media-count cap applied per modality). The end-to-end condition shows the same single-cause pattern (6,184 image, 587 video, 2,533 audio; 9,304 total).
Challenge (required)ScoreN
Image — Background detail0.670284
Image — Fine-grained attribute0.801333
Image — Spatial configuration0.714185
Video — Action ordering0.77364
Video — Action repetition count0.66656
Video — State transition0.69764
Appendix
Table 32: Per-challenge judge score for the best end-to-end system (ReAct over the native MiMo-V2.5 stack). An item may contribute to multiple rows, so the strata overlap and do not sum to 784.
FamilyMean scoreRange
Native-MM ReAct0.7030.674–0.732
Native-MM RAG0.6510.614–0.687
A-MEM (text architectures)0.6170.611–0.623
Text-only ReAct0.4170.415–0.419
Text-only RAG0.3830.375–0.390
MemoryOS (text architectures)0.2490.243–0.255
Appendix
Table 33: Method family ranking by mean answerable judge score across the twelve Track B runs (two configurations per family).
Figure 9: Mean judge score on the 784 answerable items for every Track B system, with persona-clustered bootstrap 95% intervals. Colour encodes the architecture family. The native-multimodal configurations occupy the top of the ranking, and the two MemoryOS configurations the bottom.
Figure 10: Judge score by system ×\times question type, with AR the correct-refusal rate on the 139 refusal items. Knowledge Update is the hardest answerable type for essentially every system. The AR column runs opposite to every answerable column, which is the indiscriminate-abstention pattern quantified in § 6.4 .
Figure 11: Judge score by question type: six panels, each ranking all twelve systems for that type. Colour encodes the architecture family. The answerable panels share a near-identical ordering, while the AR panel reverses it.
Figure 12: Birding-log photo from item mm_msr_selection_ac58aedd_002 . The answer-critical detail (receipt price $189.99 under the black 10x42 pair) is only readable in the image; the chat text never states the price.
More in this topic

Multimodal Memory

Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.

See all 71 papers

Explore similar work

CardsList

  1. M3^3Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

    Jun 5, 2026Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6Multimodal UnderstandingMultimodal Grounding

    The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4

  2. CUE-Mem: Benchmarking Long-Term User Memory via Implicit Cues in Multimodal Conversations

    Sep 26, 2026Yulin Hu, Yanyan Zhao, Zimo Long +6Multimodal MemoryMemory Agent Benchmarks

    Harbin Institute of Technology · Huawei Technologies Co., Ltd.

  3. According to Me: Long-Term Personalized Referential Memory QA

    Mar 2, 2026Jingbiao Mei, Jinghong Chen, Guangyu Yang +3LLM MemoryMemory Agent Benchmarks

    Department of Engineering, University of Cambridge, United Kingdom · Department of Physics, University of Cambridge, United Kingdom · Independent Researcher