Retrieved but Not Delivered: Multimodal Memory Delivery for Long-Term Agents

Yuhang Jiang; Qingwei Liao; Kaize Yin; Xingling Liu; Luca Cuomo; Silvio Bacci

cs.CVSep 26, 2026

Retrieved but Not Delivered: Multimodal Memory Delivery for Long-Term Agents

Authors: Yuhang Jiang, Qingwei Liao, Kaize Yin, Xingling Liu, Luca Cuomo, Silvio Bacci

Organizations: Huawei Technologies Ltd.

Abstract

Work on memory for multimodal agents optimizes what is written, updated and retrieved. Between retrieval and the answer, however, is a stage that multimodal memory evaluations do not isolate: what of the retrieved memory reaches the model, and in what form. We call it delivery, and a controlled decomposition on MemLens locates the remaining room there. With the retrieved evidence set exactly fixed, delivering the original pixels instead of withholding them raises accuracy by 13.87 points on an 8B backbone, whereas making retrieval perfect on those same messages improves it by 2.31. Delivery is the larger term on all three MemLens backbones and grows with backbone strength; retrieval grows too, without closing the gap. We propose DeliverMem, an instantiation of delivery as three decisions: keep the original modality, give each item a readable identity, and state when it was seen, with a retrieval-side adapter for the one property delivery cannot supply. Each is measured against a delivery-matched control that alters only its own variable. DeliverMem leads the strongest published memory agent on MemLens at all four context lengths, and beats DMV-Bench's own strongest method at every setting on both backbones. On MemLens it does this on a tenth to a seventieth of the input. Each decision helps only where the question lacks what it supplies, and is null elsewhere. A single fixed configuration nonetheless leads both benchmarks, without training any component or modifying the stored records. Project page: https://avalon-s.github.io/DeliverMem/

Figures & tables

Figure 1: Left: one MemLens question from our runs, answered with the pixels withheld and with them delivered. Right: the two quantities this paper separates, with the arrow marking the difference between them.

Retrieved set	Pixels delivered	Pixels withheld	$\Delta_{D}$	W/L, $p$
Gold messages ( $R^{\star}$ )	$43.93$	$30.06$	$+13.87$	31/7, $1.2\times 10^{-4}$
Our retriever, text channel ( $R$ )	$41.62$	$27.17$	$+14.45$	30/5, $2.2\times 10^{-5}$
$\Delta_{R}$	$+2.31$	$+2.89$	interaction $-0.58$
W/L, $p$	17/13, $0.58$	15/10, $0.42$	12/14, $0.85$

Table 1: Delivery against retrieval on MemLens 32K (

n=173

non-refusal questions, benchmark judge). Rows vary what is retrieved and columns whether the retrieved items keep their pixels, each row paired within its own run. W/L counts questions fixed and broken, and Appendix C.1 defines the conditions.

Figure 2: The memory pipeline, with the stage this paper isolates. MLLM is the answering model

M

	32K	64K	128K	256K
Best published agent at each length	32.82	31.28	31.28	30.77
DeliverMem, images withheld	35.38	35.38	32.82	30.26
DeliverMem, text-channel retrieval	48.72	44.10	44.10	38.46
DeliverMem	51.28	44.10	42.05	38.97
Margin over the best published agent	$+18.46$	$+12.82$	$+10.77$	$+8.20$
Input ratio, full context to delivered	$10.0\times$	$19.2\times$	$36.6\times$	$72.9\times$

Table 2: MemLens at all four released context lengths (benchmark judge, canonical

195

questions, Qwen3-VL-8B,

K{=}10

). “Images withheld” delivers the retrieved messages without their pixels; “text-channel retrieval” keeps the pixels and drops the image channel from retrieval. The shaded row is the one configuration deployed on both benchmarks, applied to every question; a per-type selection rule fitted on the dev split is reported separately (Appendix C.5 ).

	Qwen2.5-VL-7B				Gemini 2.5 Flash
Memory	$J{=}5$	$J{=}10$	$J{=}15$	$J{=}50$	$J{=}5$	$J{=}10$	$J{=}15$	$J{=}50$
NoMemory	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
TextOnly	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
WorldMM-lite ( Yeo et al., 2026 )	37.3	34.1	29.7	27.5	43.5	41.1	38.2	37.0
MMA-lite ( Lu et al., 2026 )	47.7	41.0	39.4	35.2	46.1	44.8	35.8	36.9
Caption	67.9	64.5	62.3	58.1	58.9	55.6	50.5	47.7

Table 3: DMV-Bench task success rate (%) on both backbones. The upper block is the benchmark’s own pool, with NoMemory, TextOnly and Caption as its baselines and the -lite rows as its re-implementations of published systems. The lower block is ours, probe-pooled. The

J{=}50

columns use the benchmark’s Monte-Carlo probe construction (Appendix E.2 ).

Component	Control alters	Benchmark	Unit ( $n$ )	$\Delta$	Agreement, $p$
(D1) original modality	pixel phase randomised	MemLens	question (173)	$+11.56$	28/8, $0.0012$
(D2) addressability	rank $\to$ tag map shuffled	DMV	chain (10)	$+2.63$	8 of 10, $0.072$
(D2) addressability ∗	tags made identical	DMV	chain (10)	$+11.85$	10 of 10, $0.002$
(D2) addressability	tag removed entirely	DMV	chain (10)	$+9.88$	10 of 10, $0.002$
(D2) addressability	tag in text, pixels intact	DMV	chain (10)	$+7.89$	9 of 10, $0.0039$
(D2) addressability ∗	tags made identical	MemLens	question (195)	$+0.51$	4/3, $1.00$

Table 4: Each delivery component against a delivery-matched control, and (R1) inside a

2\times 2

. Agreement is wins/losses over discordant questions on MemLens and chains agreeing in sign on DMV-Bench. MemLens rows are at 32K, the (D3) row on the temporal questions of all

789

released; DMV rows use Qwen2.5-VL-7B at

J{=}10

, (R1) at

J{=}50

. The first three DMV rows nest, each control removing more than the one above. ∗ Against a constant tag rather than no tag.

Component	DMV-Bench	MemLens
Delivery
(D1) original modality	—	positive $+11.56$
(D2) addressability	positive $+3.96$	null
(D3) temporal position	null	positive $+8.76^{\dagger}$
Binding (not deployed)	—	positive $+6.36$
Retrieval

Table 5: Component by benchmark, with the contrast behind each entry given in the text. On DMV-Bench, (D2) and (R1) are main effects in the

2\times 2

design with Qwen2.5-VL-7B at

J{=}50

, whose delivery condition moves the tag and the ordinal together, and (D2) is null with Gemini 2.5 Flash;

\dagger

marks (D3) against no date, on temporal-reasoning questions. A dash marks a component the benchmark cannot separate or did not measure.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	MemLens	DMV-Bench
Store	every conversation message with its images, $32$ K to $256$ K tokens per conversation	every product page visited over $J$ sessions, in DualMem’s image and text banks
Text channel	bge-small-en-v1.5 ( Xiao et al., 2024 ) on the message text	all-MiniLM-L6-v2 ( Reimers & Gurevych, 2019 ) on Gemini 2.5 Flash captions
Image channel (R1)	siglip2-base-patch16-384 , whole image plus $2\times 2$ overlapping tiles, scored by MaxSim	the same encoder and tiles, in place of DualMem’s whole-image vector
Fusion	image scores min-max normalised within the question and averaged with the text score; a message without an image scores $0.5$ on the image channel	DualMem’s weighted fusion at $\alpha{=}0.75$
Items delivered	$K{=}10$ messages	the five items the benchmark specifies
Delivery (D1–D3)	original images (D1), a tag in the pixels (D2) and the message date (D3)	a tag in the pixels (D2) and [seen # $k$ of $N$ ] (D3); DualMem already delivers images

Appendix

Table 6: Where DeliverMem acts in each benchmark, in pipeline order. Shaded rows are the stages DeliverMem changes.

	all	non-AR
System	(195)	(173)	IE	MSR	TR	KU	AR
M2A ( Feng et al., 2026 ) (same backbone)	15.38	14.45	14.75	8.57	2.08	0.00	22.73
M3C ( Jang et al., 2025 )	18.46	18.50	8.20	25.71	31.25	10.34	18.18
M3-Agent ( Long et al., 2025 )	19.49	20.23	18.03	22.86	29.17	6.90	13.64
Memory-T1 ( Du et al., 2025 )	28.72	31.21	18.03	25.71	62.50	13.79	9.09
MemOS ( Li et al., 2025 )	30.26	25.44	18.03	22.86	39.58	24.14	68.18

Appendix

Table 7: MemLens at 32K by question type (canonical

195

questions, benchmark judge), with every published memory agent in the benchmark’s pool. IE, MSR, TR, KU and AR are information extraction, multi-session reasoning, temporal reasoning, knowledge update and answer refusal. Published rows are transcribed as reported, and the shaded row is the configuration of Table 2 .

	32K	64K	128K	256K
Qwen3.5-122B ( Qwen Team, 2026 )	63.59	59.49	49.23	n/a
Qwen3-VL-30B-A3B ( Bai et al., 2025a )	58.46	52.31	50.77	n/a
Gemini-3.1-Pro ( Google DeepMind, 2026 )	55.38	58.46	58.46	n/a
GPT-5.4 ( OpenAI, 2026 )	51.28	56.92	51.79	n/a
Qwen3-VL-8B ( Bai et al., 2025a ) (our backbone)	50.77	47.69	34.36	n/a
Qwen3-VL-2B ( Bai et al., 2025a )	38.46	35.38	27.69	n/a

Appendix

Table 8: Direct LVLMs reading the entire MemLens context, as reported by the benchmark ( Ren et al., 2026 ) , with our own row from Table 2 . Bold and underline mark the best and second-best value in each of the first three columns.

Backbone	Setting	DeliverMem	$n_{r}$	Published	$\Delta$
Qwen2.5-VL-7B	$J{=}5$	88.22	433	81.2	$+7.02$
	$J{=}10$	89.67	1,888	77.8	$+11.87$
	$J{=}15$	88.87	4,359	74.6	$+14.27$
	$J{=}50$ (exh.)	85.19	48,701	68.3	$+16.89$
	$J{=}50$ (MC)	84.96	4,820	68.3	$+16.66$
Gemini 2.5 Flash	$J{=}5$	93.29	686	82.7	$+10.59$

Appendix

Table 9: DeliverMem against the numbers DMV-Bench reports for DualMem ( Tang et al., 2026 ) , the strongest system in its published pool. Scores are probe-pooled task success rate (%),

n_{r}

is the number of probes pooled, and

\Delta

is DeliverMem minus published. The settings marked exh. and MC use the benchmark’s exhaustive and Monte-Carlo probe constructions.

	text says nothing	text names the tag
pixels unstamped	$79.67$	$82.75$
pixels stamped	$81.00$	$90.64$

Appendix

Table 10: Where the identity lives, on DMV-Bench at

J{=}10

(task success rate %, seed-averaged over ten chains). Columns are whether the delivered text names the tag, rows whether the pixels carry it.

Figure 3: One retrieved MemLens memory delivered eleven ways. The top block is its accompanying text, with the (D3) timestamp in brackets and the (D2) tag in the pixels. Each row group is a separate experiment with its base condition named at the right, and tiles compare only within a group.

Condition	What reaches the model	Answer
cap10	ten retrieved over the text channel, with pixels	“To the right…”
img10	ten retrieved over both channels, with pixels	“To the right…”
oracle10	the gold messages padded to ten, with pixels	“To the right…”
goldonly	the gold messages alone, no distractors	“To the right…”
oracle10_noimg	the gold messages padded to ten, pixels withheld	“On top…”

Appendix

Table 11: One MemLens question through five conditions (32K, benchmark judge), with retrieval improving down the first four rows.

Condition	What the delivered pixels carry	$\Delta$	W/L	$p$
oracle10_scramble	objects legible, positions permuted	$+6.94$	17/5	$0.017$
oracle10_phasescram	same statistics, no objects	$+11.56$	28/8	$0.0012$
oracle10_decoy	another question’s photograph	$+15.03$	33/7	$4.2\times 10^{-5}$
oracle10_noimg	nothing	$+15.03$	34/8	$6.9\times 10^{-5}$
Between counterfactual images
decoy $-$ noimg	a real photograph versus none	$+0.00$	6/6	$1.00$

Appendix

Table 12: The counterfactual-image ladder on MemLens (32K, benchmark judge,

n=173

non-refusal questions, five conditions run and paired within one batch).

\Delta

is oracle10 minus the condition, so a larger

\Delta

means a larger loss. The shaded row, the phase-randomised image, keeps the most of the original; Section 4 reports it.

Question type	$n$	$\Delta$	W/L	$p$
Information extraction	61	$+40.98$	28/3	$4.6\times 10^{-6}$
Multi-session reasoning	35	$+2.86$	2/1	$1.00$
Temporal reasoning	48	$+0.00$	1/1	$1.00$
Knowledge update	29	$+0.00$	3/3	$1.00$
Non-refusal (pooled)	173	$+15.03$	34/8	$6.9\times 10^{-5}$

Appendix

Table 13: Delivering the correct pixels rather than none, by question type (MemLens 32K, benchmark judge, the batch of Table 12 , paired per question).

Subset	$n$	$\Delta_{R}$	W/L	$\Delta_{D}$	W/L
Information extraction	$61$	$-1.64$	5/6	$+40.98$	27/2
Multi-session reasoning	$35$	$+5.71$	2/0	$-2.86$	1/2
Temporal reasoning	$48$	$+0.00$	4/4	$+2.08$	1/0
Knowledge update	$29$	$+10.34$	6/3	$-3.45$	2/3
Gold messages $5$ – $9$	$133$	$+4.51$	15/9	$+18.80$	31/6
Gold messages $\geq 10$	$38$	$-2.63$	2/3	$-2.63$	0/1

Appendix

Table 14: The two quantities of Table 1 by question type and by number of gold messages (MemLens 32K, its

173

non-refusal questions, paired within one run).

\Delta_{R}

is the gold messages minus our retriever, both with pixels, and

\Delta_{D}

is the gold messages with pixels minus without. The last row reproduces Table 1 .

Backbone	Quantity	$\Delta$	W/L	$p$
$8$ B dense	Delivery	$+13.87$	31/7	$1.2\times 10^{-4}$
	Retrieval	$+2.31$	17/13	$0.58$
30B-A3B	Delivery	$+15.61$	36/9	$6.6\times 10^{-5}$
(MoE)	Retrieval	$+9.83$	21/4	$9.1\times 10^{-4}$
$32$ B dense	Delivery	$+17.92$	32/1	$7.9\times 10^{-9}$
	Retrieval	$+11.56$	26/6	$5.4\times 10^{-4}$

Appendix

Table 15: The two quantities of Section 4 on three Qwen3-VL backbones (MemLens 32K,

n=173

non-refusal questions, benchmark judge, paired within run). Delivery is the gold messages with pixels minus without, and retrieval is the gold messages minus our retriever, both with pixels. Backbones are ordered by total parameters.

Backbone	Setting	Same-batch	DeliverMem	$\Delta$	same dir.	$p$
Qwen2.5-VL-7B	$J{=}5$	79.88	87.51	$+7.63$	8/10	$0.020$
	$J{=}10$	77.80	89.55	$+11.75$	10/10	$0.002$
	$J{=}15$	76.59	88.94	$+12.34$	10/10	$0.002$
	$J{=}50$ (exh.)	72.34	85.20	$+12.86$	10/10	$0.002$
	$J{=}50$ (MC)	72.17	84.96	$+12.79$	10/10	$0.002$
Gemini 2.5 Flash	$J{=}5$	80.68	93.26	$+12.58$	6/6	$0.031$

Appendix

Table 16: Same-batch paired comparison against our own run of DualMem at its strongest setting. Estimates are seed-averaged, and

p

is an exact cluster permutation whose floor is

0.002

k{=}10

chains and

0.031

k{=}6

Retrieval	delivery off	delivery on
DualMem	$65.99$	$66.29$
(R1) region multi-vector	$87.99$	$\mathbf{88.61}$

Appendix

Table 17: The

2\times 2

J{=}50

on Gemini 2.5 Flash. Task success rate (%) over the probes common to all four cells. The delivery column adds the tag and the ordinal together.

Figure 4: Paired difference against the same-batch DualMem run as memory load grows, for Qwen2.5-VL-7B (

k{=}10

chains, left) and Gemini 2.5 Flash (

k{=}6

, right). Bands are

95\%

t

intervals across chains. At

J{=}50

Qwen’s point is the exhaustive construction with the Monte-Carlo value ringed, and Gemini was run under Monte-Carlo only.

	32K	64K	128K	256K
Our delivered input (tokens)	2,405	2,421	2,460	2,450
Full context (tokens)	24,001	46,476	89,940	178,558
Input ratio	$10.0\times$	$19.2\times$	$36.6\times$	$72.9\times$
DeliverMem	51.28	44.10	42.05	38.97
Direct inference, same backbone	50.77	47.69	34.36	n/a
Accuracy difference	$+0.51$	$-3.59$	$+7.69$	n/a

Appendix

Table 18: Input cost and accuracy against direct inference with the same backbone. “Input ratio” is median full-context tokens over median delivered tokens. The benchmark runs direct LVLMs at three lengths only, since many of them do not reach

256

K, so the last column has no direct-inference reference for any model ( Ren et al., 2026 ) .

Figure 5: (R1) on DMV-Bench’s cued frames. Top: one frame with its inserted cue, the overlapping tiles and the tile MaxSim scores highest. Bottom: offline Recall@1 at four granularities with one encoder, the deployed setting in the bottom row.

Question type	$n$	Full	Binding shuffled	$\Delta$	W/L
Information extraction	61	59.02	50.82	$+8.20$	6/1
Multi-session reasoning	35	22.86	20.00	$+2.86$	2/1
Temporal reasoning	48	50.00	43.75	$+6.25$	3/0
Knowledge update	29	37.93	31.03	$+6.90$	3/1
Answer refusal	22	100.00	100.00	$+0.00$	0/0
Non-refusal (pooled)	173	45.66	39.31	$+6.36$	14/3

Appendix

Table 19: The binding control by MemLens question type (32K, benchmark judge). The control permutes only which image follows which message (Appendix C.1 ), and

\Delta

is paired within one run.

Selection	32K	128K
Query-conditioned (ours)	51.28	42.05
Recency, no query	22.05	14.36
Random, no query	17.44	14.87

Appendix

Table 20: Selection without the query (MemLens judge score, canonical

195

questions). All rows deliver ten messages through one pipeline and differ only in which ten are selected.

Cell	probes/seed	obs. SD	binomial SD	residual
Qwen $J{=}5$	43	8.01	6.09	5.20
Qwen $J{=}10$	189	6.31	3.02	5.54
Qwen $J{=}15$	436	5.21	2.03	4.80
Qwen $J{=}50$	4870	3.15	0.64	3.09

Appendix

Table 21: Per-seed variance on DMV-Bench with Qwen2.5-VL-7B, split into a binomial term and a residual.

Figure 6: The first of the three, with the delivered set shown without its stamps. Five items reach the model and the target is second. The box marks the cue, located by differencing the frame against its uncued original.

Multimodal Memory

Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.

See all 71 papers

MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents
Sep 7, 2026Xian Gao, Jinpeng Wang, Jiacheng Ruan +3Multimodal Memory Long-Horizon LLM Agents
Shanghai Jiao Tong University · Tsinghua University · Institute of Automation of the Chinese Academy of Sciences
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Agent Evaluation Multimodal Memory
University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3
V-Mem: Modality-Routed Retrieval for Long-Term Multimodal Agentic Memory
Aug 2, 2026Dingyi Kang, Dongming Jiang, Yi Li +2Multimodal IR Multimodal Memory
Department of Computer Science, The University of Texas at Dallas · Department of Electrical and Computer Engineering, University of Florida

Sep 7, 2026J/K move · Enter open · S save

MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents

Xian Gao, Jinpeng Wang, Jiacheng Ruan, Guangyu Cao+2

Shanghai Jiao Tong University · Tsinghua University · Institute of Automation of the Chinese Academy of Sciences

Multimodal Memory Long-Horizon LLM Agents Agent Memory Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Long-running LLM agents rely on external memory to store and reuse information beyond a single context window, yet there is a fundamental tension between the continuous accumulation of interaction trajectories and the limited context capacity. The key challenge in agent memory is therefore not only to retrieve relevant records, but also to select necessary evidence under a given budget and organize it in an appropriate modality. Existing memory readout methods mainly use textual or visual forms. Text preserves high fidelity, but its linear token representation makes contents with different importance compete for the limited context at nearly uniform unit cost. Visual readout renders text into document-like images, which can use two-dimensional layouts to expose structure and emphasize key information, but it may lose fine-grained details during rendering and compression. To address this issue, we propose MEMO, a multimodal evidence memory organization method for LLM agents. MEMO first uses a trained evidence extractor to select relevant memory blocks and form evidence units with source information and presentation requirements. A trained query-conditioned memory manager assigns each unit to a textual, visual, or dual-channel carrier and selects a layout that matches the evidence structure. A deterministic memory construction module then generates the textual package and visual pages. The memory manager is trained with feedback from an offline reader that measures the utility of the guided memory plan, so that retention and presentation decisions align with downstream usage. We evaluate MEMO on four benchmarks, HotpotQA, 2WikiMultiHopQA, LoCoMo, and ALFWorld, with multiple reader backends. The results show that MEMO presents memory more efficiently with fewer memory tokens, improves downstream task performance, and builds more effective working memory under constrained budgets.

Abstract

Multimodal Memory

Explore similar work

MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

V-Mem: Modality-Routed Retrieval for Long-Term Multimodal Agentic Memory