MemTransfer: Benchmarking Memory Beyond Matched Experience in Embodied Decision-Making

Haiming Tang; Xianjie Dai; Gujie Shao; Zuyi Guo; Jingguang Li; Kailang Ma; Yihong Tang; Heye Huang

cs.ROSep 26, 2026

MemTransfer: Benchmarking Memory Beyond Matched Experience in Embodied Decision-Making

Authors: Haiming Tang, Xianjie Dai, Gujie Shao, Zuyi Guo, Jingguang Li, Kailang Ma, Yihong Tang, Heye Huang

Organizations: National University of Singapore · Korea Advanced Institute of Science and Technology · McGill University

Abstract

Memory lets an embodied agent reuse past experience, yet retaining useful information does not ensure that the agent can apply it when conditions change. We present MemTransfer, a benchmark comparing six memory representations, a working-memory baseline and five representations of past experience, under a shared frozen vision-language-model policy. It comprises 100 navigation cases across ten task types in a simulated warehouse, with expert demonstrations supplying the history. Three comparisons vary the starting pose, route availability, and amount and task relevance of history. With one demonstration per task, Full-context and Episodic memory reach 95.3% and 100.0% success at the original demonstration start, but lose 48-49 percentage points at a new test start. Summary changes little between these two test starts, yet with four demonstrations per task it retains a smaller fraction of its unchanged-route success after blocking (39.3%) than Working memory (44.8%) or the two trajectory memories (56-58%). At the new test start, increasing from one to four relevant demonstrations raises Episodic success by 14.3 percentage points, while the other evaluated representations gain no more than 1.3 percentage points. Replacing half of the relevant histories with other-task experience lowers success for both trajectory memories. These results show that robustness to one kind of mismatch does not imply robustness to another, motivating evaluation of both stored information and its use at decision time.

Figures & tables

Fig. 1 : Three questions about reusing past experience. Left: a new starting pose (RQ1), a blocked route (RQ2), and multiple histories including experience from other tasks (RQ3). Right: a shared policy and working-memory window, with five representations of past experience.

Fig. 2 : Overview. (A) The shared VLM policy receives the task, observation, pose, action feedback, recent interactions and historical memory. (B) Warehouse layout, start poses, target instances and route obstructions. (C) Demonstration starting poses and the six memory representations, with examples of their contents and input sizes. Section III-B defines the comparisons used for each RQ.

Basic search
Type	Task and route	Task requirement
1	Single object search, $O\to A$	Reach one instance of $A$
2	Category object search, $O\to C$	Reach one item in $C$
Multi-object search
3	Exhaustive single object search, $O\to A_{1}\to\dots\to A_{k}$	Visit every instance of $A$
4	Exhaustive category search, $O\to C_{1}\to\dots\to C_{m}$	Visit every item in $C$

TABLE I : Navigation tasks used in the three RQs.

Representation	Memory organization	Construction	Decision-time access
Working memory	Ordered recent interactions	Keep the last $h$ steps	Entire recent window
Full-context	Chronological actions and keyframes	Retain actions; select keyframes	All retained actions and keyframes
Summary	Object locations, landmarks and routes	Sequential summarization	Entire summary document
Semantic	Generalized spatial and navigation knowledge	Extract, consolidate and refine knowledge from demonstrations	Entire knowledge document
Hybrid	Summary and Semantic documents	Concatenate the two documents	Both documents
Episodic	Frames linked to expert continuations	Encode frames; retain trajectories	View-matched local continuation

TABLE II : Construction and use of the six memory representations. Each historical representation supplements the working-memory window used by the baseline.

Test start			Original		New		New		New
Route condition			Unchanged		Unchanged		Unchanged		Blocked
Demonstrations per task			1		1		4		4
Representation	Img	Mtok	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Working memory	$7$	$34$	$15.2\pm 3.0$	$0.023$	$22.3\pm 2.9$	$0.051$	$22.3\pm 2.9$	$0.051$	$10.0\pm 2.6$	$0.025$
Full-context	$47$	$96$ / $138$	$95.3\pm 0.6$	$0.792$	$46.7\pm 2.5$	$0.157$	$48.0\pm 2.6$	$0.167$	$28.0\pm 3.5$	$0.067$
Summary	$7$	$163$ / $588$	$43.0\pm 2.7$	$0.136$	$44.3\pm 3.8$	$0.149$	$44.0\pm 2.6$	$0.173$	$17.3\pm 3.1$	$0.042$

TABLE III : Memory performance across the three RQs.

Test: original start; memory: one demonstration per task
Representation	Original start	Opposite heading	Nearby start	Distant start
Full-context	$95.3\pm 0.6$	$83.0\pm 3.0$	$77.7\pm 3.1$	$33.7\pm 4.7$
	$0.792$	$0.487$	$0.487$	$0.076$
Episodic	$\mathbf{100.0\pm 0.0}$	$\mathbf{96.3\pm 3.1}$	$\mathbf{90.0\pm 1.7}$	$43.0\pm 2.6$
	$\mathbf{0.849}$	$\mathbf{0.701}$	$\mathbf{0.605}$	$0.117$
Summary	$43.0\pm 2.7$	$51.7\pm 5.1$	$46.7\pm 1.5$	$\mathbf{45.7\pm 4.5}$

TABLE IV : RQ1: Changing starting pose.

Representation	$\Delta$	reten.	Follow [%]	Recovery [%]
Working memory	$0$	$0.45$	$64.7\pm 3.1$	$11.8\pm 4.8$
Full-context	$-7.7$	$\mathbf{0.58}$	$71.0\pm 1.0$	$\mathbf{30.5\pm 2.9}$
Summary	$-14.3$	$\mathbf{0.39}$	$\mathbf{80.3\pm 0.6}$	$\mathbf{15.3\pm 3.8}$
Semantic	$-0.3$	$0.25$	$59.0\pm 2.0$	$6.3\pm 2.7$
Hybrid	—	—	—	—
Episodic	$-16.7$	$0.56$	$63.3\pm 2.1$	$29.6\pm 4.7$

TABLE V : RQ2: Responses to a blocked route.

Fig. 3 : RQ3: Success under mixed histories. New start, unchanged route, and four demonstrations per task; mixed conditions replace two with other-task histories (Sec. III-B ). Mean SR

\pm

std over three 100-case runs.

Representation	Basic	Multi-object	Loop	T9
Working memory	$53$ / $30$	$22$ / $9$	$10$ / $1$	$0$ / $0$
Full-context	$85$ / $72$	$60$ / $28$	$18$ / $7$	$17$ / $3$
Summary	$72$ / $35$	$55$ / $25$	$19$ / $1$	$20$ / $0$
Semantic	$38$ / $10$	$18$ / $4$	$8$ / $1$	$0$ / $3$
Episodic	$\mathbf{93}$ / $\mathbf{67}$	$\mathbf{78}$ / $\mathbf{45}$	$\mathbf{22}$ / $\mathbf{11}$	$\mathbf{93}$ / $\mathbf{23}$

TABLE VI : Task completion across navigation requirements.

Memory Agent Benchmarks

Memory agent benchmarks evaluate how AI agents store, retrieve, update, and use information across interactions and changing tasks.

See all 162 papers

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks
Sep 23, 2026Lizhou Liang, Xinyu Zhong, Miao Pan +7Embodied Agent Benchmarks Long-Term Memory for Embodied Agents
Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences
What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents
Oct 8, 2026Junxiang Wang, Yuchen Liu, Yi Wu +6Long-Term Memory for Embodied Agents Agent Memory
Dexmate · Carnegie Mellon University · Purdue University
When Successful Memories Mislead Embodied Agents:Memory Adaption For Task-Conditioned Execution
Sep 20, 2026Quanquan Li, Hongbo Zhang, Yihe Chi +5Agent Memory Memory-Augmented Language Model Agents
East China Normal University, Shanghai, China · Peking University, Beijing, China · University of Science and Technology of China, Hefei, China

Sep 23, 2026J/K move · Enter open · S save

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

Lizhou Liang, Xinyu Zhong, Miao Pan, Xiaohe Zhou+6

Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences

Embodied Agent Benchmarks Long-Term Memory for Embodied Agents Long-Horizon Agent Tasks Embodied AI Memory-Augmented Agents

Open paper AskPDF arXiv

Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter change. Yet current agents struggle to maintain such memory reliably. Our analysis traces this limitation to four key deficiencies: weak fine-grained visual memory, unreliable dynamic world-state tracking, failing to record world state revealed by interaction outcomes, and limited generalization from prior experience. However, existing benchmarks do not directly assess these memory capabilities during long-horizon embodied interaction. To address this gap, we introduce EmbodiedMemory-Bench (EMem-Bench), comprising 2,554 interactive episodes across four task families. EMem-Bench requires agents to build and update memory from interaction history, then use it to complete a later task by acting in the environment. We further present Embodied-Memorizer (EMem), an external memory system that organizes embodied experience into spatial, event, and scene memories. We also train EMem-8B, an 8B policy that manages and uses these memories. We evaluate a diverse range of open-source and proprietary MLLMs and representative multimodal memory systems. Results show that current models remain weak and uneven across the four challenges. Under matched backbones, EMem achieves the best overall performance among the evaluated memory systems and improves both open-source and proprietary models, while EMem-8B further improves over its backbone. Project page: https://zju-omniai.github.io/EmbodiedMemoryBench/

Abstract

Memory Agent Benchmarks

Explore similar work

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents

When Successful Memories Mislead Embodied Agents:Memory Adaption For Task-Conditioned Execution