ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control

Yize Liu; Ke Wang; Mac Schwager; Yiqing Xu; Jiajun Wu

cs.ROSep 30, 2026

ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control

Authors: Yize Liu, Ke Wang, Mac Schwager, Yiqing Xu, Jiajun Wu

Organizations: Stanford University

Abstract

A robot may lose sight of an object it must later retrieve, need to recall what a person demonstrated earlier, or track which steps of a task it has already completed. Current vision-language-action (VLA) policies often fail once the information needed for action disappears from the current observation, making memory critical for long-horizon robot behavior. Existing approaches typically provide longer histories or learn implicit memory from observation-action trajectories. But action supervision tells a policy how to act, not what to remember: it does not specify which past facts should persist or how they should change as new evidence arrives. We therefore separate maintaining an evidence-grounded account of the past from learning how to act on it. This insight motivates Explicit Concept Memory (ECoMEM), which represents task-relevant history with a reusable library of grounded concepts. An evidence-based Writer selects and updates these records, while a learned Reader turns them into memory tokens that directly condition the VLA. Across 16 RoboMME tasks, ECoMEM leads the evaluated robot policies on 15 tasks. On two new real-robot tasks, the same memory library either transfers directly or requires only one new concept, achieving 86.1% success versus 8.6% for a no-memory VLA. These results show that explicit concepts provide a reusable and extensible memory interface for robot control. Project website: https://ecomem.github.io/

Figures & tables

Figure 2: ECoMEM separates memory construction from memory use. The Writer selects task-relevant concepts, recognizes them from visual and robot-state evidence, and maintains structured records. The learned Reader encodes these records as memory tokens that condition the VLA alongside image and language tokens.

Figure 3: A shared concept library supports diverse memory tasks. Columns show concepts grouped into four semantic families, and rows show the 16 RoboMME tasks and two real-robot tasks. Filled cells indicate the concepts used by each task, revealing extensive reuse of shared concepts across otherwise different memory demands.

Method	Counting				Permanence				Reference				Imitation				AVG
	Bin Fill	Pick Xtimes	Swing Xtimes	Stop Cube	Video Umsk	Button Umsk	Video UmskS	Button UmskS	Pick HighL	Video Repick	Video PicBtn	Video PicOrd	Move Cube	Insert Peg	Pattern Lock	Route Stick
Human Performance	96.00	100.00	80.00	78.00	90.00	92.00	92.00	90.00	92.00	92.00	98.00	90.00	90.00	98.00	84.00	86.00	90.50
FrameSamp+Modul	39.56	87.33	92.00	42.00	32.67	25.11	24.44	18.22	22.89	30.44	60.00	32.00	77.78	7.56	53.56	66.67	44.51
$\pi_{0.5}$	30.00	42.89	35.56	6.67	20.44	22.22	18.67	6.67	11.33	0.44	31.11	25.78	26.00	1.56	2.89	4.67	17.93
$\pi_{0.5}$ w/ past actions	26.67	58.33	26.67	4.67	30.67	23.67	20.67	16.00	12.33	8.67	24.00	18.67	34.00	1.00	4.00	5.67	19.73
SAM2Act+	40.00	76.00	25.33	0.00	27.33	32.00	18.00	26.67	17.33	5.33	24.67	20.00	29.33	0.00	0.00	0.00	21.37

Table 1: Success rates (%) on the 16 RoboMME memory-dependent tasks. AVG is the task-equal mean across all tasks. Baseline and human results are reported by RoboMME ( Dai et al., 2026 ) .

Family	Concept extracted	Structured	QA	Concept Memory
Entity & Spatial	Picked-object identity	71.4%	71.4%	100.0%
Entity & Spatial	Highlighted-object set	63.2%	63.2%	97.4%
State & Relation	Currently highlighted	100.0%	97.4%	97.4%
State & Relation	Currently holding queried object	100.0%	96.7%	100.0%
Event & Progress	Pick–place cycle count	86.7%	93.3%	100.0%
Event & Progress	Region-entry count	95.0%	90.0%	95.0%

Table 2: Concept extraction across all four memory families. We evaluate eight whole-video concepts on 95 RoboMME test videos. Structured and QA are separately fine-tuned Qwen3-VL-8B-Instruct models; Concept Memory uses our evidence-based Writer. Cell shading encodes extraction accuracy (darker is higher); bold denotes the best result in each row. See Appendix H for protocols and limitations.

	ScoopPour			CupSwitch		Pooled
Method	$1\times$	$2\times$	$3\times$	$2\times$	$3\times$	succ./trials	rate
ECoMEM	16/20	19/23	11/12	11/12	11/12	68/79	86.1%
$\pi_{0.5}$ (no memory)	4/12	0/12	0/11	0/12	1/11	5/58	8.6%

Table 3: Real-robot success (successes/trials) by requested number of cycles; success requires exactly that count. Pooled uses the targets both policies were run on.

Figure 7: Concept memory transfers through reuse and compositional extension on a real robot. Blue : reused base concepts; orange : the new SCOOP concept. Both tasks require counting, but ECoMEM represents progress at the task-relevant granularity: CupSwitch advances after each completed round trip, while ScoopPour composes SCOOP with the same counter so that only successful scoops carrying beans advance progress. An empty scoop therefore leaves the count unchanged and automatically triggers a retry. Records are produced online by the Writer.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Concept / arguments	Meaning and representation
Entity & Spatial Grounding
Entity identity and language binding	Episode-local entity keys link observations, relations, and events. Groundings retain instruction phrases and aliases; the policy interface encodes entity slots rather than alias strings.
role:cube , role:container , role:object , role:region , role:robot	The five grounding-role codes in the base policy vocabulary. Each grounding records an entity key, role, aliases, confidence, and source.
Entity location and endpoint anchor	Grounded entity or tool-endpoint locations are supplied to the Reader as row and column cells on a $32\times 32$ image grid. Cell indices are 1-based; zero denotes an unspecified position. Selected goal records also carry the subject’s target cell.
occupies_region (entity, region)	Current occupancy of a grounded region. Entry and exit are represented by separate transition events.
side:left , side:right	Entity-bound categorical insertion-side predicates, distinct from the demonstrated endpoint’s spatial anchor.

Appendix

Table 4: Semantic inventory of the base Concept Memory interface, grouped into four families. Monospaced names are record codes; italicized entries are fields or derived structures. Predicate arguments name the subject and, where applicable, the object. Gripper identity is stored separately as agent metadata.

Figure 8: Concept recognition and memory update in ECoMEM. A pick-and-place interaction illustrates how language-grounded entities and robot-state evidence jointly establish an event. The writer merges concept evidence, updates grounded records, and advances progress once per distinct completion.

Field	Encoding
Kind	Record type: belief, event, goal, grounding, or geometry; zero denotes padding.
Predicate	Shared identifiers for predicates, transition events, entity roles, and categorical attributes.
Subject, object	Episode-local entity identifiers; zero denotes an unspecified argument.
Truth	True, false, or unknown; zero denotes an inapplicable value.
Status	Tentative, confirmed, or stale; zero denotes an inapplicable value.
Confidence	$\operatorname{round}(9p)$ for confidence $p\in[0,1]$ .

Appendix

Table 5: Structured fields supplied to the memory Reader. Subject/object and row/column each denote two fields.

Task	Median C	Median A	A/C	Mean C	Mean A
VideoUnmask	2.23	40.47	$18.1\times$	3.87	41.41
VideoRepick	4.38	52.66	$12.0\times$	5.51	56.61

Appendix

Table 6: Memory initialization on matched episodes using the same pair of RTX 3090 GPUs. Times are in seconds; each condition uses five episodes. C denotes task-conditioned selection and A denotes No Selection.

Task	Matches C	Matches A	Successes C	Successes A
VideoUnmask	98/100	20/100	43/50	12/50
PatternLock	86/98	0/98	36/50	1/50

Appendix

Table 7: Language-grounding results. C uses specific entity queries; A uses generic role-level queries. Matches denote agreement with pipeline-generated reference records for the target cell (VideoUnmask) or complete ordered sequence (PatternLock). Successes denote closed-loop task completion.

		Concept Memory		No-memory $\pi_{0.5}$
Task	Target	Success / Trials	Rate (%)	Success / Trials	Rate (%)	Gain (pp)
ScoopPour	$1\times$	16 / 20	80.00	4 / 12	33.33	+46.67
	$2\times$	19 / 23	82.61	0 / 12	0.00	+82.61
	$3\times$	11 / 12	91.67	0 / 11	0.00	+91.67
CupSwitch	$2\times$	11 / 12	91.67	0 / 12	0.00	+91.67
	$3\times$	11 / 12	91.67	1 / 11	9.09	+82.58

Appendix

Table 8: Real-robot success rates by task and target count. Targets count scoop-and-pour cycles for ScoopPour and complete round trips for CupSwitch. Gain is measured in percentage points (pp).

			No-memory $\pi_{0.5}$ outcomes
Task	Target	Concept Memory failures	Recorded cycle counts	Other outcomes
ScoopPour	$1\times$	4 failures	1 (success): 4; 2: 2; 6: 1	Too many cycles: 2; timeout: 1; freeze: 1; miss: 1
	$2\times$	4 failures	0: 2; 1: 6; 3: 1; 4: 1	Timeout: 2
	$3\times$	1 failure	2: 2; 4: 2	Timeout: 7
CupSwitch	$2\times$	1 timeout	0: 6; prolonged 0-cycle execution: 1	Timeout: 4; miss: 1
	$3\times$	1 timeout	0: 1; 1: 2	Too many cycles: 2; timeout: 3; miss: 1; 3 cycles then timeout: 1; success: 1

Appendix

Table 9: Manual real-robot outcome annotations. Numeric entries give the number of trials. Baseline outcomes include successes and failures.

Explore similar work

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs
Sep 28, 2026Tanguy Dieudonné, Jack B. Jedlicki, Heng YangRobotic Manipulation Persistent State
Harvard University · ETH Zürich
MemBodied: Recurrent Associative Memory for Vision-Language-Action Models
Sep 23, 2026Tej Deep Pala, Navonil Majumder, Bryce Goh +4Generalizable Vision-Language-Action Policies Embodied
Nanyang Technological University · Griffin Labs · École Centrale de Lyon
Simple Agentic Memory for Generalist Robot Policies
Sep 29, 2026Yuyou Zhang, Yunbei Zhang, Miao Li +4Agentic Memory Robot Policies
Carnegie Mellon University · Tulane University · New York University +2

Sep 28, 2026, cs.ROJ/K move · Enter open · S save

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

Tanguy Dieudonné, Jack B. Jedlicki, Heng Yang

Harvard University · ETH Zürich

Robotic Manipulation Persistent State Ledger Objects Frame Rate

Open paper AskPDF arXiv

Memory is essential for long-horizon, partially observed robotic manipulation: a robot must remember which object was placed in a drawer, whose cup it moved, or how many action cycles have elapsed. Recent vision-language-action (VLA) models embed memory directly inside the policy, but benchmarks show no single in-policy mechanism covers all spatio-temporal dimensions, trailing oracle methods by a wide margin. We argue that memory type dictates where memory should reside: short-term perceptual memory (repetition, timing, retracing) belongs inside the policy, while long-term object memory (persistent spatial state, containment, event history) belongs outside as an explicit, readable record. We present Ledger, a harness that realizes this split over a single fine-tuned

π_{0.5}

policy by pairing an in-policy frame-sampling memory with an external spatio-temporal object memory, the ledger, built from a SAM3 tracker and a VLM captioner of the demonstration and read by an LLM planner that decides at step boundaries. On RoboMME, Ledger reaches the highest four-suite average among the evaluated methods, 64.3% (vs. 45.9% for the strongest prior method under identical evaluation), leading object reference (60.7% vs. 40.3%) and object permanence (86.7% vs. 56.2%) using a single set of weights. Choosing the memory source at runtime, from the instruction and the record, removes the need for a task-level router.

Abstract

Explore similar work

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

MemBodied: Recurrent Associative Memory for Vision-Language-Action Models

Simple Agentic Memory for Generalist Robot Policies