Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulation

Xinying Guo; Chenxi Jiang; Hyun Bin Kim; Yuhang Han; Ying Sun; Yang Xiao; Jianfei Yang

cs.ROMar 25, 2026

Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulation

Authors: Xinying Guo, Chenxi Jiang, Hyun Bin Kim, Yuhang Han, Ying Sun, Yang Xiao, Jianfei Yang

Organizations: MARS Lab, Nanyang Technological University · Institute for Infocomm Research, A*STAR · National University of Singapore

Abstract

Robots often observe information that determines a future action long before that action is executed. In a shell game, for example, a robot first sees which cup hides the ball, watches the cups move, and only later needs to choose the correct cup. The final observation alone is not enough for a decision: the correct action depends on an earlier event. We refer to this temporal gap as observation-action delay. It makes memory a policy-facing problem: a policy must keep similar histories distinct, retrieve the past event relevant to the current decision, and convert that recall into an action-ready state. We call these requirements separability, addressability, and prospectiveness. We introduce Chameleon, a ~60M visuomotor policy for control-indexed prospective memory. Chameleon writes embodied event memory, preserves separable histories, retrieves control-relevant traces, and trains the resulting working state to be prospective. We also introduce Camo-Dataset, a real-robot benchmark that isolates observation-action delay by making the decision scene visually ambiguous, so the correct action must be inferred from earlier observations. Chameleon improves decision/end-to-end success on Camo-Dataset from 22.5%/21.3% to 80.8%/71.3%. On public long-horizon memory benchmarks, it achieves 87.1% +/- 0.8% on LIBERO-10, 97.3% +/- 4.5% on MemoryBench, and 75.1% +/- 1.4% on MIKASA-Robo, setting the state of the art for same-size models and exceeding multiple larger VLA baselines under the reported protocols. Probes and ablations show that Chameleon learns separable, addressable, and prospective memory, and that these properties drive its performance gains.

Figures & tables

Figure 1: Observation–action delay requires control-indexed memory. Current: Semantic summaries or similarity-based visual retrieval can select a plausible but wrong trace. Ours & Human: Effective memory requires three capabilities: separability, addressability, and prospectiveness.

Figure 2: Chameleon overview. Chameleon writes embodied event tokens, propagates them as token-grounded traces, addresses them with a learned control index, and consolidates the recalled trace into the prospective policy state $h_{t}$ . The state conditions a rectified-flow action policy, while Control-JEPA trains it to predict future control context.

Figure 3: Evaluation benchmarks. Left: public long-horizon and memory benchmarks. Right: Camo-Dataset real-robot tasks. The frame marked ∗ denotes the non-Markovian decision stage, where the correct action depends on the earlier episode variable $z$ rather than the current observation alone.

Matched imitation baselines
Method	Clean a specified plate	Play shell game	Add various seasonings	Avg. DSR	Avg. MSR	Avg. SR
	DSR/SR	DSR/SR	DSR/SR
Diffusion Policy Chi et al. (2023)	33.3/30.6	34.3/33.3	0.0/0.0	22.5	67.6	21.3
ACT Zhao et al. (2023)	28.0/19.4	35.5/30.6	0.0/0.0	21.2	51.8	16.7
Flow Matching Lipman et al. (2022) ; Liu et al. (2022)	30.0/25.0	25.7/25.0	0.0/0.0	18.6	62.0	16.7
Ours and mechanism ablations

Table 4

Benchmark	Protocol	Chameleon	Published context
MemoryBench Fang et al. (2025)	3 task-specific policies	97.3 $\pm$ 4.5	RVT-2 54.0 Goyal et al. (2024) ; Fang et al. (2025) ; SAM2Act 55.0 Fang et al. (2025) ; SAM2Act+ 94.3 Fang et al. (2025) ; ReMem-VLA 94.5 ∗ Li et al. (2026) .
LIBERO-10 Liu et al. (2023)	10-task mixed policy	87.1 $\pm$ 0.8	DP-T 51.0 Chi et al. (2023) ; Reuss et al. (2025) ; QueST 69.0 Reuss et al. (2025) ; DP-CNN 73.0 Chi et al. (2023) ; Reuss et al. (2025) ; MoDE 92.0 Reuss et al. (2025) ; OpenVLA 53.7 Kim et al. (2024) ; Shi et al. (2026) ; CoT-VLA 69.0 Zhao et al. (2025) ; Shi et al. (2026) ; TriVLA 73.2 Liu et al. (2025) ; Shi et al. (2026) ; $\pi_{0}$ 85.2 Black et al. (2024) ; Shi et al. (2026) ; 4D-VLA 86.5 Zhang et al. (2026) ; Shi et al. (2026) ; MemoryVLA 93.4 Shi et al. (2026) .
MIKASA-Robo Cherepanov et al. (2026)	5-task mixed policy	75.1 $\pm$ 1.4	CronusVLA 18.0 Li et al. (2025a) ; Shi et al. (2026) ; SpatialVLA 21.0 Qu et al. (2025) ; Shi et al. (2026) ; OpenVLA-OFT 28.4 Kim et al. (2025) ; Shi et al. (2026) ; $\pi_{0}$ 29.4 Black et al. (2024) ; Shi et al. (2026) ; MemoryVLA 41.2 Shi et al. (2026) ; GMP 67.8 Gao et al. (2026) .
MIKASA-Robo Cherepanov et al. (2026)	2 task-specific policies	95.6 $\pm$ 1.0	DP 19.5 Chi et al. (2023) ; Lei et al. (2026) ; DP-PTP 15.0 Lei et al. (2026) ; MaIL 19.5 Jia et al. (2024) ; Lei et al. (2026) ; DP-VPWEM 86.5 Lei et al. (2026) .

Table 2: Public long-horizon and memory benchmarks. Success rate (%). All entries follow the protocol of the cited source. Benchmark details and per-task results are provided in the Appendix.

Figure 4: Mechanistic probes on Camo-Dataset. (a) Separability : hidden-variable decoding from current tokens and memory states. (b) Addressability : counterfactual trace edits under a fixed control query. (c) Prospectiveness : future-control decoding from $h_{t}$ with and without Control-JEPA.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Shape	Description
$d$	scalar	token / hidden dimension
$V$	scalar	number of camera views
$N_{v}$	scalar	patch tokens from view $v$
$N$	scalar	total event tokens, $N=\sum_{v}N_{v}+2$
$L$	scalar	number of memory layers
$H$	scalar	action prediction horizon

Appendix

Table S1: Notation and tensor shapes. Shapes are per timestep

t

and per layer

\ell

unless noted.

B

denotes batch size and is omitted from the shape column.

Benchmark	#Tasks	Policy	#Demos/task	#Rollouts/task	#Seeds	Protocol followed
LIBERO-10	10	mixed	50	50	3	LIBERO Liu et al. [2023]
MemoryBench	3	task-specific	100	25	4	SAM2Act Fang et al. [2025]
MIKASA-Robo (mixed)	5	mixed	250	30	3	MIKASA-Robo Cherepanov et al. [2026]
MIKASA-Robo (special.)	2	task-specific	250	30	3	DP-VPWEM Lei et al. [2026]

Appendix

Table S2: Simulation benchmark protocols. Configuration used for each public benchmark. “Policy” indicates whether a single mixed-task policy or one policy per task is trained. “Protocol” names the source whose evaluation setup we match.

Figure S1: Real-robot setup for Camo-Dataset. UR5 workspace with the $2$ camera viewpoints annotated.

Task	Hidden episode variable	Aliased decision point	Diagnostic role	$p_{e}$
Clean a specified plate	Which visually similar plate was used or contaminated earlier.	Candidate plates appear interchangeable when the robot selects the target.	Event-object binding under delayed relevance.	$1/3$
Play shell game	The hidden object’s location after occlusion and swaps.	Cups are visually identical after rearrangement.	Spatial tracking under occlusion and distractor motion.	$1/3$
Add various seasonings	Which subgoals have already been completed in a repeated sequence.	The workspace returns to a similar state after each seasoning action.	Sequential progress memory and prevention of repetition or omission.	$1/27$

Appendix

Table S3: Task design in Camo-Dataset. The dataset contains three real-robot tasks that instantiate delayed control relevance through complementary hidden variables.

p_{e}

is the chance decision rate used for chance-adjusted DSR. For Add various seasonings, DSR is scored at the complete sequence level.

Component	#Params	Trainable?
Visual encoder (DP-style patch encoder)	22.9M	yes
Text encoder (DistilBERT)	66.4M	frozen
Memory layers ( $L$ control-indexed cells)	20.3M	yes
– SSM ${}_{\text{slow}}$ / SSM ${}_{\text{fast}}$	5.1M	yes
– Mixer / control index / attention	15.2M	yes
Rectified-flow action head	23.1M	yes

Appendix

Table S4: Parameter budget of Chameleon. Breakdown by component. The text encoder is frozen during training. “Trainable” excludes it. The “

\sim

60M” claim in the main text refers to the trainable-parameter budget, not the total parameter count.

Model	Approx. scale	Model type
Chameleon (ours)	100–500M	policy model
Diffusion Policy Chi et al. [2023]	100–500M	policy model
ACT Zhao et al. [2023]	10–100M	policy model
Flow Matching Lipman et al. [2022] , Liu et al. [2022]	100–500M	policy model
DP-T / DP-CNN Chi et al. [2023] , Reuss et al. [2025]	100–500M	policy model
QueST Reuss et al. [2025]	100–500M	policy model

Appendix

Table S5: Reported model scale. We report approximate total/backbone model scale, not benchmark-specific trainable parameters. Ranges indicate commonly reported configurations when exact counts are implementation-dependent.

Shared model and training hyperparameters
Feature dimension $d$	512
Memory layers $L$	2
Mixer attention heads	8
SSM ${}_{\text{slow}}$ state dim	128
SSM ${}_{\text{fast}}$ state dim	32
RF head depth / heads	6 / 8

Appendix

Table S6: Hyperparameters of Chameleon. Top block: model settings shared across benchmarks; optimization rows that vary are reported in the order LIBERO-10 / MemoryBench / MIKASA / Camo. Bottom block: input/output configuration.

\alpha

and

K

are fixed across all experiments.

MemoryBench Fang et al. [2025] – SOTA: 2 strongest baselines (SAM2Act+ Fang et al. [2025] , ReMem-VLA Li et al. [2026] )
Task	Chameleon (ours)	DP-family baseline	VLA baseline
Reopen Drawer	92.0 $\pm$ 13.5	84.0 $\pm$ 0.0	100.0
Put Block Back	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	93.0
Rearrange Block	100.0 $\pm$ 0.0	99.0 $\pm$ 2.0	99.0
Average	97.3 $\pm$ 4.5	94.3	94.5
MIKASA-Robo Cherepanov et al. [2026] (mixed, 5-task) – SOTA: 2 strongest baselines (GMP Gao et al. [2026] , MemoryVLA Shi et al. [2026] )

Appendix

Table S7: Per-task results on public benchmarks. Success rate (%). Each block uses the baselines selected per the convention in the text. Bold = best in block.

Method	Clean plate	Shell game	Seasoning	Avg. DSR / SR	Ablated property
	DSR/SR	DSR/SR	DSR/SR
Chameleon (full)	91.2/86.1	86.1/86.1	65.2/41.7	80.8 / 71.3	—
w/o memory	26.7/22.2	34.4/30.6	0.0/0.0	20.4 / 17.6	separability (all memory)
similarity retrieval bank	41.4/33.3	28.6/22.2	0.0/0.0	23.3 / 18.5	addressability (vs. control-indexed)
Vanilla Mamba memory	27.6/22.2	30.0/25.0	50.0/19.4	35.9 / 22.2	separability (token-grounded traces)
w/o control index	40.7/30.6	45.8/30.6	60.0/16.7	48.8 / 26.0	addressability

Appendix

Table S8: Ablation study on Camo-Dataset. Avg. columns aggregate over tasks. The “Ablated property” column names the functional capability each variant removes.

Task	Chance	Current $Z^{0}_{t}$	Memory $h^{L}_{t}$
Play shell game	33.3	46.7	83.3
Add various seasonings	3.7	37.4	98.5

Appendix

Table S9: Separability probe accuracy (%). Decoding the hidden variable

z

from current tokens vs. memory state at

t_{d}

. Chance is the per-task

p_{e}

	Full	Swap	Mask+	Mask-
Choice accuracy	93	87	40	90

Appendix

Table S10: Addressability counterfactual choice accuracy (%). The control query

u^{\ell}_{t}

is fixed while the trace bank

E^{\ell}_{t}

is edited in Add various seasonings.

Memory-Augmented Agents

Memory-augmented agents use stored experience and knowledge to act coherently across long tasks, changing environments, and repeated interactions.

See all 348 papers

MemBodied: Recurrent Associative Memory for Vision-Language-Action Models
Sep 23, 2026Tej Deep Pala, Navonil Majumder, Bryce Goh +4Robotic Manipulation Vision-Language-Action Models
Nanyang Technological University · Griffin Labs · École Centrale de Lyon
ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control
Sep 30, 2026Yize Liu, Ke Wang, Mac Schwager +2Robotic Control Long-Horizon Robotic Manipulation
Stanford University
MEM: Multi-Scale Embodied Memory for Vision Language Action Models
Mar 4, 2026Marcel Torne, Karl Pertsch, Homer Walke +14Memory-Augmented Agents Long-Term Memory for Embodied Agents
Physical Intelligence · Stanford University · UC Berkeley +1

Sep 23, 2026J/K move · Enter open · S save

MemBodied: Recurrent Associative Memory for Vision-Language-Action Models

Tej Deep Pala, Navonil Majumder, Bryce Goh, Raphael Yee+3

Nanyang Technological University · Griffin Labs · École Centrale de Lyon

Robotic Manipulation Vision-Language-Action Models VLMs for Robotics Long-Term Memory for Embodied Agents

Open paper AskPDF arXiv

Vision-Language-Action models provide a strong foundation for general-purpose robot control, yet a vast majority of policies do not preserve and leverage episode-level information beyond the current observation. This limitation is consequential in history-dependent manipulation tasks that depend on information available only in past observations. Retaining past observations in context can aid in recovering this information, but at the significant cost of ever-growing, bloated context and inference latency. We thus introduce MemBodied, a fixed-size episodic memory with two complementary components: an associative state that records interactions across policy calls and an episode anchor that preserves a compact representation of the initial scene as a reference. At each policy call, the model conditions action generation on the current input and the memory components, rather than directly using past observations. Across five evaluated RMBench tasks requiring memory, MemBodied achieves

7.81\times

the mean success rate of a stateless policy and

2.98\times

of vanilla recurrent memory, while outperforming the strongest memory-augmented baseline by

1.3\times

with

10\times

fewer added parameters. On the fully observable LIBERO-Long suite, it reached 90.6%, a 5.4% improvement over the stateless

π_0

policy. These findings support MemBodied as a practical alternative to expanding the policy context for history-dependent manipulation.

Abstract

Memory-Augmented Agents

Explore similar work

MemBodied: Recurrent Associative Memory for Vision-Language-Action Models

ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control

MEM: Multi-Scale Embodied Memory for Vision Language Action Models