LTM: Latent Transition Based In-Context Memory for Vision-Language-Action Models

Qixin Sun; WenLong Gao; Xiansheng Chen; Zhongyuan Wang; Si Liu; Pengwei Wang

cs.ROOct 15, 2026

LTM: Latent Transition Based In-Context Memory for Vision-Language-Action Models

Authors: Qixin Sun, WenLong Gao, Xiansheng Chen, Zhongyuan Wang, Si Liu, Pengwei Wang

Organizations: Beihang University · BAAI

Abstract

Robotic manipulation often requires remembering interactions that are no longer visible. We introduce Latent Transition Memory (LTM), which stores history as compact latent transitions between consecutive observations. Instantiated on $π_{0.5}$ as LTM- $π$ , our method uses a frozen latent action encoder, a lightweight adapter, and causal KV caching for incremental inference without memory-specific semantic annotations or external reasoning-model calls. With DreamDojo transitions, LTM- $π$ achieves 58.54% success on RoboMME and 80.5% on LIBERO-Mem, performing best among baselines without external reasoning models or memory-specific annotations. Ablations support the contribution of transition content beyond temporal cues and the efficiency of delayed memory admission. Scaling RoboMME demonstrations tenfold improves success from 57.67% to 87.50%, while a Cup Swap diagnostic highlights the role of process supervision in temporal composition. Under a 200-token memory budget, success remains 84.00%, versus 87.50% without compression. Finally, real-robot counting success improves from 11.25% to 82.50%. Together, these results demonstrate the effectiveness, scalability, and efficiency of latent transition memory, highlighting its capacity for complex temporal information, bounded-context operation, and real-world robotic control.

Figures & tables

Figure 1: Success rates (%) on the 16 RoboMME tasks, grouped by track. The legend shows data scales and overall success rates; scores follow Tables 1 and 5 .

Figure 2: LTM- $\pi$ encodes consecutive observations with a frozen latent action model and maps each transition to a memory token. The trainable policy conditions on the instruction, transition history, and current observation. The schematic omits the fixed lag and the visual anchors used for video demonstrations.

Method	Counting				Permanence				Reference				Imitation				Avg.
	BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS
Privileged-information and human references (not ranked)
SimpleSG+Oracle	85.78	99.78	100.00	44.67	33.11	22.00	15.56	15.56	44.00	27.78	31.33	26.00	87.33	10.00	95.33	55.11	49.58
GroundSG+Oracle	85.78	100.00	100.00	49.67	98.78	95.00	99.22	80.22	83.33	97.33	100.00	100.00	87.78	15.56	97.00	55.56	84.08
WeaveLA (oracle)	28.00	34.00	50.00	12.00	24.00	28.00	22.00	4.00	12.00	2.00	22.00	14.00	16.00	2.00	8.00	12.00	18.12
Human	96.00	100.00	80.00	78.00	90.00	92.00	92.00	90.00	92.00	92.00	98.00	90.00	90.00	98.00	84.00	86.00	90.50

Table 1: Success rates (%,

\uparrow

) on all 16 RoboMME tasks. Deployable methods are divided into two blocks according to whether they avoid both external reasoning-model calls and memory-specific annotations. Within each block, bold and underline indicate the best and second-best results, respectively. Oracle and human references are excluded from ranking.

Method	T0	T1	T2	T3	T4	T5	T6	T7	T8	T9	Avg.
$\pi_{0.5}$ (original)	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
SlotVLA ( $h=8$ )	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Naive E-SlotSSM (oracle)	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
MemoryVLA	55.0	0.0	35.0	0.0	5.0	0.0	0.0	0.0	0.0	0.0	9.5
GMP	85.0	45.0	20.0	5.0	40.0	0.0	0.0	0.0	0.0	0.0	19.5
MemoryVAM (Pathway 1 only)	85.0	40.0	35.0	10.0	15.0	0.0	5.0	0.0	30.0	65.0	28.5

Table 2: Success rates (SR, %) on LIBERO-Mem. Bold indicates the best result in each column.

Method	Counting				Permanence				Reference				Imitation				Avg.
	BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS
Full	64.67	100.00	93.33	87.33	68.67	98.00	30.67	31.33	22.67	30.00	38.67	22.67	80.00	20.67	64.00	70.00	57.67
LT-only	66.00	100.00	94.00	90.00	24.00	44.00	30.00	16.00	26.00	30.00	40.00	36.00	90.00	14.00	46.00	40.00	49.13
Anchor-only	34.00	32.00	42.00	0.00	66.00	58.00	24.00	8.00	12.00	2.00	26.00	10.00	32.00	2.00	12.00	10.00	23.13
Fixed Random LT	72.00	74.00	98.00	58.00	82.00	94.00	30.00	38.00	18.00	26.00	26.00	20.00	60.00	2.00	10.00	4.00	44.50

Table 3: Ablation study on the effect of latent transitions and visual anchors on RoboMME. Success rates (%,

\uparrow

) are reported on all 16 tasks. Fixed Random LT removes transition-specific semantics while retaining temporal information from the memory sequence.

$k$	Counting				Permanence				Reference				Imitation				Avg.	Runtime (ms, $\downarrow$ )
	BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS		Policy	Sim.	LAM update	LAM block
0	62.00	98.00	94.00	90.00	70.00	94.00	30.00	34.00	28.00	26.00	38.00	24.00	82.00	20.00	66.00	70.00	57.88	66.71	117.01	114.25 (10p, sync.)	114.25
5	64.67	100.00	93.33	87.33	68.67	98.00	30.67	31.33	22.67	30.00	38.67	22.67	80.00	20.67	64.00	70.00	57.67	66.71	117.01	67.97 (5p, async.)	9.47
10	58.00	98.00	90.00	86.00	68.00	96.00	28.00	32.00	20.00	30.00	36.00	24.00	76.00	24.00	64.00	72.00	56.38	66.71	117.01	114.25 (10p, async.)	0.00

Table 4: Ablation study on lag

k

on RoboMME. Success rates (%,

\uparrow

) are reported on all 16 tasks. Reported LAM blocking is an estimated critical-path cost,

\max(0,T_{\mathrm{LAM}}-T_{\mathrm{overlap}})

Method	Data scale	Counting				Permanence				Reference				Imitation				Avg.	Gain ratio
		BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS		(pp/ $\times$ )
PonderPounce (9B)	$1\times$	50.67	93.33	72.00	82.67	94.67	97.33	25.33	34.00	85.33	40.67	83.33	79.33	81.33	6.00	12.67	34.67	60.83	–
PonderPounce (9B)	$9\times$	74.00	98.00	93.33	60.00	96.67	100.00	73.33	50.67	92.67	88.67	92.00	97.33	93.33	18.67	25.33	54.67	75.54	1.84
LTM- $\pi$ (ours)	$1\times$	64.67	100.00	93.33	87.33	68.67	98.00	30.67	31.33	22.67	30.00	38.67	22.67	80.00	20.67	64.00	70.00	57.67	–
LTM- $\pi$ (ours)	$10\times$	80.00	100.00	100.00	100.00	100.00	100.00	74.00	62.00	70.00	66.00	84.00	84.00	96.00	88.00	98.00	98.00	87.50	3.31

Table 5: Task-level data-scaling results on RoboMME (success rate, %,

\uparrow

). We report PonderPounce (9B) with

1\times

and

9\times

data and LTM-

\pi

with

1\times

and

10\times

data. Gain ratio is the increase in average success rate divided by the additional data multiple relative to

1\times

, measured in percentage points per additional

1\times

data. The data, training, and supervision budgets differ between methods; gain ratio is descriptive, not a controlled estimate of sample efficiency.

Figure 3: Our Cup Swap diagnostic benchmark and two supervision methods. The arrows are for illustration only and are not included in the actual dataset. <LT_i> denotes a latent transition; the illustration uses zero-indexed latent labels.

Supervision	Individual Swap Acc.	Final-state Acc.
Vanilla supervision (final state only)	–	2.40
Process supervision (CoT-like)	99.78	97.93

Table 6: Diagnostic results (%) on the six-cup, multi-ball Cup Swap task. Final-state accuracy requires exact cup occupancy. The Vanilla value is its best reported checkpoint result; its final checkpoint reaches 1.75%.

Statistic	Uncompressed	Compressed ( $Q=200$ )
Mean	323.93	$\leq 200$
Median	265	200
P90	747	200
P95	893	200
Maximum	1,789	200
Queries $>200$ (%)	61.98	0.00

Table 7: Execution-transition tokens per query (

k=10

). Uncompressed values are measured; compressed counts follow

\min(N,200)

. The compressed mean is an upper bound.

Memory	Counting				Permanence				Reference				Imitation				Avg.
	BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS
Uncompressed	80.00	100.00	100.00	100.00	100.00	100.00	74.00	62.00	70.00	66.00	84.00	84.00	96.00	88.00	98.00	98.00	87.50
Compressed ( $Q=200$ )	84.00	100.00	100.00	82.00	100.00	100.00	62.00	58.00	74.00	56.00	86.00	80.00	94.00	76.00	96.00	96.00	84.00
$\Delta$ (pp)	+4.00	0.00	0.00	-18.00	0.00	0.00	-12.00	-4.00	+4.00	-10.00	+2.00	-4.00	-2.00	-12.00	-2.00	-2.00	-3.50

Table 8: DreamDojo-based LTM-

\pi

on RoboMME-10x with and without execution-token compression (success rate, %). The uncompressed row reproduces the

10\times

reference in Table 5 ; the compressed model uses

Q=200

during training and inference and is evaluated over 50 rollouts per task.

\Delta

is compressed minus uncompressed success in percentage points. Task abbreviations follow Table 1 .

Figure 4: Task and environment for real-world experiments.

Method	$x=1$	$x=2$	$x=3$	$x=4$	Avg.
$\pi_{0.5}$ (no memory)	5.0	20.0	20.0	0.0	11.25
LTM- $\pi$ (ours)	95.0	90.0	70.0	75.0	82.50

Table 9: Real-world success rate (%,

\uparrow

) for picking and placing

x

objects. Avg. is the unweighted mean across the four target quantities.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	$\pi_{0.5}$ (no memory)				LTM- $\pi$ (ours)
Trial	$x=1$	$x=2$	$x=3$	$x=4$	$x=1$	$x=2$	$x=3$	$x=4$
1	0	1	4	5	1	2	3	4
2	2	1	5	7	1	2	3	3
3	2	2	5	8	1	1	3	3
4	2	3	4	6	1	2	4	4
5	3	3	6	8	1	2	4	4

Appendix

Table 10: Per-trial transferred-object counts in real-world evaluation. Bold entries match the instructed quantity; “–” denotes an unperformed trial. Trial indices are local to each model–quantity setting.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 159 papers

Jul 8, 2026

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6

Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Memory-Augmented VLMs Robotic Manipulation Vision-Language-Action Models Memory-Augmented Agents Reasoning for VLA Models Temporal Modeling for VLA Models

Ask PDF

Oct 15, 2026

Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions. Existing approaches typically preserve raw historical observations, compressed visual tokens, or language summaries, but these representations either expose the action policy to increasingly long histories or risk discarding task-relevant temporal, semantic, and perceptual information. We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control. Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory. To this end, it organizes history via multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question-answering supervision and key-frame DINO feature reconstruction, and employs memory-first two-stage training to encourage greater use of memory by the action policy. The resulting memory captures both high-level task state and fine-grained perceptual evidence while providing a bounded interface to the action policy. Across RoboMME, RMBench, and four real-world manipulation tasks, Mem4Act achieves state-of-the-art performance on memory-dependent control, with success rates of 49.88%, 61.0%, and 81.5%, respectively. Extensive ablations further demonstrate the importance of temporal aggregation, complementary memory supervision, and memory-first training.

Yun Zhang, Zhiyu Huang, Tianhui Cai +3

University of California, Los Angeles · North Carolina State University

Long-Horizon Robotic Manipulation Memory-Augmented Agents

Ask PDF

Jun 8, 2026

MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models

Temporal modeling is essential for robotic manipulation, as effective control requires both memory of past interactions and imagination of future states. However, most VLA models rely primarily on the current observation and therefore struggle with long-horizon, temporally dependent tasks. Cognitive science suggests that humans rely on working memory to buffer short-lived context, the hippocampal system to preserve episodic memory of past experience, and internal models to imagine possible future state evolution. Inspired by these mechanisms, we propose MemoryVLA++, a full temporal modeling framework that equips VLA models with memory and imagination for robotic manipulation. A pretrained VLM encodes the current observation into perceptual and cognitive tokens, forming working memory. These tokens query a Perceptual-Cognitive Memory Bank to retrieve relevant historical context. This bank stores low-level details and high-level semantics from past interactions, and is updated through redundancy-aware consolidation. A world model imagines future states in a denoising latent space, and the imagined latents are integrated under memory guidance to form full temporal-aware tokens. The resulting tokens condition a diffusion action expert to predict temporally consistent action sequences. We conduct extensive experiments on 5 simulation benchmarks and 3 categories of real-robot tasks across 3 robots, covering general manipulation, long-horizon temporal tasks, robustness, and generalization. Our method achieves strong performance across Libero, SimplerEnv, Mikasa-Robo, Calvin, Libero-Plus, and diverse real-robot tasks, validating the effectiveness of full temporal modeling with memory and imagination. For example, on real robots, it achieves +9%, +26%, +28% gains on general, memory-dependent, and imagination-dependent tasks. Project Page: https://shihao1895.github.io/MemoryVLA-PP-Web

Hao Shi, Weiye Li, Bin Xie +6

Department of Automation, BNRist, Tsinghua University, Beijing, China · Department of Computer Science, The University of Hong Kong, Hong Kong, China · Dexmal, Beijing, China +1

Long-Horizon Robotic Manipulation World Models for Robotics Temporal Modeling for VLA Models Vision-Language-Action Models Memory-Augmented Agents

Ask PDF

Abstract

Memory-Augmented VLMs

Explore similar work

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models