LTM: Latent Transition Based In-Context Memory for Vision-Language-Action Models

Qixin Sun; WenLong Gao; Xiansheng Chen; Zhongyuan Wang; Si Liu; Pengwei Wang

cs.ROOct 15, 2026

LTM: Latent Transition Based In-Context Memory for Vision-Language-Action Models

Authors: Qixin Sun, WenLong Gao, Xiansheng Chen, Zhongyuan Wang, Si Liu, Pengwei Wang

Organizations: Beihang University · BAAI

Abstract

Robotic manipulation often requires remembering interactions that are no longer visible. We introduce Latent Transition Memory (LTM), which stores history as compact latent transitions between consecutive observations. Instantiated on $π_{0.5}$ as LTM- $π$ , our method uses a frozen latent action encoder, a lightweight adapter, and causal KV caching for incremental inference without memory-specific semantic annotations or external reasoning-model calls. With DreamDojo transitions, LTM- $π$ achieves 58.54% success on RoboMME and 80.5% on LIBERO-Mem, performing best among baselines without external reasoning models or memory-specific annotations. Ablations support the contribution of transition content beyond temporal cues and the efficiency of delayed memory admission. Scaling RoboMME demonstrations tenfold improves success from 57.67% to 87.50%, while a Cup Swap diagnostic highlights the role of process supervision in temporal composition. Under a 200-token memory budget, success remains 84.00%, versus 87.50% without compression. Finally, real-robot counting success improves from 11.25% to 82.50%. Together, these results demonstrate the effectiveness, scalability, and efficiency of latent transition memory, highlighting its capacity for complex temporal information, bounded-context operation, and real-world robotic control.

Figures & tables

Figure 1: Success rates (%) on the 16 RoboMME tasks, grouped by track. The legend shows data scales and overall success rates; scores follow Tables 1 and 5 .

Figure 2: LTM- $\pi$ encodes consecutive observations with a frozen latent action model and maps each transition to a memory token. The trainable policy conditions on the instruction, transition history, and current observation. The schematic omits the fixed lag and the visual anchors used for video demonstrations.

Method	Counting				Permanence				Reference				Imitation				Avg.
	BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS
Privileged-information and human references (not ranked)
SimpleSG+Oracle	85.78	99.78	100.00	44.67	33.11	22.00	15.56	15.56	44.00	27.78	31.33	26.00	87.33	10.00	95.33	55.11	49.58
GroundSG+Oracle	85.78	100.00	100.00	49.67	98.78	95.00	99.22	80.22	83.33	97.33	100.00	100.00	87.78	15.56	97.00	55.56	84.08
WeaveLA (oracle)	28.00	34.00	50.00	12.00	24.00	28.00	22.00	4.00	12.00	2.00	22.00	14.00	16.00	2.00	8.00	12.00	18.12
Human	96.00	100.00	80.00	78.00	90.00	92.00	92.00	90.00	92.00	92.00	98.00	90.00	90.00	98.00	84.00	86.00	90.50

Table 1: Success rates (%,

\uparrow

) on all 16 RoboMME tasks. Deployable methods are divided into two blocks according to whether they avoid both external reasoning-model calls and memory-specific annotations. Within each block, bold and underline indicate the best and second-best results, respectively. Oracle and human references are excluded from ranking.

Method	T0	T1	T2	T3	T4	T5	T6	T7	T8	T9	Avg.
$\pi_{0.5}$ (original)	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
SlotVLA ( $h=8$ )	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Naive E-SlotSSM (oracle)	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
MemoryVLA	55.0	0.0	35.0	0.0	5.0	0.0	0.0	0.0	0.0	0.0	9.5
GMP	85.0	45.0	20.0	5.0	40.0	0.0	0.0	0.0	0.0	0.0	19.5
MemoryVAM (Pathway 1 only)	85.0	40.0	35.0	10.0	15.0	0.0	5.0	0.0	30.0	65.0	28.5

Table 2: Success rates (SR, %) on LIBERO-Mem. Bold indicates the best result in each column.

Method	Counting				Permanence				Reference				Imitation				Avg.
	BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS
Full	64.67	100.00	93.33	87.33	68.67	98.00	30.67	31.33	22.67	30.00	38.67	22.67	80.00	20.67	64.00	70.00	57.67
LT-only	66.00	100.00	94.00	90.00	24.00	44.00	30.00	16.00	26.00	30.00	40.00	36.00	90.00	14.00	46.00	40.00	49.13
Anchor-only	34.00	32.00	42.00	0.00	66.00	58.00	24.00	8.00	12.00	2.00	26.00	10.00	32.00	2.00	12.00	10.00	23.13
Fixed Random LT	72.00	74.00	98.00	58.00	82.00	94.00	30.00	38.00	18.00	26.00	26.00	20.00	60.00	2.00	10.00	4.00	44.50

Table 3: Ablation study on the effect of latent transitions and visual anchors on RoboMME. Success rates (%,

\uparrow

) are reported on all 16 tasks. Fixed Random LT removes transition-specific semantics while retaining temporal information from the memory sequence.

$k$	Counting				Permanence				Reference				Imitation				Avg.	Runtime (ms, $\downarrow$ )
	BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS		Policy	Sim.	LAM update	LAM block
0	62.00	98.00	94.00	90.00	70.00	94.00	30.00	34.00	28.00	26.00	38.00	24.00	82.00	20.00	66.00	70.00	57.88	66.71	117.01	114.25 (10p, sync.)	114.25
5	64.67	100.00	93.33	87.33	68.67	98.00	30.67	31.33	22.67	30.00	38.67	22.67	80.00	20.67	64.00	70.00	57.67	66.71	117.01	67.97 (5p, async.)	9.47
10	58.00	98.00	90.00	86.00	68.00	96.00	28.00	32.00	20.00	30.00	36.00	24.00	76.00	24.00	64.00	72.00	56.38	66.71	117.01	114.25 (10p, async.)	0.00

Table 4: Ablation study on lag

k

on RoboMME. Success rates (%,

\uparrow

) are reported on all 16 tasks. Reported LAM blocking is an estimated critical-path cost,

\max(0,T_{\mathrm{LAM}}-T_{\mathrm{overlap}})

Method	Data scale	Counting				Permanence				Reference				Imitation				Avg.	Gain ratio
		BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS		(pp/ $\times$ )
PonderPounce (9B)	$1\times$	50.67	93.33	72.00	82.67	94.67	97.33	25.33	34.00	85.33	40.67	83.33	79.33	81.33	6.00	12.67	34.67	60.83	–
PonderPounce (9B)	$9\times$	74.00	98.00	93.33	60.00	96.67	100.00	73.33	50.67	92.67	88.67	92.00	97.33	93.33	18.67	25.33	54.67	75.54	1.84
LTM- $\pi$ (ours)	$1\times$	64.67	100.00	93.33	87.33	68.67	98.00	30.67	31.33	22.67	30.00	38.67	22.67	80.00	20.67	64.00	70.00	57.67	–
LTM- $\pi$ (ours)	$10\times$	80.00	100.00	100.00	100.00	100.00	100.00	74.00	62.00	70.00	66.00	84.00	84.00	96.00	88.00	98.00	98.00	87.50	3.31

Table 5: Task-level data-scaling results on RoboMME (success rate, %,

\uparrow

). We report PonderPounce (9B) with

1\times

and

9\times

data and LTM-

\pi

with

1\times

and

10\times

data. Gain ratio is the increase in average success rate divided by the additional data multiple relative to

1\times

, measured in percentage points per additional

1\times

data. The data, training, and supervision budgets differ between methods; gain ratio is descriptive, not a controlled estimate of sample efficiency.

Figure 3: Our Cup Swap diagnostic benchmark and two supervision methods. The arrows are for illustration only and are not included in the actual dataset. <LT_i> denotes a latent transition; the illustration uses zero-indexed latent labels.

Supervision	Individual Swap Acc.	Final-state Acc.
Vanilla supervision (final state only)	–	2.40
Process supervision (CoT-like)	99.78	97.93

Table 6: Diagnostic results (%) on the six-cup, multi-ball Cup Swap task. Final-state accuracy requires exact cup occupancy. The Vanilla value is its best reported checkpoint result; its final checkpoint reaches 1.75%.

Statistic	Uncompressed	Compressed ( $Q=200$ )
Mean	323.93	$\leq 200$
Median	265	200
P90	747	200
P95	893	200
Maximum	1,789	200
Queries $>200$ (%)	61.98	0.00

Table 7: Execution-transition tokens per query (

k=10

). Uncompressed values are measured; compressed counts follow

\min(N,200)

. The compressed mean is an upper bound.

Memory	Counting				Permanence				Reference				Imitation				Avg.
	BF	PX	SX	SC	VU	BU	VUS	BUS	PH	VR	VPB	VPO	MC	IP	PL	RS
Uncompressed	80.00	100.00	100.00	100.00	100.00	100.00	74.00	62.00	70.00	66.00	84.00	84.00	96.00	88.00	98.00	98.00	87.50
Compressed ( $Q=200$ )	84.00	100.00	100.00	82.00	100.00	100.00	62.00	58.00	74.00	56.00	86.00	80.00	94.00	76.00	96.00	96.00	84.00
$\Delta$ (pp)	+4.00	0.00	0.00	-18.00	0.00	0.00	-12.00	-4.00	+4.00	-10.00	+2.00	-4.00	-2.00	-12.00	-2.00	-2.00	-3.50

Table 8: DreamDojo-based LTM-

\pi

on RoboMME-10x with and without execution-token compression (success rate, %). The uncompressed row reproduces the

10\times

reference in Table 5 ; the compressed model uses

Q=200

during training and inference and is evaluated over 50 rollouts per task.

\Delta

is compressed minus uncompressed success in percentage points. Task abbreviations follow Table 1 .

Figure 4: Task and environment for real-world experiments.

Method	$x=1$	$x=2$	$x=3$	$x=4$	Avg.
$\pi_{0.5}$ (no memory)	5.0	20.0	20.0	0.0	11.25
LTM- $\pi$ (ours)	95.0	90.0	70.0	75.0	82.50

Table 9: Real-world success rate (%,

\uparrow

) for picking and placing

x

objects. Avg. is the unweighted mean across the four target quantities.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	$\pi_{0.5}$ (no memory)				LTM- $\pi$ (ours)
Trial	$x=1$	$x=2$	$x=3$	$x=4$	$x=1$	$x=2$	$x=3$	$x=4$
1	0	1	4	5	1	2	3	4
2	2	1	5	7	1	2	3	3
3	2	2	5	8	1	1	3	3
4	2	3	4	6	1	2	4	4
5	3	3	6	8	1	2	4	4

Appendix

Table 10: Per-trial transferred-object counts in real-world evaluation. Bold entries match the instructed quantity; “–” denotes an unperformed trial. Trial indices are local to each model–quantity setting.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 159 papers

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation
Oct 15, 2026Yun Zhang, Zhiyu Huang, Tianhui Cai +3Long-Horizon Robotic Manipulation Memory-Augmented Agents
University of California, Los Angeles · North Carolina State University
MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models
Jun 8, 2026Hao Shi, Weiye Li, Bin Xie +6Long-Horizon Robotic Manipulation World Models for Robotics
Department of Automation, BNRist, Tsinghua University, Beijing, China · Department of Computer Science, The University of Hong Kong, Hong Kong, China · Dexmal, Beijing, China +1

Jul 8, 2026J/K move · Enter open · S save

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang+5

Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Memory-Augmented VLMs Robotic Manipulation Vision-Language-Action Models Memory-Augmented Agents Reasoning for VLA Models Temporal Modeling for VLA Models

Open paper AskPDF arXiv

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

Abstract

Memory-Augmented VLMs

Explore similar work

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models