Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

Tanguy Dieudonné; Jack B. Jedlicki; Heng Yang

cs.ROSep 28, 2026

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

Authors: Tanguy Dieudonné, Jack B. Jedlicki, Heng Yang

Organizations: Harvard University · ETH Zürich

Abstract

Memory is essential for long-horizon, partially observed robotic manipulation: a robot must remember which object was placed in a drawer, whose cup it moved, or how many action cycles have elapsed. Recent vision-language-action (VLA) models embed memory directly inside the policy, but benchmarks show no single in-policy mechanism covers all spatio-temporal dimensions, trailing oracle methods by a wide margin. We argue that memory type dictates where memory should reside: short-term perceptual memory (repetition, timing, retracing) belongs inside the policy, while long-term object memory (persistent spatial state, containment, event history) belongs outside as an explicit, readable record. We present Ledger, a harness that realizes this split over a single fine-tuned $π_{0.5}$ policy by pairing an in-policy frame-sampling memory with an external spatio-temporal object memory, the ledger, built from a SAM3 tracker and a VLM captioner of the demonstration and read by an LLM planner that decides at step boundaries. On RoboMME, Ledger reaches the highest four-suite average among the evaluated methods, 64.3% (vs. 45.9% for the strongest prior method under identical evaluation), leading object reference (60.7% vs. 40.3%) and object permanence (86.7% vs. 56.2%) using a single set of weights. Choosing the memory source at runtime, from the instruction and the record, removes the need for a task-level router.

Figures & tables

Panel of Figure 1 — Figure 1: Ledger Overview . Two memories serve one $\pi_{0.5}$ policy. Outside the policy , a SAM3 tracker seeded from the first frame records where each object rests and when it is covered, and a video VLM transcribes the demonstration when there is one. Inside the policy , a frame-sampling modulator over past observations carries repetition, timing, and motion. At each step boundary the LLM planner reads the instruction, the record, an annotated frame, and the completed steps, and either grounds a coordinate target or defers to the policy’s own memory with an empty subgoal.

Method	Counting				Permanence				Reference				Imitation				Avg
	Bin Fill	Pick Xtimes	Swing Xtimes	Stop Cube	Video Umsk	Button Umsk	Video UmskS	Button UmskS	Pick HighL	Video Repick	Video PlcBtn	Video PlcOrd	Move Cube	Insert Peg	Pattern Lock	Route Stick
As reported by Dai et al. [5] : three seeds $\times$ three checkpoints
Human	96.0	100.0	80.0	78.0	90.0	92.0	92.0	90.0	92.0	92.0	98.0	90.0	90.0	98.0	84.0	86.0	90.5
Oracle	85.8	100.0	100.0	49.7	98.8	95.0	99.2	80.2	83.3	97.3	100.0	100.0	87.8	15.6	97.0	55.6	84.1
QwenVL	52.0	92.7	7.3	0.0	88.7	24.0	30.7	14.0	15.1	25.3	54.0	31.8	71.6	3.3	6.7	6.0	32.7
FrameSamp	39.6	87.3	92.0	42.0	32.7	25.1	24.4	18.2	22.9	30.4	60.0	32.0	77.8	7.6	53.6	66.7	44.5

Table 1: Main results. RoboMME success rate (%) on the test split, 50 episodes per task, mean over evaluation seeds; Avg is the mean over the 16 tasks. Upper block: as reported by Dai et al. [5] . Lower block: methods run by us under the protocol of Sec. 4 . Oracle and QwenVL are GroundSG with ground-truth subgoals and with the benchmark’s fine-tuned Qwen3-VL subgoal generator; FrameSamp is FrameSamp+Modul; Tracker is the tracker-only variant of our harness. Red marks the best method within each block; Human and Oracle are reference rows. Seed spreads are in Appendix C .

Decision loop over the ledger	Unmask	UnmaskSwap	Repick	PlaceButton	PlaceOrder
Plan once at the start of the episode	90	68	60	58	60
+ cover state, re-decide after each step	92	86	28	66	66
+ remaining plan carried across decisions	92	92	54	66	56

Table 2: Ablation of the decision loop on the demonstration-guided tasks (success %, one evaluation seed, 50 episodes per cell, checkpoint

80

k; a cell’s binomial standard error is up to

7

points). The last row is the configuration of Table 1 , which averages three seeds. Re-deciding at step boundaries lets the planner react to events that occur during execution; carrying the remaining plan across decisions keeps the count of completed repetitions.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: First frame of one test episode of each RoboMME task, from our evaluation runs (front camera, $256\times 256$ , the resolution the policy and the tracker see). Rows are the four suites: Counting, Permanence, Reference, Imitation. Table 3 gives the instruction of each task.

Task	Instruction
BinFill	put one red cube into the bin, then press the button to stop
PickXtimes	pick up the green cube and place it on the target, repeating this action three times, then press the button to stop
SwingXtimes	pick up the green cube, move it to the top of the right-side target, then move it to the top of the left-side target, repeating this back-and-forth motion two times, finally press the button to stop
StopCube	press the button to stop the cube just as it reaches the target for the fourth time
VideoUnmask	watch the video carefully, then pick up the container hiding the green cube
ButtonUnmask	first press the button, then pick up the container hiding the red cube

Appendix

Table 3: One instruction instance per task, as given to the planner and to the low-level policy; object colours, counts, and targets vary across episodes. Tasks that begin with “watch the video carefully” start with a demonstration video; the others start from a single observed frame.

Method	Counting	Permanence	Reference	Imitation	Avg
FrameSamp	$69.2\pm 2.9$	$25.7\pm 1.3$	$37.3\pm 3.2$	$51.3\pm 2.5$	$45.9\pm 0.7$
MemER	$49.2\pm 1.6$	$56.2\pm 0.6$	$40.3\pm 1.9$	$27.3\pm 2.0$	$43.3\pm 0.7$
Tracker	$66.0\pm 1.8$	$87.2\pm 5.1$	$32.7\pm 0.8$	$47.2\pm 0.8$	$58.3\pm 1.8$
Ledger	$66.8\pm 2.8$	$86.7\pm 3.5$	$60.7\pm 5.2$	$43.0\pm 2.2$	$64.3\pm 0.9$

Appendix

Table 4: Mean

\pm

standard deviation over the three evaluation seeds of the suite means and of the 16-task average (success %).

Explore similar work

Remember What You Did: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies
Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Mamba
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Generalist Vision--Language--Action Vision-Language-Action Framework
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Sep 30, 2026Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov +1Goal-Conditioned Dynamic Manipulation Robocasa
AXXX, Moscow, Russia · MIRIAI, Moscow, Russia · HSE University, Moscow, Russia

Sep 29, 2026, cs.ROJ/K move · Enter open · S save

Remember What You Did: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

Yaxin Zhao, Dianye Huang, Chenwei Wang, Chenguang Yang+1

Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

Mamba

Open paper AskPDF arXiv

Vision-language-action (VLA) models have driven rapid progress in robotic manipulation, demonstrating strong fine-grained control and promising performance on long-horizon tasks. However, many existing VLAs lack explicit access to interaction history, making them vulnerable to perceptual aliasing: similar current observations and robot states at different task stages may induce action ambiguity and lower success rate. Existing methods incorporate temporal or progress cues through feature conditioning, action-prior modification, or sampling guidance. However, methods that jointly fine-tune memory modules and the base VLA incur additional policy-training costs, motivating the separation of trainable history-conditioned steering from frozen base-policy refinement. We propose ActMem-VLA, a dual-expert handover architecture that augments a frozen, fine-tuned VLA with a memory plugin comprising a Mamba-based memory module and a lightweight PreAction Expert (PAE). Specifically, Mamba encodes executed-action history into memory that conditions PAE alongside current context. With these inputs, PAE steers task progression during early, high-noise denoising, then passes the partially denoised action to the frozen Action Expert (AE) to refine action details during the remaining low-noise steps. The fine-tuned base VLA remains frozen throughout training, while only the Mamba module and PAE are jointly optimized. On LIBERO-Mem, ActMem-VLA achieves 80.8% average success across all ten tasks, compared with 65.2% for

π_{0.5}

and 49.5% for MemoryVLA, while introducing only 3.45% additional parameters. Across four real-world tasks, it improves the average success rate over

π_{0.5}

by 28.8%.

Abstract

Explore similar work

Remember What You Did: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation