LogLook: Execution-Grounded Memory for Frontier-Model Robot Planners

Tianyi Jiang; Tianyao Tong; Zhenshan Bing

cs.ROOct 23, 2026

LogLook: Execution-Grounded Memory for Frontier-Model Robot Planners

Authors: Tianyi Jiang, Tianyao Tong, Zhenshan Bing

Organizations: State Key Laboratory for Novel Software Technology and School of Intelligence Science and Technology, Nanjing University (Suzhou Campus), Suzhou, China · School of Computation, Information and Technology, Technical University of Munich, Garching, Germany

Abstract

Many manipulation tasks require a robot to remember what it has done and what it has seen. Frontier vision-language models can plan such tasks without task-specific training, but how such a planner should remember an episode is an open question. We argue that its memory must keep track of what the planner has done and learned, and must let the planner choose what to look at, since every frame costs tokens; a note rewritten at every step meets both poorly. We propose LogLook, whose execution-grounded memory lets the system keep the record, an execution log and the model's earlier notes, while the model decides where to look: frames are returned only where it expects a change, and demonstrations are skimmed before it inspects the parts it chooses. Measured with the benchmark's motion planner, LogLook matches the success of a note-only memory with 43% fewer tokens and less than half the images, and fails far fewer episodes through what it remembered or saw. With direct control, LogLook solves 93.5% of the RoboMME test set.

Figures & tables

Figure 2: Overview of execution-grounded memory. At every decision, GPT-6 Astra receives the observation, the instruction and the memory, and returns a note and an action, optionally with image points to watch; the motion planner or $\pi_{0.5}$ executes the action. The memory has two parts. The system keeps the record: one log line per executed action with the measured gripper state, and every note the model has written. The model chooses what to see: after an action only the recorded frames that changed near the watched points are returned, and a demonstration is skimmed before the model inspects the ranges it asks for. All snippets come from one MoveCube test episode: the observation, a verbatim (shortened) decision, the resulting log lines and notes, the three frames returned during the watched push, and the demonstration range the model inspected.

System	Decision maker	Executor (action)	Beyond RGB + proprio.	Trained on RoboMME
MemER, FrameSamp-Modul	learned memory policy	$\pi_{0.5}$ (joint actions)	–	memory and executor
GroundSG, oracle subgoals	simulator ground truth	$\pi_{0.5}$ (joint actions)	privileged subgoals	executor
Human [ 1 ]	human player	oracle skill interface	privileged skills	–
Ours, $\pi_{0.5}$ executor	GPT-6 Astra	$\pi_{0.5}$ (subgoal phrases)	depth, calibration, task notes	executor only (same checkpoint)
Ours, direct control	GPT-6 Astra	motion planner (waypoints)	depth, calibration, task notes	nothing

Table 1: What each compared system decides with, how its actions are executed, which inputs it receives beyond the RGB images and proprioception, and which parts are trained on RoboMME.

Figure 3: Execution-grounded memory against note-only memory, with everything else fixed. With either executor the two memories reach similar success: with the learned

\pi_{0.5}

executor, executor failures decide most episodes in which they disagree, and with the motion planner, motion-planning failures do. In both settings our memory sends the model far fewer tokens and images; in (c), the lighter part of each bar counts demonstration frames and the darker part frames from execution.

Executor (episodes)	Memory	Success (%)	Failures: planning / other	Ours wins / losses	Tokens / ep.	Calls / ep.
Motion planner (256)	execution-grounded	237/256 (92.6)	15 / 4	19 / 16	40.5k	7.3
Motion planner (256)	note-only	234/256 (91.4)	10 / 12	19 / 16	71.3k	9.2
$\pi_{0.5}$ (128)	execution-grounded	104/128 (81.2)	–	7 / 7	57.4k	11.5
$\pi_{0.5}$ (128)	note-only	104/128 (81.2)	–	7 / 7	118.6k	12.0

Table 2: Execution-grounded vs. note-only memory (same code, prompts, executor, task notes and budgets). Motion planner: test episodes 4–19 of every task;

\pi_{0.5}

: episodes 4–11. Failures are split into motion-planning failures and all other failures.

Figure 4: Two failures of the note-only memory, same episode under both memories (direct control). Quotes are verbatim excerpts of the planner’s notes. (a) MoveCube: in the demonstration the cube slides under the closed gripper. Our planner skims the demonstration, inspects frames 40–64 and 72–92 closely and reads a push; with the full demonstration at a fixed stride, the note-only memory reads a carry. (b) ButtonUnmask: the colours are visible only before the containers cover the cubes, and the later notes of the note-only memory no longer contain them.

Memory	Success	Failures: planning / other	Full wins / losses	Tokens	Images (exec. / demo)		Calls
All 16 tasks, 64 episodes
Ours (full)	56/64	8 / 0	–	50.3k	19	30	6.7
without execution log	58/64	3 / 3	6 / 8	40.0k	21	30	7.4
without note history	61/64	1 / 2	2 / 7	45.8k	23	29	8.0
without gated frames	58/64	4 / 2	5 / 7	61.1k	83	28	7.6
Note-only memory	53/64	4 / 7	10 / 7	92.6k	90	52	8.7

Table 3: Switching off one memory component at a time, motion planner, hard test episodes (7, 11, 15 and 19 of every task). Failures are split into motion-planning failures and all other failures. The skim-then-inspect demonstration viewer only matters for tasks with a demonstration and is evaluated on those. Tokens, images and calls are per episode; images are split into frames seen during execution (including the current front and wrist images) and demonstration frames. “Gated frames”: expectation-gated frames.

Figure 5: Success against tokens per episode on the hard episodes, for our memory, our memory with one component switched off, and the note-only memory, each with and without task notes. Success counts motion-planning failures as failures, which differ by a few episodes between configurations at random ( Table 3 splits the failures). The arrow marks replacing the note-only memory by ours; the dashed line the same without task notes. Skim-then-inspect only applies to tasks with a demonstration and is shown in Table 3 .

Figure 6: Fraction of frames within eight frames of a ground-truth demonstration event, for the frames the model asked to inspect and for frames drawn uniformly from the same skimmed segments. All direct-control test episodes with at least one closer look, in the five tasks whose demonstrations contain discrete object events.

Figure 7: Success on the RoboMME test set, averaged and per suite, for $\pi_{0.5}$ -based systems: MemER, oracle subgoals and our planner drive the same RoboMME $\pi_{0.5}$ checkpoint, while $\pi_{0.5}$ without memory and FrameSamp-Modul are separately trained $\pi_{0.5}$ policies. Baselines are the averages published by RoboMME [ 1 ] ; the dashed line is human performance, obtained with RoboMME’s oracle skill interface. SAM2Act+ and GroundSG with Qwen3-VL subgoals are listed in Table 4 . Below: frames from our episodes.

	Counting				Permanence				Reference				Imitation
Method	Bin Fill	Pick Xtimes	Swing Xtimes	Stop Cube	Video Umsk	Button Umsk	Video UmskS	Button UmskS	Pick HighL	Video Repick	Video PlcBtn	Video PlcOrd	Move Cube	Insert Peg	Pattern Lock	Route Stick	Avg.
Human	96.0	100.0	80.0	78.0	90.0	92.0	92.0	90.0	92.0	92.0	98.0	90.0	90.0	98.0	84.0	86.0	90.5
$\pi_{0.5}$ (no memory)	30.0	42.9	35.6	6.7	20.4	22.2	18.7	6.7	11.3	0.4	31.1	25.8	26.0	1.6	2.9	4.7	17.9
SAM2Act+	40.0	76.0	25.3	0.0	27.3	32.0	18.0	26.7	17.3	5.3	24.7	20.0	29.3	0.0	0.0	0.0	21.4
MemER	56.7	79.3	59.3	0.0	81.3	72.0	38.0	21.3	70.7	25.3	30.0	26.0	82.7	6.7	16.7	12.0	42.4
GroundSG, Qwen3-VL sg.	52.0	92.7	7.3	0.0	88.7	24.0	30.7	14.0	15.1	25.3	54.0	31.8	71.6	3.3	6.7	6.0	32.7

Table 4: Per-task success (%) on the RoboMME test set (50 episodes per task). Baselines as reported by RoboMME [ 1 ] ; bold: best per column excluding humans. ‡ Subgoals from simulator ground truth.

Planner	Memory	Success (%)	Ours wins / losses	Tokens / ep.	Calls / ep.
MemER (official run)	keyframes	341/800 (42.6)	327 / 13	–	–
FrameSamp (official run)	sampled frames	372/800 (46.5)	327 / 44	–	–
Oracle subgoals	simulator	658/800 (82.2)	59 / 62	–	–
Per-chunk planner	state note, rewritten	601/800 (75.1)	89 / 35	203k	22.8
Ours	execution-grounded	655/800 (81.9)	–	51k	10.5

Table 5:

\pi_{0.5}

-based systems on all 800 test episodes. MemER, oracle subgoals, the per-chunk planner and ours drive the same RoboMME

\pi_{0.5}

checkpoint; FrameSamp-Modul is a separately trained

\pi_{0.5}

policy. “Official run”: per-episode results released with RoboMME, which can differ slightly from the published averages in Table 4 . “Oracle subgoals”: subgoals from simulator ground truth, our run of RoboMME’s evaluation code (RoboMME reports 84.1%). Wins and losses count the episodes in which only one of the two planners succeeds.

Figure 8: Our planner with direct control on all 800 RoboMME test episodes, averaged and per suite, next to the human performance reported by RoboMME, which was obtained with the oracle skill interface. Shown separately from Figure 7 because the executor differs. Below: frames from our episodes.

Task notes	Success (%)	Failures: planning / other	Full notes wins / losses	Calls / ep.
Ours, full notes	237/256 (92.6)	15 / 4	–	7.3
Without completion criteria	236/256 (92.2)	9 / 11	14 / 13	8.5
Without task notes	228/256 (89.1)	13 / 15	19 / 10	8.7

Table 6: Task notes with direct control, test episodes 4–19 of every task. Removing the completion criteria keeps the description of each task; removing the notes leaves only the instruction. Wins and losses count the episodes in which only one of the two settings succeeds.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	$\pi_{0.5}$ executor, 8 episodes per task					Motion planner, 16 ep.
Task	MemER	FrameSamp	Oracle sg.	Per-chunk	Ours	Exec.-gr.	Note-only
BinFill	4	4	6	6	7	16	15
PickXtimes	5	6	8	8	8	15	14
SwingXtimes	4	8	8	8	8	14	16
StopCube	0	2	4	1	3	13	15
VideoUnmask	7	3	7	8	8	16	16

Appendix

Table 7: Per-task successes on the subsets used in the memory swap ( Table 2 ); bold marks our planner, not the best entry. Left: learned

\pi_{0.5}

executor, test episodes 4–11 (MemER and FrameSamp: official runs; “Per-chunk”: the per-chunk planner). Right: memory swap with direct control, test episodes 4–19.

Memory swap, motion planner, test episodes 4–19
Difficulty	MemER	FrameSamp-Modul	Oracle subgoals	Ours, direct control
Easy	201/416 (48.3)	241/416 (57.9)	365/416 (87.7)	384/416 (92.3)
Medium	90/192 (46.9)	83/192 (43.2)	153/192 (79.7)	187/192 (97.4)
Hard	50/192 (26.0)	48/192 (25.0)	140/192 (72.9)	177/192 (92.2)
Difficulty	Note-only memory		Execution-grounded
Easy	119/128		118/128

Appendix

Table 8: Success by difficulty level. Top: all 800 test episodes; MemER and FrameSamp-Modul from the official per-episode runs released with RoboMME; oracle subgoals from our run of RoboMME’s evaluation code. Bottom: the memory swap with the motion planner.

Episode (task, test ep.)	Succeeds with	Cause of the failure
Motion planner, test episodes 4–19
BinFill 7	execution-grounded	count drift: a deposited cube is not counted
ButtonUnmaskSwap 19	execution-grounded	fact dropped: swap tracking lost in a rewrite
ButtonUnmaskSwap 5	execution-grounded	motion planning failure of the other memory (no path found; counted as a failure, as in RoboMME)
ButtonUnmaskSwap 6	execution-grounded	motion planning failure of the other memory (no path found; counted as a failure, as in RoboMME)
ButtonUnmaskSwap 9	execution-grounded	motion planning failure of the other memory (no path found; counted as a failure, as in RoboMME)

Appendix

Table 9: Every episode of the memory swap in which exactly one memory succeeds, with the cause of the other memory’s failure, read from the logs and checked against the executed-step feedback. A motion-planning failure ends the episode as a failure, as in RoboMME. “Replay”: the logged commands re-executed without model calls.

Figure 9: The 16 RoboMME tasks. For each task, the left frame shows what has to be remembered (a frame of the episode’s demonstration video, or the scene at the first decision for tasks without a video), and the right frame is taken from a successful episode of our planner with direct control. The line below names what the planner has to keep track of.

Figure 10: Selected frames and notes from one episode of our planner with direct control (ButtonUnmaskSwap, test episode 8, success). Top: front frames at the given environment steps, from an exact replay of the logged commands; the green circle marks the container that the model’s note says hides the green cube. Bottom: four of the nine notes, quoted verbatim (positions in the 512-pixel image). The model watched the containers during each move; after the six watched actions the system returned 14, 7, 6, 7, 8 and 9 of the 79 recorded frames.

Figure 11: Examples of the failure causes in Table 9 . Each panel shows two frames of the failed episode (d $N$ : the scene at the $N$ -th decision; demo: a frame of the demonstration) and an excerpt of the planner’s note, quoted verbatim. Top: the note-only memory with direct control, in episodes that our memory solves. Bottom: a failure of the learned executor, and two failures of our planner with direct control.

Parameter	Value
Model and inputs
Planner model	GPT-6 Astra through its API, reasoning effort “high”
Maximum output per call	6000 tokens
Images at each decision	front and wrist camera, $512\times 512$
Demonstration frames	$256\times 256$
Budgets per episode

Appendix

Table 10: Implementation details. All values are identical for every task; the note-only memory differs from ours only in the last block.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 159 papers

Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?
Oct 12, 2026Jack B. Jedlicki, Amish Sethi, Heng YangRobotic Manipulation Long-Horizon Robotic Manipulation
Harvard University
Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs
Sep 28, 2026Tanguy Dieudonné, Jack B. Jedlicki, Heng YangMemory-Augmented VLMs Long-Horizon Robotic Manipulation
Harvard University · ETH Zürich
Memory for Attention: Language-Conditioned Re-Perception with a Vision--Language--Motion Map
Jul 26, 2026Dibyendu GhoshActive Perception Visual Attention
Dibyendu Ghosh

Oct 12, 2026J/K move · Enter open · S save

Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?

Jack B. Jedlicki, Amish Sethi, Heng Yang

Harvard University

Robotic Manipulation Long-Horizon Robotic Manipulation VLMs for Robotics ICL for Robotics

Open paper AskPDF arXiv

We use a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy on RoboMME, a benchmark of 16 memory-dependent manipulation tasks spanning temporal, spatial, object, and procedural memory. The model keeps the whole episode in context and outputs every end-effector waypoint and gripper command, with no RoboMME-specific training. On the 800 test episodes it succeeds in 92.1%, above the previously reported human score of 90.5%, which was obtained with an oracle skill interface, and far ahead of the best leaderboard entry (60.8%) and of a hierarchical system that pairs the same model with a RoboMME-trained policy (79.1%). It matches or exceeds the human score on 13 of the 16 tasks. Its 63 failures concentrate in two tasks and are mostly premature completions and mistimed or unintended button presses. These results blur the usual division between long-horizon planning, memory and end-effector control, and raise the question of how much of this stack a single general-purpose model can take over in embodied systems. Project website: https://embodied-astra.github.io. Code, prompts and all episode logs: https://github.com/jackbjed/embodied-astra-robomme.

Abstract

Memory-Augmented VLMs

Explore similar work

Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

Memory for Attention: Language-Conditioned Re-Perception with a Vision--Language--Motion Map