OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation

Jack B. Jedlicki; Tanguy Dieudonné; Heng Yang

cs.ROSep 23, 2026

OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation

Authors: Jack B. Jedlicki, Tanguy Dieudonné, Heng Yang

Organizations: Harvard University · ETH Zürich

Abstract

Long-horizon manipulation often requires reasoning about state absent from the current view, such as a vanished object's location, temporal identity, or the contents of a shuffled container. We present OCC4M ("Occam"), an object-centric 4D memory that maintains persistent tracks in a shared world frame and explicitly represents temporal, motion, and containment relations. A vision-language model (VLM) queries this structured memory to select actionable targets for history-free low-level execution. Across seven simulation conditions and 350 episodes, OCC4M achieves 96.6% memory success and 88.9% end-to-end success, versus 54.6% and 57.7% for FrameSamp, a raw-history VLM baseline using Gemini 3.7 Flash with the complete observation history and the same executor. In a controlled viewpoint-transfer test, OCC4M maintains 100% memory and 98% end-to-end success after a viewpoint change, while full-history FrameSamp falls to near-zero success. On 20 fixed-camera Franka episodes, OCC4M reaches 85% joint memory accuracy, versus at most 30% for FrameSamp across context sizes from $K=16$ to the complete history, and completes 45% of full two-stage tasks. These results support explicit object-centric memory for persistent spatiotemporal reasoning in long-horizon manipulation. Qualitative videos are available at https://occ4m-sup.github.io/occ4m-supplementary/.

Figures & tables

	Capability	Required decision
T1	Spatial persistence	Place the red cube at the vanished green cube’s last location.
T1c	Viewpoint transfer	After the target vanishes and the base moves $0.30\,\mathrm{m}$ , express its location in the current view.
T2	Temporal identity	Place the blue cube at the location of the marker that appeared first, and the red cube at the location of the marker that appeared second.
T3	Event history	Three cubes move sequentially; pick the second mover and place it at the first mover’s pre-motion pose.
T4	Relational persistence	After identical covers are shuffled, pick the cover containing the red cube.
T5	Cross-view association	Retrieve a vanished sign location after driving to a second table and back.

Table 1: Six task families and the viewpoint-transfer variant. T1c combines a new camera pose with a current-frame grounding requirement.

Method	T1	T1c	T2	T3	T4	T5	T6	Mean
A. Memory success
OCC4M	100	100	92	98	88	98	100	96.6
OCC4M + oracle actions	100	100	92	98	88	98	100	96.6
OCC4M + oracle memory	100	100	100	100	100	100	100	100.0
OCC4M + oracle memory+plan	100	100	100	100	100	100	100	100.0
FrameSamp ( $K{=}16$ , Gemini 3.7)	100	6	72	32	54	100	2	52.3

Table 2: Memory and end-to-end success (%) on

N=50

evaluation episodes per condition. Means average the seven task-condition rates. All learned-action rows use the same frozen executor. The principal raw-history baseline receives the complete history (budget

K=32

Figure 2: T1c viewpoint-transfer example. The green cube disappears, the base translates $0.30\,\mathrm{m}$ , and the robot must place the red cube at the green cube’s remembered world location from the new viewpoint. Yellow ring: OCC4M prediction; pink ring: FrameSamp (Gemini 3.7 Flash) prediction; green ring: ground-truth target.

Condition	FrameSamp	OCC4M
T1: historical-frame grounding ( $N=50$ )	100 / 96	100 / 94
T1b: current frame, fixed viewpoint ( $N=50$ )	100 / 92	100 / 94
T1c: current frame + $0.30\,\mathrm{m}$ base motion ( $N=50$ )	6 / 6	100 / 98

Table 3: Viewpoint-transfer control. Entries report memory / end-to-end success (%). T1b changes only the required output frame; T1c additionally moves the base after the target vanishes. FrameSamp uses Gemini 3.7 Flash with

K=16

, which covers the full history for T1/T1b/T1c.

Figure 3: Real-world compositional evaluation. The robot must first identify and pick the cup covering the red cube after the cups are shuffled, and then pick the cup occupying the white mug’s remembered initial location. The yellow rings are rendered by OCC4M into the visual input of the low-level policy to indicate the target.

Figure 4: Closed-loop identity-bound grounding under perturbation. After planning, the target cup is moved during execution. OCC4M preserves its identity and updates its location without re-querying the VLM, enabling a successful grasp at the new location. Magenta: memory estimate (not policy input); yellow: cue sent to $\pi_{0.5}$ at each action chunk.

	Memory accuracy			Physical execution
Method	S1	S2	Both	S1	S2	E2E
OCC4M	85%	95%	85%	75%	55%	45%
FrameSamp ( $K=16$ )	30%	95%	30%	–	–	–
FrameSamp ( $K=32$ )	25%	100%	25%	–	–	–
FrameSamp (all frames)	25%	90%	20%	–	–	–

Table 4: Real-world compositional manipulation. Memory accuracy measures selection of the correct target cup for each subtask. S1 denotes shuffled-containment recall and S2 denotes recall of the white mug’s original location; “Both” requires both memory decisions to be correct in the same episode. Physical execution success is reported only for OCC4M, and “E2E” requires successful completion of both subtasks. All methods are evaluated on the same

N=20

recorded episodes.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Varying demonstrations per task (20k steps)
Demonstrations	80	250	400	–
Mean success	8.3	5.0	9.3	–
Varying training duration (250 demonstrations/task)
Training steps	10k	20k	30k	40k
Mean success	1.7	5.0	3.3	3.3

Appendix

Table 5: Robustness of the multi-task FrameSamp+Modul policy to demonstration count and training duration. Entries are mean end-to-end success (%) on the six original task conditions (T1–T6, excluding T1c).

Task	Pick target (GT)	Place target (GT)	Additional constraints
T1	red cube (current)	remembered green-cube location	N/A
T1c	red cube (current)	same remembered location, expressed after base motion	place prediction uses final frame
T2	blue cube, red cube	blue $\rightarrow$ first marker; red $\rightarrow$ second marker	labels and order correct
T3	second cube that moved (current)	pre-motion location of first mover	pick is second mover
T4	current correct grey cover	N/A (pick-only)	correct cover among three, pick prediction uses final frame
T5	query-color cube on Table B	remembered query-color sign location on Table A	queried color correct

Appendix

Table 6: Task-specific definitions of the executor-independent memory metric.

Figure 5: T1: Place the red cube where the green cube was.

Figure 6: T1c: After the green cube disappears, the mobile base translates $0.30\,\mathrm{m}$ , and the robot must place the red cube at the green cube’s remembered location from the new viewpoint. Yellow ring: OCC4M prediction; pink ring: FrameSamp (Gemini 3.7 Flash) prediction; green ring: ground-truth target.

Figure 7: T2: Place the blue cube at the location of the marker that appeared first, and the red cube at the location of the marker that appeared second.

Figure 8: T3: Pick the second mover and place it at the first mover’s pre-motion location.

Figure 9: T4: Pick the grey cover containing the red cube after shuffling.

Figure 10: T5: Place the query-color cube at the remembered sign location on the first table.

Figure 11: T6: After recoloring, sort the cubes according to their original colors.

Explore similar work

Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action
May 21, 2026Pengteng Li, Weiyu Guo, He Zhang +4Vision-Language-Action Models Real-World Manipulation Tasks
MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models
Jun 8, 2026Hao Shi, Weiye Li, Bin Xie +6Vision-Language-Action Models Long-Horizon Planning
MemoryWAM: Efficient World Action Modeling with Persistent Memory
Jun 18, 2026Sizhe Yang, Juncheng Mu, Tianming Wei +8Efficient World-Action Model World Models

May 21, 2026 · cs.ROJ/K move · Enter open · S save

Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

Pengteng Li, Weiyu Guo, He Zhang, Tiefu Cai+3

Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou), China · AI2Robotics · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology Hong Kong SAR, China.

We introduce SOMA, the Spatial Memory framework for Out-of-Vision Manipulation in Vision-Language-Action (VLA) models. Most existing VLAs implicitly assume that task-relevant objects are always visible, leading to brittle and reactive behaviors when targets fall outside the camera's field of view. SOMA addresses this limitation by equipping VLAs with a persistent spatial memory constructed from multi-view observations acquired via a movable head camera, enabling reasoning beyond the current visual frustum. The framework consists of three components: Spatial Memory Construction, which aggregates angular-wise observations into a unified spatial-semantic representation through scanning; Dynamic Memory Refinement, which maintains global consistency over time; and Contextual Memory Retrieval, which activates instruction-relevant spatial cues during manipulation. We evaluate SOMA on five challenging real-world out-of-vision manipulation tasks, including multi-step and dual-arm scenarios where target objects are initially invisible. Experimental results show that SOMA not only improves task success rates, but also induces qualitatively different manipulation behaviors, with faster target localization, reduced viewpoint search, and near one-shot grasping under partial observability. Additional experiments on RoboCasa GR1 and SimplerEnv further validate the effectiveness of SOMA's memory design under conventional fully observable settings. Code will be released soon.

Vision-Language-Action Models Real-World Manipulation Tasks Spatial Memory Multiple Vision Tasks Manipulation

Open paper Ask about it PDF

Abstract

Explore similar work

Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models

MemoryWAM: Efficient World Action Modeling with Persistent Memory