3D-Anchored Lookahead Planning for Persistent Robotic Scene Memory via World-Model-Based MCTS

Bronislav Sidik; Dror Mizrahi

cs.ROApr 13, 2026

3D-Anchored Lookahead Planning for Persistent Robotic Scene Memory via World-Model-Based MCTS

Authors: Bronislav Sidik, Dror Mizrahi

Organizations: Huawei Technologies, Israel R&D Center

Abstract

We present 3D-Anchored Lookahead Planning (3D-ALP), a System 2 reasoning engine for robotic manipulation that combines Monte Carlo Tree Search (MCTS) with a 3D-consistent world model as the rollout oracle. Unlike reactive policies that evaluate actions from the current camera frame only, 3D-ALP maintains a persistent camera-to-world (c2w) anchor that survives occlusion, enabling accurate replanning to object positions that are no longer directly observable. On a 5-step sequential reach task requiring spatial memory (Experiment E3), 3D-ALP achieves 0.650 0.109 success rate on memory-required steps versus 0.006 0.008 for a greedy reactive baseline (Δ=+0.645), while step 5 success reaches 0.822 against 0.000 for greedy. An ablation study (30 episodes, 3 seeds) isolates tree search spatial memory as the primary driver (+0.533, 82% of gain) with additional benefit from deeper lookahead (+0.111, 17%). We also identify and resolve four structural failure modes in applying UCT-MCTS (Upper Confidence Bounds applied to Trees [10]) to continuous robotic manipulation.

Figures & tables

Figure 1: 3D-ALP overview. Left: The memory gap between reactive policies and 3D-ALP on occluded-object tasks. Greedy reactive agents collapse to 0.6% SR; 3D-ALP’s persistent SE(3) anchor enables 82.2% SR on the hardest chained-memory step. Right: The 3D-ALP architecture: a persistent SE(3) spatial anchor updated via FK, a world-model-based MCTS tree that imagines future frames, and a hybrid geometric-semantic scorer that corrects VLM depth blindness. The line chart confirms the performance gap is concentrated on memory-required steps 4–5, where the greedy baseline collapses to zero.

Variant	Non-Mem SR	Memory SR	$\boldsymbol{\Delta}$	Step 5
Greedy	$0.748\pm 0.029$	$0.006\pm 0.008$	—	$0.000$
MCTS $D{=}1$	$0.389\pm 0.024$	$0.539\pm 0.064$	$+0.533$	$0.622$
3D-ALP $D{=}2$	$0.463\pm 0.064$	$\mathbf{0.650\pm 0.109}$	$\mathbf{+0.645}$	$\mathbf{0.822}$

Table 1: Ablation study: 30 episodes

\times

3 seeds (mean

\pm

std). Non-Mem SR = steps 1–3; Memory SR = steps 4–5.

\Delta

is vs. greedy. Bold = best.

Figure 2: 3D-ALP qualitative results. Top row: MCTS planning tree at steps 1, 3, and 5 ( $\bigstar$ ). Each node is a candidate c2w pose; colour encodes Q-value (blue=low, teal=high); edge width = visit count; gold border = selected action. At step 5 ( $\bigstar$ ), the planner navigates to a position that is no longer visible in the current frame — the c2w anchor retains its coordinates in the persistent tree. Bottom left: 3D EE trajectory; $\bigstar$ = memory steps 4–5; red crosses = target positions (occluded at steps 4–5). Bottom right: Per-step SR; gold shading = memory-required steps where greedy collapses to zero while 3D-ALP maintains 60–83% SR via persistent 3D spatial memory.

Robotic Spatial Memory

See all 21 papers

Memory as Plans: World-Action Modeling with Memory-Grounded Planning
Sep 10, 2026Sizhe Zhao, Haozhe Xie, Weiyu Zhao +5Long-Horizon Robotic Manipulation World Models for Robotics
Harbin Institute of Technology, China · Nanyang Technological University, Singapore · Shandong University, China +1
GaussMemory: Task-Driven 3D Gaussian Scene Memory for Long-Horizon Robotic Manipulation
Aug 15, 2026Zhiqiang Hu, Shouren Huang, Masatoshi IshikawaRobotic Spatial Memory Memory-Augmented Agents
Research Institute for Science & Technology, Tokyo University of Science
RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics
Mar 13, 2026Yuzhi Huang, Jie Wu, Weijue Bu +9Long-Horizon Robotic Manipulation Robot Task Planning
Shenzhen International Graduate School, Tsinghua University · YuanxingGuangnian Robotics · China University of Mining and Technology +3

Sep 10, 2026J/K move · Enter open · S save

Memory as Plans: World-Action Modeling with Memory-Grounded Planning

Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang+4

Harbin Institute of Technology, China · Nanyang Technological University, Singapore · Shandong University, China+1

Long-Horizon Robotic Manipulation World Models for Robotics Robot Task Planning Memory-Augmented Agents

Open paper AskPDF arXiv

Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows.

Abstract

Robotic Spatial Memory

Explore similar work

Memory as Plans: World-Action Modeling with Memory-Grounded Planning

GaussMemory: Task-Driven 3D Gaussian Scene Memory for Long-Horizon Robotic Manipulation

RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics