Robots manipulating in cluttered spaces must often plan from partial views where foreground objects occlude the target, surrounding geometry, and usable free space. Reactive policies often miss these hidden constraints, while exhaustive search becomes computationally prohibitive in dense clutter. We present SCOUT, an occlusion-aware planning framework that bridges high-level deliberation with grounded physical rollouts. SCOUT leverages an object-centric scene representation to reconstruct hidden geometry and derive a relational planning state capturing blocker hierarchies and accessibility corridors. To navigate the vast action space in clutter, an LLM proposes structured action candidates which are then rigorously evaluated via Monte Carlo Tree Search (MCTS) using a learned action-conditioned world model. Evaluations on 'Reveal' and 'Placement' tasks show that SCOUT outperforms Vision-Language-Action (VLA) policies, embodied agents, and 3D world models, with significant gains in long-horizon tasks requiring reasoning over occluded geometry. We further validate the framework's robustness through real-world robotic manipulation in a cluttered shelf environment.
Figures & tables
Figure 1: Overview of SCOUT. From partial RGB-D observations, GHOST reconstructs an object-centric hidden scene representation, which is converted into a relational planning state. An LLM generates structured action proposals, and MCTS with a learned world model evaluates simulated rollouts to guide selection of a multi-step plan.
Figure 2: GHOST scene representation pipeline. Multi-view RGB-D observations are fused into object-centric slots, refined via slot interactions, and rendered into a unified 3D scene representation.
Figure 3: Relational planning state derived from object-centric slots.
Figure 4: WM-Guided Planning: Starting from the reconstructed scene state, future states are recursively predicted by the action-conditioned world model during MCTS rollouts, and the resulting summaries guide final action selection.
Figure 5: Representative examples from the four task families used for evaluation.
Figure 6: Results on the simulated shelf task suite. Success rate (in %) is averaged across 3 seeds and uncertainty is given by the standard error. Scout consistently outperforms the adapted baselines across in-distribution, novel-object, and novel-view settings.
Task
Steps
Δ Visibility
Δ Access.
Invalid Rate
Reveal
6.3 ± 1.4
+0.46 ± 0.07
+0.41 ± 0.06
3.8%
Reveal_Multi
8.7 ± 1.8
+0.59 ± 0.09
+0.53 ± 0.08
5.9%
Place
6.0 ± 1.2
+0.34 ± 0.06
+0.37 ± 0.05
3.2%
Place_Multi
9.1 ± 2.0
+0.51 ± 0.08
+0.49 ± 0.07
6.4%
Table 1: Trajectory rollout metrics.
Figure 7: Failure analysis of Scout . (a) Failure decomposition as percentages of all rollouts. Most failures stem from hidden-scene reconstruction errors, followed by transition-prediction and geometric-validity errors. (b) Transition-prediction failure: the selected action does not create enough free space to retrieve the target. (c) Hidden-scene failure: a small back-row target remains too occluded to be reconstructed from partial observations. (d) Geometric-validity failure: dense clutter leaves no feasible retrieval corridor under the current primitive set.
Figure 8: Success rate as clutter density increases.
Variant
Avg. Succ. (%)
Avg. Act.
Single-view input
69.7 ± 4.6
7.9 ± 0.5
w/o relational projection
68.1 ± 4.4
8.2 ± 0.5
w/o rollout search
70.6 ± 4.2
6.8 ± 0.4
Full Scout
80.8 ± 3.5
7.5 ± 0.4
Table 2: Ablation study of Scout components.
Figure 9: Real-world rollouts of Scout on the Reveal and Place tasks in cluttered shelf scenes.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Targets
Success condition
Reveal
1
Target visibility and accessibility exceed threshold
Reveal_Multi
2
Both targets satisfy Reveal predicate
Place
1
Target object lies inside assigned goal region
Place_Multi
2
Both target objects lie inside the shared goal region
Appendix
Table 3: Task construction details for the simulated shelf benchmark.
Parameter
Values / Range
Candidate views
16
Selected views
up to 5
Distance from shelf opening
0.34 – 0.46 m
Yaw
{−20,−12,0,12,20}∘
Pitch
{0,−8}∘
Lateral offset
{−0.05,0.0,0.05} m
Appendix
Table 4: Camera sampling ranges used for simulated RGB-D observations.
Figure 10: Example sparse RGB-D observations used by GHOST. Each row shows a different front-facing view of the same cluttered shelf scene, with corresponding RGB image, depth map, and object segmentation mask.
Primitive
Axis
Sign
Termination
shift_left
y
−1
wall contact
shift_right
y
+1
wall contact
shift_back
x
+1
wall contact
pull_forward
x
−1
front boundary
Appendix
Table 5: Discrete manipulation primitives used by Scout .
Method
Initial views only
Re-observation
π0.5
55.5
63.2±4.1
Long-VLA
52.3
64.0±4.0
ParticleFormer
72.8
82.3±3.9
Scout
80.8±3.5
86.3±3.0
Appendix
Table 6: Simulation success (%) with initial observations only and re-observation after every primitive.
Figure 11: Real-world shelf observation setup. Left: Spot observes the cluttered shelf using its wrist-mounted camera. Right: representative robot-captured RGB views and corresponding depth maps used for scene reconstruction.
Robotic manipulation in cluttered environments presents a critical challenge for automation. Recent large-scale, end-to-end models demonstrate impressive capabilities but often lack the data efficiency and modularity required for retrieving objects in dense clutter. In this work, we argue for a paradigm of specialized, decoupled systems and present Unveiler, a framework that explicitly separates high-level spatial reasoning from low-level action execution. Unveiler's core is a lightweight, transformer-based Spatial Relationship Encoder (SRE) that sequentially identifies the most critical obstacle for removal. This discrete decision is then passed to a rotation-invariant Action Decoder for execution. We demonstrate that this decoupled architecture is not only more computationally efficient in terms of parameter count and inference time, but also significantly outperforms both classic end-to-end policies and modern, large-model-based baselines in retrieving targets from dense clutter. The SRE is trained in two stages: imitation learning from heuristic demonstrations provides sample-efficient initialization, after which PPO fine-tuning enables the policy to discover removal strategies that surpass the heuristic in dense clutter. Our results, achieving up to 97.6% success in partially occluded and 90.0% in fully occluded scenarios in simulation, make a case for the power of specialized, object-centric reasoning in complex manipulation tasks. Additionally, we demonstrate that the SRE's spatial reasoning transfers zero-shot to real scenes, and validate the full system on a physical robot requiring only geometric workspace calibration; no learned components are retrained.
Vision-Language-Action (VLA) models have demonstrated strong performance in robotic manipulation, yet most rely primarily on 2D visual observations and language instructions, leaving scene geometry and obstacle relationships to be inferred implicitly. This limitation is particularly problematic in cluttered environments, where small spatial errors can lead to unintended contacts or collisions. We propose MOSAIC, a modular spatial- and obstacle-aware framework that can be integrated with pretrained VLA policies to improve object-level scene understanding and action generation. Our method augments the VLA backbone with an auxiliary semantic grounding task that predicts role-specific masks for the target object, destination region, and surrounding obstacles. We further combine the target and destination masks with depth estimated from 3D scene reconstruction to construct role-specific geometric tokens that are injected directly into the action expert. In addition, we propose obstacle-aware attention supervision that emphasizes obstacles near the robot body during task execution. The resulting attention features are subsequently injected into later action-expert layers to further guide action generation. Together, these components provide explicit semantic, geometric, and obstacle-aware cues while preserving the underlying pretrained VLA architecture. We evaluate our framework on pretrained VLA model using the RoboPRO benchmark, including visual perturbation tests, and further validate it on real-world manipulation tasks. MOSAIC improves the success rate by 5.1% in cluttered scenes, and by 9.7% under visual perturbations, with gains of up to 25.0% in the most challenging environment. We will release our code and dataset upon acceptance at the following Github page: https://mmahdavian.github.io/MOSAIC/
Richard Shi, Mohammad Mahdavian, Mehrdad Momeni Zadeh +2
Huawei Noah’s Ark Lab · University of Toronto · Simon Fraser University
Robots operating in cluttered environments must often manipulate objects whose locations are only partially observable. A central challenge is deciding whether to acquire another observation or to first manipulate objects that may occlude the target. Conventional task and motion planning (TAMP) approaches typically make this decision using symbolic action costs or expensive geometric planning, neither of which adequately captures how likely an observation is to reveal an occluded target. We introduce Imagine-TAMP, an interleaved planning and execution framework that uses semantic and geometric imagination to compare alternative task-level strategies under partial observability before committing to expensive motion planning. A vision-language model shapes a particle belief over target locations using commonsense relationships between the target and visible objects, while a generative scene model estimates plausible geometry in unobserved regions. Given a target hypothesis and imagined scene, Imagine-TAMP generates multiple symbolic plan skeletons and assigns non-unit costs that approximate both manipulation effort and target visibility from sensing actions, distinguishing a short but poorly informative observation strategy from a longer strategy that first manipulates an occluder to better expose the target. The selected skeleton is then refined into a feasible continuous plan and executed, with new observations updating the belief and triggering replanning when necessary. Experiments show that imagination-guided evaluation improves observation-versus-manipulation decisions: in viewpoint-constrained shelf scenes, non-unit geometric evaluation increases success from 46.0% to 84.0%, while semantic belief shaping further reduces manipulation and replanning. On a real robot, the complete system reduces planning time by 32% relative to a geometry-only ablation.
Antareep Singha, Shivaram Kumar, Yoonwoo Kim +1
Nanyang Technological University · The University of Texas at Austin