Building Explicit World Model for Zero-Shot Open-World Object Manipulation

Xiaotong Li; Gang Chen; Javier Alonso-Mora

cs.ROMar 14, 2026

Building Explicit World Model for Zero-Shot Open-World Object Manipulation

Authors: Xiaotong Li, Gang Chen, Javier Alonso-Mora

Organizations: Department of Cognitive Robotics (CoR), Delft University of Technology, 2628CD Delft, The Netherlands.

Abstract

Open-world object manipulation remains a fundamental challenge in robotics. While Vision-Language-Action (VLA) models have demonstrated promising results, they rely heavily on large-scale robot action demonstrations, which are costly to collect and can hinder out-of-distribution generalization. In this paper, we propose an explicit-world-model-based framework for open-world manipulation that achieves zero-shot generalization by constructing a physically grounded digital twin of the environment. The framework integrates open-set perception, digital-twin reconstruction, sampling and evaluation of interaction strategies. By constructing a digital twin of the environment, our approach efficiently explores and evaluates manipulation strategies in physic-enabled simulator and reliably deploys the chosen strategy to the real world. Experimentally, the proposed framework is able to perform multiple open-set manipulation tasks without any task-specific action demonstrations, proving strong zero-shot generalization on both the task and object levels. Project Page: https://bojack-bj.github.io/projects/thesis/

Figures & tables

Fig. 1: Overview of our manipulation task framework with explicit world model construction. The system takes an RGB-D observation $I$ and a natural language task instruction $C$ as input and proceeds through four main stages: (A) Open-set segmentation , (B) Open-set object grasping , (C) Digital twin reconstruction , (D) Manipulation strategy sampling . The outcome is the most promising manipulation strategy, which is then executed on the real robot.

Fig. 2: The proposed digital twin construction module, containing the mesh generation and two-stage pose alignment. We first generate a textured mesh from the masked RGB image via Hunyuan3D 2.0 [ 24 ] . During coarse alignment, we render RGB and depth images from a set of hypopaper poses and compare their similarities with real-world observation in DINO [ 25 ] feature space, and select the one that best matches the real-world observation. The resulting coarse pose is then refined using RANSAC and ICP on the partial point cloud back-projected from the depth image.

Fig. 3: Our manipulation strategy sampling module. (a) Interaction area segmentation : We use GPT-4o and Grounded-SAM to segment the interaction area to constrain the sample space of translation. (b) Interaction strategy sampling : Different rotations of the dynamic object are sampled. The outcomes are simulated in the Isaac Sim. (c) Result checking : The results are rendered in the simulator. We query the GPT-4o again to check which samples fulfill the task requirements, and feed the results to a Gaussian Process classifier to get the success probabilities.

objects	valid sample	our method		direct alignment
		success rate ↑	RMSE ↓	success rate	RMSE
without grasp in the gripper
mug	11	90.91%	0.00457	27.27%	0.00390
box	5	100.00%	0.00564	100.00%	0.00684
foam box container	5	40.00%	0.00665	20.00%	0.00745
bottle	8	100.00%	0.00432	62.50%	0.00367

TABLE I: Comparison of mesh alignment performance between our two-stage alignment pipeline and direct alignment.

Fig. 4: Qualitative results of the proposed two-stage mesh alignment method. The left image shows the aligned meshes overlaid with the RGB-D observation. The right image illustrates the corresponding real-world setup.

Fig. 5: Representative tasks used for experimental validation. Frames with the same color represent the same task, where the top one denotes the initial scene and the bottom one denotes the final scene.

Task	Success Rate
Put the banana into the basket	83.3% (10/12)
Put the lemon into the white cup	75.0% (9/12)
Put the yellow cube into the white box	91.7% (11/12)
Put the yellow cube into the blue can	75.0% (9/12)
Stack the green cube onto the yellow cube	50.0% (6/12)
Put the long cutlery box on the two boxes, like a bridge	75.0% (9/12)

TABLE II: Task success rate of different manipulation scenarios.

Fig. 6: Sankey diagram illustrating the stage-wise success and failure flow across all 96 real-world trials. Each block represents one functional module in the proposed framework.

Fig. 7: Failure type distribution across all 9 tasks. Each cell represents the number of occurrences of a specific failure type for a given task.

Zero-Shot Robotic Manipulation

Zero-shot robotic manipulation enables robots to perform unfamiliar tasks and handle unseen objects, scenes, or embodiments without task-specific training.

See all 95 papers

LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation
Apr 9, 2026Jingjing Wang, Zhengdong Hong, Chong Bao +3Generalization in Robotic Manipulation Robotic Manipulation
State Key Lab of CAD&CG, Zhejiang University
World Action Models are Zero-shot Policies
Feb 17, 2026Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng +33World Action Models Zero-Shot Robotic Manipulation
NVIDIA
MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation
Mar 17, 2026Abhay Deshpande, Maya Guru, Rose Hendrix +23Sim-to-Real Transfer Zero-Shot Robotic Manipulation
Allen Institute for AI · Technische Universität Darmstadt · University of Washington +2

Apr 9, 2026J/K move · Enter open · S save

LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation

Jingjing Wang, Zhengdong Hong, Chong Bao, Yuke Zhu+2

State Key Lab of CAD&CG, Zhejiang University

Generalization in Robotic Manipulation Robotic Manipulation Zero-Shot Robotic Manipulation 3D Representation Learning

Open paper AskPDF arXiv

Human-like generalization in open-world remains a fundamental challenge for robotic manipulation. Existing learning-based methods, including reinforcement learning, imitation learning, and vision-language-action-models (VLAs), often struggle with novel tasks and unseen environments. Another promising direction is to explore generalizable representations that capture fine-grained spatial and geometric relations for open-world manipulation. While large-language-model (LLMs) and vision-language-model (VLMs) provide strong semantic reasoning based on language or annotated 2D representations, their limited 3D awareness restricts their applicability to fine-grained manipulation. To address this, we propose LAMP, which lifts image-editing as 3D priors to extract inter-object 3D transformations as continuous, geometry-aware representations. Our key insight is that image-editing inherently encodes rich 2D spatial cues, and lifting these implicit cues into 3D transformations provides fine-grained and accurate guidance for open-world manipulation. Extensive experiments demonstrate that \codename delivers precise 3D transformations and achieves strong zero-shot generalization in open-world manipulation. Project page: https://zju3dv.github.io/LAMP/.

Abstract

Zero-Shot Robotic Manipulation

Explore similar work

LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation

World Action Models are Zero-shot Policies

MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation