LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation

Jingjing Wang; Zhengdong Hong; Chong Bao; Yuke Zhu; Junhan Sun; Guofeng Zhang

cs.CVApr 9, 2026

LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation

Authors: Jingjing Wang, Zhengdong Hong, Chong Bao, Yuke Zhu, Junhan Sun, Guofeng Zhang

Organizations: State Key Lab of CAD&CG, Zhejiang University

Abstract

Human-like generalization in open-world remains a fundamental challenge for robotic manipulation. Existing learning-based methods, including reinforcement learning, imitation learning, and vision-language-action-models (VLAs), often struggle with novel tasks and unseen environments. Another promising direction is to explore generalizable representations that capture fine-grained spatial and geometric relations for open-world manipulation. While large-language-model (LLMs) and vision-language-model (VLMs) provide strong semantic reasoning based on language or annotated 2D representations, their limited 3D awareness restricts their applicability to fine-grained manipulation. To address this, we propose LAMP, which lifts image-editing as 3D priors to extract inter-object 3D transformations as continuous, geometry-aware representations. Our key insight is that image-editing inherently encodes rich 2D spatial cues, and lifting these implicit cues into 3D transformations provides fine-grained and accurate guidance for open-world manipulation. Extensive experiments demonstrate that \codename delivers precise 3D transformations and achieves strong zero-shot generalization in open-world manipulation. Project page: https://zju3dv.github.io/LAMP/.

Figures & tables

Figure 2: Overview. Given the RGB-D observation and a language instruction, the Image-editing generates an edited state, which is used for registration to extract the inter-object transformation in reasoning stage. This transformation is converted into target pose for execution.

Figure 3: Illustration of the 2D-3D hierarchical point-cloud filtering . Colorful points in block (c) and (d-e) represent $\mathcal{P_{\text{obs}}}$ and $\mathcal{P_{\text{edit}}}$ with DINO features visualized via PCA, respectively. (a) Task : observed (top) and edited (bottom) images for stamping and insertion . (b) Spatial space : flying-edge points (gray boxes) of the stamp and vase are spatially proximal to valid points (orange boxes). (c) Feature space : flying-edge points (gray boxes) are distant from valid points (orange boxes) with similar PCA colors. (d) Spatial clustering : it fails when the stamp in horizontal or the stick is misaligned with the vase opening. (e) Hierarchical filtering : it successfully removes flying-edge points and recovers the correct spatial alignment.

Figure 4: Illustration of scale alignment . Colorful points are $\mathcal{P_{\text{edit}}}$ with DINO features after PCA, while green points are $\mathcal{P_{\text{obs}}}$ . (a) : Observation and edited image of task “cover the lid onto the holder”. (b) Without alignment : the two parts (lid and holder) drift apart in the world frame when transforming back under different scale from the edited frame. (c) With alignment : enforcing a consistent scale maintains the stable spatial relationship between the parts when transformed back to the world frame.

Figure 5: Edited-informed grasping . (a) Candidate grasps (blue) generated by AnyGrasp on the observed point cloud of the pencil. (b) Transformed grasps (blue) derived from the candidate set using the edit-informed transformation. (c) Collision convex hull (gray mesh) of the holder. (d) Filtered grasps : red grasps indicate collisions with the holder, while green grasps denote valid task-specific candidates.

Figure 6: Mesh of objects scanned by AR-Code App.

Figure 7: Qualitative results of point cloud registration across diverse manipulation tasks. LAMP consistently aligns active and passive objects under various task configurations, showcasing strong generalization and robustness to noisy, partial real-world point clouds.

Tasks	Lid covering		Toast insertion		Block assembly		Tea pouring		Drawer opening
Tasks	2BY2	Ours	2BY2	Ours	2BY2	Ours	2BY2	Ours	2BY2	Ours
RMSE(t) $\downarrow$	0.091	0.003	0.095	0.015	/	0.005	/	0.014	/	0.017
RMSE(R) $\downarrow$	16.54	8.736	35.12	11.10	/	30.05	/	21.53	/	2.614

Table 1: Quantitative results of point cloud registration.

Tasks	VoxPoser	CoPa	Rekep	Ours
Egg placing	2/10	2/10	4/10	6/10
Coin insertion	0/10	0/10	0/10	5/10
Pencil insertion	0/10	4/10	3/10	7/10
Toast insertion	0/10	0/10	0/10	6/10
Lid covering	0/10	3/10	4/10	8/10
Pen-cap covering	0/10	1/10	2/10	6/10

Table 2: Success rate of 13 real-world manipulation tasks. ‘/’ indicates the method is not applicable for that task.

Figure 8: Qualitative comparison of different manipulation representations. The blue to orange arrows indicate the target manipulation pose. Voxposer [ 38 ] grounds manipulation at the center of the object, ReKep [ 37 ] uses keypoints, CoPa [ 35 ] uses keypoints and vectors, and our approaches uses a full 3D inter-object transformation.

Figure 9: Qualitative results on real-world insertion tasks ( Toast, Coin ). Voxposer [ 38 ] fails to infer rotations; ReKep [ 37 ] misidentifies keypoints and rotations; CoPa [ 35 ] cannot reliably capture vector constraints; our method recovers precise inter-object 3D transformations.

Figure 10: Qualitative analysis of articulated manipulation.

Figure 11: Example rollouts of long-horizon manipulation tasks. Bottom right corner shows the edited prior for each step.

Figure 12: Qualitative analysis of camera‑viewpoint effects.

View- point	ReKep	CoPa	Ours (wo filter)	Ours (wo scale)	Ours (full)
$0^{\circ}$	1/10	1/10	2/10	3/10	6/10
$45^{\circ}$	3/10	4/10	6/10	3/10	8/10
$90^{\circ}$	2/10	6/10	7/10	4/10	8/10

Table 3: Quantative results of viewpoint influence ( ring stacking ).

Egg placing	move the egg onto the green holder
Coin insertion	insert the coin into the piggy bank
Pencil insertion	insert the pencil into the holder
Toast insertion	insert the toast into the toaster
Lid covering	move the lid onto the teapot
Pen-cap covering	cover the pen with the pen cap
Tea pouring	teapot pours into the cup

Table 4: A list of 13 open-world manipulation tasks . We provide the prompt used to generate the edited image in our experiment.

Figure 13: More visualization results. The first and second columns show the original observation and the edited state. The third and fourth columns show the registered point clouds in the edited frame and the world frame. The colored point clouds are $\mathcal{P}^{a/p}_{\text{edit}}$ after PCA. The last column shows the filtered grasp and the transformed active object.

Figure 14: Visualization of closed-loop execution rollout.

Tasks	Ours(QWen)	Ours(Gemini)
Egg placing	6/10	5/10
Toast insertion	1/10	6/10
Pen-cap covering	1/10	6/10
Toast cutting	8/10	5/10
Ring stacking	8/10	3/10

Table 5: Ablation on open-world manipulation between different image editing models.

Tasks	Lid covering	Pencil Insertion	Pen-cap covering	Ring stacking	Drawer closing
GoalVLA	3/10	1/10	0/10	4/10	1/10
Ours(LAMP)	8/10	7/10	6/10	8/10	7/10

Table 6: Real-world comparison with GoalVLA [ 13 ] . Their neglect of scale consistency between active and passive objects throughout the pipeline results in significant spatial offsets. Consequently, their approach suffers from a remarkably low success rate in precision-demanding tasks such as fine-grained manipulation.

Figure 17: Comparison of Alignment with GoalVLA. While GoalVLA [ 13 ] treats the passive object (e.g., the gray holder) as a static reference and aligns the active object (pencil) using an independent scale $s^{\prime}$ , this decoupling fails to account for global scene consistency. As illustrated in the right figure, such independent scaling distorts the relative spatial relationship between the two objects when transformed back to world coordinates. In contrast, our method enforces a unified scale across both the pencil and holder, strictly preserving their spatial configuration and ensuring the pencil remains correctly centered within the holder in 3D space.

Figure 18: Robustness comparison of viewpoint variation. GoalVLA (row 2) exhibits noticeable scene shifts relative to the observation point cloud (dark) under different perspectives, our method (row 3) achieves stable 3D alignment. This demonstrates that our 3D-based registration is invariant to camera viewpoint changes, whereas 2D-based scale estimation is highly sensitive to perspective distortion.

Figure 19: Comparison of edited manipulation state with different editing models.

Figure 20: Comparison between edited-image priors and video-generation priors for long-horizon manipulation. Edited-image priors provide stronger semantic adherence with better subject and background consistency.

Generalization in Robotic Manipulation

Generalization in robotic manipulation studies how robots transfer grasping and interaction skills to new objects, scenes, tasks, and embodiments.

See all 343 papers

Building Explicit World Model for Zero-Shot Open-World Object Manipulation
Mar 14, 2026Xiaotong Li, Gang Chen, Javier Alonso-MoraZero-Shot Robotic Manipulation Robotic Manipulation
Department of Cognitive Robotics (CoR), Delft University of Technology, 2628CD Delft, The Netherlands.
ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation
Mar 14, 2026You Wu, Zixuan Chen, Cunxu Ou +9Robotic Manipulation VLMs for Robotics
School of Computer Science, Nanjing University · School of Intelligence Science and Technology, Nanjing University · School of Electronic Science and Engineering, Nanjing University +2
3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints
Sep 8, 2026Ziqin Huang, Yingyue Li, Chenyangguang Zhang +6Generalization in Robotic Manipulation Trajectory Representation Learning
Tsinghua University · ETH Zürich · University of California, Berkeley

Mar 14, 2026J/K move · Enter open · S save

Building Explicit World Model for Zero-Shot Open-World Object Manipulation

Xiaotong Li, Gang Chen, Javier Alonso-Mora

Department of Cognitive Robotics (CoR), Delft University of Technology, 2628CD Delft, The Netherlands.

Zero-Shot Robotic Manipulation Robotic Manipulation World Models for Robotics

Open paper AskPDF arXiv

Open-world object manipulation remains a fundamental challenge in robotics. While Vision-Language-Action (VLA) models have demonstrated promising results, they rely heavily on large-scale robot action demonstrations, which are costly to collect and can hinder out-of-distribution generalization. In this paper, we propose an explicit-world-model-based framework for open-world manipulation that achieves zero-shot generalization by constructing a physically grounded digital twin of the environment. The framework integrates open-set perception, digital-twin reconstruction, sampling and evaluation of interaction strategies. By constructing a digital twin of the environment, our approach efficiently explores and evaluates manipulation strategies in physic-enabled simulator and reliably deploys the chosen strategy to the real world. Experimentally, the proposed framework is able to perform multiple open-set manipulation tasks without any task-specific action demonstrations, proving strong zero-shot generalization on both the task and object levels. Project Page: https://bojack-bj.github.io/projects/thesis/

Abstract

Generalization in Robotic Manipulation

Explore similar work

Building Explicit World Model for Zero-Shot Open-World Object Manipulation

ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation

3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints