PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing

Ruihang Xu; Dewei Zhou; Xiaolong Shen; Fan Ma; Yi Yang

cs.CVApr 8, 2026

PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing

Authors: Ruihang Xu, Dewei Zhou, Xiaolong Shen, Fan Ma, Yi Yang

Organizations: Zhejiang University, Hangzhou, China

Abstract

Achieving physically accurate object manipulation in image editing is essential for its potential applications in interactive world models. However, existing visual generative models often fail at precise spatial manipulation, resulting in incorrect scaling and positioning of objects. This limitation primarily stems from the lack of explicit mechanisms to incorporate 3D geometry and perspective projection. To achieve accurate manipulation, we develop PhyEdit, an image editing framework that leverages explicit geometric simulation as contextual 3D-aware visual guidance. By combining this plug-and-play 3D prior with joint 2D--3D supervision, our method effectively improves physical accuracy and manipulation consistency. To support this method and evaluate performance, we present a real-world dataset, RealManip-40K, for 3D-aware object manipulation featuring paired images and depth annotations. We also propose ManipEval, a benchmark with multi-dimensional metrics to evaluate 3D spatial control and geometric consistency. Extensive experiments show that our approach outperforms existing methods, including strong closed-source models, in both 3D geometric accuracy and manipulation consistency.

Figures & tables

Figure 1. Comparison of image editing results on three physical scenarios between our PhyEdit and Nano Banana Pro ( DeepMind, 2025 ) A three-row comparison with inputs and transformation instructions at left, PhyEdit outputs in the center, and Nano Banana Pro outputs at right. The rows show a robot hand moving a red pen to the lower-left of a sheet of paper, a smiling man thrusting and slightly tilting a graphics card, and selected LEGO pieces assembled into a pen at a specified position.

Figure 2. Overview of PhyEdit. User and target inputs are first processed via the 3D transformation module. The resulting conditions are then fed into the backbone for the training forward pass, followed by 2D and 3D level joint supervision. A left-to-right architecture diagram. User and target inputs provide a source image, a source object with a target 3D position, a target image, and a prompt. A frozen 3D foundation model estimates depth, unprojects a point cloud, relocates the object, and projects a reference image. The backbone encodes the target image, reference image, and prompt and processes them with LoRA-adapted MM-DiT blocks. Joint supervision combines flow-matching loss with pixel-level depth SILog loss computed after VAE decoding.

Figure 3. Samples from RealManip-40K. The objects are annotated with red and orange boxes. The normalized 3D object coordinates $(x,y,z)$ follow an image-aligned coordinate system with the origin at the top-left-near corner. Nine source-target examples arranged in three groups. The top group shows vehicle, cat, and can motions with large depth changes; the middle group shows a girl, toy car, and hiker becoming more or less occluded; and the bottom group shows paired relocations of donuts, people, and a man with a bag. Red and orange boxes identify manipulated objects, and each example lists source and target 3D coordinates.

Figure 4. Dataset Construction Pipeline. The workflow consists of four key stages: data source filtering, camera-static clip extraction, depth and mask processing, and depth-aware frame pair selection. At the bottom of the second column, the symbols and represent 3D viewing frustums, denoting that Cluster 1 and Cluster 2 have different camera viewpoints. A four-stage pipeline from left to right. Videos from tracking and general-purpose datasets are filtered by motion and quality. A 3D foundation model clusters camera tokens to retain camera-static clips. Object detection, tracking, segmentation, and depth estimation produce masks and depth maps. Frame-wise 3D reconstruction then selects the image pair with the largest valid object-coordinate change, followed by VLM validation.

Method	DIoU $\uparrow$	Mask IoU $\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}$ $\uparrow$	Chamfer $\downarrow$	Centroid $\downarrow$	RA-DINO $\uparrow$	DeQA $\uparrow$	Phys-VLM $\uparrow$
GeoDiffuser	51.72	20.10	67.64	31.40	73.41	64.56	14.78	56.72	54.62
DiffusionHandles	56.22	18.88	57.36	31.73	73.00	59.65	17.08	61.83	46.90
Move-and-Act	46.89	10.97	68.47	29.30	64.59	63.62	13.01	60.79	60.85
OBJect 3DIT	46.63	10.18	66.93	32.75	62.30	64.14	14.31	48.01	66.65
GoodDrag	51.07	13.29	69.60	34.19	51.27	58.39	20.85	74.08	85.08
PixelMan	49.48	11.49	66.46	35.10	50.18	57.13	21.17	72.79	73.60

Table 1. Quantitative comparison on ManipEval. All metrics are linearly normalized to

[0,100]

. Methods are sorted by Chamfer distance in descending order. Proprietary commercial models are marked with

\dagger

Figure 5. Qualitative comparison on ManipEval. The top two rows highlight manipulation accuracy and geometric consistency; the middle two rows focus on preserving object identity and fine details; the last two rows show multi-object manipulation. In the first four rows, source and target positions are annotated with orange boxes. Objects that are misplaced or still left at the source are marked with red boxes, and correctly manipulated objects are indicated by green boxes. A six-row qualitative comparison with columns for the source, target, 3D view, four baseline methods, and PhyEdit. The rows include moving a toy train around a building, inserting a spatula into a pitcher, relocating a metal bowl and a patterned mug, repositioning multiple toy cars, and changing the positions of several people. Orange boxes mark source and target regions, red boxes mark incorrect or incomplete baseline edits, and green boxes mark successful placements.

Figure 6. Continuous object manipulation along a trajectory. Given the initial image and a trajectory, PhyEdit performs continuous editing (solid borders). Intermediate frames are further interpolated by a video model (dashed borders). Notably, this robotic arm is out of our training distribution, demonstrating our strong generalization. A source image and a 3D trajectory are followed by six frames of a robot arm moving a red pen along a curved path toward the lower-left of a sheet of paper. Solid borders denote PhyEdit keyframes, and dashed borders denote interpolated frames.

Figure 7. Architectures of different depth supervision methods. The core difference is highlighted by a red dashed box. Three parallel depth-supervision designs. The latent-to-latent design predicts a depth latent with eight trainable DiT layers and uses MSE loss. The latent-to-depth design adds a DPT head to predict pixel depth and uses SILog loss. The pixel-level design decodes the denoised latent with a frozen VAE, estimates depth with a frozen 3D foundation model, and applies SILog loss against target-image depth.

Method	DIoU $\uparrow$	AbsRel $\downarrow$	Chamfer $\downarrow$	RA-DINO $\uparrow$
w/o Supervision	62.37	50.92	24.52	33.53
Latent-to-Latent	60.48	49.78	28.61	32.29
Latent-to-Depth	64.19	49.73	20.87	35.93
Ours (pixel-level)	65.33	49.53	18.93	36.91

Table 2. Quantitative comparison of different depth supervision methods.

Figure 8. Qualitative comparison of methods with and without depth supervision. A two-by-two example showing the source, target, output without depth supervision, and output with depth supervision. The task moves a small glue bottle from the right side of a workshop table to a target position near the saw. Without supervision the bottle is a blurred, distorted trace; with supervision it is recognizable and positioned close to the target. Enlarged crops highlight the small object.

Reference	Training	DIoU $\uparrow$	AbsRel $\downarrow$	Chamfer $\downarrow$	RA-DINO $\uparrow$
✗	✗	45.58	68.22	55.38	18.85
✗	✓	46.16	67.99	52.93	18.87
✓	✗	53.28	64.63	46.31	26.09
✓	✓	65.33	49.53	18.93	36.91

Table 3. Quantitative comparison of the effects of reference image and reference-aware training.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9. More qualitative results on ManipEval. For baseline methods, misplaced objects and objects still left at the source are marked with red boxes, and correctly manipulated objects are indicated by green boxes. Eight qualitative comparisons arranged by row, with columns for the source, target, 3D view, four baseline methods, and PhyEdit. The examples move objects including a bottle, a wooden assembly, people, boxes, cookware, and multiple toys. Orange boxes mark source and target regions, red boxes mark incorrect or incomplete edits, and green boxes mark successful placements.

Method	DIoU $\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}$ $\uparrow$	Chamfer $\downarrow$	Geo. $\Delta$ $\uparrow$
Nano Banana Pro	59.97	55.02	46.11	25.33	+0.00
Ours (DA2)	61.26	51.53	47.93	22.80	+2.28
Ours (MoGe)	60.68	49.81	51.14	21.82	+3.62
Ours (VGGT)	60.92	48.95	48.11	21.62	+3.18
Ours (Pi3X)	62.71	51.21	50.21	20.23	+3.94
Ours (DA3)	65.33	49.53	51.08	18.93	+5.55

Appendix

Table 4. Robustness to different 3D foundation models. Scores are normalized to

[0,100]

. Geo.

\Delta

is the average signed improvement over Nano Banana Pro across the four geometry metrics after aligning metric directions.

Figure 10. Representative imperfect 3D-transformed previews and their corresponding final outputs. Red dashed boxes mark the manipulated regions. The examples cover boundary and mask errors, distortion under large depth changes, occlusion-order errors, and noisy or fragmented projected surfaces. Pairs of imperfect 3D-transformed previews and final edited images for several object-manipulation cases.

Baseline	Place/Scale	Identity	Contact	Overall
Nano Banana Pro	81.3	69.0	74.0	78.0
Qwen-Image-2.0-Pro	85.3	78.7	83.0	85.3

Appendix

Table 5. Blind human preference. Entries are PhyEdit preference rates (%) over each baseline, with ties assigned half weight.

Method	General-edit success (%)				3D accuracy
	Color $\uparrow$	Pattern $\uparrow$	Local $\uparrow$	All $\uparrow$	DIoU $\uparrow$	AbsRel $\downarrow$	Chamfer $\downarrow$
Qwen-Image-Edit	89.9	88.6	86.8	88.8	53.28	64.63	46.31
Ours	88.6	86.4	86.1	87.5	63.68	47.79	20.97

Appendix

Table 6. General-edit success (%) and 3D accuracy. Qwen-Image-Edit uses separate editing-only and manipulation-only runs; PhyEdit performs both in the same generation.

Figure 11. Examples of applying additional appearance or local editing instructions during 3D manipulation. The two added edits in each row are, from top to bottom: flipping the shoe and adding a swoosh; changing the man’s gesture and the box material; adding a baseball bat and opening the window blinds. Faces are anonymized except where facial state is necessary to visualize the requested edit. Examples in which PhyEdit moves an object in 3D while also changing appearance, adding a local detail, or modifying a nearby scene attribute.

Component or setting	Runtime (s)	Peak VRAM (GB)
DA3 depth/camera	0.124	9.26
Preview projection	0.028	–
3D preprocessing total	0.152	–
Base Qwen-Image-Edit	7.67	57.7
PhyEdit end-to-end	11.78	59.0

Appendix

Table 7. Single-GPU inference profile. Runtime is the mean compute time per image; memory is peak VRAM.

Method	DIoU $\uparrow$	Mask IoU $\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}$ $\uparrow$	Chamfer $\downarrow$	Centroid $\downarrow$	RA-DINO $\uparrow$	DeQA $\uparrow$	Phys-VLM $\uparrow$
GeoDiffuser	51.36	18.70	85.67	24.72	93.34	73.95	15.40	60.23	66.08
Object 3DIT	45.34	10.03	87.47	26.78	91.11	77.96	14.64	48.70	69.70
Move-and-Act	50.54	12.65	83.88	26.76	73.83	68.99	15.32	64.20	66.70
LightningDrag	51.99	17.94	74.68	27.84	66.65	64.31	21.93	72.80	92.00
Qwen-Image-Edit	51.41	12.70	86.77	36.54	66.43	64.38	25.56	74.76	92.80
GoodDrag	51.05	13.65	88.85	31.55	65.55	67.10	22.69	74.20	91.50

Appendix

Table 8. Quantitative comparison on ManipEval on single-object items, sorted by Chamfer distance in descending order.

Method	DIoU $\uparrow$	Mask IoU $\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}$ $\uparrow$	Chamfer $\downarrow$	Centroid $\downarrow$	RA-DINO $\uparrow$	DeQA $\uparrow$	Phys-VLM $\uparrow$
DiffusionHandles	53.72	17.55	46.81	35.74	79.35	59.70	13.35	59.42	41.10
GeoDiffuser	52.07	21.49	49.79	38.01	61.58	55.14	14.17	53.24	43.27
Move-and-Act	43.25	9.29	53.05	31.85	56.91	56.97	10.69	57.39	55.00
Object 3DIT	47.92	10.33	46.39	38.72	40.29	51.06	13.99	47.31	63.60
GoodDrag	51.08	12.92	50.16	36.85	37.65	49.17	18.98	73.96	78.59
PixelMan	48.87	11.17	48.79	35.55	37.62	47.63	19.81	72.96	63.90

Appendix

Table 9. Quantitative comparison on ManipEval on multi-object items, sorted by Chamfer distance in descending order.

ObjectMorpher: 3D-Aware Image Editing via Deformable 3DGS Models
Mar 30, 2026Yuhuan Xie, Aoxuan Pan, Yi-Hua Huang +43D Shape Editing Deformable Object Manipulation
The University of Hong Kong
BoxCtrl: 3D-Aware Visual Prompting for Geometric Image Editing
Jun 22, 2026Feifei Wang, Shiyuan Yang, Xiaoyu Li +1Image Editing Visual Prompt Learning
City University of Hong Kong, China · Tencent, China
GeoEdit: Geometry-Aware Object Editing via Dual-Branch Denoising
Jun 29, 2026Yi He, Jiangming Wang, Xinyu Wang +5Diffusion Guidance Diffusion-Based Image Editing
Shenzhen International Graduate School, Tsinghua University · Sun Yat-sen University · Peking University +3

Mar 30, 2026J/K move · Enter open · S save

ObjectMorpher: 3D-Aware Image Editing via Deformable 3DGS Models

Yuhuan Xie, Aoxuan Pan, Yi-Hua Huang, Chirui Chang+3

The University of Hong Kong

3D Shape Editing Deformable Object Manipulation 3D Gaussian Splatting Diffusion-Based Image Editing

Open paper AskPDF arXiv

Achieving precise, object-level control in image editing remains challenging: 2D methods lack 3D awareness and often yield ambiguous or implausible results, while existing 3D-aware approaches rely on heavy optimization or incomplete monocular reconstructions. We present ObjectMorpher, a unified, interactive framework that converts ambiguous 2D edits into geometry-grounded operations. ObjectMorpher lifts target instances with an image-to-3D generator into editable 3D Gaussian Splatting (3DGS), enabling fast, identity-preserving manipulation. Users drag control points; a graph-based non-rigid deformation with as-rigid-as-possible (ARAP) constraints ensures physically sensible shape and pose changes. A composite diffusion module harmonizes lighting, color, and boundaries for seamless reintegration. Across diverse categories, ObjectMorpher delivers fine-grained, photorealistic edits with superior controllability and efficiency, outperforming 2D drag and 3D-aware baselines on KID, LPIPS, SIFID, and user preference.

Abstract

Explore similar work

ObjectMorpher: 3D-Aware Image Editing via Deformable 3DGS Models

BoxCtrl: 3D-Aware Visual Prompting for Geometric Image Editing

GeoEdit: Geometry-Aware Object Editing via Dual-Branch Denoising