PhysInsert: Physically Plausible Object Insertion with Geometric Priors and Bidirectional Object-Scene Interactions

Yue Gong; Le Zhang; Rui Chen; Song Chang; Xiaochao Qu; Luoqi Liu; Ting Liu; Yebin Liu

cs.CVOct 29, 2026

PhysInsert: Physically Plausible Object Insertion with Geometric Priors and Bidirectional Object-Scene Interactions

Authors: Yue Gong, Le Zhang, Rui Chen, Song Chang, Xiaochao Qu, Luoqi Liu, Ting Liu, Yebin Liu

Organizations: School of Computer Science and Engineering, Beihang University · MT Lab, Meitu Inc. · Department of Automation, Tsinghua University

Abstract

We present PhysInsert, a framework for physically plausible reference-based object insertion that jointly adapts objects and their surrounding scenes. Existing methods often rely on masked inputs, generation-based pairs, or training pairs with limited viewpoint variation, leaving pose and illumination adaptation and bidirectional object-scene interactions insufficiently constrained. We construct the Insert World dataset with strictly aligned Blender-rendered counterfactual pairs and object references across viewpoints and lighting conditions. Building on these data, Geometry-Guided Adaptation couples a VGGT geometry stream with image and text streams under target-scene feature supervision to guide spatially consistent insertion. Complementary 2D Object-Scene Spatial Supervision jointly supervises object occupancy and environmental changes while grounding spatial instructions in object regions through image-text attention. We introduce PhysInsert-Bench with dedicated cross-environment and cross-view cases requiring illumination and pose adaptation beyond appearance copying. Experiments on PhysInsert-Bench and DreamBooth demonstrate improved image fidelity; a user study further shows more accurate placement and improved lighting integration, perspective, and pose plausibility.

Figures & tables

Figure 1: (a–c) PhysInsert simultaneously supports cross-environment adaptation, bidirectional object–scene interactions, and cross-view adaptation. (d) InsertWorld offers broad coverage of scenes, object properties, and interaction types.

Figure 2: Overview of InsertWorld construction and PhysInsert-Bench composition.

Figure 3: PhysInsert combines a VGGT geometry stream with image and text streams. Target-scene features supervise geometry-token states, while 2D-OSS aligns object/effect predictions and spatial-text attention with image regions. Auxiliary supervision is used only during training.

Method	DIoU (%) $\uparrow$	Mask IoU (%) $\uparrow$	DINO-Obj. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}$ (%) $\uparrow$	Phy-VLM $\uparrow$
Proprietary Models
Nano Banana Pro ( Google DeepMind, 2025 )	54.58	48.34	0.7023	22.85	0.8231	0.1527	46.55	0.2066	77.08	$\underline{3.48}\pm 0.09$
GPT-Image-2 ( OpenAI, 2026 )	38.02	31.92	0.6746	19.39	0.7017	0.2949	53.46	0.3129	65.48	$3.17\pm 0.11$
Open-Source Models
OmniPaint + PhysW. ( Yu et al., 2025 )	37.69	33.94	0.5828	20.67	0.8302	0.1733	73.54	0.2401	67.30	$2.06\pm 0.13$
Insert-Anything + PhysW. ( Song et al., 2025 )	22.66	21.46	0.6275	19.45	0.7711	0.3128	64.39	0.2651	67.79	$2.36\pm 0.13$

Table 2: Text-guided results on PhysInsert-Bench using one object-reference image per case for all reported metrics. Phy-VLM obtains three valid ratings per output with GPT-5.5, retrying null responses, and reports mean

\pm

95% confidence-interval half-width. Unless otherwise specified, the following conventions apply to this and subsequent tables: +PhysW. denotes fine-tuning on our InsertWorld dataset with the same training budget as ours under the corresponding guidance setting; bold / underline mark the best/second-best across methods.

Proprietary Models
Method	Mask IoU (%) $\uparrow$	DINO-Obj. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}$ (%) $\uparrow$
Nano Banana Pro ( Google DeepMind, 2025 )	52.94	0.7152	22.53	0.8056	0.1609	45.32	0.1846	81.03
GPT-Image-2 ( OpenAI, 2026 )	39.61	0.6798	21.45	0.7716	0.2049	50.68	0.2860	71.24
Open-Source Models
AnyDoor ( Chen et al., 2024a )	55.92	0.5379	20.42	0.8158	0.2413	63.50	0.2070	67.01
Insert-Anything ( Song et al., 2025 )	55.09	0.6507	22.70	0.8735	0.1608	52.23	0.1911	74.99

Table 3: Box-guided results on PhysInsert-Bench. Ours is adapted from the text-guided model for an additional 4,000 box-guided steps; +PhysW. baselines use the same training budget with our method.

\dagger

For PhysInsert-Bench, DIRECT uses GT-image-assisted pose fitting

Proprietary Models
Method	PhysInsert-Bench-Wild						DreamBooth
	1-view			3-view			1-view
	PSNR $\uparrow$	FID $\downarrow$	DINO-Obj. $\uparrow$	PSNR $\uparrow$	FID $\downarrow$	DINO-Obj. $\uparrow$	PSNR $\uparrow$	FID $\downarrow$	DINO-Obj. $\uparrow$
Nano Banana Pro ( Google DeepMind, 2025 )	15.78	123.2	0.7881	15.72	115.3	0.7909	15.58	83.74	0.8650
GPT-Image-2 ( OpenAI, 2026 )	16.74	113.4	0.7457	16.88	112.9	0.7865	14.68	87.77	0.8821
Open-Source Models

Table 4: Results on PhysInsert-Bench-Wild with one or three reference images, and DreamBooth with one reference image.

Figure 4: User-study result. Error bars show 95% confidence intervals.

Figure 5: Qualitative comparisons on PhysInsert-Bench (top), DreamBooth and PhysInsert-Bench-Wild (middle), and the user-study set (bottom).

Figure 6: Mechanism probes and qualitative ablations. (a) Spatial-text attention, object/effect predictions, and VGGT feature evolution under identical GT-noised inputs ( $\sigma=0.5$ ). (b) Separate ablations of SSG, GGA, and COSS for placement and scale, perspective, and floor reflection. (c) Text-guided pose and placement control.

Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	DIoU (%) $\uparrow$	DINO-Obj. $\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}$ (%) $\uparrow$
Init.	19.34	0.7840	0.2260	60.99	48.32	0.6786	0.2207	73.35
PhysW.	22.53	0.8529	0.1442	48.61	60.19	0.7062	0.1372	84.18
w/o GGA	23.83	0.8702	0.1226	43.94	61.45	0.7206	0.1354	85.15
w/o SSG	23.73	0.8702	0.1203	44.97	62.51	0.7157	0.1287	86.06
w/o COSS	23.90	0.8722	0.1168	42.68	61.58	0.7232	0.1315	86.40
Full model	24.40	0.8773	0.1118	41.96	62.99	0.7276	0.1195	86.56

Table 5: Ablations on PhysInsert Bench.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Proprietary Models
Method	Optical Adaptation				Pose Adaptation
	1-view				1-view			3-view
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FID $\downarrow$	DINO-Obj. $\uparrow$	PSNR $\uparrow$	FID $\downarrow$	DINO-Obj. $\uparrow$	PSNR $\uparrow$
Nano Banana Pro ( Google DeepMind, 2025 )	22.92	0.8153	0.1488	56.67	59.70	0.6886	22.20	60.85	0.6824	22.05
GPT-Image-2 ( OpenAI, 2026 )	22.01	0.8045	0.1785	66.55	64.31	0.6566	20.99	66.53	0.6404	21.06
Open-Source Models

Appendix

Table S1 : Box-guided results on 180 Optical Adaptation and 220 Pose Adaptation cases.

Metric	Evaluation scope	Preferred
PSNR	Full-image pixel fidelity	Higher
SSIM	Full-image structural similarity	Higher
LPIPS	Full-image perceptual distance	Lower
FID	Full-image feature distributions	Lower
DIoU (%)	Mask-box overlap and center distance	Higher
Mask IoU (%)	Visible foreground overlap	Higher

Appendix

Table S2 : Evaluation metrics and their scope.

Setting	Value
Base-model training
Optimizer	AdamW
Base learning rate	$1\times 10^{-4}$
Weight decay	$0.01$
Learning-rate scheduler	PyTorch ConstantLR (defaults)
Effective batch size	$8$

Appendix

Table S3 : Base-model training and inference hyperparameters. The additional box-guided training budget is listed separately.

Module settings
Parameter	Value
Flow Matching Loss coefficient	$1$
Geometry Alignment Loss, $\lambda_{\mathrm{geo}}$	$0.02$
Counterfactual Supervision Loss, $\lambda_{\mathrm{cf}}$	$0.01$
Object Grounding Loss, $\lambda_{\mathrm{ground}}$	$0.005$
VGGT feature layer, $\ell_{v}$	$23$

Appendix

Table S4 : Auxiliary-loss weights and module hyperparameters for single-stage training.

Method	Denoiser params (B)	Pipeline params (B)	Time/image (s)	Peak memory (GiB)
Qwen-Image-Edit-2511	20.430	28.849	$79.40\pm 0.06$	55.92
Ours (MLP)	20.815	30.143	$120.13\pm 0.26$	60.80

Appendix

Table S5 : Inference efficiency on one NVIDIA L20X GPU at

1024\times 1024

, batch size 1, 40 denoising steps, and CFG 4.0. Parameter counts are obtained from the loaded inference modules. Latency is the mean and sample standard deviation over five cases after two warmup calls; memory is the maximum PyTorch-allocated GPU memory over the timed calls. B denotes

10^{9}

parameters.

Figure S1 : Selected box-guided comparisons on PhysInsert-Bench with one reference image. DIRECT additionally receives GT pose.

Figure S2 : Selected text-guided comparisons on PhysInsert-Bench with one reference image. References are center-cropped only for visualization.

Figure S3 : Additional qualitative comparisons on DreamBooth (1/2). Columns show the background, object reference, ground-truth target (GT), OmniPaint, InsertAnything, DIRECT, GPT-Image2, Nano Banana Pro, and Ours. DIRECT uses a GT-pose RGB proxy; GPT-Image2 and Nano Banana Pro use text guidance.

Figure S4 : Additional qualitative comparisons on DreamBooth (2/2). The column order and method settings are the same as in Figure S3 .

Figure S5 : Additional qualitative comparisons on 11 cases from the in-the-wild user-study set. Columns show the object reference, background scene, OmniPaint, InsertAnything, DIRECT, Nano Banana Pro, and Ours. White boxes in the scene column indicate approximate insertion regions. OmniPaint is included for qualitative comparison only and was not part of the user-study ratings.

Figure S6 : Five InsertWorld training examples. Each row contains the with-object scene (With), the corresponding object-free scene (Without), and the isolated target rendered from the scene viewpoint on a white display background (Removed-only). The right panel shows the same target under two lighting conditions (rows L1 and L2) and four corresponding fixed viewpoints (columns). Lighting conditions and viewpoints are selected separately for each example.

Figure S7 : PhysInsert-Bench samples (1/4). The original interaction tag appears at left in readable form, followed by the object-free scene (Without), with-object target (GT), and actual single-view object input. Case IDs refer to the benchmark manifest.

Figure S8 : PhysInsert-Bench samples (2/4). Column meanings follow Figure S7 . The reference is the actual one-view benchmark input, including its original background; it is neither a multiview collage nor a newly rendered reference.

Figure S9 : PhysInsert-Bench samples (3/4). Column meanings follow Figure S7 . The examples illustrate cloth, suspension, leaning, flat support, and depth ordering using the original benchmark annotations.

Figure S10 : PhysInsert-Bench samples (4/4). Column meanings follow Figure S7 . These examples cover partial occlusion, placement relations, scale adaptation, perspective foreshortening, and viewpoint gaps.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 474 papers

Place-it-R1: Unlocking Environment-aware Reasoning Potential of MLLM for Video Object Insertion
Mar 6, 2026Bohai Gu, Taiyi Wu, Dazhao Du +5Video Object Insertion Video Editing
The Hong Kong University of Science and Technology · AI Technology Center, Tencent Video, Tencent · Peking University
Direct 3D-Aware Object Insertion via Decomposed Visual Proxies
Jun 4, 2026Jingbo Gong, Yikai Wang, Yushi Lan +6Image Editing Diffusion Models
VCIP, School of Computer Science, Nankai University · Zhongguancun Academy · S-Lab, Nanyang Technological University +2
ElasticFit: Fit-Aware 3D Object Insertion via VLM Reasoning and Generative Adaptation
Oct 5, 2026Tzu-Hsin Hsieh, Ricardo Marroquim3D Scene Editing 3D Visual Grounding
Delft University of Technology Delft, Netherlands

Mar 6, 2026J/K move · Enter open · S save

Place-it-R1: Unlocking Environment-aware Reasoning Potential of MLLM for Video Object Insertion

Bohai Gu, Taiyi Wu, Dazhao Du, Jian Liu+4

The Hong Kong University of Science and Technology · AI Technology Center, Tencent Video, Tencent · Peking University

Video Object Insertion Video Editing Physical Consistency in Video Generation Video Diffusion Models Direct Preference Optimization

Open paper AskPDF arXiv

Video object insertion is fundamental to video editing, yet existing diffusion methods often produce visually plausible but physically inconsistent results. We present Place-it-R1, an end-to-end framework for physically plausible video object insertion driven by environment-aware MLLM reasoning. Rather than treating reasoning as a generic text prompt, Place-it-R1 uses the MLLM to analyze the target environment, infer object-scene interactions, and determine where an insertion is physically valid. The resulting reasoning is translated into two complementary forms of guidance for video diffusion: semantic guidance that describes the intended physical interaction and spatial guidance that provides a valid insertion region in each frame. To further align generation with local physical realism, we introduce Spatial Direct Preference Optimization, which leverages an MLLM to rank generated candidates, and introduces a region-aware preference objective that explicitly localizes physical-violation penalties to the inserted-object region. Place-it-R1 further offers flexible and standard modes to trade off environment adaptation and scene preservation. Extensive experiments show that Place-it-R1 produces more physically coherent and visually natural insertions than state-of-the-art methods and achieves competitive results against commercial systems.

Abstract

Image Generation

Explore similar work

Place-it-R1: Unlocking Environment-aware Reasoning Potential of MLLM for Video Object Insertion

Direct 3D-Aware Object Insertion via Decomposed Visual Proxies

ElasticFit: Fit-Aware 3D Object Insertion via VLM Reasoning and Generative Adaptation