Controllable Multi-Object Insertion for Detection Data Augmentation

Jiyang Huang; Tianwei Zhang; Xiaoqin Zhang; Ling Shao; Shijian Lu

cs.CVOct 17, 2026

Controllable Multi-Object Insertion for Detection Data Augmentation

Authors: Jiyang Huang, Tianwei Zhang, Xiaoqin Zhang, Ling Shao, Shijian Lu

Organizations: Nanyang Technological University · Zhejiang University of Technology · University of the Chinese Academy of Sciences

Abstract

Controllable object insertion provides a targeted approach to augmenting object detection datasets that require category--count fidelity, annotation alignment, and source-content preservation beyond visual realism. Several approaches have been proposed to address this grand challenge, but most suffer from different constraints in precise category--count requests and visual consistency, limited coordination across categories and instances, and poor localization of multiple objects of specific category and count. We design SceneP2F, a two-stage trainable framework that connects scene-aware joint planning with layout-aligned generation leveraging explicit instance-level layouts. SceneP2F consists of two novel designs. The first is ScenePlan which allocates one class-labeled query per requested instance, jointly predicts scene-compatible object centers, and separately estimates box sizes conditioned on the planned centers and scene context. The second is SceneAlign-Fill which combines geometry- and appearance-aware conditioning, box--class-guided regional attention, and an annotation-region-weighted flow-matching objective to render the planned objects while limiting unintended changes to source content. Extensive experiments on three challenging benchmarks UAVDT, VisDrone, and BDD100K show that SceneP2F outperforms representative general-purpose editors and object-insertion methods consistently, demonstrating the value of coordinating instance-level planning and generation for category-targeted detection augmentation.

Figures & tables

Figure 1: Given a source image and an object insertion request, representative image editing and object insertion approaches may fail to satisfy the requested categories or counts, alter existing objects, or generate implausible placements, scales, and appearances, as highlighted by the red boxes. SceneP2F, a two-stage framework comprising ScenePlan and SceneAlign-Fill, first predicts a scene-compatible, class-labeled box layout and then jointly synthesizes the requested instances within the planned regions, aiming to improve those questions in multi-object insertion task.

Figure 2: Overview of the SceneP2F framework, comprising ScenePlan (left) and SceneAlign-Fill (right). ScenePlan first jointly predicts object centers from image features, existing boxes, and category–count queries, while a size predictor estimates the corresponding box extents. Given the resulting class-labeled layout, SceneAlign-Fill uses geometric, categorical, and scene-style conditioning to synthesize the requested objects while limiting changes outside the target regions.

Figure 3: Removal dataset construction (left) and pretrained size predictor (right). The removal-data construction pipeline uses SAM2 to generate masks and ObjectClear to remove the target objects, while a verification module triggers iterative reprocessing when the check fails. Size predictor sub-model estimates target-box dimensions from image features, context boxes, and input center.

Figure 4: Qualitative comparison of SceneP2F with image-editing models (FLUX.2-dev and Qwen-Image-Edit) and object-insertion baselines (Add-SD and Add-it) on the joint multi-object insertion task. The first three rows present results on UAVDT, the subsequent two rows on VisDrone, and the final row on BDD100K. The leftmost column shows the requested categories and counts. For BDD100K (last row), Diffree replaces Add-SD, which was not trained on this dataset.

Dataset	Method Category	Method	Fine- tuned	Request Fidelity		Visual Consistency			Outside Preservation
Dataset	Method Category	Method	Fine- tuned	mTCE $\downarrow$	mCCE $\downarrow$	LPIPS $\downarrow$	DINO Sim $\uparrow$	CLIP-I $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
UAVDT ( Du et al., 2018 )	Image Editing	FLUX.2-dev ( Labs, 2025 )		7.001	3.076	0.5561	0.4753	0.8131	18.9585	0.8405
		Qwen-Image-Edit ( Wu et al., 2025 )		5.596	2.458	0.5588	0.4635	0.8188	20.6050	0.8973
		Qwen-Image-Edit ( Wu et al., 2025 )	$\checkmark$	14.959	7.603	0.5051	0.4499	0.8452	19.3728	0.8408
	Object Insertion	Add-it ( Tewel et al., 2025 )	$\checkmark$	8.529	4.594	0.5164	0.4508	0.8410	21.5469	0.8987
	Object Insertion	Add-SD ( Yang et al., 2024 )	$\checkmark$	2.744	2.105	0.5751	0.4295	0.8356	22.2962	0.9010

Table 1: Quantitative comparison of general image-editing and object-insertion methods on UAVDT, VisDrone, and BDD100K. The best results are shown in bold, and the second-best are underlined.

Dataset	Inserted Objects	Method	Inserted Class			All Classes
Dataset	Inserted Objects	Method	Recall	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{50\text{:}95}$	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{50\text{:}95}$
UAVDT ( Du et al., 2018 )	2 buses	Original dataset	0.403	0.229	0.168	0.341	0.208
		Copy-and-Paste	0.408	0.268	0.190	0.358	0.211
		X-Paste	0.464	0.283	0.210	0.351	0.212
		Ours (SceneP2F)	0.522	0.310	0.226	0.371	0.224
	2 trucks	Original dataset	0.246	0.0718	0.045	0.341	0.208

Table 2: Downstream detection performance of rare-class augmentation methods on UAVDT using YOLO11-L. In two separate settings, each method augments 40% of the training images with either two buses or two trucks per image, and the resulting samples are added to the original training set.

Figure 7: Qualitative comparison of Random, Dataset-Prior, and ScenePlan object placements on VisDrone and UAVDT, illustrating differences in predicted locations and scales.

Table 9

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Visualization of Removal Dataset Processed images. The left figure is the original image with target removed bbox and corresponding class. while the right is processed image.

Figure 9: Visualization of plausible region mask among three datasets generated by SAM3. The black and white image on the right of each RGB image is the binary map of the result segmentation.

Group	ID	Class-balanced log-area L1 $\downarrow$	Class-balanced aspect-ratio L1 $\downarrow$	Average $\downarrow$	Description
Ours		0.4651	0.2730	0.3691	Selected configuration
Global feature	GF0	0.5057	0.2732	0.3895	All global features
	GF1	0.4621	0.2933	0.3777	Distance feature
	GF2	0.4774	0.2876	0.3825	Median-area feature
Relative feature	RF0	0.5149	0.3550	0.4350	All relative features
	RF1	0.4834	0.2715	0.3775	Class embedding

Appendix

Table 5: Ablation study of the ScenePlan size predictor. We evaluate alternative global-feature and relative-feature designs, MLP head architectures, and training settings using class-balanced log-area and aspect-ratio L1 errors. The average is the arithmetic mean of the two errors.

Figure 10: Visual Comparison of different Image Inpainting Methods with SceneAlign-Fill.

Dataset	Recall	mAP50	mAP50–95
BDD100K ( Yu et al., 2020 )	94.91	65.54	38.04
UAVDT ( Du et al., 2018 )	96.64	91.71	70.75
VisDrone ( Zhu et al., 2021 )	78.88	48.29	28.59

Appendix

Table 6: Evaluation results of GroundingDINO on BDD100K, UAVDT and Visdrone test dataset

Figure 11: Visualization of inserted object detected by GroundingDINO. The figure in the left in original image and right is the generated. Green, red, and blue boxes indicate matched, unpreserved, and newly inserted objects, respectively; inserted-object metrics are computed on the blue boxes.

Figure 12: Qualitative results for the two-bus and two-truck insertion settings on UAVDT.

Figure 13: Visualized Comparison between different methods.

Synthetic Data Augmentation

Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.

See all 452 papers

PDA++: Field-Aligned Planning and Scene-Adaptive Insertion in Remote Sensing
Sep 16, 2026Xianchi Dong, Yingyan Hou, Chao Ren +6Remote Sensing Image Understanding Synthetic Data Augmentation
InsertFuse: A Unified Framework for Multi-Category Reference-Guided Image Insertion
Aug 6, 2026Guangzhao Li, Qingyan Wei, Huayu Zheng +7Conditional Image Generation Diffusion-Based Image Generation
1Shanghai Jiao Tong University · 2Shanghai Innovation Institute · 4Xi’an Jiaotong University +2
Direct 3D-Aware Object Insertion via Decomposed Visual Proxies
Jun 4, 2026Jingbo Gong, Yikai Wang, Yushi Lan +6Image Editing Diffusion Models
VCIP, School of Computer Science, Nankai University · Zhongguancun Academy · S-Lab, Nanyang Technological University +2

Sep 16, 2026J/K move · Enter open · S save

PDA++: Field-Aligned Planning and Scene-Adaptive Insertion in Remote Sensing

Xianchi Dong, Yingyan Hou, Chao Ren, Wanxuan Lu+5

Remote Sensing Image Understanding Synthetic Data Augmentation

Open paper AskPDF arXiv

Remote sensing recognition is often constrained by scarce observations of rare targets and costly annotations, making realistic synthetic augmentation particularly valuable for few-shot and long-tailed scenarios. Object insertion provides an efficient way to increase target diversity while preserving authentic background scenes, but realistic insertion in overhead imagery requires the generated target to adapt coherently to its surrounding environment. To this end, we propose PDA++, a unified environment-aware object insertion framework organized as Plan, Decouple, and Assimilate. Planning determines scene-compatible poses through an affordance field that combines geometric clearance with structure- and scale-aware cues. Decoupling introduces a pose-conditioned background that provides precise spatial guidance together with target-scene context, allowing the reference object to preserve its identity while adapting to the target observation. This construction also naturally provides pixel-level masks for segmentation augmentation. Assimilation further improves local coherence by aligning multi-scale texture distributions through optimal transport. On the optical benchmark, PDA++ achieves a whole-image FID of 6.28 and improves average few-shot recognition mAP50 by 17.69 points, corresponding to a 28.8% relative gain over the real-data baseline. On SAR imagery, it improves ship detection by 4.10 mAP50 points and remains effective under cross-dataset transfer and amorphous-target insertion. Code is available at https://github.com/lisheyu972/PDA_PLUS.

Abstract

Synthetic Data Augmentation

Explore similar work

PDA++: Field-Aligned Planning and Scene-Adaptive Insertion in Remote Sensing

InsertFuse: A Unified Framework for Multi-Category Reference-Guided Image Insertion

Direct 3D-Aware Object Insertion via Decomposed Visual Proxies