Organizations: University of Auckland, Auckland, New Zealand · Dcar, Beijing, China · Singapore University of Technology and Design, Singapore · WIZYELAB PTE. LTD., Singapore · Baidu Inc., Beijing, China · Adelaide University, Adelaide, Australia
Synthetic data from diffusion models can improve object detection, but long-tail performance is still limited by scarce rare classes and inflexible conditioning (e.g., text-only or layout-only inputs). In this paper, we propose Tail-SD, a layout-instruction conditioned stable diffusion framework that supports text, layouts, or their hybrid, enabling controllable synthesis from boxes, masks, and/or prompts to better cover the long-tail objects. To train a single model across heterogeneous conditions, we introduce a multi-task cross-attention module and multi-task embedding tokens, improving generation quality and diversity. Extensive experiments on COCO and LVIS demonstrate that diffusion synthesis with hybrid conditioning consistently benefits detection, e.g., on LVIS val, it achieves +4.89% / +5.68% rare Box/Mask AP, highlighting strong potential for addressing long-tail challenges.
Figures & tables
Fig. 1 : Our Tail-SD is a flexible and versatile image generation pipeline, supporting text, layouts and a hybrid of both as inputs. The last three synthetic images are sampled from different seeds. (Zoom in for best view)
Fig. 2 : (a) Overview of Tail-SD. The model supports instruction, layout, and hybrid conditioning, including two components: (i) MTCA that applies format-aware attention control, and (ii) MTET that indicates the conditioning mode to the diffusion model. The autoencoder is omitted for clarity. (b) Visualization of multi-task cross-attention maps in our Tail-SD: for each original image, we show two MTCA maps emphasizing corresponding objects and the generated image.
Method
AP b
AP m
Baseline
46.00
39.80
+ Copy-Paste [ 11 ]
46.40
39.80
+ X-Paste [ 55 ]
46.60
39.90
+ FreeStyleNet [ 49 ]
46.67
40.33
+ Gen2Det [ 40 ]
47.18
40.44
+ Tail-SD (ours)
47.06
40.60
TABLE I : Comparison on COCO val using CenterNet2. Synthetic images are used to augment detector training.
Method
Box
Mask
AP
AP r
AP c
AP f
AP
AP r
AP c
AP f
Baseline
33.80
20.84
32.84
40.58
29.98
18.36
29.64
35.46
+ InstructPix2Pix [ 4 ]
24.88
17.40
24.00
29.16
22.13
15.49
21.60
25.64
+ Copy-Paste [ 11 ]
34.31
21.19
34.32
40.07
30.29
19.97
30.43
34.67
+ X-Paste [ 55 ]
34.34
21.05
33.86
40.71
30.18
18.77
30.11
35.28
+ Gen2Det [ 40 ]
34.70
23.78
33.71
40.61
30.82
21.24
30.32
35.59
TABLE II : Comparison on LVIS val using CenterNet2. For InstructPix2Pix [ 4 ] , we generate synthetic images using the same instruction templates as Tail-SD for a fair comparison.
Method
L1 ↓
L2 ↓
CLIP-I ↑
CLIP-T ↑
DINO ↑
Instruction
InstructPix2Pix [ 4 ]
0.302
0.311
0.863
0.227
0.780
MagicBrush [ 54 ]
0.238
0.259
0.900
0.225
0.841
MGIE [ 10 ]
0.325
0.421
0.861
0.228
0.783
Tail-SD (ours)
0.195
0.161
0.955
0.221
0.970
Layout
FreestyleNet [ 49 ]
0.946
1.448
0.762
-
0.633
Tail-SD (ours)
0.355
0.413
0.891
-
0.865
TABLE III : Quantitative comparison on image editing with instruction-based and layout-based inputs.
Fig. 3 : Qualitative comparison of Tail-SD with representative image-editing baselines under different conditioning formats. Tail-SD supports both layout- and instruction-based control while better preserving background consistency, producing more realistic edits. (Best viewed in color and with zoom.)
Method
Task
COCO
LVIS
AP b
AP m
AP
AP r
Baseline
-
46.00
39.80
33.80
20.84
+ Tail-SD
instruction
46.80
40.36
33.96
22.26
layout
46.58
40.37
34.05
19.81
multi-task
47.06
40.60
34.97
25.73
TABLE IV : Effect of multi-task learning on detection (CenterNet2 [ 56 ] ). “instruction” and “layout” denote single-format training, while “multi-task” denotes multi-task training.
Method
COCO
LVIS
AP b
AP m
AP
AP r
Baseline
46.00
39.80
33.80
20.84
+ Tail-SD w/o MTET
47.02
40.43
33.98
20.51
+ Tail-SD
47.06
40.60
34.97
25.73
TABLE V : Effect of MTET on detection (CenterNet2 [ 56 ] ).
Data utilization
Box
Mask
instruction
layout
AP
AP50
AP
AP50
×
×
46.00
-
39.80
-
✓
×
46.51
65.40
40.24
62.22
×
✓
46.72
65.71
40.33
62.58
✓
✓
47.06
66.01
40.60
62.66
TABLE VI : Effect of synthetic images generated with different conditioning formats on COCO [ 24 ] using CenterNet2 [ 56 ] .
Model
Method
Box
Mask
AP
AP50
AP
AP50
CenterNet2 [ 56 ]
Baseline
46.00
-
39.80
-
+ Tail-SD
47.06
66.01
40.60
62.66
Faster R-CNN [ 32 ]
Baseline
40.22
61.02
-
-
+ Tail-SD
40.47
61.09
-
-
TABLE VII : Effect of Tail-SD synthetic augmentation across different detectors on COCO [ 24 ] .
Diffusion-based image synthesis has emerged as a promising source of synthetic training data for AI-based object detection and classification. In this work, we investigate whether images generated with diffusion can improve military vehicle detection under low-data conditions. We fine-tuned the text-to-image diffusion model FLUX.1 [dev] using LoRA with only 8 or 24 real images per class across 15 vehicle categories, resulting in class-specific diffusion models, which were used to generate new samples from automatically generated text prompts. The same real images were used to fine-tune the RF-DETR detector for a 15-class object detection task. Synthetic datasets generated by the diffusion models were then used to further improve detector performance. Importantly, no additional real data was required, as the generative models leveraged the same limited training samples. FLUX-generated images improved detection performance, particularly in the low-data regime (up to +8.0% mAP50 with 8 real samples). To address the limited geometric control of text prompt-based diffusion, we additionally generated structurally guided synthetic data using ControlNet with Canny edge-map conditioning, yielding a FLUX-ControlNet (FLUX-CN) dataset with explicit control over viewpoint and pose. Structural guidance further enhanced performance when data is scarce (+4.1% mAP50 with 8 real samples), but no additional benefit was observed when more real data is available. This study demonstrates that object-specific diffusion models are effective for improving military object detection in a low-data domain, and that structural guidance is most beneficial when real data is highly limited. These results highlight generative image data as an alternative to traditional simulation pipelines for the training of military AI systems.
Ella P. Fokkinga, Jan Erik van Woerden, Thijs A. Eker +4
TNO - Intelligent Imaging, Oude Waalsdorperweg 63, the Hague, the Netherlands
Training data for class-conditional image synthesis often exhibit a long-tailed distribution with limited amount of images for tail classes. Such an imbalance causes mode collapse and reduces the diversity of synthesized images for tail classes. For class-conditional diffusion models trained with imbalanced data, we aim to improve the diversity and fidelity of tail class images without compromising the quality of head class images. We propose contrastive conditional-unconditional alignment (CCUA), which comprises two synergistic loss functions. Our first loss is an Alignment Loss (AL) that aligns class-conditional generation with unconditional generation at large timesteps. Alignment loss makes the denoising process insensitive to class conditions for the initial steps, which enriches tail classes through knowledge sharing from head classes. Secondly, we diversify unconditional generation via an Unsupervised Contrastive Loss (UCL) to increase the distance/dissimilarity among synthetic images. We combine the two losses to implicitly diversify conditional generation. Our framework is easy to implement as demonstrated on both U-Net based architecture and Diffusion Transformer. Our method outperforms vanilla denoising diffusion probabilistic models, score-based diffusion model, and alternative contrastive methods for class-imbalanced image generation across various datasets, in particular ImageNet-LT with 256×256 resolution.
Fang Chen, Alex Villa, Gongbo Liang +3
University of California Merced · Texas A&M University-San Antonio · Oregon State University +1
Synthetic images are increasingly used to augment scarce real data for object detection. However, not all synthetic sets help equally, and the only way to know a set's value is to train a detector on it, which is slow and demands dense annotation. We ask whether a training-free metric can instead rank candidate synthetic training sets by their downstream utility. Existing image-set metrics such as FID, KID, and MMD compare two feature distributions with a single global statistic, which we show is mis-specified for detection-data selection in two ways: it is blind to per-image composition (object count, box scale, class mix), and even at fixed composition its global averaging washes out the appearance differences that separate high-mAP pools from low-mAP ones. We propose Conditional-Composition Domain Match (CCDM), which converts any feature-space distance into a composition-stratified comparison, matching candidate and target within metadata-defined strata without training a detector. On COCO and VisDrone-DET, the best CCDM variant ranks 19 candidate training sets in strong agreement with YOLOv8 mAP (Spearman \r{ho} = 0.97 and 0.96), outperforming FID, KID, and MMD. Furthermore, CCDM holds when reference metadata comes from detector pseudo-labels rather than ground-truth boxes.