Organizations: University of Auckland, Auckland, New Zealand · Dcar, Beijing, China · Singapore University of Technology and Design, Singapore · WIZYELAB PTE. LTD., Singapore · Baidu Inc., Beijing, China · Adelaide University, Adelaide, Australia
Synthetic data from diffusion models can improve object detection, but long-tail performance is still limited by scarce rare classes and inflexible conditioning (e.g., text-only or layout-only inputs). In this paper, we propose Tail-SD, a layout-instruction conditioned stable diffusion framework that supports text, layouts, or their hybrid, enabling controllable synthesis from boxes, masks, and/or prompts to better cover the long-tail objects. To train a single model across heterogeneous conditions, we introduce a multi-task cross-attention module and multi-task embedding tokens, improving generation quality and diversity. Extensive experiments on COCO and LVIS demonstrate that diffusion synthesis with hybrid conditioning consistently benefits detection, e.g., on LVIS val, it achieves +4.89% / +5.68% rare Box/Mask AP, highlighting strong potential for addressing long-tail challenges.
Figures & tables
Fig. 1 : Our Tail-SD is a flexible and versatile image generation pipeline, supporting text, layouts and a hybrid of both as inputs. The last three synthetic images are sampled from different seeds. (Zoom in for best view)
Fig. 2 : (a) Overview of Tail-SD. The model supports instruction, layout, and hybrid conditioning, including two components: (i) MTCA that applies format-aware attention control, and (ii) MTET that indicates the conditioning mode to the diffusion model. The autoencoder is omitted for clarity. (b) Visualization of multi-task cross-attention maps in our Tail-SD: for each original image, we show two MTCA maps emphasizing corresponding objects and the generated image.
Method
AP b
AP m
Baseline
46.00
39.80
+ Copy-Paste [ 11 ]
46.40
39.80
+ X-Paste [ 55 ]
46.60
39.90
+ FreeStyleNet [ 49 ]
46.67
40.33
+ Gen2Det [ 40 ]
47.18
40.44
+ Tail-SD (ours)
47.06
40.60
TABLE I : Comparison on COCO val using CenterNet2. Synthetic images are used to augment detector training.
Method
Box
Mask
AP
AP r
AP c
AP f
AP
AP r
AP c
AP f
Baseline
33.80
20.84
32.84
40.58
29.98
18.36
29.64
35.46
+ InstructPix2Pix [ 4 ]
24.88
17.40
24.00
29.16
22.13
15.49
21.60
25.64
+ Copy-Paste [ 11 ]
34.31
21.19
34.32
40.07
30.29
19.97
30.43
34.67
+ X-Paste [ 55 ]
34.34
21.05
33.86
40.71
30.18
18.77
30.11
35.28
+ Gen2Det [ 40 ]
34.70
23.78
33.71
40.61
30.82
21.24
30.32
35.59
TABLE II : Comparison on LVIS val using CenterNet2. For InstructPix2Pix [ 4 ] , we generate synthetic images using the same instruction templates as Tail-SD for a fair comparison.
Method
L1 ↓
L2 ↓
CLIP-I ↑
CLIP-T ↑
DINO ↑
Instruction
InstructPix2Pix [ 4 ]
0.302
0.311
0.863
0.227
0.780
MagicBrush [ 54 ]
0.238
0.259
0.900
0.225
0.841
MGIE [ 10 ]
0.325
0.421
0.861
0.228
0.783
Tail-SD (ours)
0.195
0.161
0.955
0.221
0.970
Layout
FreestyleNet [ 49 ]
0.946
1.448
0.762
-
0.633
Tail-SD (ours)
0.355
0.413
0.891
-
0.865
TABLE III : Quantitative comparison on image editing with instruction-based and layout-based inputs.
Fig. 3 : Qualitative comparison of Tail-SD with representative image-editing baselines under different conditioning formats. Tail-SD supports both layout- and instruction-based control while better preserving background consistency, producing more realistic edits. (Best viewed in color and with zoom.)
Method
Task
COCO
LVIS
AP b
AP m
AP
AP r
Baseline
-
46.00
39.80
33.80
20.84
+ Tail-SD
instruction
46.80
40.36
33.96
22.26
layout
46.58
40.37
34.05
19.81
multi-task
47.06
40.60
34.97
25.73
TABLE IV : Effect of multi-task learning on detection (CenterNet2 [ 56 ] ). “instruction” and “layout” denote single-format training, while “multi-task” denotes multi-task training.
Method
COCO
LVIS
AP b
AP m
AP
AP r
Baseline
46.00
39.80
33.80
20.84
+ Tail-SD w/o MTET
47.02
40.43
33.98
20.51
+ Tail-SD
47.06
40.60
34.97
25.73
TABLE V : Effect of MTET on detection (CenterNet2 [ 56 ] ).
Data utilization
Box
Mask
instruction
layout
AP
AP50
AP
AP50
×
×
46.00
-
39.80
-
✓
×
46.51
65.40
40.24
62.22
×
✓
46.72
65.71
40.33
62.58
✓
✓
47.06
66.01
40.60
62.66
TABLE VI : Effect of synthetic images generated with different conditioning formats on COCO [ 24 ] using CenterNet2 [ 56 ] .
Model
Method
Box
Mask
AP
AP50
AP
AP50
CenterNet2 [ 56 ]
Baseline
46.00
-
39.80
-
+ Tail-SD
47.06
66.01
40.60
62.66
Faster R-CNN [ 32 ]
Baseline
40.22
61.02
-
-
+ Tail-SD
40.47
61.09
-
-
TABLE VII : Effect of Tail-SD synthetic augmentation across different detectors on COCO [ 24 ] .