DEIG: Detail-Enhanced Instance Generation with Fine-Grained Semantic Control

Shiyan Du; Conghan Yue; Xinyu Cheng; Dongyu Zhang

cs.CVFeb 20, 2026

DEIG: Detail-Enhanced Instance Generation with Fine-Grained Semantic Control

Authors: Shiyan Du, Conghan Yue, Xinyu Cheng, Dongyu Zhang

Organizations: Sun Yat-sen University, Guangzhou, China · Fudan University, Shanghai, China · Yale University, New Haven, United States

Abstract

Multi-Instance Generation has advanced significantly in spatial placement and attribute binding. However, existing approaches still face challenges in fine-grained semantic understanding, particularly when dealing with complex textual descriptions. To overcome these limitations, we propose DEIG, a novel framework for fine-grained and controllable multi-instance generation. DEIG integrates an Instance Detail Extractor (IDE) that transforms text encoder embeddings into compact, instance-aware representations, and a Detail Fusion Module (DFM) that applies instance-based masked attention to prevent attribute leakage across instances. These components enable DEIG to generate visually coherent multi-instance scenes that precisely match rich, localized textual descriptions. To support fine-grained supervision, we construct a high-quality dataset with detailed, compositional instance captions generated by VLMs. We also introduce DEIG-Bench, a new benchmark with region-level annotations and multi-attribute prompts for both humans and objects. Experiments demonstrate that DEIG consistently outperforms existing approaches across multiple benchmarks in spatial consistency, semantic accuracy, and compositional generalization. Moreover, DEIG functions as a plug-and-play module, making it easily integrable into standard diffusion-based pipelines.

Figures & tables

Figure 1: Fine-Grained Generation. Given bounding boxes and detailed descriptions, our method accurately generate multi-attribute instances, while existing methods fail to preserve fine-grained semantic details.

Figure 2: Overview of the DEIG pipeline. (a) DEIG enables the use of a frozen large text encoder to extract raw instance embeddings, which are refined by the IDE and fused into the UNet via the DFM for fine-grained control. (b) Structure of the IDE, which refines learnable queries via time-aware self- and cross-attention to produce compact instance embeddings.

Figure 3: Workflow Visualization of Fine-Grained Instance Generation. (a) Instance-based masked attention mechanism divides the attention map into four sub-regions, applying masks to restrict cross-instance interactions and prevent semantic leakage. (b) Visualization of aggregated semantic embeddings across different semantic dimensions.

	Qwen2.5-VL									InternVL3
Method	$\textbf{MAA}_{\textbf{human}}$ $\uparrow$				$\textbf{MAA}_{\textbf{obj}}$ $\uparrow$					$\textbf{MAA}_{\textbf{human}}$ $\uparrow$				$\textbf{MAA}_{\textbf{obj}}$ $\uparrow$					mIoU $\uparrow$
	C1	C2	C3	AVG	L1	L2	L3	L4	AVG	C1	C2	C3	AVG	L1	L2	L3	L4	AVG
GLIGEN	0.23	0.05	0.02	0.10	0.20	0.08	0.08	0.03	0.10	0.28	0.12	0.05	0.15	0.32	0.17	0.17	0.11	0.19	0.71
MIGC	0.51	0.11	0.03	0.22	0.63	0.29	0.29	0.20	0.36	0.60	0.23	0.11	0.31	0.75	0.54	0.47	0.36	0.54	0.72
InstanceDiffusion	0.54	0.17	0.05	0.25	0.53	0.24	0.32	0.20	0.33	0.61	0.23	0.12	0.32	0.62	0.49	0.46	0.36	0.49	0.75
ROICtrl	0.61	0.23	0.09	0.31	0.56	0.32	0.27	0.16	0.33	0.68	0.33	0.16	0.39	0.67	0.48	0.42	0.30	0.47	0.71

Table 1: Quantitative Results on DEIG-Bench. We report MAA for human (C1–C3) and object (L1–L4) instances under two VLMs, Qwen2.5-VL ( Bai et al. 2025 ) and InternVL3 ( Zhu et al. 2025 ) , as well as spatial alignment via mIoU .

Method	Instance Success Rate (%) $\uparrow$						mIoU $\uparrow$
Level	L2	L3	L4	L5	L6	AVG	L2	L3	L4	L5	L6	AVG
GLIGEN	41.56	32.29	28.13	25.38	29.79	29.91	36.70	29.10	24.92	23.37	27.22	27.03
MIGC	75.00	65.83	66.88	62.63	64.79	65.84	64.29	55.94	56.68	53.63	56.25	56.44
InstanceDiffusion	68.44	58.96	59.84	55.75	56.77	58.63	61.91	54.63	53.71	50.17	51.30	53.06
ROICtrl	72.19	65.63	64.69	59.88	60.94	63.25	61.75	57.32	56.28	52.68	53.56	55.27
DEIG (Ours)	75.23	72.00	70.02	71.00	73.13	72.25	60.43	66.50	62.01	61.65	62.68	62.64

Table 2: Quantitative results on MIG-Bench. MIG-Bench focuses on color-centric attribute evaluation via Instance Success Rate , which checks color match within a predefined gamut, and assesses spatial alignment using mIoU with Grounding-DINO.

Method	$\textbf{Acc}_{\textbf{c.}}$	$\textbf{CLIP}_{\textbf{c.}}$	$\textbf{Acc}_{\textbf{t.}}$	$\textbf{CLIP}_{\textbf{t.}}$	AP	$\textbf{AP}_{50}$
GLIGEN	25.0	0.217	15.8	0.205	0.20	0.35
MIGC	52.3	0.252	23.2	0.221	0.22	0.40
InstanceDiffusion	53.4	0.252	25.9	0.227	0.40	0.57
ROICtrl	56.9	0.255	23.7	0.223	0.26	0.51
DEIG (Ours)	58.8	0.258	26.1	0.228	0.34	0.57

Table 3: Quantitative results on InstDiff-Bench. Attribute alignment is assessed using Accuracy and CLIP scores for color (

\text{Acc}_{\text{c.}}

\text{CLIP}_{\text{c.}}

) and texture (

\text{Acc}_{\text{t.}}

\text{CLIP}_{\text{t.}}

), while spatial precision is measured by AP and

\text{AP}_{50}

using YOLOv8.

Figure 4: Detail-Enriched caption generation pipeline. Instances are described by a VLM from cropped images and filtered using CLIP scores and human review.

Figure 5: Qualitative comparison on DEIG-Bench. DEIG exhibits accurate generation of fine-grained, multi-attribute instances across varying levels of complexity, demonstrating superior compositional control and semantic alignment.

IDE	DFM	Cap.	mIoU $\uparrow$	$\textbf{MAA}_{\textbf{human}}$ $\uparrow$	$\textbf{MAA}_{\textbf{obj}}$ $\uparrow$
	✓	✓	0.73	0.51	0.35
✓		✓	0.75	0.70	0.41
✓	✓		0.70	0.31	0.29
✓	✓	✓	0.79	0.75	0.44

Table 4: Ablation study of DEIG components. We evaluate the individual contributions of IDE, DFM, and Captions supervision by measuring MAA with Qwen2.5-VL and mIoU.

Figure 6: Trade-off Between Semantic Precision and Computational Cost. 3D visualization of how the aggregated semantic dimension $S$ affects MAA and GPU memory usage under FP16 precision.

Figure 7: Plug-and-play adaptation to community diffusion models. DEIG preserves fine-grained semantic control when integrated into community diffusion models, demonstrating strong compatibility and generalization.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: The pipeline of constructing DEIG-Bench.

Figure A2: More qualitative results on MIG-Bench. MIG-Bench focuses on spatial layout and color consistency.

Figure A3: More qualitative results on InstDiff-Bench. InstDiff-Bench imposes strict spatial constraints and contains a larger number of instances per image.

Figure A4: More qualitative results using Free-Form generation. The examples illustrate that our model effectively handles open-ended descriptions across diverse scenes, producing visually coherent outputs with fine-grained instance attributes and spatial relationships without relying on predefined templates.

Figure A5: Illustration of typical limitations. Examples show cases where overlapping objects, dense scenes, or small instances cause incomplete generation or loss of fine-grained details.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

InstanceControl: Controllable Complex Image Generation without Instance Labeling
Jun 30, 2026Xiaoyu Liu, Huan Wang, Fan Li +4Controllable Image Generation
Harbin Institute of Technology, Harbin, China · HUAWEI Noah’s Ark Lab, Shenzhen, China
TIMI: Training-Free Image-to-3D Multi-Instance Generation with Spatial Fidelity
Mar 2, 2026Xiao Cai, Pengpeng Zeng, Ji Zhang +33D Scene Generation 3D Diffusion Models
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Computer Science and Technology, Tongji University, Shanghai, China · School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China +1
Delta-K: Boosting Multi-Instance Generation via Cross-Attention Augmentation
Mar 10, 2026Zitong Wang, Zijun Shen, Haohao Xu +2Diffusion Model Guidance T2I Generation
School of Software Engineering, Sun Yat-sen University · Nanjing University · College of Management and Economics, Tianjin University

Jun 30, 2026J/K move · Enter open · S save

InstanceControl: Controllable Complex Image Generation without Instance Labeling

Xiaoyu Liu, Huan Wang, Fan Li, Zhixin Wang+3

Harbin Institute of Technology, Harbin, China · HUAWEI Noah’s Ark Lab, Shenzhen, China

Controllable Image Generation

Open paper AskPDF arXiv

Controllable image generation methods, such as ControlNet, have demonstrated a remarkable capacity to introduce visual conditions(e.g., depth maps) to guide image generation. However, these methods often struggle with complex multi-instance scenes, frequently leading to attribute confusion among instances. While recent approaches attempt to mitigate this via manual instance labeling, such requirements are labor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for instance labeling. We identify the primary bottleneck in existing methods as the inability to accurately associate instance descriptions with their corresponding regions within visual conditions. To address this, we leverage the Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions. Specifically, the VLM automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. Furthermore, since the predicted masks may contain noise, we introduce an adaptive mask refinement strategy that dynamically refines these instance masks during the generation process. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.

Abstract

Image Generation

Explore similar work

InstanceControl: Controllable Complex Image Generation without Instance Labeling

TIMI: Training-Free Image-to-3D Multi-Instance Generation with Spatial Fidelity

Delta-K: Boosting Multi-Instance Generation via Cross-Attention Augmentation