EchoGen: Cycle-Consistent Learning for Unified Layout-Image Generation and Understanding
Authors: Kai Zou, Hongbo Liu, Dian Zheng, Jianxiong Gao, Zhiwei Zhao, Bin Liu
Organizations: School of Cyber Science and Technology, University of Science and Technology of China · Anhui Province Key Laboratory of Digital Security · Tongji University · Sun Yat-sen University · Fudan University · Hefei University of Technology
In this work, we present EchoGen, a unified framework for layout-to-image generation and image grounding, capable of generating images with accurate layouts and high fidelity to text descriptions (e.g., spatial relationships), while grounding the image robustly at the same time. We believe that image grounding possesses strong text and layout understanding abilities, which can compensate for the corresponding limitations in layout-to-image generation. At the same time, images generated from layouts exhibit high diversity in content, thereby enhancing the robustness of image grounding. Jointly training both tasks within a unified model can promote performance improvements for each. However, we identify that this joint training paradigm encounters several optimization challenges and results in restricted performance. To address these issues, we propose progressive training strategies. First, the Parallel Multi-Task Pre-training (PMTP) stage equips the model with basic abilities for both tasks, leveraging shared tokens to accelerate training. Next, the Dual Joint Optimization (DJO) stage exploits task duality to sequentially integrate the two tasks, enabling unified optimization. Finally, the Cycle RL stage eliminates reliance on visual supervision by using consistency constraints as rewards, significantly enhancing the model's unified capabilities via the GRPO strategy. Extensive experiments demonstrate state-of-the-art results on both layout-to-image generation and image grounding benchmarks, and reveal clear synergistic gains from optimizing the two tasks together.
Figures & tables
Figure 1: Our unified approach faithfully parses complex prompts—e.g., color attributes and relative spatial relations—and, combined with layout conditions, achieves more accurate semantic alignment. In contrast, specialized methods (e.g., Gligen and MIGC ) often fail to handle such constraints. For instance, in the first row of MIGC results, the model generates a vertical spatial relation (top–middle–bottom), whereas the prompt specifies a depth-based relation (front–middle–back).
Figure 2: Overview of EchoGen . Left—Parallel Multi‑task Pre‑training: a unified autoregressive transformer is trained on layout‑to‑image and image grounding tasks in parallel, yielding fast acquisition of base capabilities. Middle—Dual Joint Optimization: the image tokens generated by the generation forward pass are directly reused as the input to grounding, forming a single joint objective, which strengthens layout → image → layout cycle consistency. Right—Cycled RL: we execute the layout → image → layout loop and treat the discrepancy between input and recovered layouts as a continuous reward, enabling self-supervised Reinforcement Learning without explicit supervision of intermediate visual outputs.
Figure 3: Attention Mask of parallel multi-task pre-training.
Method
MS COCO
LayoutSAM-Eval
AP↑
AP50↑
AP75↑
CLIP ↑
FID ↓
Spatial ↑
Color ↑
Texture ↑
Shape ↑
Gen. only
GLIGEN
30.99
41.44
36.89
24.81
27.93
77.53
49.41
55.29
52.72
MIGC
46.16
58.29
50.21
25.04
25.35
85.66
66.97
71.24
69.06
InstanceDiffusion
49.97
61.33
52.65
25.15
25.00
87.99
69.16
72.78
71.08
IFAdapter
43.31
56.29
52.76
26.39
23.45
84.32
68.32
71.37
68.37
Table 1: Quantitative results on MS COCO and LayoutSAM-Eval benchmark. Method marked as Gen. only perform layout-to-image generation only, while Und. and Gen. denotes unified layout-to-image generation and grounding models.
Stage
Samples
Steps
Parallel Multi‑task Pre‑training
4M
125K
Dual Joint Optimization
2M
60K
Cycled RL
50K
50K
Table 2: Training recipe.
Figure 4: Quantitative comparison on layout-to-image generation. The first column lists the global text prompts; the second column visualizes the layout conditions, rendered as (referring-expression, bounding-box) pairs. We highlight a subset of negatives and positives with red and green boxes; note that negatives may correspond to incorrect spatial locations or attributes (e.g., color) .
Method
Ref-L4
Acc0.5
Acc0.75
Acc0.9
mAcc
Und. only
KOSMOS-2
48.53
38.34
17.54
34.72
Qwen-VL-Chat
73.80
58.05
37.16
55.94
CogVLM-g.
81.70
70.77
48.35
66.09
Und. and Gen.
Table 3: Quantitative results on Ref-L4. Method marked as Und. only perform image grounding only, while Und. and Gen. denotes unified layout-to-image generation and grounding models.
Stage 1
Stage 2
Stage 3
AP ↑
FID ↓
#1
✓
47.26
21.98
#2
✓
✓
52.38
22.23
#3
✓
✓
48.01
20.32
#4
✓
✓
✓
54.61
20.12
Table 4: Ablation over training stages.
Figure 5: Ablation on stage transition points. Left: gains when switching from Stage 1 → Stage 2; Right: gains when switching from Stage 2 → Stage 3. The x - axis records the AP at entry to the current stage (determined by the preceding stage’s training budget), while the y - axis reports the AP improvement achieved by that stage.
Shanghai Jiao Tong University, Shanghai, China · Institute of Artificial Intelligence, China Telecom (TeleAI), Shanghai, China · University of Science and Technology of China, Hefei, China