Planning and Rendering in Concert: DeepFusion of Autoregressive Layouts and Diffusion for Visual Text Generation
Authors: Guanqiao Chen, Jingru Tan, Dongxing Mao, Catherine Chen, Zijian Du, Libo Qin, Hu Jian Guo, Alex Jinpeng Wang
Organizations: University of Science and Technology of China · Central South University · Louisiana State University · Arizona State University · Sun Yat-sen University
Generating text-rich images from prompts requires both textual fidelity and the coherent integration of text into the surrounding image. An explicit layout can provide structured guidance about what text should appear and where, but a well-formed plan alone does not guarantee that the renderer will realize it faithfully. Existing layout-based AR-diffusion systems typically optimize planning and rendering separately, preventing the planner's representations from being adapted jointly with image synthesis. We introduce DuetGen, an autonomous visual text generator built on DeepFusion, which jointly learns autoregressive planning and continuous diffusion rendering. DeepFusion conditions a diffusion transformer on the planner's prompt and bbox-content hidden states, allowing rendering supervision to shape the representations connecting textual plans with visual outputs. Its joint objective combines autoregressive plan supervision, text-region-weighted diffusion learning, and auxiliary coordinate supervision to maintain structured planning, emphasize text-bearing regions, and improve the spatial precision of planner representations. During inference, Phase-Aware Attention Modulation strengthens the correspondence between image regions and their matched coordinate and content states, facilitating region-specific execution of the generated plan. With a 2B planner and a 4B single-stream DiT, DuetGen achieves 0.8293 word accuracy on CVTG-2K and 0.938 accuracy on LongText-Bench, closely matching the substantially larger Qwen-Image on both benchmarks. These results demonstrate the value of jointly learned planning representations and region-specific rendering for autonomous visual text generation.
Figures & tables
Figure 1: Paradigms for autonomous layout-aware visual text generation. (a) Separated AR–diffusion systems separately optimize planning and continuous rendering in two stages. (b) Fully autoregressive systems jointly model layout and image tokens, but rely on discrete visual representations. (c) DeepFusion jointly optimizes AR planning and continuous diffusion rendering through shared planner states and region-specific execution.
Figure 2: Overview of D u e t G e n . (a) DeepFusion jointly optimizes planning and rendering through prompt-and-plan states, autoregressive plan supervision, text-region-weighted diffusion learning, and auxiliary coordinate supervision. (b) During inference, Phase-Aware Attention Modulation strengthens attention from in-region patches to their matched bbox and content states in middle DiT layers and denoising steps.
Method
Word Accuracy ↑
NED ↑
CLIPScore ↑
2
3
4
5
Avg.
SD3.5 Large ( Esser et al., 2024 )
0.7293
0.6825
0.6574
0.5940
0.6548
0.8470
0.7797
FLUX.1 dev ( BlackForest, 2024 )
0.6089
0.5531
0.4661
0.4316
0.4965
0.6879
0.7401
AnyText ( Tuo et al., 2023 )
0.0513
0.1739
0.1948
0.2249
0.1804
0.4675
0.7432
TextDiffuser-2 ( Chen et al., 2024a )
0.5322
0.3255
0.1787
0.0809
0.2326
0.4353
0.6765
RAG-Diffusion ( Chen et al., 2024c )
0.4388
0.3316
0.2116
0.1910
0.2648
0.4498
0.7797
Table 1: Quantitative results on CVTG-2K ( Du et al., 2025 ) . Word accuracy is grouped by the number of text regions. Best and second-best results are bold and underlined .
Method
Acc. ↑
Method
Acc. ↑
BLIP3-o ( Chen et al., 2025b )
0.021
OmniGen2 ( Wu et al., 2025b )
0.561
Show-o2 ( Xie et al., 2025a )
0.119
FLUX.1 ( BlackForest, 2024 )
0.607
Janus-Pro ( Chen et al., 2025f )
0.019
TextDiffuser-2 ( Chen et al., 2024a )
0.371
Kolors 2.0 ( team, 2025 )
0.543
HiDream-I1-Full ( Cai et al., 2025 )
0.543
RAG-Diffusion ( Chen et al., 2024c )
0.321
TextCrafter ( Du et al., 2025 )
0.683
Qwen-Image ( Wu et al., 2025a )
0.943
D u e t G e n
0.938
Table 2: Text accuracy on LongText-Bench ( Geng et al., 2025 ) . Best and second-best results are bold and underlined .
Figure 3: Qualitative comparison on prompts containing multiple text regions and extended text.
Figure 6Table 7Figure 8
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Pairs
Share
Primary coverage
CC12M
4.0M
26.7%
Real-world images and long-tail concepts
LAION (high-quality subset)
3.5M
23.3%
Visual concepts, styles, and scenes
COYO (high-quality subset)
2.5M
16.7%
Web scenes and contemporary imagery
RedCaps
2.0M
13.3%
Natural user-authored descriptions
JourneyDB
2.0M
13.3%
Aesthetic synthetic images and detailed prompts
WIT
1.0M
6.7%
Entity, cultural, and limited multilingual coverage
Appendix
Table 6: Composition of the 15M-pair generic text-to-image pretraining mixture. The high-quality LAION and COYO subsets are sampled before pretraining.
Method
Vis. Aes. ↑
Text Qual. ↑
Spatial Harm. ↑
AnyText
7.12
7.85
7.43
TextDiffuser-2
7.45
8.21
7.92
FLUX.1 dev
8.92
8.64
8.15
D u e t G e n
8.75
9.12
8.95
Appendix
Table 7: Human evaluation on 50 randomly sampled CVTG-2K prompts. Scores range from 1 to 10; higher is better.
Model
Sng. Obj.
Tw. Obj.
Cntng
Clrs
Pstn
Clr Attr.
Overall
D u e t G e n
0.99
0.95
0.88
0.90
0.92
0.79
0.91
Appendix
Table 8: GenEval ( Ghosh et al., 2023 ) results. Sng. Obj.: single object; Tw. Obj.: two objects; Cntng: counting; Clrs: colors; Pstn: position; Clr Attr.: color attribution.
Condition
Target shift r
Target IoU ↑
Center Error ↓
Word Acc. ↑
CLIPScore ↑
Clean self-generated plan
0.00
0.9176
0.068
0.8309
0.7925
Forced valid-shift plan
0.35
0.9042
0.074
0.8295
0.7941
Appendix
Table 9: Counterfactual bbox-following results on the four-region CVTG-2K subset. The forced-shift plan preserves box size, stays in bounds, and contains no overlapping pair; only coordinate-span condition tokens differ from the clean run.
Figure 8: Additional qualitative results in diverse design settings. Panel (a) shows poster and comic examples; panel (b) shows book-cover and cinematic examples.
Figure 9: Qualitative examples on Korean (top) and Japanese (bottom) prompts, which are underrepresented in the current training distribution. D u e t G e n preserves the overall composition and script-like visual appearance, but may produce inaccurate strokes or character identities.
Text-to-image generation has advanced rapidly with diffusion models, progressing from CLIP and T5 conditioning to unified systems where a single LLM backbone handles both visual understanding and generation. Despite the architectural unification, these systems frequently fail to faithfully align complex prompts during synthesis, even though they remain highly accurate at verifying whether an image satisfies those same prompts. We formalize this as the understanding-generation gap and propose UniReasoner, a framework that leverages the LLM as a universal reasoner to convert its understanding strength into direct generation guidance. Given a prompt, the LLM first produces a coarse visual draft composed of discrete vision tokens. It then performs a self-critique by evaluating the draft for prompt consistency, producing a grounded textual evaluation that pinpoints what needs to be corrected. Finally, a diffusion model is conditioned jointly on the prompt, the visual draft, and the evaluation, ensuring that generation is guided by explicit corrective signals. Each signal addresses a limitation of the other: the draft provides a concrete, scene-level anchor that reduces under-specification in text-only conditioning, while the evaluation turns verification into grounded, actionable constraints that correct omissions, hallucinations, and relational errors. Experiments show that UniReasoner improves compositional alignment and semantic faithfulness under the same diffusion backbone while maintaining image quality, demonstrating a practical way to exploit LLM reasoning to close the understanding-generation gap.
Current text-to-image systems typically employ a "text encoder plus diffusion decoder" paradigm, in which text semantics directly modulate continuous latent noise. Despite their success, these methods lack an explicit, interpretable intermediate representation that effectively bridges high-level linguistic semantics and low-level visual signals. In this paper, we propose Vision-of-Thought (VoT), a framework that introduces a discrete visual-thinking layer between vision-language models (VLMs) and diffusion transformers (DiTs). Instead of treating VLMs merely as text encoders, we use them as multimodal planners that generate discrete VoT tokens representing high-level visual plans, such as objects and layouts, before rendering pixels. We train a specialized VoT tokenizer in the VLM semantic space with a closed-loop objective that combines VLM alignment, feature reconstruction, and vector-quantization losses. These objectives make the tokens semantically readable by the VLM while preserving the visual information needed for generation. Experimental results demonstrate that VoT improves semantic alignment and provides a structured interface for interpretable and controllable generation.
Text-to-image diffusion models like Stable Diffusion generate high-quality images from text, but lack a way to inject visual guidance (e.g. sketches, styles) at inference without retraining. Existing methods either require computationally expensive fine-tuning or rely on style transfer techniques that risk semantic misalignment with textual prompts. We introduce Visual Concept Fusion (VCF), the first method offering dual conditioning on both an image and text prompt at inference time without any concept-specific training. VCF enables visual concept injection into Stable Diffusion by aligning CLIP image features with the text embedding space. VCF consists of three components: (1) a lightweight aligner that maps image tokens to the text embedding manifold using InfoNCE and cross-attention reconstruction losses, (2) a fusion strategy that preserves both textual and visual semantics, and (3) an optional Prompt-Noise Optimization (PNO) module for test-time refinement. Our experiments demonstrate that VCF successfully transfers visual attributes including style, composition, and color palette from reference images while maintaining prompt adherence. Quantitative results show a trade-off between text alignment (CLIP score) and visual correspondence (LPIPS), with VCF outperforming baselines in reference fidelity.