Recent text-to-image models can generate high-quality images from natural-language prompts, yet controlling typography remains challenging: requested typographic appearance is often ignored or only weakly followed. We address this limitation with a data-centric approach that trains image generation models using targeted supervision derived from a structured annotation pipeline specialized for typography. Our pipeline constructs a large-scale typography-focused dataset, FontUse, consisting of about 70K images annotated with user-friendly prompts, text-region locations, and OCR-recognized strings. The annotations are automatically produced using segmentation models and multimodal large language models (MLLMs). The prompts explicitly combine font styles (e.g., serif, script, elegant) and use cases (e.g., wedding invitations, coffee-shop menus), enabling intuitive specification even for novice users. Fine-tuning existing generators with these annotations allows them to consistently interpret style and use-case conditions as textual prompts without architectural modification. For evaluation, we introduce a Long-CLIP-based metric that measures alignment between generated typography and requested attributes. Experiments across diverse prompts and layouts show that models trained with our pipeline produce text renderings more consistent with prompts than competitive baselines. The source code for our annotation pipeline is available at https://github.com/xiaxinz/FontUSE.
Figures & tables
Figure 2: Dataset construction pipeline. (a) Typography-focused images are collected from public design resources. (b) Hi-SAM [ 33 ] detects text regions and outputs bounding boxes. (c)-(e) An MLLM performs text recognition, font-style annotation, and use-case annotation.
Figure 3: Example of the input prompt and an MLLM responses. See Appendix A for details of ⟨example format⟩.
Figure 4: Prompt used for text recognition. The MLLM is instructed to output only the extracted word (case-sensitive), returning “-” if no text is detected and “#” for non-Roman characters.
Figure 5: Top-40 style and use-case families after consolidation, illustrating their frequency distributions.
Figure 6: Cumulative distributions of use-case and style families. Dashed lines indicate 50%, 80%, and 90% cumulative mass.
AnyText
TD-2
SD-3
base
FT
base
FT
base
FT
CLIP ↑
0.3644
0.3241
0.4364
0.4550
0.4189
0.4355
Long-CLIP ↑
0.3946
0.4464
0.4765
0.5170
0.4696
0.5618
Fine-tuned Long-CLIP ↑
0.5858
0.6368
0.6097
0.7414
0.5068
0.6297
GPT selections ↑
12/60
48/60
9/60
51/60
16/60
44/60
Gemini selections ↑
14/60
46/60
11/60
49/60
16/60
44/60
Table 1: Comparison of CLIP, original Long-CLIP and fine-tuned Long-CLIP scores (linearly normalized to [0,1] ), MLLM preference counts, and legibility (measured by the average character error rate, CER). Boldface indicates the best result within each backbone. FT denotes “fine-tuned.”
Figure 7: Qualitative comparison with AnyText [ 28 ] and FontUse-tuned model (AnyText fine-tuned using our dataset). In the text prompts, the target string to be rendered as a text image is shown in green , the font style in brown , and the font use case in blue .
Figure 8: Qualitative comparison with Stable Diffusion 3 (SD-3) [ 27 ] , TextDiffuser-2 [ 5 ] , and FontUse-tuned models (SD-3 and TextDiffuser-2 fine-tuned using our dataset).
Figure 9: Comparison of outputs across different use-case conditions. Underlined text indicates prompt differences. Input image and edit region omitted.
Figure 10: Comparison of controllability under color and effect specifications (shown in purple ). Input image and edit region omitted.
CLIP
Long-CLIP
Fine-tuned Long-CLIP
(a) corresponding
22.13
65.31
72.76
(b) non-corresponding
21.71
61.40
62.72
(c) unrelated text
19.88
38.98
33.56
Gain (a vs. b) ↑
+1.9 %
+6.4 %
+16.0 %
Gain (a vs. c) ↑
+11.3 %
+67.5 %
+116.8 %
Table 2: Validation of the alignment metrics. We report the average similarity scores, computed by each metric between images and texts under three conditions: (a) corresponding use-case text, (b) non-corresponding use-case text, and (c) unrelated text.
Method
Average CER ↓
Tesseract OCR [ 26 ]
71.88%
EasyOCR [ 14 ]
36.23%
PaddleOCR(v3) [ 16 ]
12.44%
olmOCR-7B [ 22 ]
3.78%
Ours
2.18%
Table 3: OCR performance measured by average CER.
Figure 11: Our failure cases. There is a trade-off between stylistic complexity and legibility.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Details of the required ⟨example format⟩ to enforce strict compliance with the target format.
Figure 13: Prompt for the MLLM-based evaluation. The detailed instructions are crafted to produce reliable typography-focused preference judgments.
ID
Prompt
1
Suitable for romantic and playful design projects, fun and flirty which fit brandings of dating services or Valentine’s Day promotions. Cute, pretty, vague, nostalgic
2
The font has a liquid, distorted look, making it ideal for creative projects that need a rugged and organic feel
3
A hand-drawn layered font, it has a unique, crooked look with irregular strokes and outlines, making it perfect for logos & branding, stationery, book covers, magazine design, clothing prints
4
For advertising campaigns focused on youthful, casual consumers, targeting parties or social gatherings
Suitable for: modern and artistic design projects. Ideal applications: [minimalistic aesthetics] branding for lifestyle brands, cafes, or boutique stores emphasizing simplicity and elegance. Font styles: bold, modern, sans-serif, clean, minimal, elegant
Appendix
Table 4: Evaluation prompts (ID: 1-15). Font styles are indicated in brown and use cases in blue .
ID
Prompt
16
Handwritten, rough, textured and brush script. Used in social media graphics, promotional content, and blog headers for lifestyle, travel, or personal brands
17
Suitable for posters or prints with motivational quotes for decor or gifts. Creative branding for businesses in arts, crafts, or entertainment
18
Youth-oriented, for merchandise like t-shirts, stickers, and accessories for a younger audience
19
Playful and vibrant, rounded font. Used for restaurant branding, pizza menus, and promotional materials
20
Sans-serif font, modern, wide, sleek and geometric shape. Suitable for futuristic and tech-oriented design projects, as technology companies or innovative startups branding, and futuristic products
21
Elegant, serif, modern. Logos for high-end clothing brands, upscale boutiques, elegant fashion lines. Books for lifestyle or beauty brands, product packaging
Appendix
Table 5: Evaluation prompts (ID: 16-30). Font styles are indicated in brown and use cases in blue .
Typography generation in diffusion models faces a persistent trade-off: enabling precise font control typically degrades text legibility, while maintaining readability often sacrifices typographic fidelity. We present FontFusion, a plug-and-play conditioning framework for Diffusion Transformer (DiT) architectures that resolves this dilemma through three core innovations: (1) a hierarchical token representation establishing explicit text-font relationships at multiple granularities, (2) position-aware embeddings creating spatial bindings between typography and image content, and (3) a multi-level token dropping strategy improving both computational efficiency and generalization to unseen fonts. Our systematic evaluation of font embedding spaces reveals that a dual encoder combining DeepFont and DINOv2 outperforms any single encoder for typography tasks. FontFusion demonstrates 76% relative improvement on challenging decorative fonts over single-encoder baselines and font consistency gains exceeding approximately 68-76% over unconditioned models, while integrating into existing DiT architectures without retraining.
Marian Lupascu, Nipun Jindal, Ionut Mironica +1
Adobe Research · Department of Computer Science, University of Bucharest, Romania
Vector glyphs are the atomic units of digital typography, yet most learning-based pipelines still depend on carefully curated exemplar sheets and raster-to-vector postprocessing, which limits accessibility and editability. We introduce VecGlypher, a single multimodal language model that generates high-fidelity vector glyphs directly from text descriptions or image exemplars. Given a style prompt, optional reference glyph images, and a target character, VecGlypher autoregressively emits SVG path tokens, avoiding raster intermediates and producing editable, watertight outlines in one pass. A typography-aware data and training recipe makes this possible: (i) a large-scale continuation stage on 39K noisy Envato fonts to master SVG syntax and long-horizon geometry, followed by (ii) post-training on 2.5K expert-annotated Google Fonts with descriptive tags and exemplars to align language and imagery with geometry; preprocessing normalizes coordinate frames, canonicalizes paths, de-duplicates families, and quantizes coordinates for stable long-sequence decoding. On cross-family OOD evaluation, VecGlypher substantially outperforms both general-purpose LLMs and specialized vector-font baselines for text-only generation, while image-referenced generation reaches a state-of-the-art performance, with marked gains over DeepVecFont-v2 and DualVector. Ablations show that model scale and the two-stage recipe are critical and that absolute-coordinate serialization yields the best geometry. VecGlypher lowers the barrier to font creation by letting users design with words or exemplars, and provides a scalable foundation for future multimodal design tools.
Faithful text rendering remains a persistent weakness of large text-to-image generative models, as it requires both semantic instruction following and fine-grained glyph-level structure. Prior methods often improve this ability through architecture-specific modules or encoder modifications, which complicate deployment across foundation models. We study text rendering as a post-training preference-alignment problem and propose TextAlign, a non-invasive framework that keeps the generator architecture unchanged. The key component is a hierarchical vision-language model (VLM)-based reward that decomposes rendering errors into global, word, and glyph levels, then converts binary defect judgments into a scalar preference signal. The resulting signal supports both Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO). Experiments on FLUX.1-dev and Z-Image-Turbo show consistent gains in OCR-based text accuracy without degrading general generation quality. Compared with strong foundation and text-rendering baselines, including SD3.5, Qwen-Image, AnyText, and TextDiffuser, these results indicate that reward design offers a scalable alternative to model redesign for improving text rendering.
Mingxuan Cui, Jingpu Yang, Fengxian Ji +7
Mohamed bin Zayed University of Artificial Intelligence · Chinese Academy of Sciences Institute of Automation · Northeastern University +1