Multimodal large language models (MLLMs) have made significant progress in understanding and interpreting mul- timedia content. However, their ability to generate me- dia remains limited. Recent approaches have attempted to bridge this gap by translating the hidden representations of token sequences into the embedding space of visual models or directly into raw image data. However, these methods often represent each image using multiple specialised to- kens which significantly increases the input length. This be- comes a major limitation for tasks such as graphic design generation where the output typically involves a seamless blend of thousands of tokens across text, multiple images, and layout information. To address this challenge, a novel architecture is proposed that maps hidden token represen- tations to the embedding space of visual models, such as CLIP ViT-L/14, using a single [IMG] token per image. The architecture employs two shallow MLP blocks, each with a separate compression module followed by a shared expan- sion module, trained with six distinct loss functions. One block aids the other during training and is omitted during inference, resulting in a lightweight solution. Strong perfor- mance is demonstrated in both image-to-design and text-to- design generation tasks.
Figures & tables
Figure 1: Example results from the proposed architecture. The model generates visually coherent and stylistically consistent graphic designs from either text or image inputs.
Figure 2: The proposed Giraffe architecture overview. It enables mapping from hidden token representations to the embedding space of visual models, such as CLIP ViT-L/14, using only a single [IMG] token. This is achieved using two shallow MLP blocks where each includes a seperate compression module followed by a shared expansion module, trained with six distinct loss functions (left). The assisting MLP block aids the main block in learning the mapping and is ignored during inference (right), leading to a more lightweight solution.
Model
Text-to-Design
Image-to-Design
(FID)
(Cosine)
Proposed
66.85
0.85±0.05
Baseline
81.61
-
Table 1: FID score and CLIP cosine similarity for text-to-design and image-to-design tasks, respectively.
Figure 3: Qualitative results for the text-to-design task. The generated graphic designs closely follow the prompts while remaining visually coherent, stylistically consistent, and of high overall quality. The model often produces complex designs with multiple images that interact harmoniously. Numbers in parentheses indicate the number of images used in each design.
Figure 4: Qualitative results for the baseline model. Generated results have limited layout variation and less consistent visual style. Numbers in parentheses indicate the number of images used in each design.
Figure 5: Qualitative results for the image-to-design task. In each pair, the left image shows the input generated by GPT-4o, and the right image shows the model’s output.
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +1