Layout understanding, or the interpretation of element organization, is essential for document analysis, user interface (UI) creation, and graphic design. While recent vision-language models (VLMs) excel at interpreting atomic layouts composed of independent elements, they struggle with compositional layouts that require reasoning over visually entangled elements within hierarchical multi-layer structures. In this paper, we introduce a new task, compositional layout understanding, and present CoDeLayout, a VQA dataset of ~20K real-world multi-layer layouts annotated with compositional element pairs and design intent. Through empirical analysis on CoDeLayout, we identify two key challenges for existing VLMs: semantic drift between textual metadata and visual content, and structural ambiguity in hierarchical inter-element relationships. To address these challenges, we propose MASON, a post-training paradigm that integrates multimodal alignment (MA) and structural perception (SP). MA enhances element interpretation by grounding metadata-defined elements to their visual counterparts, mitigating semantic drift, while SP models layer-aware inter-element spatial relationships to improve hierarchical understanding and reduce structural ambiguity. Experiments reveal substantial gaps in existing VLMs: even the strongest baseline, GPT-o3, achieves only 79.68% accuracy, whereas Qwen2.5-VL 7B with MASON reaches 91.66%. Notably, MASON surpasses full-data Direct Finetune using only 30% of the training data and scales better with additional data.
Figures & tables
Figure 1: Compositional layouts in graphic design. ( 1(a) – 1(d) ) show representative compositional layouts, with green boxes marking key elements. ( 1(a) ) Clipping extracts a region from a base image as a separate layer, allowing insertion of new elements between the base and the cutout; ( 1(b) ) Blending integrates different crops of the same image into stylized shapes for aesthetic coherence; ( 1(c) ) Overlaying combines an element with its outline or shadow to emphasize visual hierarchy; and ( 1(d) ) Morphing replaces text with graphics to create expressive visual effects. ( 1(e) ) Compositional layout understanding is critical for design assistants, as handling such layouts often requires coordinated edits across interrelated elements to faithfully implement user intent.
Figure 2: Data format of a compositional design. Each instance includes a rendered layout image, element-level metadata, and QA-style annotations specifying compositional element pairs and their associated design intents.
Dataset
Annotation
Structure
Domain
# Samples
# Layers
Layout Generation
RICO [ 10 ]
N/A
Flat
UI
∼ 72K
N/A
Magazine [ 52 ]
N/A
Layering
Graphic
∼ 3.9K
N/A
GenPoster-100K [ 13 ]
N/A
Layering
Graphic
∼ 100K
2–20
DesignerIntention [ 21 ]
N/A
Layering
Graphic
∼ 100K
3–10
Crello [ 47 ]
N/A
Layering
Graphic
∼ 20K
2–50
Table 1: Comparison of representative layout datasets. In contrast to existing datasets that primarily focus on atomic layouts or generative objectives, CoDeLayout provides multi-layer layouts with explicit annotations of inter-element compositional relationships and the design intent.
Figure 3: Impact of semantic drift (a) and structural ambiguity (b) on compositional layout understanding. Higher grounding performance and spatial relation perception accuracy consistently correspond to better compositional element identification accuracy, indicating that multimodal alignment and structural perception are essential for compositional layout understanding.
Figure 4: Overview of MASON. MASON mitigates semantic drift and structural ambiguity through multimodal alignment (MA) and structural perception (SP). MA introduces an element grounding objective to align textual metadata with visual content under visual entanglement. SP augments metadata with layer-aware spatial relationships, enabling accurate modeling of hierarchical inter-element structures.
Element Grounding Prompt Pground
Inputs:
- Full design image: I
- Layout metadata: M={Me1,Me2,…,Men}
- Element ID: {element_id}
- Image crop and metadata of the element: (Ie,Me)
Table 2: Prompt template Pground for generating element grounding QA pairs in multimodal alignment.
Table 3: Schema of layer-aware inter-element spatial relationships used for structural perception.
Model
Overlaying
Clipping
Blending
Morphing
Weighted Acc.
Average Acc.
Heuristic Baselines
Max-Overlap
61.74 (N/A)
69.70 (N/A)
23.08 (N/A)
36.17 (N/A)
44.27
47.67
Nearest-Neighbor
58.26 (N/A)
62.12 (N/A)
23.72 (N/A)
38.30 (N/A)
42.44
45.60
Open-Source VLMs
Qwen2.5-VL
66.09 (3.51/3.65/23.94)
37.88 (2.85/1.16/18.94)
53.85 (3.37/2.48/22.28)
36.17 (3.22/2.16/21.42)
52.60
48.49
Qwen3-VL
86.09 (4.52/7.63/32.77)
71.21 (3.85/3.81/26.12)
79.49 (4.11/4.59/28.55)
55.32 (3.49/2.22/23.04)
77.08
73.02
Table 4: Performance on CoDeLayout across four compositional categories. Each cell reports Accuracy (GPT-Score/BLEU/ROUGE). Accuracy ( ↑ ) evaluates compositional element identification, while GPT-Score/BLEU/ROUGE ( ↑ ) measure explanation quality. Bold and underline indicate the best and second-best results. Weighted Accuracy is computed using test sample proportions, and Average Accuracy is the unweighted mean across categories. The results highlight challenges in compositional layout understanding for existing VLMs and MASON’s effectiveness and data efficiency.
Model
Overlaying
Clipping
Blending
Morphing
Weighted Acc.
Average Acc.
DF
86.09
74.24
80.77
72.34
80.21
78.36
DF + MA
87.83
81.82
81.41
70.21
82.03
80.32
DF + SP
89.57
83.33
82.69
74.47
83.85
82.51
MASON
86.96
81.82
83.97
80.85
84.11
83.40
Table 5: Module ablation on CoDeLayout under a type-balanced setting. DF denotes Direct Finetune, MA denotes Multimodal Alignment, and SP denotes Structural Perception. Results report accuracy ( ↑ ) for each compositional category, along with Weighted and Average accuracy. Weighted accuracy accounts for category proportions, while Average accuracy is the unweighted mean. Bold indicates the best result in each column. Results show that MA and SP provide complementary improvements, with their combination achieving the strongest overall performance.
Figure 5: Data Scale Ablation. Performance of Direct Finetune and MASON trained on 10%, 15%, 30%, and 100% of the CoDeLayout training set. The x-axis denotes training data scale, the left y-axis shows test set weighted accuracy, and the right y-axis shows accuracy gain of MASON over Direct Finetune. MASON consistently outperforms across all scales, with a clear advantage under low-resource settings.
Table 11
Figure 6: Case studies across four compositional categories. Each example shows the full design, the query element, and the predictions of Direct Finetune and MASON.