cs.CVAug 27, 2026

Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models

Authors: Yiyang Huang, Zhaowen Wang, Simon Jenni, Jing Shi, Yitian Zhang, Yizhou Wang, Yun Fu

Organizations: Northeastern University, Boston, USA · Adobe Research, San Jose, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Layout understanding, or the interpretation of element organization, is essential for document analysis, user interface (UI) creation, and graphic design. While recent vision-language models (VLMs) excel at interpreting atomic layouts composed of independent elements, they struggle with compositional layouts that require reasoning over visually entangled elements within hierarchical multi-layer structures. In this paper, we introduce a new task, compositional layout understanding, and present CoDeLayout, a VQA dataset of ~20K real-world multi-layer layouts annotated with compositional element pairs and design intent. Through empirical analysis on CoDeLayout, we identify two key challenges for existing VLMs: semantic drift between textual metadata and visual content, and structural ambiguity in hierarchical inter-element relationships. To address these challenges, we propose MASON, a post-training paradigm that integrates multimodal alignment (MA) and structural perception (SP). MA enhances element interpretation by grounding metadata-defined elements to their visual counterparts, mitigating semantic drift, while SP models layer-aware inter-element spatial relationships to improve hierarchical understanding and reduce structural ambiguity. Experiments reveal substantial gaps in existing VLMs: even the strongest baseline, GPT-o3, achieves only 79.68% accuracy, whereas Qwen2.5-VL 7B with MASON reaches 91.66%. Notably, MASON surpasses full-data Direct Finetune using only 30% of the training data and scales better with additional data.

Figures & tables

Figure 1: Compositional layouts in graphic design. ( 1(a) – 1(d) ) show representative compositional layouts, with green boxes marking key elements. ( 1(a) ) Clipping extracts a region from a base image as a separate layer, allowing insertion of new elements between the base and the cutout; ( 1(b) ) Blending integrates different crops of the same image into stylized shapes for aesthetic coherence; ( 1(c) ) Overlaying combines an element with its outline or shadow to emphasize visual hierarchy; and ( 1(d) ) Morphing replaces text with graphics to create expressive visual effects. ( 1(e) ) Compositional layout understanding is critical for design assistants, as handling such layouts often requires coordinated edits across interrelated elements to faithfully implement user intent.
Figure 2: Data format of a compositional design. Each instance includes a rendered layout image, element-level metadata, and QA-style annotations specifying compositional element pairs and their associated design intents.
DatasetAnnotationStructureDomain# Samples# Layers
Layout Generation
RICO [ 10 ]N/AFlatUI∼\sim 72KN/A
Magazine [ 52 ]N/ALayeringGraphic∼\sim 3.9KN/A
GenPoster-100K [ 13 ]N/ALayeringGraphic∼\sim 100K2–20
DesignerIntention [ 21 ]N/ALayeringGraphic∼\sim 100K3–10
Crello [ 47 ]N/ALayeringGraphic∼\sim 20K2–50
Table 1: Comparison of representative layout datasets. In contrast to existing datasets that primarily focus on atomic layouts or generative objectives, CoDeLayout provides multi-layer layouts with explicit annotations of inter-element compositional relationships and the design intent.
Figure 3: Impact of semantic drift (a) and structural ambiguity (b) on compositional layout understanding. Higher grounding performance and spatial relation perception accuracy consistently correspond to better compositional element identification accuracy, indicating that multimodal alignment and structural perception are essential for compositional layout understanding.
Figure 4: Overview of MASON. MASON mitigates semantic drift and structural ambiguity through multimodal alignment (MA) and structural perception (SP). MA introduces an element grounding objective to align textual metadata with visual content under visual entanglement. SP augments metadata with layer-aware spatial relationships, enabling accurate modeling of hierarchical inter-element structures.
Element Grounding Prompt Pground\mathcal{P}_{\text{ground}}
Inputs:
- Full design image: I\mathcal{I}
- Layout metadata: M={Me1,Me2,…,Men}\mathcal{M}=\{\mathcal{M}_{e_{1}},\mathcal{M}_{e_{2}},\dots,\mathcal{M}_{e_{n}}\}
- Element ID: {element_id}
- Image crop and metadata of the element: (Ie,Me)(\mathcal{I}_{e},\mathcal{M}_{e})
 
Table 2: Prompt template Pground\mathcal{P}_{\text{ground}} for generating element grounding QA pairs in multimodal alignment.
Spatial Relationship Schema
spatialRelationship: {
hasOverlap: <True ∣\mid False>, overlapPercentage: <float>,
containment: <contain ∣\mid contained ∣\mid none>, centerDistance: <float>,
relativePosition: <left ∣\mid right ∣\mid above ∣\mid below>, layerOrder: <above_given ∣\mid below_given>
}
Table 3: Schema of layer-aware inter-element spatial relationships used for structural perception.
ModelOverlayingClippingBlendingMorphingWeighted Acc.Average Acc.
Heuristic Baselines
Max-Overlap61.74 (N/A)69.70 (N/A)23.08 (N/A)36.17 (N/A)44.2747.67
Nearest-Neighbor58.26 (N/A)62.12 (N/A)23.72 (N/A)38.30 (N/A)42.4445.60
Open-Source VLMs
Qwen2.5-VL66.09 (3.51/3.65/23.94)37.88 (2.85/1.16/18.94)53.85 (3.37/2.48/22.28)36.17 (3.22/2.16/21.42)52.6048.49
Qwen3-VL86.09 (4.52/7.63/32.77)71.21 (3.85/3.81/26.12)79.49 (4.11/4.59/28.55)55.32 (3.49/2.22/23.04)77.0873.02
Table 4: Performance on CoDeLayout across four compositional categories. Each cell reports Accuracy (GPT-Score/BLEU/ROUGE). Accuracy ( ↑\uparrow ) evaluates compositional element identification, while GPT-Score/BLEU/ROUGE ( ↑\uparrow ) measure explanation quality. Bold and underline indicate the best and second-best results. Weighted Accuracy is computed using test sample proportions, and Average Accuracy is the unweighted mean across categories. The results highlight challenges in compositional layout understanding for existing VLMs and MASON’s effectiveness and data efficiency.
ModelOverlayingClippingBlendingMorphingWeighted Acc.Average Acc.
DF86.0974.2480.7772.3480.2178.36
DF + MA87.8381.8281.4170.2182.0380.32
DF + SP89.5783.3382.6974.4783.8582.51
MASON86.9681.8283.9780.8584.1183.40
Table 5: Module ablation on CoDeLayout under a type-balanced setting. DF denotes Direct Finetune, MA denotes Multimodal Alignment, and SP denotes Structural Perception. Results report accuracy ( ↑\uparrow ) for each compositional category, along with Weighted and Average accuracy. Weighted accuracy accounts for category proportions, while Average accuracy is the unweighted mean. Bold indicates the best result in each column. Results show that MA and SP provide complementary improvements, with their combination achieving the strongest overall performance.
Figure 5: Data Scale Ablation. Performance of Direct Finetune and MASON trained on 10%, 15%, 30%, and 100% of the CoDeLayout training set. The x-axis denotes training data scale, the left y-axis shows test set weighted accuracy, and the right y-axis shows accuracy gain of MASON over Direct Finetune. MASON consistently outperforms across all scales, with a clear advantage under low-resource settings.
Table 11
Figure 6: Case studies across four compositional categories. Each example shows the full design, the query element, and the predictions of Direct Finetune and MASON.
More in this topic

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Explore similar work

CardsList
Sep 29, 2026

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.

L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5

NIT Silchar · MBZUAI · BML Munjal University +1

VLM EvaluationVisual Question AnsweringVision-Language GroundingVLM ReasoningCompositional Reasoning in VLMs
AskPDF
Mar 17, 2026

LICA: Layered Image Composition Annotations for Graphic Design Research

We introduce LICA (Layered Image Composition Annotations), a large scale dataset of 1,550,244 multi-layer graphic design compositions designed to advance structured understanding and generation of graphic layouts. In addition to rendered PNG images, LICA represents each design as a hierarchical composition of typed components including text, image, vector, and group elements, each paired with rich per-element metadata such as spatial geometry, typographic attributes, opacity, and visibility. The dataset spans 20 design categories and 971,850 unique templates, providing broad coverage of real-world design structures. We further introduce graphic design video as a new and largely unexplored challenge for current vision-language models through 27,261 animated layouts annotated with per-component keyframes and motion parameters. Beyond scale, LICA establishes a new paradigm of research tasks for graphic design, enabling structured investigations into problems such as layer-aware inpainting, structured layout generation, controlled design editing, and temporally-aware generative modeling. By representing design as a system of compositional layers and relationships, the dataset supports research on models that operate directly on design structure rather than pixels alone.

Elad Hirsch, Shubham Yadav, Mohit Garg +1

Compositional Image Generation
AskPDF
May 19, 2026

Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding

Vision-Language Models (VLMs) parse documents end-to-end but frequently break down on layouts unlike those seen in training. We attribute this to a two-hop bottleneck: before the decoder can extract content (Hop 2), it must first classify and localize the enclosing layout entity (Hop 1), and when the first hop fails the second collapses into omissions, malformed structure, or autoregressive repetition. We pre-resolve Hop 1 outside the decoder by running a lightweight RT-DETR detector, serializing its outputs in the parser's native DocTags vocabulary, and injecting them into the prompt alongside the full page image. Unlike analyze-then-parse approaches that crop the page, or prior prompt-level priors written in plain text, our prior shares the decoder's generation space and leaves the global image in view as a fallback when detections are noisy. On a 10k-page structural out-of-distribution benchmark, markdown F1 rises from 0.370.37 to 0.920.92; on the Chinese subset of OmniDocBench, table TEDS rises from 0.010.01 to 0.360.36; and on the 26k-page ViDoRe V3 benchmark, infinite-loop decoding failures drop across every industrial domain tested. These gains cost 15%15\% wall-clock latency and a median of 7474 prompt tokens, with no architectural change to the base VLM. An attention-level analysis further reveals a bimodal phase shift in which the decoder attends to injected layout tokens when emitting structure and to image patches when emitting content, consistent with the two-hop bottleneck being alleviated. Model weights will be released to support reproducibility.

Peter El Hachem, Ahmed Nassar, A. Said Gurbuz +2

ETH Zurich Rämistrasse, Switzerland · IBM Research Rüschlikon, Switzerland

VLM RobustnessLLM PromptingDocument Layout AnalysisDocument Image Understanding
AskPDF