AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards
Authors: Xin Gao, Yifan Yang, Xin Li, Pan Wang, Yan Li, Muzhao Tian, Mingxi Cheng, Bei Liu, Kai Qiu, Qi Dai, Dong Chen, Yue Dong, Leandros Tassiulas, Rex Ying, Ji Li, Chong Luo
Organizations: Yale University · Microsoft · Nanyang Technological University · University of Pittsburgh · Shanghai Jiao Tong University · Fudan University
Generating information-rich visual artifacts such as slides, posters, and dashboards as code is appealing because the outputs are structured, editable, and verifiable. Yet code models cannot see how their choices of layout and color come together on the canvas, often limiting spatial planning and aesthetic quality. Image generators excel at visual design but can misrender requested content, making their outputs unreliable targets for direct reproduction. We therefore introduce AesCode, which generates code grounded in the content prompt while using an image generated from the same prompt as an aesthetic guidance. To do so, AesCode represents the target as a design graph that describes the whole canvas and makes each property attributable. From this graph, we derive complementary cross-modal rewards: deterministic verifiers score properties parsable from code and rendering, while a VLM judge assesses the non-parsable ones with a sample-specific Visual Graph Rubric (VGR) tied to graph elements and relations. We optimize these decoupled rewards through reinforcement learning with per-channel normalization to jointly improve spatial planning, aesthetic quality, and verifiable correctness. To evaluate this setting, we build a benchmark that scores verifiable correctness and visual quality separately, with visual scores highly consistent with human judgments. We find that reference images substantially improve the visual quality of code generated by existing models, and ablations show that AesCode benefits from references both as model input and as a source of visual rewards. Relative to same-scale Qwen3-VL baselines given the same references, AesCode-8B and AesCode-32B improve visual quality by 31.1 and 22.4 percentage points, reaching scores comparable to those of closed-source frontier models such as GPT-5.5 and Claude Opus 4.8.
Figures & tables
Figure 1: Information-rich infographics generated as code. Prompts specify content, while image-generator references offer visual design guidance. The examples show a landscape dashboard and portrait poster rendered from generated HTML/CSS.
Figure 2: Overview of AesCode. Prompt requirements and reference design cues yield graph-based supervision. Conditioned on p and Iref , the policy generates HTML/CSS code c , rendered as I^ .
Model
Text
Bound
Chart
Rule
Content
Layout
Style
Visual
Overall
Frontier Closed-Source
GPT-5.5
88.67
86.18
82.85
85.90
79.27
68.09
52.80
66.72
76.31
GPT-5.4
76.37
86.14
65.30
75.94
71.05
63.29
47.31
60.55
68.24
GPT-5.1
80.42
78.64
73.68
77.58
73.17
59.76
43.12
58.68
68.13
GPT-4o
73.64
78.37
55.64
69.22
51.34
33.33
15.95
33.54
51.38
Claude Opus 4.8
91.40
85.03
87.69
88.04
84.98
64.88
46.72
65.53
76.78
Table 1: Results on our infographic benchmark of 300 instances. Models in the first two blocks receive only the prompt; those in the third, including AesCode, also receive a generated reference image. Rule averages the programmatic Text, Bound, and Chart checks; Visual averages the Content, Layout, and Style scores as defined in Section 3.4 . Overall is the mean of Rule and Visual. Scores (%) are averaged over three generations per prompt; higher is better. Bold and underlined values indicate the best and second-best scores in each column.
Figure 3: Qualitative comparisons selected by the three-rollout mean Overall gap. Each row shows the reference and outputs from GPT-5.5 and AesCode-32B. Bars report evaluation scores; red dashed boxes mark visible GPT-5.5 errors.
GDB/lica
SlidesBench
Design2Code
Method
NIMA
CLIP
Pick
FID ↓
Harm.
OCR
Cont.
Layout
Color
Block
Text
Pos.
Color
CLIP
GPT-5.5
4.362
30.2
19.53
0.439
70.6
71.9
4.54
2.85
3.63
92.3
98.5
94.2
95.3
94.3
GPT-4o
2.589
24.1
17.82
0.673
69.3
43.3
3.41
2.00
2.81
77.5
93.0
77.9
82.1
89.5
GLM-4.5V
3.682
26.4
18.49
0.541
70.3
65.1
4.22
2.14
3.00
76.7
88.2
73.8
84.7
90.3
Qwen2.5-VL-72B
3.564
26.0
18.40
0.537
69.1
62.6
4.03
2.02
2.83
82.6
97.3
78.0
78.8
88.9
Qwen3-VL-30B-A3B
3.461
26.1
18.38
0.532
68.8
57.6
3.95
2.10
2.97
86.1
95.6
80.5
91.3
91.4
Table 2: Transfer to three external benchmarks. GDB/lica evaluates text-to-image generation; SlidesBench and Design2Code evaluate image-to-code reconstruction. All models output rendered HTML/CSS under each benchmark’s official protocol. SlidesBench uses a 0–5 scale; Design2Code and GDB/lica CLIP, Harm., and OCR use 0–100. Higher is better except FID.
Model
Text
Bound
Chart
Rule
Content
Layout
Style
Visual
Overall
Reference input removed at inference only
GPT-5.5
88.67 (−0.6)
86.18 (−0.7)
82.85 (+1.5)
85.90 (+0.1)
79.27 (−3.2)
68.09 (−21.8)
52.80 (−5.1)
66.72 (−10.0)
76.31 (−5.0)
Qwen3-VL-32B
66.00 (−5.2)
80.31 (+0.7)
41.36 (−9.4)
62.55 (−4.6)
49.81 (−12.2)
34.85 (−29.7)
19.34 (−19.1)
34.67 (−20.4)
48.61 (−12.5)
Qwen3-VL-8B
58.32 (−5.8)
77.23 (+6.2)
43.30 (+1.0)
59.61 (+0.5)
38.33 (−14.6)
23.97 (−27.4)
13.22 (−16.7)
25.17 (−19.6)
42.39 (−9.5)
AesCode-8B
93.74 (−0.3)
88.19 (−0.2)
82.36 (−5.4)
88.09 (−2.0)
84.92 (−1.5)
86.84 (−1.0)
52.63 (−0.6)
74.80 (−1.0)
81.45 (−1.5)
Reference input removed during training and inference
Table 3: Reference images during training and inference. All rows omit reference input at inference; the lower row also omits it from policy inputs during training while retaining reference-based rewards. Gray values show changes from reference-conditioned results.
Figure 4: Human evaluation. (a) Agreement with training VGR and evaluation-checklist judgments. (b) Content satisfaction and aesthetic equivalence between AesCode-32B and GPT-5.5. (c) Agreement between metric rankings and human preference by normalized score gap.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure B1 : One cold-start SFT demonstration: (a) source design, (b) excerpt of the target HTML with elisions marked, (c) its render in the evaluation sandbox.
Split
Items
Reference image
Cold-start SFT
3,000
original human design
RL, 8B and 32B
7,408
GPT-Image-2 from the prompt
Benchmark
300
GPT-Image-2 from the prompt
Benchmark composition
CLIP clusters covered
25
9 to 20 items each
Orientation
235 landscape, 56 portrait, 9 square
Appendix
Table B1 : Composition of the three splits. Both backbones use the same RL file.
Figure B2 : Six benchmark items, each with its GPT-Image-2 reference, the opening of its generation prompt, and its first few verifiable requirements.
Figure C1 : Per-channel training rewards under GDPO. Solid: single rollout; dashed: mean over eight rollouts; blue: training set; red: held-out set.
Model
Text
Bound
Chart
Rule
Content
Layout
Style
Visual
Overall
Image-to-code systems
Design2Code
39.60
54.13
0.92
31.55
19.26
9.75
2.79
10.60
21.08
UI2Code n
80.10
62.38
7.56
50.01
69.70
68.11
41.93
59.92
54.96
WebSight
25.28
62.48
0.06
29.27
27.46
20.56
7.43
18.48
23.88
Reconstruction objective, our data, Qwen3-VL-8B
Qwen3-VL-8B (backbone)
64.07
70.99
42.35
59.14
52.90
51.33
29.92
44.72
51.93
Appendix
Table C1 : Reconstruction objectives. Upper: published image-to-code systems. Lower: the backbone, prompts, and sampler of AesCode-8B with the reward replaced by a reconstruction objective. Evaluation as in Table 1 ; all rows receive Iref ; scores in %.
Figure C2 : Portrait and square outputs of AesCode-32B on held-out benchmark prompts.
Figure C3 : Landscape outputs of AesCode-32B on held-out benchmark prompts.
Figure D1 : Reward channels. Six deterministic verifiers read the rendered DOM and screenshot. Execution : render success, decayed by uncaught JavaScript errors. Text : required strings found, weighted by overlap and in-canvas visibility. Boundary : in-canvas fraction of overflowing elements and aspect-ratio match. Layout : proximity of text blocks matched to the reference. Table/Chart : parsed table rows and ECharts series against the required data. Whitespace : cosine similarity of occupancy grids with the reference. VGR : pass rate of sample-specific yes/no predicates answered from the candidate render. Each channel is masked by applicability and normalized separately by GDPO. Values are from two rollouts on one prompt.
Channel
Applicability
Recorded evidence
Output
Execution
all rollouts
render status, timeout, blank-page and console flags
pass / severity score
Text
required text exists
required, matched, missing, and unsupported strings
normalized fidelity
Boundary
rendered DOM exists
element and canvas boxes, overflow events, aspect ratios
containment and shape score
TableChart
table or chart required
parsed cells or ECharts type, categories, series, and values
structural accuracy
SemanticLayout
matched text blocks exist
block matches, normalized centers, distances, and coverage
spatial proximity
Whitespace
usable screenshots exist
coarse/fine occupancy grids and per-cell differences
multi-scale similarity
Appendix
Table D1 : Fields of the per-rollout reward record.
Figure D2 : Table/Chart. Outlines mark the tables and charts found in the DOM. Top: the baseline emits no table. Bottom: two rollouts on a chart prompt, with and without live ECharts instances.
Figure D3 : SemanticLayout. Top: OCR text blocks. Bottom: arrows from each reference block to its matched candidate block, coloured by proximity; hollow circles are unmatched reference blocks.
Figure D4 : Whitespace. Rendering, content mask, and coarse and fine occupancy grids; sws averages the two grid cosines with the reference.
Figure D5 : Complete generation prompt and reference for the VGR example (instance sesimic_folder_2_original_14504 ). Both inputs match the frozen final-evaluation instance. Reference-derived design criteria appear in the VGR questions, not in the generation prompt.
ID
Target
Question
Pass
q1
H
Is the large headline block positioned in the upper-left quadrant of the poster?
Yes
q2
P
Is the central product image slot the largest visual element on the page?
Yes
q3
B
Are the two badge/logo-like tiles arranged as a horizontal pair below the headline area?
Yes
q4
A
Are the two accessory image slots placed along the lower half of the poster rather than near the top?
Yes
q5
P
Is the central product image slot fully visible without being cropped by the page edges?
Yes
q6
L
Are the three left-side callout blocks readable as separate blocks with clear spacing between them?
Yes
Appendix
Table 21
ID
Target
Question
Pass
q1
Canvas
Is the overall composition a vertical poster with all major content stacked from top to bottom?
Yes
q2
G
Are the six FAQ cards arranged as a 3-column by 2-row grid?
Yes
q3
C
Is the contact information strip a single wide rounded panel spanning most of the poster width?
Yes
q4
F
Does the bottom area use a two-part layout with the warning block on the left and the QR-code slot on the right?
Yes
q5
G
Is each FAQ card fully visible within its rounded card boundary?
Yes
q6
Q
Is the QR-code placeholder area fully visible as a square slot in the lower-right section?
Yes
Appendix
Table 22
Figure D6 : Portrait FAQ example (instance poster_v2_00018 ). The complete Chinese generation prompt and reference match the frozen evaluation inputs. The reference is shown once, with display annotations locating the stored groups and relations.
Figure E1 : Complete generation prompt and illustrative reference for the evaluation example (instance vendor_data_original_903 ). The prompt comes from the frozen final-evaluation data.
Content
ID
Target
Question
Pass
N10
Canvas
Are there more than zero chart visualizations in the content sections?
No
N11
P1–P3
Are all three numbered policy pillars drawn whole, with no part of any pillar severed by the canvas edge?
Yes
N12
S3
Is the cross-provincial direct settlement content, including “98.2%”, drawn wholly inside the canvas rather than cut off by a page edge?
Yes
N14
S3
Are all digits of “98.2%” fully visible, with none clipped by a page edge or by the boundary of whatever element holds the value?
Yes
N15
S1
Is the subsidies group, including the “670 ¥” metric and its unit, drawn whole rather than cut off by a page edge?
Yes
Appendix
Table 25
Layout
ID
Target
Question
Pass
q1
H
Is the title block, carrying “XX Health Insurance Bureau” and “2024 Key Policy Interpretation”, encountered before the three numbered policy pillars?
Yes
q2
G / P1–P3
Does the section heading “Three Pillars of Policy” read as introducing the three numbered pillars, rather than attaching to the title block or to the later content groups?
Yes
q3
P1–P3
Does each numbered policy pillar keep its own supporting text with it? Answer no if any pillar’s supporting text sits closer to a different pillar than to its own.
Yes
q4
S1–S3
Do the three metric content groups read as one later section, encountered after the three numbered pillars rather than interleaved with them?
Yes
q6
S1–S3
Are all three post-pillar content groups present and legible on the page?
Yes
Appendix
Table 26
Style
ID
Target
Question
Pass
H06
S1
In the subsidies group, is the unit “/person/Year” rendered with the “670 ¥” metric so it clearly belongs to that value? Answer no if the unit is not fully readable, or if it appears to label a different metric.
Yes
SL01
Canvas
Is there a visible hierarchy across the page, and do the title block, the section heading “Three Pillars of Policy”, the three numbered pillars, and the three content groups read as distinct levels rather than one undifferentiated block? Answer no if any of the four reads at the same weight as its neighbours, or if a reader cannot tell where one level ends and the next begins.
Yes
SL02
P1–P3
Does every numbered policy pillar keep its number clearly associated with its own pillar title, and does the visual reading order of the pillars match 1, 2, 3? Answer no if any number appears closer to a different pillar title, or if the numbers are encountered out of order.
Yes
SL03
Canvas
Do the title block, every numbered policy pillar, and every post-pillar content group use coordinated but distinct visual treatment so the pillars read as one set and the content groups as another? Answer no if any member of either set is given visibly different treatment from its peers.
Yes
SL04
S1–S3
Do all specified metric values in the three post-pillar content groups read with their own unit or label as clear metric pairings? Answer no if any metric pairing is unreadable or visually attaches to the wrong content group.
Yes
Appendix
Table 27
ID
Target
Question
Pass
DG01
Canvas
Is the colour work already at that standard, so that a designer taking this over would change nothing about it? Answer no if the palette is a default or a safe choice rather than a considered one, if colour only fills shapes instead of carrying structure and meaning, if the page falls back on plain white and grey because no palette was committed to, if any hue is off, or if the tints and neutrals have not been tuned to sit together.
Yes
DG02
Canvas
Is the typography already at that standard, so that a designer taking this over would change nothing about it? Answer no if the type reads as browser defaults dressed up, if the scale is a few sizes chosen by eye rather than a system, if two sizes do the same job, if the page never commits to a display size where it needs one, or if line length, line spacing or optical alignment would be corrected in review.
Yes
DG03
Canvas
Has the material on this page actually been designed rather than merely placed, so that a designer would neither add anything nor take anything away? Answer no if a section is a plain list or a plain block that a designer would have given form to, if a region is empty because nothing was developed for it rather than because the composition wanted the rest, if anything reads as a placeholder, if filler is occupying space, or if something important was given less room than it deserves.
Yes
DG04
Canvas
Is this page composed, so that the eye is led first, second and third exactly as intended and nothing would be adjusted? Answer no if the page is only stacked in a reasonable order without being composed, if nothing on it is genuinely dominant, if the first thing seen is not the most important thing, if two elements compete for the same rank, or if any part goes flat and stops directing the eye.
Yes
DG05
Canvas
Is every repeated component resolved to the standard of released work, so that none of them would be revisited? Answer no if any card, row, tile, chip, label or control is a default container rather than a designed one, if one instance is less resolved than its neighbours, if spacing inside repeated elements varies without reason, or if a designer picking this up would rebuild the components rather than adjust them.
Yes
Appendix
Table 28
Metric
<5% of range
5 – 15% of range
≥15% of range
Our Visual
66.72%
80.00%
93.76%
Design2Code
44.67%
66.84%
82.50%
SlidesBench
–
74.00%
91.25%
GDB/lica
52.63%
74.67%
90.23%
Appendix
Table E5 : Human agreement with each metric’s score ordering, by relative score gap.
Mode (scope)
Metric
AesCode-32B
GPT-5.5
Severe boundary ( 300 )
recurrent
4.3%
34.7%
Relational subgraph ( 46 )
recurrent
23.9%
10.9%
mean pass
75.4%
87.0%
Semantic element ( 30 )
recurrent
23.3%
3.3%
mean pass
74.3%
92.0%
Appendix
Table F1 : Residual failure modes. Scope: number of prompts to which the mode applies. Recurrent: failure in ≥2 of 3 rollouts (lower is better); mean pass: higher is better.
Figure F1 : Additional instances of the three residual failure modes, with recurrence rates above each column.
We study whether vision-language models (VLMs) can reconstruct structured, graphics-based images by writing executable code to generate them. Existing benchmarks on image-to-code generation are generally limited in the domains they cover, rely on reference source code for evaluation, and do not provide metrics to quantify domain-specific issues. To address this, we introduce Vision2Code, a multi-domain, reference-code-free benchmark containing 539 test examples from 15 different datasets, spanning charts and plots, geometry, graphs, scientific diagrams, documents and 3D spatial scenes. Evaluation is done by executing the generated programs and comparing their rendered outputs to the source images using dataset- and domain-specific rubrics for a VLM judge. We find that the final scores obtained in this manner have substantially higher correlation with human ratings than those obtained via generic VLM rubrics or standard pixel-based, embedding-based or perceptual similarity metrics. We further demonstrate the reliability of our evaluation under repeated sampling from the VLM judges, varying rubric weights, and in the presence of controlled errors. Our results highlight the variability across domains of existing image-to-code generation models, with accuracy on charts and graphs being significantly higher than other domains. Lastly, we demonstrate the utility of our scoring method for rejection sampling of generated programs for fine-tuning, improving Qwen3.5-9B by 16% and Phi-4-MM by 28% compared to their base checkpoints. Our code and data are publicly available at https://image2code.github.io/vision2code/.
Existing screenshot-to-code systems face a trade-off between flexibility and controllability. Direct multimodal generation can hallucinate visible details, whereas structured pipelines reduce such errors through component-wise decomposition, predefined templates, and customized intermediate representations. These structures, however, introduce additional generative orchestration and restrict outputs to designs covered by the representation. We investigate whether selective tool grounding can improve the fidelity--efficiency trade-off of direct widget-to-code generation. We introduce WidgetGen, a lightweight tool-grounded framework that extracts observable text and color evidence, performs high-level layout and optional chart reasoning, and directly generates executable JavaScript XML (JSX). This design reduces reliance on component-wise generation while avoiding a fixed UI schema. Across six multimodal models and 1,000 held-out widgets, WidgetGen outperforms direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Finally, reconstruction-derived image-code pairs improve six Qwen-family open-weight models across every reported metric through supervised fine-tuning. These results establish WidgetGen as a strong lightweight baseline and show that selective evidence grounding offers an effective alternative to extensive representation constraints.
Houston H. Zhang, Tao Zhang, Li Gu +5
McMaster University · University of Toronto · Concordia University
Code-generating large language models (LLMs) increasingly produce visual artifacts such as charts, web pages, and slides by writing programs that are executed by non-differentiable renderers, committing to code before observing the render. As a result, otherwise executable code often yields artifacts with visually salient defects, including overlapping elements, clipped text, broken alignment, low contrast, and overflow. We study visual-feedback self-distillation for code-generated visual artifacts. We propose Visual-SDPO, a self-distillation policy-optimization framework that treats rendered visual feedback as privileged context for a weight-sharing teacher and distills this feedback into a coding student. To make supervision spatially targeted rather than uniform, we introduce Visual-Grounded Code Credit Weighting, which traces each detected defect back to the code statements responsible for the affected elements and amplifies the distillation signal on those statements. A sequence-level GRPO (Group Relative Policy Optimization) term complements the dense token-level objective by rewarding executable, visually high-quality rollouts, while failed executions remain learnable through the self-distillation path by passing execution errors as privileged context to the teacher. We instantiate Visual-SDPO for chart, web/UI, and slide generation with a unified Qwen3-VL-8B-Instruct backbone. Across chart-to-code, UI-to-code, and slide-generation benchmarks (ChartMimic, Design2Code, and AeSlides), Visual-SDPO improves over the zero-shot base by more than 10 absolute points in the primary metric and over GRPO by at least 2.4 points, with fewer training steps and no added inference-time cost.