AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

Xin Gao; Yifan Yang; Xin Li; Pan Wang; Yan Li; Muzhao Tian; Mingxi Cheng; Bei Liu

cs.AIOct 13, 2026

AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

Authors: Xin Gao, Yifan Yang, Xin Li, Pan Wang, Yan Li, Muzhao Tian, Mingxi Cheng, Bei Liu,

Organizations: Yale University · Microsoft · Nanyang Technological University · University of Pittsburgh ·

Abstract

Generating information-rich visual artifacts such as slides, posters, and dashboards as code is appealing because the outputs are structured, editable, and verifiable. Yet code models cannot see how their choices of layout and color come together on the canvas, often limiting spatial planning and aesthetic quality. Image generators excel at visual design but can misrender requested content, making their outputs unreliable targets for direct reproduction. We therefore introduce AesCode, which generates code grounded in the content prompt while using an image generated from the same prompt as an aesthetic guidance. To do so, AesCode represents the target as a design graph that describes the whole canvas and makes each property attributable. From this graph, we derive complementary cross-modal rewards: deterministic verifiers score properties parsable from code and rendering, while a VLM judge assesses the non-parsable ones with a sample-specific Visual Graph Rubric (VGR) tied to graph elements and relations. We optimize these decoupled rewards through reinforcement learning with per-channel normalization to jointly improve spatial planning, aesthetic quality, and verifiable correctness. To evaluate this setting, we build a benchmark that scores verifiable correctness and visual quality separately, with visual scores highly consistent with human judgments. We find that reference images substantially improve the visual quality of code generated by existing models, and ablations show that AesCode benefits from references both as model input and as a source of visual rewards. Relative to same-scale Qwen3-VL baselines given the same references, AesCode-8B and AesCode-32B improve visual quality by 31.1 and 22.4 percentage points, reaching scores comparable to those of closed-source frontier models such as GPT-5.5 and Claude Opus 4.8.

Figures & tables

Figure 1: Information-rich infographics generated as code. Prompts specify content, while image-generator references offer visual design guidance. The examples show a landscape dashboard and portrait poster rendered from generated HTML/CSS.

Figure 2: Overview of AesCode. Prompt requirements and reference design cues yield graph-based supervision. Conditioned on $p$ and $I_{\mathrm{ref}}$ , the policy generates HTML/CSS code $c$ , rendered as $\hat{I}$ .

Model	Text	Bound	Chart	Rule	Content	Layout	Style	Visual	Overall
Frontier Closed-Source
GPT-5.5	88.67	86.18	82.85	85.90	79.27	68.09	52.80	66.72	76.31
GPT-5.4	76.37	86.14	65.30	75.94	71.05	63.29	47.31	60.55	68.24
GPT-5.1	80.42	78.64	73.68	77.58	73.17	59.76	43.12	58.68	68.13
GPT-4o	73.64	78.37	55.64	69.22	51.34	33.33	15.95	33.54	51.38
Claude Opus 4.8	91.40	85.03	87.69	88.04	84.98	64.88	46.72	65.53	76.78

Table 1: Results on our infographic benchmark of 300 instances. Models in the first two blocks receive only the prompt; those in the third, including AesCode, also receive a generated reference image. Rule averages the programmatic Text, Bound, and Chart checks; Visual averages the Content, Layout, and Style scores as defined in Section 3.4 . Overall is the mean of Rule and Visual. Scores (%) are averaged over three generations per prompt; higher is better. Bold and underlined values indicate the best and second-best scores in each column.

Figure 3: Qualitative comparisons selected by the three-rollout mean Overall gap. Each row shows the reference and outputs from GPT-5.5 and AesCode-32B. Bars report evaluation scores; red dashed boxes mark visible GPT-5.5 errors.

	GDB/lica						SlidesBench			Design2Code
Method	NIMA	CLIP	Pick	FID ↓	Harm.	OCR	Cont.	Layout	Color	Block	Text	Pos.	Color	CLIP
GPT-5.5	4.362	30.2	19.53	0.439	70.6	71.9	4.54	2.85	3.63	92.3	98.5	94.2	95.3	94.3
GPT-4o	2.589	24.1	17.82	0.673	69.3	43.3	3.41	2.00	2.81	77.5	93.0	77.9	82.1	89.5
GLM-4.5V	3.682	26.4	18.49	0.541	70.3	65.1	4.22	2.14	3.00	76.7	88.2	73.8	84.7	90.3
Qwen2.5-VL-72B	3.564	26.0	18.40	0.537	69.1	62.6	4.03	2.02	2.83	82.6	97.3	78.0	78.8	88.9
Qwen3-VL-30B-A3B	3.461	26.1	18.38	0.532	68.8	57.6	3.95	2.10	2.97	86.1	95.6	80.5	91.3	91.4

Table 2: Transfer to three external benchmarks. GDB/lica evaluates text-to-image generation; SlidesBench and Design2Code evaluate image-to-code reconstruction. All models output rendered HTML/CSS under each benchmark’s official protocol. SlidesBench uses a 0–5 scale; Design2Code and GDB/lica CLIP, Harm., and OCR use 0–100. Higher is better except FID.

Reference input removed at inference only
Model	Text	Bound	Chart	Rule	Content	Layout	Style	Visual	Overall
GPT-5.5	88.67 $(-0.6)$	86.18 $(-0.7)$	82.85 $(+1.5)$	85.90 $(+0.1)$	79.27 $(-3.2)$	68.09 $(-21.8)$	52.80 $(-5.1)$	66.72 $(-10.0)$	76.31 $(-5.0)$
Qwen3-VL-32B	66.00 $(-5.2)$	80.31 $(+0.7)$	41.36 $(-9.4)$	62.55 $(-4.6)$	49.81 $(-12.2)$	34.85 $(-29.7)$	19.34 $(-19.1)$	34.67 $(-20.4)$	48.61 $(-12.5)$
Qwen3-VL-8B	58.32 $(-5.8)$	77.23 $(+6.2)$	43.30 $(+1.0)$	59.61 $(+0.5)$	38.33 $(-14.6)$	23.97 $(-27.4)$	13.22 $(-16.7)$	25.17 $(-19.6)$	42.39 $(-9.5)$
AesCode-8B	93.74 $(-0.3)$	88.19 $(-0.2)$	82.36 $(-5.4)$	88.09 $(-2.0)$	84.92 $(-1.5)$	86.84 $(-1.0)$	52.63 $(-0.6)$	74.80 $(-1.0)$	81.45 $(-1.5)$
Reference input removed during training and inference

Table 3: Reference images during training and inference. All rows omit reference input at inference; the lower row also omits it from policy inputs during training while retaining reference-based rewards. Gray values show changes from reference-conditioned results.

Panel of Figure 4 — Figure 4: Human evaluation. (a) Agreement with training VGR and evaluation-checklist judgments. (b) Content satisfaction and aesthetic equivalence between AesCode-32B and GPT-5.5. (c) Agreement between metric rankings and human preference by normalized score gap.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B1 : One cold-start SFT demonstration: (a) source design, (b) excerpt of the target HTML with elisions marked, (c) its render in the evaluation sandbox.

Split	Items	Reference image
Cold-start SFT	$3{,}000$	original human design
RL, 8B and 32B	$7{,}408$	GPT-Image-2 from the prompt
Benchmark	$300$	GPT-Image-2 from the prompt
Benchmark composition
CLIP clusters covered	$25$	$9$ to $20$ items each
Orientation	$235$ landscape, $56$ portrait, $9$ square

Appendix

Table B1 : Composition of the three splits. Both backbones use the same RL file.

Figure B2 : Six benchmark items, each with its GPT-Image-2 reference, the opening of its generation prompt, and its first few verifiable requirements.

Figure C1 : Per-channel training rewards under GDPO. Solid: single rollout; dashed: mean over eight rollouts; blue: training set; red: held-out set.

Model	Text	Bound	Chart	Rule	Content	Layout	Style	Visual	Overall
Image-to-code systems
Design2Code	39.60	54.13	0.92	31.55	19.26	9.75	2.79	10.60	21.08
UI2Code n	80.10	62.38	7.56	50.01	69.70	68.11	41.93	59.92	54.96
WebSight	25.28	62.48	0.06	29.27	27.46	20.56	7.43	18.48	23.88
Reconstruction objective, our data, Qwen3-VL-8B
Qwen3-VL-8B (backbone)	64.07	70.99	42.35	59.14	52.90	51.33	29.92	44.72	51.93

Appendix

Table C1 : Reconstruction objectives. Upper: published image-to-code systems. Lower: the backbone, prompts, and sampler of AesCode-8B with the reward replaced by a reconstruction objective. Evaluation as in Table 1 ; all rows receive

I_{\mathrm{ref}}

; scores in %.

Figure C2 : Portrait and square outputs of AesCode-32B on held-out benchmark prompts.

Figure C3 : Landscape outputs of AesCode-32B on held-out benchmark prompts.

Figure D1 : Reward channels. Six deterministic verifiers read the rendered DOM and screenshot. Execution : render success, decayed by uncaught JavaScript errors. Text : required strings found, weighted by overlap and in-canvas visibility. Boundary : in-canvas fraction of overflowing elements and aspect-ratio match. Layout : proximity of text blocks matched to the reference. Table/Chart : parsed table rows and ECharts series against the required data. Whitespace : cosine similarity of occupancy grids with the reference. VGR : pass rate of sample-specific yes/no predicates answered from the candidate render. Each channel is masked by applicability and normalized separately by GDPO. Values are from two rollouts on one prompt.

Channel	Applicability	Recorded evidence	Output
Execution	all rollouts	render status, timeout, blank-page and console flags	pass / severity score
Text	required text exists	required, matched, missing, and unsupported strings	normalized fidelity
Boundary	rendered DOM exists	element and canvas boxes, overflow events, aspect ratios	containment and shape score
TableChart	table or chart required	parsed cells or ECharts type, categories, series, and values	structural accuracy
SemanticLayout	matched text blocks exist	block matches, normalized centers, distances, and coverage	spatial proximity
Whitespace	usable screenshots exist	coarse/fine occupancy grids and per-cell differences	multi-scale similarity

Appendix

Table D1 : Fields of the per-rollout reward record.

Figure D2 : Table/Chart. Outlines mark the tables and charts found in the DOM. Top: the baseline emits no table. Bottom: two rollouts on a chart prompt, with and without live ECharts instances.

Figure D3 : SemanticLayout. Top: OCR text blocks. Bottom: arrows from each reference block to its matched candidate block, coloured by proximity; hollow circles are unmatched reference blocks.

Figure D4 : Whitespace. Rendering, content mask, and coarse and fine occupancy grids; $s_{\mathrm{ws}}$ averages the two grid cosines with the reference.

Figure D5 : Complete generation prompt and reference for the VGR example (instance sesimic_folder_2_original_14504 ). Both inputs match the frozen final-evaluation instance. Reference-derived design criteria appear in the VGR questions, not in the generation prompt.

ID	Target	Question	Pass
q1	H	Is the large headline block positioned in the upper-left quadrant of the poster?	Yes
q2	P	Is the central product image slot the largest visual element on the page?	Yes
q3	B	Are the two badge/logo-like tiles arranged as a horizontal pair below the headline area?	Yes
q4	A	Are the two accessory image slots placed along the lower half of the poster rather than near the top?	Yes
q5	P	Is the central product image slot fully visible without being cropped by the page edges?	Yes
q6	L	Are the three left-side callout blocks readable as separate blocks with clear spacing between them?	Yes

Appendix

Table 21

ID	Target	Question	Pass
q1	Canvas	Is the overall composition a vertical poster with all major content stacked from top to bottom?	Yes
q2	G	Are the six FAQ cards arranged as a 3-column by 2-row grid?	Yes
q3	C	Is the contact information strip a single wide rounded panel spanning most of the poster width?	Yes
q4	F	Does the bottom area use a two-part layout with the warning block on the left and the QR-code slot on the right?	Yes
q5	G	Is each FAQ card fully visible within its rounded card boundary?	Yes
q6	Q	Is the QR-code placeholder area fully visible as a square slot in the lower-right section?	Yes

Appendix

Table 22

Panel of Figure 23 — Figure D6 : Portrait FAQ example (instance poster_v2_00018 ). The complete Chinese generation prompt and reference match the frozen evaluation inputs. The reference is shown once, with display annotations locating the stored groups and relations.

Figure E1 : Complete generation prompt and illustrative reference for the evaluation example (instance vendor_data_original_903 ). The prompt comes from the frozen final-evaluation data.

Content
ID	Target	Question	Pass
N10	Canvas	Are there more than zero chart visualizations in the content sections?	No
N11	P1–P3	Are all three numbered policy pillars drawn whole, with no part of any pillar severed by the canvas edge?	Yes
N12	S3	Is the cross-provincial direct settlement content, including “98.2%”, drawn wholly inside the canvas rather than cut off by a page edge?	Yes
N14	S3	Are all digits of “98.2%” fully visible, with none clipped by a page edge or by the boundary of whatever element holds the value?	Yes
N15	S1	Is the subsidies group, including the “670 ¥” metric and its unit, drawn whole rather than cut off by a page edge?	Yes

Appendix

Table 25

Layout
ID	Target	Question	Pass
q1	H	Is the title block, carrying “XX Health Insurance Bureau” and “2024 Key Policy Interpretation”, encountered before the three numbered policy pillars?	Yes
q2	G / P1–P3	Does the section heading “Three Pillars of Policy” read as introducing the three numbered pillars, rather than attaching to the title block or to the later content groups?	Yes
q3	P1–P3	Does each numbered policy pillar keep its own supporting text with it? Answer no if any pillar’s supporting text sits closer to a different pillar than to its own.	Yes
q4	S1–S3	Do the three metric content groups read as one later section, encountered after the three numbered pillars rather than interleaved with them?	Yes
q6	S1–S3	Are all three post-pillar content groups present and legible on the page?	Yes

Appendix

Table 26

Style
ID	Target	Question	Pass
H06	S1	In the subsidies group, is the unit “/person/Year” rendered with the “670 ¥” metric so it clearly belongs to that value? Answer no if the unit is not fully readable, or if it appears to label a different metric.	Yes
SL01	Canvas	Is there a visible hierarchy across the page, and do the title block, the section heading “Three Pillars of Policy”, the three numbered pillars, and the three content groups read as distinct levels rather than one undifferentiated block? Answer no if any of the four reads at the same weight as its neighbours, or if a reader cannot tell where one level ends and the next begins.	Yes
SL02	P1–P3	Does every numbered policy pillar keep its number clearly associated with its own pillar title, and does the visual reading order of the pillars match 1, 2, 3? Answer no if any number appears closer to a different pillar title, or if the numbers are encountered out of order.	Yes
SL03	Canvas	Do the title block, every numbered policy pillar, and every post-pillar content group use coordinated but distinct visual treatment so the pillars read as one set and the content groups as another? Answer no if any member of either set is given visibly different treatment from its peers.	Yes
SL04	S1–S3	Do all specified metric values in the three post-pillar content groups read with their own unit or label as clear metric pairings? Answer no if any metric pairing is unreadable or visually attaches to the wrong content group.	Yes

Appendix

Table 27

ID	Target	Question	Pass
DG01	Canvas	Is the colour work already at that standard, so that a designer taking this over would change nothing about it? Answer no if the palette is a default or a safe choice rather than a considered one, if colour only fills shapes instead of carrying structure and meaning, if the page falls back on plain white and grey because no palette was committed to, if any hue is off, or if the tints and neutrals have not been tuned to sit together.	Yes
DG02	Canvas	Is the typography already at that standard, so that a designer taking this over would change nothing about it? Answer no if the type reads as browser defaults dressed up, if the scale is a few sizes chosen by eye rather than a system, if two sizes do the same job, if the page never commits to a display size where it needs one, or if line length, line spacing or optical alignment would be corrected in review.	Yes
DG03	Canvas	Has the material on this page actually been designed rather than merely placed, so that a designer would neither add anything nor take anything away? Answer no if a section is a plain list or a plain block that a designer would have given form to, if a region is empty because nothing was developed for it rather than because the composition wanted the rest, if anything reads as a placeholder, if filler is occupying space, or if something important was given less room than it deserves.	Yes
DG04	Canvas	Is this page composed, so that the eye is led first, second and third exactly as intended and nothing would be adjusted? Answer no if the page is only stacked in a reasonable order without being composed, if nothing on it is genuinely dominant, if the first thing seen is not the most important thing, if two elements compete for the same rank, or if any part goes flat and stops directing the eye.	Yes
DG05	Canvas	Is every repeated component resolved to the standard of released work, so that none of them would be revisited? Answer no if any card, row, tile, chip, label or control is a default container rather than a designed one, if one instance is less resolved than its neighbours, if spacing inside repeated elements varies without reason, or if a designer picking this up would rebuild the components rather than adjust them.	Yes

Appendix

Table 28

Metric	$<5\%$ of range	$5$ – $15\%$ of range	$\geq 15\%$ of range
Our Visual	66.72%	80.00%	93.76%
Design2Code	44.67%	66.84%	82.50%
SlidesBench	–	74.00%	91.25%
GDB/lica	52.63%	74.67%	90.23%

Appendix

Table E5 : Human agreement with each metric’s score ordering, by relative score gap.

Mode (scope)	Metric	AesCode-32B	GPT-5.5
Severe boundary ( $300$ )	recurrent	$\mathbf{4.3\%}$	$34.7\%$
Relational subgraph ( $46$ )	recurrent	$23.9\%$	$\mathbf{10.9\%}$
	mean pass	$75.4\%$	$\mathbf{87.0\%}$
Semantic element ( $30$ )	recurrent	$23.3\%$	$\mathbf{3.3\%}$
	mean pass	$74.3\%$	$\mathbf{92.0\%}$

Appendix

Table F1 : Residual failure modes. Scope: number of prompts to which the mode applies. Recurrent: failure in

\geq 2

3

rollouts (lower is better); mean pass: higher is better.

Figure F1 : Additional instances of the three residual failure modes, with recurrence rates above each column.

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 313 papers

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
May 11, 2026Ajay Vikram Periasami, Junlin Wang, Bhuwan DhingraVLM Evaluation Image-to-Code Generation
Duke University
From Visual Widgets to UI Code: Efficient Tool-Grounded Generation
Aug 12, 2026Houston H. Zhang, Tao Zhang, Li Gu +5Image-to-Code Generation Code Generation
McMaster University · University of Toronto · Concordia University
Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts
Jun 9, 2026Haoyu DongRL for Code Generation Code Generation
Microsoft

May 11, 2026J/K move · Enter open · S save

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

Ajay Vikram Periasami, Junlin Wang, Bhuwan Dhingra

Duke University

VLM Evaluation Image-to-Code Generation Code Generation Code Generation Evaluation Code Generation Benchmarks

Open paper AskPDF arXiv

We study whether vision-language models (VLMs) can reconstruct structured, graphics-based images by writing executable code to generate them. Existing benchmarks on image-to-code generation are generally limited in the domains they cover, rely on reference source code for evaluation, and do not provide metrics to quantify domain-specific issues. To address this, we introduce Vision2Code, a multi-domain, reference-code-free benchmark containing 539 test examples from 15 different datasets, spanning charts and plots, geometry, graphs, scientific diagrams, documents and 3D spatial scenes. Evaluation is done by executing the generated programs and comparing their rendered outputs to the source images using dataset- and domain-specific rubrics for a VLM judge. We find that the final scores obtained in this manner have substantially higher correlation with human ratings than those obtained via generic VLM rubrics or standard pixel-based, embedding-based or perceptual similarity metrics. We further demonstrate the reliability of our evaluation under repeated sampling from the VLM judges, varying rubric weights, and in the presence of controlled errors. Our results highlight the variability across domains of existing image-to-code generation models, with accuracy on charts and graphs being significantly higher than other domains. Lastly, we demonstrate the utility of our scoring method for rejection sampling of generated programs for fine-tuning, improving Qwen3.5-9B by 16% and Phi-4-MM by 28% compared to their base checkpoints. Our code and data are publicly available at https://image2code.github.io/vision2code/.

Abstract

Code Generation

Explore similar work

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

From Visual Widgets to UI Code: Efficient Tool-Grounded Generation

Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts