Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization

Peng Zhang; Xuefeng Li; Xiaoqi Wang; Han-Wei Shen; Yifan Hu

cs.LGApr 3, 2026

Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization

Authors: Peng Zhang, Xuefeng Li, Xiaoqi Wang, Han-Wei Shen, Yifan Hu

Organizations: Northeastern University, United States · Bosch AI Research, United States · The Ohio State University, United States

Abstract

Network visualization has traditionally relied on heuristic metrics, such as stress, under the assumption that optimizing them leads to aesthetic and informative layouts. However, no single metric consistently produces the most effective results. A data-driven alternative is to learn from human preferences, where annotators select their favored visualization among multiple layouts of the same graphs. These human-preference labels can then be used to train a generative model that approximates human aesthetic preferences. However, obtaining human labels at scale is costly and time-consuming. As a result, this generative approach has so far been tested only with machine-labeled data. In this paper, we explore the use of large language models (LLMs) and vision models (VMs) as proxies for human judgment. Through a carefully designed user study involving 27 participants, we curated a large set of human preference labels. We used this data both to better understand human preferences and to bootstrap LLM/VM labelers. We show that prompt engineering that combines few-shot examples and diverse input formats, such as image embeddings, significantly improves LLM-human alignment, and additional filtering by the confidence score of the LLM pushes the alignment to human-human levels. Furthermore, we demonstrate that carefully trained VMs can achieve VM-human alignment at a level comparable to that between human annotators. Our results suggest that AI can feasibly serve as a scalable proxy for human labelers.

Figures & tables

Figure 1: A UI showing eight graph visualizations. Of the users ( $n=43$ ), 56% preferred the starred option, and 16% chose the bottom-left.

$\begin{subarray}{c}\textbf{Unique}\\ \textbf{Layouts}\end{subarray}$	Human	Human*	incl. LLM*	incl. VM*
1	5.15	4.50	2.95	2.02
2	40.79	39.70	31.72	38.24
3	38.55	41.5	43.93	42.16
4	13.49	11.8	17.27	14.81
5	1.94	2.50	3.56	2.63
6	0.08	0.00	0.56	0.18

Table 1: Distribution (%) of unique preferred layouts per graph. *Results based on the 1,000-graph test set.

You are an expert in human aesthetics. I want you to evaluate the graph’s layout primarily from an aesthetic perspective. Based on your aesthetic judgment, select the layout that best aligns with human preferences. Let’s think step by step . Give me the answer with the style: "Reason: Result: layout ?" Just give me plain text, do not add emphasize markdown.

Table 2: Prompt for 0-shot evaluation of layout aesthetics

… To systematically analyze the layout preferences, we provided multiple exemplar images for layout assessment and human aesthetic preference exploration: In the first set of eight layouts, the human selected layout 2. In the second set of eight layouts, the human selected layout 7. In the third set of eight layouts, the human selected layout 8. In the fourth set of eight layouts, the human selected layout 4. In the fifth set of eight layouts, the human selected layout 8. We request a comprehensive analysis of the aesthetic criteria that influenced these layout selections. It is important to note that the preference is independent of the absolute layout numbering. …

Table 3: An example prompt for 5-shot evaluation of layout aesthetics; this is added to the prompt in Table 2 , just before “Let’s think step by step”.

… Below are examples of layouts labeled as "good" or "bad" based on human feedback: {examples} Now, please evaluate the new graph: Graph Structure: {graph structural information} Layout Coordinates to Evaluate: {features} …

Table 4: Prompt for few-shot with embeddings evaluation of layout aesthetics. This is added to the prompt in Table 2 , just before “Let’s think step by step”.

Figure 2: Demonstration of memory bank vs graph representation. The green route shows the graph representation, while the blue route shows the image-embedding–based memory bank approach.

You are an expert in human aesthetics. Below are examples of layouts labeled as "good" or "bad" based on human feedback: Good Layouts: {good_layouts} Now, evaluate the following layouts and select the one that is most aligned with human aesthetics: Features: {features} And the last sentence must start with "In conclusion, I will pick layout X".

Table 5: Prompt for image-embedding memory bank approach

Method	0 - Shot	1 - Shot	5 - Shot
Image Only*	$19.70$	$19.52$	$18.79$
Edgelist	$16.76$	$16.21$	$14.35$
Adjacency	$16.37$	$16.92$	$15.77$
Node2Vec	$17.67$	$15.01$	—
Spectral	$17.21$	$15.71$	$14.54$
MB ResNet50*	$10.45$	$11.37$	$14.45$

Table 6: Alignment (%) of LLM with human labels. Note that 10 shots for Node2Vec experiments are missing due to token length limits. Rows marked with * do not use coordinates, and the rest use coordinates.

Model	Alignment (%)
Qwen3 (235b-a22b-2507)	11.77
Claude Sonet 4.5	12.70
Gemini 2.5 Flash	14.81
GPT-5	27.80
GPT-4o mini	33.11

Table 7: LLM -human alignment (%) for DINOv2 Memory Bank 1-shot across different LLMs.

Method	ResNet	DINOv2
Frozen feature extractor
Baseline	16.82	27.82
Augment (10×)	18.66	32.82
Soft Multiclass	30.41	32.80
Fine-tuned backbone
Tune + Baseline	17.36	33.07

Table 8: VM-human alignment (%) under different training strategies for ResNet and DINOv2

Model	Size	Embedding Dim	Alignment (%)
DINOv2-Small	21M	384	35.05
DINOv2-Base	86M	768	36.81
DINOv2-Large	300M	1024	35.90

Table 9: Ablation study of VM-human alignment with three DINOv2 models of different sizes, under fine-tuning setting (2 layers)

Model	Num. of fine-tune layers	Alignment (%)
DINOv2-Base	1	32.10
DINOv2-Base	2	36.80
DINOv2-Base	3	34.76
DINOv2-Base	4	36.04

Table 10: Ablation study for number of layers to fine-tune

Figure 3: Left: Human-human and LLM-human alignment as a function of VM confidence. Right: Human-human and VM-human alignment as a function of VM confidence. The number of test samples remaining after threadholding is given at the top of each figure.

Layout	LLM Count	LLM %	VM Count	VM %	Human Count	Human %
fa2	31	3.10	5	0.50	269	4.84
fdp	193	19.30	36	3.60	448	8.06
Kamada-Kawai	709	70.90	495	49.50	2364	42.54
neato	22	2.20	436	43.60	1935	34.82
pmds	9	0.90	4	0.40	72	1.30
sfdp	8	0.80	15	1.50	292	5.25

Table 11: Choice of layout algorithms by LLM (ChatGPT-40-mini), VM (DINOv2 Tune

+

Soft Multiclass) and Human. Note that the “Human Count” column has a total of 5557 labels, because each of the 1000 test graphs has, on average, 5.56 labels by different human labelers.

Figure 4: Human-human, LLM-human and VM-human alignment visualization. Here we pick the seven most prolific human labelers.

Panel of Figure 5 — Figure 5: Examples illustrating agreement and disagreement between LLM and human preferences.

Panel of Figure 6 — Figure 6: Examples illustrating agreement and disagreement between VM and human preferences.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: A screenshot showing the training provided to the participants

Figure 8: Distribution of layout algorithms behind the visualizations chosen by human subjects as being the most pleasing

Reason	Description	Freq
Symmetry	Layouts with symmetric arrangements, balanced proportions, or bilateral harmony.	42
No edge crossings	Absence of edge intersections or overlaps, leading to planar and clean visuals.	35
Clear structure	Readable, clear overall structure or preserved graph shape.	31
Good spacing	Even node spacing, well-spread layout, or efficient use of space.	29
Geometric harmony	Harmonious proportions, stability, or appealing geometric metaphors (e.g., cubic, spherical).	25
Uniform edge lengths	Consistent or even edge lengths without variation.	22

Appendix

Table 12: Top reasons for liking a visualization & frequency

Reason	Description	Freq
Asymmetry or imbalance	Lopsided, unbalanced, or asymmetrical layout.	45
Edge crossings	Presence of intersecting or crossing edges causing ambiguity.	38
Dense or crowded areas	Clustered, dense, or congested regions with too much proximity.	36
Overlapping edges/nodes	Edges or nodes overlapping, leading to confusion or invisibility.	32
Distorted or skewed shape	Warped, distorted, or irregular overall shape.	26
Uneven edge lengths	Inconsistent, irregular, or varying edge lengths.	24

Appendix

Table 13: Top reasons for disliking a visualization & frequency

Threshold	User–User (%)	VM–User (%)	LLM–User (%)
0.00	100.00	100.00	100.00
0.50	96.17	96.87	95.48
0.60	90.71	92.51	88.36
0.70	83.25	84.69	79.67
0.80	71.66	72.50	67.64
0.85	65.83	67.09	61.47

Appendix

Table 14:

Agreement_{\alpha}

Summary at Different Thresholds

Figure 9: A screenshot showing the preference of humans vs that of the LLM for graph 1138_bus ( $|V|=1138,|E|=1452$ ).

Figure 10: A screenshot showing the preference of human vs that of the VM for graph 1138_bus( $|V|=1138,|E|=1452$ )

Figure 11: A screenshot showing the preference of human vs that of the LLM for graph USPowerGrid( $|V|=4591,|E|=6594$ )

Figure 12: A screenshot showing the preference of human vs that of the VM for graph USPowerGrid ( $|V|=4591,|E|=6594$ )

Data Visualization

Data visualization studies how visual representations help people explore, interpret, communicate, and reason about data.

See all 120 papers

DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation
Feb 26, 2026Varun Gopal, Rishabh Jain, Aradhya Mathur +6Human Preference Modeling Reward Modeling
MDSR, Adobe
Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective
Mar 1, 2026Arctanx An, Shizhao Sun, Danqing Huang +5VLM Evaluation
Peking University · Microsoft Research Asia · Microsoft +1
UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment
Feb 23, 2026Yecheng Zhang, Rong Zhao, Zhizhou Sha +10VLM Calibration Vision-Language Alignment
Tsinghua University · University College London · Hong Kong University of Science and Technology +6

Feb 26, 2026J/K move · Enter open · S save

DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation

Varun Gopal, Rishabh Jain, Aradhya Mathur, Nikitha SR+5

MDSR, Adobe

Human Preference Modeling Reward Modeling Vision-Language Model-as-a-Judge

Open paper AskPDF arXiv

Graphic layouts serve as an important and engaging medium for visual communication across different channels. While recent layout generation models have demonstrated impressive capabilities, they frequently fail to align with nuanced human aesthetic judgment. Existing preference datasets and reward models trained on text-to-image generation do not generalize to layout evaluation, where the spatial arrangement of identical elements determines quality. To address this critical gap, we introduce DesignSense-10k, a large-scale dataset of 10,235 human-annotated preference pairs for graphic layout evaluation. We propose a five-stage curation pipeline that generates visually coherent layout transformations across diverse aspect ratios, using semantic grouping, layout prediction, filtering, clustering, and VLM-based refinement to produce high-quality comparison pairs. Human preferences are annotated using a 4-class scheme (left, right, both good, both bad) to capture subjective ambiguity. Leveraging this dataset, we train DesignSense, a vision-language model-based classifier that substantially outperforms existing open-source and proprietary models across comprehensive evaluation metrics (54.6% improvement in Macro F1 over the strongest proprietary baseline). Our analysis shows that frontier VLMs remain unreliable overall and fail catastrophically on the full four-class task, underscoring the need for specialized, preference-aware models. Beyond the dataset, our reward model DesignSense yields tangible downstream gains in layout generation. Using our judge during RL based training improves generator win rate by about 3%, while inference-time scaling, which involves generating multiple candidates and selecting the best one, provides a 3.6% improvement. These results highlight the practical impact of specialized, layout-aware preference modeling on real-world layout generation quality.

Abstract

Data Visualization

Explore similar work

DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation

Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective

UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment