Pixel art is widely used in games and digital content creation, requiring readable forms, coherent spatial relationships, and harmonious colors under limited resolution and palette constraints. Language models show growing promise in visual creation, yet their pixel art creation capabilities lack systematic evaluation. We introduce PixelBench, formalizing text-to-image generation and instruction-guided editing for pixel art and providing a benchmark for these tasks. We further present a creation environment in which language model agents invoke drawing tools through code to generate and edit pixel art. We assess the quality of generated artwork through human evaluation and employ a multimodal large language model as a judge, validating the agreement between its automated assessments and human preferences. We compare language models and configurations across diverse subjects and canvas sizes. In our experiments, GPT-6 Astra achieves both the highest average judge score and the best average human rank, highlighting the potential of general-purpose language models for pixel art creation. We release the tasks, all artworks, the human rankings, and the judge protocol.
Figures & tables
Figure 1: PixelBench on the task summer-seaside . (a) A task gives a written requirement and a native canvas; an editing task also gives the artwork to modify. (b) The model writes, runs, views, and revises a drawing program until it submits. The code shown is part of the program submitted by GPT-6 Astra in the evaluation. (c) The calibrated judge scores each artwork given a rubric and a reference image. Below: the task’s 12 submissions by mean rank over the ten raters, #1 to #12 reading left to right and top to bottom, each with its judge score; and leave-one-out agreement over all 100 tasks, for the judge and for the average held-out rater.
Figure 2: An example PixelBench task, summer-seaside : a 128×128 summer seaside with at most 86 colors; insets enlarge the sailboat. (a) The output of an image generation model imitates pixel art, but its pixels are not clean cells: many are not square, straddle cell boundaries, or blend several colors, and no fixed palette constrains them. (b) Pixel Snapper ( Duprez, 2025 ) infers a grid from that image and quantizes it to the task’s 86 colors; every cell is now uniform, but the grid is 203×209 rather than the required canvas. (c) GPT-6 Astra, working in PixelBench, writes a program that draws on the native canvas cell by cell; shown is its benchmark submission.
Protocol
Change
All (100)
95% CI
Generation (75)
Editing (25)
Simplest
single image, overall score
0.528
[0.484, 0.572]
0.528
0.529
+ presentation
1/16 transparent margin
0.532
[0.497, 0.568]
0.531
0.536
+ rubric
by subject and task type
0.544
[0.506, 0.581]
0.553
0.517
+ reference
same-task reference image
0.557
[0.520, 0.593]
0.563
0.538
Table 1: Calibrating the judge. Per-task mean Kendall τb against the consensus order of the ten raters, with a 95% bootstrap interval over tasks. Each row adds one change to the row above, and Appendix C gives the exact prompt of each version; the last row is the final protocol.
All (100)
Generation (75)
Editing (25)
Configuration
Score
Rank ↓
Score
Rank ↓
Score
Rank ↓
Judge
Human
Judge
Human
Judge
Human
GPT-6 Astra (high)
83.76
2.52
3.34
82.35
2.40
3.21
88.00
2.86
3.74
Claude Opus 5 (low)
78.49
5.03
4.88
77.64
5.01
5.09
81.04
5.06
4.25
Qwen3.8 Max
78.04
4.72
5.13
77.35
4.54
5.00
80.12
5.26
5.49
Gemini 3.8 Flash (high)
77.35
4.69
4.27
77.87
4.45
3.82
75.80
5.40
5.62
Table 2: Leaderboard over 12 configurations, ordered by mean judge score over all 100 tasks under the final protocol. The two rank columns are mean within-task ranks over the twelve configurations, the judge’s from its scores and the human one over the ten raters; both are lower-is-better and live on the same scale. The value in parentheses is the reasoning effort every call ran at; the two Qwen models take a thinking budget instead.
Model
Setting
All (100)
Generation (75)
Editing (25)
GPT-6 Astra
high (Table 2 )
83.8
82.3
88.0
low
81.8
80.5
85.8
medium
82.4
80.8
87.4
xhigh
84.7
83.6
87.9
no image observation
79.9
80.5
77.8
direct answer
76.9
76.5
78.1
Table 3: Judge-scored variants, each run on all 100 tasks and scored once by the judge. The rows marked Table 2 are the main-table setting; creation runs 2 and 3 repeat that setting from scratch.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Full gallery, part 1 of 6.
Figure 4: Full gallery, part 2 of 6.
Figure 5: Full gallery, part 3 of 6.
Figure 6: Full gallery, part 4 of 6.
Figure 7: Full gallery, part 5 of 6.
Figure 8: Full gallery, part 6 of 6.
Gap in the earlier pass
Comparisons
Reversed
Tie changed
≥0
2376
9.7%
8.9%
≥3
1827
5.3%
3.0%
≥5
1509
3.4%
1.8%
≥10
970
0.3%
0.7%
Appendix
Table 4: Order stability against the score gap. “Reversed” is the share of comparisons in which the later pass strictly reverses the earlier one; “tie changed” the share in which a tie becomes a strict order or the reverse.
Configuration
SE of 100-task mean
95% half-width of a rerun difference
GPT-6 Astra
0.15
0.42
Claude Opus 5
0.23
0.64
Qwen3.8 Max
0.23
0.63
Gemini 3.8 Flash
0.16
0.43
GPT-5.6 Sol
0.25
0.69
DeepSeek V4.1 Flash
0.26
0.71
Appendix
Table 5: Scoring standard error of a 100-task mean, by configuration, and the 95% half-width of the difference between two independent evaluations of the same configuration.
Judge
Reasoning effort
All
Generation
Editing
Leave-one-out
ρ
GPT-6 Astra
xhigh
0.557
0.563
0.538
0.554 (3)
0.958
Claude Opus 5
low
0.498
0.469
0.590
0.494 (1)
0.825
DeepSeek V4.1 Flash
high
0.414
0.407
0.432
0.413 (0)
0.958
Appendix
Table 6: Three judge models under the same protocol. Agreement is the per-task mean Kendall τb with the ten-rater consensus order; leave-one-out is the judge’s mean agreement with the consensus of nine raters, with the number of held-out raters below it in parentheses; ρ is the Spearman correlation of the judge leaderboard with the human one. The lower block gives the per-task paired difference in agreement, GPT-6 Astra minus the other judge, with 95% intervals. Claude Opus 5’s values cover the 99 tasks it scored in full, 24 of them editing.
Leave-one-out
Slice
Tasks
Score SD
Rater–rater
Rank SD
Judge–human
Judge
Raters
Below
Characters
13
10.4
0.492
2.38
0.609
0.614
0.482–0.683
5
Creatures
10
9.4
0.469
2.04
0.563
0.569
0.527–0.662
3
Scenes, architecture
16
10.0
0.466
2.29
0.634
0.631
0.494–0.667
7
Icons, interface
7
8.6
0.457
2.25
0.579
0.571
0.363–0.698
4
Objects
21
7.1
0.448
2.06
0.518
0.508
0.381–0.662
2
Appendix
Table 7: Tasks by subject, canvas size (longer side), and editing type. Score SD is the standard deviation of the judge scores of a task’s twelve artworks, averaged over the slice; rater–rater is the pairwise agreement among the ten raters; rank SD is the standard deviation of the twelve configurations’ mean human ranks over the slice; judge–human is the judge against the ten-rater consensus. Leave-one-out gives the judge’s mean agreement with the consensus of nine raters, the range of the ten held-out raters, and how many of them fall below the judge.
Figure 9: The ranking interface on summer-seaside , with the reference image (left) and without it (right). Cards carry random labels; the rater drags them into a Z-shaped order from best (top left) to worst (bottom right).
Configuration
Runs 1 / 2 / 3
SD of means
Per-task SD
Creation SD
ρ
GPT-6 Astra (high)
83.76 / 82.95 / 81.69
1.04
4.35
3.69
0.69–0.77
GPT-5.6 Sol (medium)
76.92 / 77.38 / 76.72
0.34
5.46
4.95
0.67–0.78
DeepSeek V4 Flash (high)
65.52 / 63.95 / 65.37
0.87
10.02
9.76
0.54–0.62
Appendix
Table 8: Three independent creation runs. Run 1 is the run of Table 2 . Per-task SD is the standard deviation of a task’s score across runs; creation SD removes the judge’s repeat-scoring variance from it; ρ is the range of the Spearman correlation of the 100 task scores between two runs, over the three pairs of runs.
Model
Level
All
Gen.
Edit
Δ [95% CI]
Formal
Output (k)
Cost
GPT-6 Astra
low
81.8
80.5
85.8
−1.9[−2.6,−1.3]
100
1.3
$0.19
medium
82.4
80.8
87.4
−1.3[−2.1,−0.6]
100
1.3
$0.17
high ∗
83.8
82.3
88.0
—
100
1.9
$0.24
xhigh
84.7
83.6
87.9
+0.9[+0.3,+1.5]
100
4.7
$0.50
DeepSeek V4 Flash
none
62.4
64.9
54.8
−3.2[−6.0,−0.3]
96
19.8
$0.05
low
61.0
62.9
55.2
−4.5[−7.8,−1.6]
100
11.0
$0.03
Appendix
Table 9: Reasoning effort, each level on all 100 tasks. Δ is the paired difference from the level of Table 2 (marked ∗ ) with a 95% task-bootstrap interval; formal counts tasks submitted by the model itself; output is the median per task, reasoning included; cost is per task at list price (Appendix J.3 ).
GPT-6 Astra (high)
DeepSeek V4 Flash (high)
Tasks
Answered
Direct
Agent
Answered
Direct
Agent
Generation, ≤32 (28)
28
80.8
81.4
25
43.2
70.9
Generation, 33–64 (33)
32
76.0
82.8
13
2.2
65.2
Generation, >64 (14)
14
69.1
83.1
0
0.0
62.9
Generation (75)
74
76.5
82.3
38
17.1
66.9
Editing (25)
24
78.1
88.0
14
23.0
61.4
Appendix
Table 10: Direct answering against the agent, by canvas size (longer side). Answered counts tasks with a valid answer; the direct-answer score counts the others as 0.
Figure 10: Sample of the analyses scored by the judge, on six tasks drawn at random (four text-to-image, two editing). (a) Each configuration created every task three times from scratch under the same settings; run 1 is the run of Table 2 . (b) The reasoning-effort levels of three models; the levels of Table 2 are GPT-6 Astra at high, DeepSeek V4 Flash at high, and GLM-5.3 Flash at max. Badges: judge score.
Configuration
Calls
Writes
Input (k)
Cached
Output (k)
Minutes
Cost
GPT-6 Astra (high)
4.6
2.0
63
88%
2.3
1.9
$0.24
Claude Opus 5 (low)
15.8
9.7
292
85%
7.5
7.3
$0.52
Qwen3.8 Max
15.9
11.1
310
91%
65.8
20.8
$0.52
Gemini 3.8 Flash (high)
27.0
13.2
1,143
79%
63.7
9.4
$0.49
GPT-5.6 Sol (medium)
5.7
2.2
82
89%
3.0
1.2
$0.13
DeepSeek V4.1 Flash (high)
27.4
15.2
1,782
98%
76.9
8.4
$0.11
Appendix
Table 11: Per-task averages over the 100 tasks. Calls are model calls; writes are source modifications; input and output are thousands of tokens, output including reasoning; cached is the share of input served from the provider’s cache.
Figure 11: Mean judge score over all 100 tasks against cost per task at list price. Colored lines join the reasoning-effort levels of one model, with an asterisk at the level of Table 2 ; grey points are the other configurations at the settings of Table 2 , including GPT-6 Astra without image observation; hollow points are the direct-answer baseline of Appendix J.2 .
Figure 12: Pixel-art techniques in the submissions, each from a generation task drawn from an empty canvas. In each panel the whole artwork is on the left, with a frame around the region enlarged on its right; enlargements use nearest-neighbor scaling with a faint pixel grid, and transparent pixels are flat grey. Thin rings mark example pixels: a 4×4 checkerboard in the halo of the sun; outline pixels whose color follows the region they bound, dark green on the cloak, a darker skin tone on the leg, and dark brown on the boot; intermediate tones at the steps of a diagonal blade; and eyes of 1×2 pixels with a one-pixel blush and pen on a 16×16 canvas. Swatches give the five bands and the shadow of the letters, and the slime’s body ramp from shadow to highlight, which shifts from blue-green toward yellow-green. The tile on the left of the last panel is the 16×16 artwork the model drew; on its right it is repeated 3×3 , with one copy outlined.