PixelBench: Evaluating Language Models for Pixel-Art Creation
Organizations: Westlake University · Mind Lab · ShanghaiTech University
Abstract
Pixel art is widely used in games and digital content creation, requiring readable forms, coherent spatial relationships, and harmonious colors under limited resolution and palette constraints. Language models show growing promise in visual creation, yet their pixel art creation capabilities lack systematic evaluation. We introduce PixelBench, formalizing text-to-image generation and instruction-guided editing for pixel art and providing a benchmark for these tasks. We further present a creation environment in which language model agents invoke drawing tools through code to generate and edit pixel art. We assess the quality of generated artwork through human evaluation and employ a multimodal large language model as a judge, validating the agreement between its automated assessments and human preferences. We compare language models and configurations across diverse subjects and canvas sizes. In our experiments, GPT-6 Astra achieves both the highest average judge score and the best average human rank, highlighting the potential of general-purpose language models for pixel art creation. We release the tasks, all artworks, the human rankings, and the judge protocol.
Figures & tables
| Protocol | Change | All (100) | 95% CI | Generation (75) | Editing (25) |
|---|---|---|---|---|---|
| Simplest | single image, overall score | 0.528 | [0.484, 0.572] | 0.528 | 0.529 |
| + presentation | 1/16 transparent margin | 0.532 | [0.497, 0.568] | 0.531 | 0.536 |
| + rubric | by subject and task type | 0.544 | [0.506, 0.581] | 0.553 | 0.517 |
| + reference | same-task reference image | 0.557 | [0.520, 0.593] | 0.563 | 0.538 |
| All (100) | Generation (75) | Editing (25) | |||||||
| Configuration | Score | Rank | Score | Rank | Score | Rank | |||
| Judge | Human | Judge | Human | Judge | Human | ||||
| GPT-6 Astra (high) | 83.76 | 2.52 | 3.34 | 82.35 | 2.40 | 3.21 | 88.00 | 2.86 | 3.74 |
| Claude Opus 5 (low) | 78.49 | 5.03 | 4.88 | 77.64 | 5.01 | 5.09 | 81.04 | 5.06 | 4.25 |
| Qwen3.8 Max | 78.04 | 4.72 | 5.13 | 77.35 | 4.54 | 5.00 | 80.12 | 5.26 | 5.49 |
| Gemini 3.8 Flash (high) | 77.35 | 4.69 | 4.27 | 77.87 | 4.45 | 3.82 | 75.80 | 5.40 | 5.62 |
| Model | Setting | All (100) | Generation (75) | Editing (25) |
|---|---|---|---|---|
| GPT-6 Astra | high (Table 2 ) | 83.8 | 82.3 | 88.0 |
| low | 81.8 | 80.5 | 85.8 | |
| medium | 82.4 | 80.8 | 87.4 | |
| xhigh | 84.7 | 83.6 | 87.9 | |
| no image observation | 79.9 | 80.5 | 77.8 | |
| direct answer | 76.9 | 76.5 | 78.1 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Gap in the earlier pass | Comparisons | Reversed | Tie changed |
|---|---|---|---|
| 2376 | 9.7% | 8.9% | |
| 1827 | 5.3% | 3.0% | |
| 1509 | 3.4% | 1.8% | |
| 970 | 0.3% | 0.7% |
| Configuration | SE of 100-task mean | half-width of a rerun difference |
|---|---|---|
| GPT-6 Astra | 0.15 | 0.42 |
| Claude Opus 5 | 0.23 | 0.64 |
| Qwen3.8 Max | 0.23 | 0.63 |
| Gemini 3.8 Flash | 0.16 | 0.43 |
| GPT-5.6 Sol | 0.25 | 0.69 |
| DeepSeek V4.1 Flash | 0.26 | 0.71 |
| Judge | Reasoning effort | All | Generation | Editing | Leave-one-out | |
|---|---|---|---|---|---|---|
| GPT-6 Astra | xhigh | 0.557 | 0.563 | 0.538 | 0.554 (3) | 0.958 |
| Claude Opus 5 | low | 0.498 | 0.469 | 0.590 | 0.494 (1) | 0.825 |
| DeepSeek V4.1 Flash | high | 0.414 | 0.407 | 0.432 | 0.413 (0) | 0.958 |
| Leave-one-out | ||||||||
| Slice | Tasks | Score SD | Rater–rater | Rank SD | Judge–human | Judge | Raters | Below |
| Characters | 13 | 10.4 | 0.492 | 2.38 | 0.609 | 0.614 | 0.482–0.683 | 5 |
| Creatures | 10 | 9.4 | 0.469 | 2.04 | 0.563 | 0.569 | 0.527–0.662 | 3 |
| Scenes, architecture | 16 | 10.0 | 0.466 | 2.29 | 0.634 | 0.631 | 0.494–0.667 | 7 |
| Icons, interface | 7 | 8.6 | 0.457 | 2.25 | 0.579 | 0.571 | 0.363–0.698 | 4 |
| Objects | 21 | 7.1 | 0.448 | 2.06 | 0.518 | 0.508 | 0.381–0.662 | 2 |
| Configuration | Runs 1 / 2 / 3 | SD of means | Per-task SD | Creation SD | |
|---|---|---|---|---|---|
| GPT-6 Astra (high) | 83.76 / 82.95 / 81.69 | 1.04 | 4.35 | 3.69 | 0.69–0.77 |
| GPT-5.6 Sol (medium) | 76.92 / 77.38 / 76.72 | 0.34 | 5.46 | 4.95 | 0.67–0.78 |
| DeepSeek V4 Flash (high) | 65.52 / 63.95 / 65.37 | 0.87 | 10.02 | 9.76 | 0.54–0.62 |
| Model | Level | All | Gen. | Edit | [95% CI] | Formal | Output (k) | Cost |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | low | 81.8 | 80.5 | 85.8 | 100 | 1.3 | $0.19 | |
| medium | 82.4 | 80.8 | 87.4 | 100 | 1.3 | $0.17 | ||
| high ∗ | 83.8 | 82.3 | 88.0 | — | 100 | 1.9 | $0.24 | |
| xhigh | 84.7 | 83.6 | 87.9 | 100 | 4.7 | $0.50 | ||
| DeepSeek V4 Flash | none | 62.4 | 64.9 | 54.8 | 96 | 19.8 | $0.05 | |
| low | 61.0 | 62.9 | 55.2 | 100 | 11.0 | $0.03 |
| GPT-6 Astra (high) | DeepSeek V4 Flash (high) | |||||
|---|---|---|---|---|---|---|
| Tasks | Answered | Direct | Agent | Answered | Direct | Agent |
| Generation, (28) | 28 | 80.8 | 81.4 | 25 | 43.2 | 70.9 |
| Generation, 33–64 (33) | 32 | 76.0 | 82.8 | 13 | 2.2 | 65.2 |
| Generation, (14) | 14 | 69.1 | 83.1 | 0 | 0.0 | 62.9 |
| Generation (75) | 74 | 76.5 | 82.3 | 38 | 17.1 | 66.9 |
| Editing (25) | 24 | 78.1 | 88.0 | 14 | 23.0 | 61.4 |
| Configuration | Calls | Writes | Input (k) | Cached | Output (k) | Minutes | Cost |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra (high) | 4.6 | 2.0 | 63 | 88% | 2.3 | 1.9 | $0.24 |
| Claude Opus 5 (low) | 15.8 | 9.7 | 292 | 85% | 7.5 | 7.3 | $0.52 |
| Qwen3.8 Max | 15.9 | 11.1 | 310 | 91% | 65.8 | 20.8 | $0.52 |
| Gemini 3.8 Flash (high) | 27.0 | 13.2 | 1,143 | 79% | 63.7 | 9.4 | $0.49 |
| GPT-5.6 Sol (medium) | 5.7 | 2.2 | 82 | 89% | 3.0 | 1.2 | $0.13 |
| DeepSeek V4.1 Flash (high) | 27.4 | 15.2 | 1,782 | 98% | 76.9 | 8.4 | $0.11 |
Image Generation
Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.
See all 502 papers