PixelBench: Can LLMs Understand and Generate Images Pixel by Pixel?
Organizations: CUHK · SCUT · NTU Singapore
Abstract
Large language models (LLMs) can generate visual assets through text-based formats such as SVG and CAD, but these symbolic representations describe images using high-level primitives rather than explicit raster values. We ask whether LLMs can instead generate pixel images directly through their existing vocabulary, without a learned image tokenizer or decoder. We introduce PixelBench, a benchmark for text-to-pixel (T2P) and image-to-pixel (I2P) generation across multiple resolutions, together with a complementary pixel-understanding evaluation. Models generate an RGB color palette followed by a grid of indices specifying the palette color of each output pixel. In T2P, the strongest models produce recognizable images directly from text. In I2P, the strongest models outperform palette-matched downsampling and specialized pixelization algorithms at low resolutions. Pixel-understanding performance correlates strongly with generation performance. A resolution-scaling study shows that pixel-generation performance often improves initially before declining at high resolutions. Finally, we introduce Pixel Reconstruction Tuning (PRT), which augments visual instruction tuning with caption-palette-grid trajectories. We post-train Molmo2-4B from its pretraining checkpoint by mixing general instruction data with our curated PRT data, improving performance across 16 visual-centric benchmarks, indicating that explicit pixel reconstruction can provide useful auxiliary supervision.
Figures & tables
| 500-way Top-1 | VLM Judge | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Valid | Flat | Avg | Avg | Agg. | ||||||
| GPT-6 Astra | 100.0 | 0.0 | 87 | 92 | 94 | 91.0 | 90.8 | 91.6 | 89.7 | 90.7 | 90.8 |
| Claude Opus 5 | 100.0 | 0.0 | 81 | 87 | 89 | 85.7 | 75.1 | 74.1 | 70.1 | 73.1 | 79.4 |
| GPT-5.6 Sol | 100.0 | 0.0 | 84 | 89 | 83 | 85.3 | 78.1 | 78.9 | 73.0 | 76.7 | 81.0 |
| Kimi K3 | 99.7 | 0.0 | 81 | 87 | 77 | 81.7 | 74.5 | 74.7 | 61.5 | 70.3 | 76.0 |
| Gemini 3.6 Flash | 100.0 | 1.3 | 87 | 78 | 72 | 79.0 | 77.8 | 71.2 | 60.4 | 69.8 | 74.4 |
| 100-way Top-1 | SigLIP cosine | DINO cosine | |||||||||||||
| Model / method | Valid | Flat | Avg | Avg | Avg | Agg. | |||||||||
| Multimodal language models | |||||||||||||||
| GPT-6 Astra | 100.0 | 0.0 | 18 | 50 | 76 | 48.0 | .615 | .652 | .695 | .654 | .023 | .177 | .326 | .175 | 80.45 |
| GPT-5.6 Sol | 100.0 | 0.0 | 44 | 61 | 63 | 56.0 | .569 | .597 | .599 | .588 | .022 | .112 | .147 | .094 | 57.27 |
| Claude Opus 5 | 100.0 | 0.0 | 26 | 49 | 47 | 40.7 | .535 | .567 | .582 | .561 | .021 | .062 | .098 | .060 | 23.16 |
| Kimi K3 | 96.3 | 0.0 | 34 | 46 | 30 | 36.7 | .558 | .562 | .501 | .541 | .020 | .064 | .038 | .040 | 8.48 |
| Comparison | Pearson | Spearman | Kendall | |
|---|---|---|---|---|
| Understanding–T2P | 7 | .871 | 1.000 | 1.000 |
| Understanding–I2P (w/o DeepSeek) | 6 | .930 | .829 | .733 |
| Benchmark | Task content | |
| General perception and reasoning | ||
| MMStar ( Chen et al., 2024 ) | Vision-essential perception and reasoning across six capability areas | 1,500 |
| MMBench ( Liu et al., 2024b ) | Multiple-choice perception and reasoning over diverse visual content | 1,292 |
| BLINK ( Fu et al., 2024 ) | Fourteen classic computer-vision tasks recast as visual question answering | 1,901 |
| BabyVision ( Chen et al., 2026a ) | Foundational visual perception and reasoning over simple scenes | 388 |
| PerceptionBench ( Lin et al., 2026 ) | Atomic perception of existence, count, position, color, and relations | 3,000 |
| Benchmark | OV | Pixel | ||
| BlindTest | 3,434 | 58.3 | 59.7 | |
| VLM Bias | 2,784 | 26.1 | 27.2 | |
| CV-Bench | 2,638 | 75.8 | 76.7 | |
| ColorBench-R. | 493 | 61.8 | 62.6 | |
| BLINK | 1,901 | 52.8 | 53.6 | |
| MMStar | 1,500 | 55.3 | 56.0 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Setting |
|---|---|
| Initialization | Molmo2-4B-Pretrain, step 32,000; Qwen3-4B language model |
| OV mixture | 500,000 single-image rows from LLaVA-OneVision-1.5-Instruct; answers over 2,048 tokens removed; image long side capped at 1,024 px |
| Pixel mixture | OV 500k plus 1k, 2k, 4k, or 8k unique Pixel trajectories |
| Schedule | One pass; OV 1,190 steps; OV Nk uses steps |
| Batching | 128 packed sequences 4,096 tokens; one image per sample; maximum answer length 1,800 tokens |
| Optimization | AdamW; LLM LR ; ViT/connector LR ; 60-step warmup; final LR multiplier 0.1; gradient clip 1.0; bf16 |
| Run | Example avg. | Benchmark avg. |
|---|---|---|
| OV-only s0 | 54.1 | 54.3 |
| OV-only s1 | 54.7 | 54.4 |
| OV-only s2 | 53.8 | 53.6 |
| OV-only s3 | 54.2 | 54.2 |
| OV 2k s0 | 55.1 | 55.1 |
| OV 2k s1 | 55.0 | 54.9 |
| Benchmark | All Pixel | ||||
|---|---|---|---|---|---|
| CV-Bench | +1.8 | +0.9 | +1.5 | +1.0 | +1.2 |
| ColorBench-Robust | +1.3 | +0.8 | +1.5 | +0.4 | +1.0 |
| VLMs Are Biased | +0.6 | +1.1 | +0.2 | +1.7 | +1.0 |
| MathVista | +0.2 | +0.5 | +1.5 | +1.3 | +0.8 |
| MMStar | +0.5 | +0.8 | +1.0 | +0.8 | +0.8 |
| HallusionBench | +0.1 | +0.2 | +2.2 | +0.5 | +0.7 |
| Model | Avg. | Base | FV | FV Pixel | |
|---|---|---|---|---|---|
| 2B | Example | 43.0 | 48.1 | 50.1 | +2.0 |
| Benchmark | 45.2 | 47.3 | 50.3 | +3.0 | |
| 4B | Example | 48.8 | 53.9 | 56.8 | +2.9 |
| Benchmark | 51.5 | 53.6 | 57.3 | +3.7 | |
| 8B | Example | 48.8 | 55.1 | 57.2 | +2.1 |
| Benchmark | 53.6 | 56.3 | 57.9 | +1.6 |
| 2B | |||||
|---|---|---|---|---|---|
| MMVP | 150 | +7.3 | +12.0 | +9.3 | +11.3 |
| BabyVision | 388 | +4.6 | +2.8 | +7.0 | +4.9 |
| MMBench | 1292 | +5.0 | +4.8 | +4.2 | +4.8 |
| CV-Bench | 2638 | +3.1 | +3.4 | +3.5 | +3.0 |
| TallyQA * | 3000 | +3.3 | +2.8 | +2.4 | +2.9 |
| Hallusion | 1129 | +5.1 | +1.2 | +3.8 | +2.8 |
| Model | Avg. | |||||
|---|---|---|---|---|---|---|
| 2B | Example | +2.43 | +2.68 | +2.01 | +2.31 | +2.18 |
| Benchmark | +3.30 | +3.20 | +3.02 | +3.31 | +3.91 | |
| 4B | Example | +2.73 | +2.08 | +2.94 | +1.92 | +2.66 |
| Benchmark | +3.33 | +3.01 | +3.70 | +2.77 | +3.61 | |
| 8B | Example | +1.59 | +1.21 | +2.07 | +0.59 | +0.99 |
| Benchmark | +1.67 | +1.04 | +1.60 | +0.90 | +0.55 |
| Model | Temperature | Reasoning configuration | Max output tokens | Samples/item |
| Text-to-Pixel (T2P) | ||||
| GPT-6 Astra | 1.0 | Default | 32,768 | 1 |
| Claude Opus 5 | 1.0 | Default (high effort) | 32,768 | 1 |
| GPT-5.6 Sol | 1.0 | High effort | 32,768 | 1 |
| Kimi K3 | 1.0 | High effort | 32,768 | 1 |
| Gemini 3.6 Flash | 0.7 | Default | 32,768 | 1 |
| 4. cat | 8. fish | 17. dog | 18. bird | 19. rabbit |
| 20. elephant | 21. turtle | 22. butterfly | 51. cow | 52. sheep |
| 53. pig | 54. chicken | 55. owl | 56. penguin | 57. frog |
| 58. snake | 59. bee | 60. snail | 1. apple | 3. banana |
| 23. orange | 24. strawberry | 25. watermelon | 26. carrot | 27. pizza |
| 28. cake | 29. bread | 30. ice cream | 61. pear | 62. pineapple |
| 63. grapes | 64. cherry | 65. lemon | 66. corn | 67. donut |
| 500. ox | 269. harp | 281. flute | 162. buckle | 286. teapot | 93. rainbow | 168. lorikeet | 421. drumstick |
| 91. axe | 31. lamp | 327. goose | 98. cactus | 375. throne | 448. Samoyed | 163. Malinois | 165. golf cart |
| 75. bed | 97. leaf | 12. heart | 36. camera | 344. tights | 154. sawmill | 446. marmoset | 362. grey wolf |
| 59. bee | 333. mask | 5. house | 26. carrot | 386. tripod | 257. snorkel | 50. mountain | 189. guacamole |
| 204. bra | 275. maze | 172. junco | 100. castle | 21. turtle | 424. spatula | 13. mushroom | 334. gyromitra |
| 40. bus | 161. mink | 72. knife | 64. cherry | 216. tusker | 140. sundial | 221. odometer | 355. hair clip |
| 7. car | 48. moon | 65. lemon | 498. collie | 191. valley | 128. T-shirt | 229. payphone | 68. hamburger |
| 260. Arctic fox | 499. coffeemaker | 249. Cairn Terrier | 119. steam locomotive |
| 263. bald eagle | 442. dung beetle | 365. china cabinet | 196. computer keyboard |
| 397. black swan | 382. fire screen | 490. digital clock | 186. impala (antelope) |
| 477. Bloodhound | 264. freight car | 336. garbage truck | 482. little blue heron |
| 383. box turtle | 207. grand piano | 166. grocery store | 295. monarch butterfly |
| 461. bridegroom | 452. great egret | 158. howler monkey | 373. Redbone Coonhound |
| 213. can opener | 234. green mamba | 427. Irish Terrier | 285. rhinoceros beetle |
| Resolution | Pool | Gem 3.6 F | GPT-5.6 Sol | GPT-6 Astra | Opus 5 | Kimi K3 | DeepSeek V4 Pro | Seed-2.1 Pro |
|---|---|---|---|---|---|---|---|---|
| 100-way | .910 | .910 | .950 | .910 | .900 | .580 | .810 | |
| 500-way | .870 | .840 | .870 | .810 | .810 | .490 | .700 | |
| 100-way | .830 | .920 | .980 | .950 | .930 | .680 | .820 | |
| 500-way | .780 | .890 | .920 | .870 | .870 | .590 | .720 | |
| 100-way | .810 | .890 | .980 | .970 | .860 | .630 | .830 | |
| 500-way | .720 | .830 | .940 | .890 | .770 | .510 | .740 |
| Resolution | Pool | Gem3.6 F | GPT-5.6 Sol | GPT-6 Astra | Opus 5 | Kimi K3 | DeepSeek V4 Pro | Seed-2.1 Pro |
|---|---|---|---|---|---|---|---|---|
| 100-way | .945 | .945 | .972 | .949 | .941 | .697 | .874 | |
| 500-way | .913 | .899 | .927 | .884 | .881 | .599 | .792 | |
| 100-way | .887 | .956 | .990 | .972 | .965 | .769 | .892 | |
| 500-way | .837 | .931 | .960 | .925 | .932 | .681 | .815 | |
| 100-way | .843 | .933 | .990 | .985 | .896 † | .718 | .898 | |
| 500-way | .785 | .893 | .970 | .945 | .826 † | .614 | .832 |
| Model | ID/20 | Parts/25 | Struct./20 | Detail/20 | Pixel/10 | Color/5 | Total/100 |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 18.83 | 22.89 | 18.16 | 16.95 | 8.94 | 4.98 | 90.75 |
| GPT-5.6 Sol | 16.11 | 19.31 | 15.77 | 13.98 | 8.21 | 4.75 | 78.13 |
| Claude Opus 5 | 15.77 | 18.52 | 15.61 | 12.60 | 7.88 | 4.75 | 75.13 |
| Kimi K3 | 15.48 | 18.21 | 15.27 | 12.90 | 7.99 | 4.69 | 74.54 |
| Gemini 3.6 Flash | 16.22 | 19.23 | 15.79 | 13.66 | 8.17 | 4.72 | 77.79 |
| DINO CLS cosine | SigLIP cosine | 100-way Top-1 (%) | |||||||
| Method | |||||||||
| Tier 0 Simple resampling | |||||||||
| Nearest + median cut | .012 | .090 | .480 | .523 | .568 | .665 | 9 | 28 | 62 |
| Box + median cut | .012 | .099 | .475 | .523 | .577 | .673 | 10 | 27 | 65 |
| Tier 1 Linear filters | |||||||||
| Bicubic + median cut | .013 | .093 | .459 | .524 | .573 | .672 | 9 | 26 | 64 |
Image Generation
Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.
See all 502 papers