cs.CVOct 19, 2026
SaveGenOlympiad: Benchmarking Frontier Image Models at Their Capability Boundaries
Organizations: Australian National University · Canva
Abstract
Text-to-image models have made tremendous progress and can produce high-quality images. However, a question which would be asked in any other discipline is where the frontier models fall short. We introduce GenOlympiad, a benchmark which provides the most challenging text prompts from nine capability axes for stress-testing frontier text-to-image models. The assessed capabilities include generating large quantities, performing deep reasoning, ensuring hard physical consistency, following complex rules and so on. The prompts are designed such that prompt alignment is visually verifiable (e.g., counting) instead of relying on subjectiveness (e.g., style). Based on this, we develop an automated evaluation framework which decomposes the prompt into easy and hard questions, and for most hard questions, draws help lines to ground evidence for accurate grading. We evaluate 20+ strong text-to-image models, including Nano Banana Pro, GPT-Image-2.5, Seedream-5.0 and so on. We report that even the best-performing T2I model (Nano Banana Pro) only achieves a strict score of 23.9 out of 100 and that coding models such as GPT-6 Astra have higher prompt alignment scores (but not as visually pleasing). We also derive insights in model evolution, alignment with Elo scores, and effectiveness of our grounding-based evaluation protocol. Code is available at https://github.com/GenOlympiad/GenOlympiad and the project page is at https://genolympiad.github.io/
Figures & tables
Figure 1: Image generation examples of the nine capability axes in GenOlympiad. In each capability, images from left to right are generated by FLUX.2 dev, GPT Image 2.5, and GPT-6 Astra, respectively. The prompts are shortened for demonstration. More examples are in Appendix Fig. 17 .
Figure 2: Example of a GenOlympiad prompt and its evaluation program. Left : prompt and a generated image. Right : Easy and hard questions compose evaluation program. Easy questions are evaluated first as a ‘gating’ check: if any fails, the final score will be zero. If all pass, the program will mark hard questions, where each hard question has partial mark with different weight.
Figure 3: Topic distribution of GenOlympiad on each axis. We show the number of tasks in each topic. A detailed distribution can be seen in Appendix Fig. 12
Figure 4: Pipeline of evidence grounding. Given a hard question and the generated image (left), we draw helping lines on the image, e.g. , bounding boxes for counting problems, boundaries for the sun, and extension lines for the compass pointer. The type of helping line is decided in the evaluation program. To make the lines accurate, we design an iterative process (middle), where we use Gemini 2.5 Flash to decide whether each line serves the purpose, until success or running out of budget. The grounding result (right) is finally used to decide whether the hard constraint is satisfied.
Figure 5: An example pipeline of deterministic scoring. We write the grounding result into a JSON state file, which, for each die, lists the bounding box height, width, and center coordinates, and the displayed number. We use a deterministic Python script to count the number of dice and the sum of the numbers for grading. The entire deterministic scoring pipeline is provided in the evaluation program. Prompt is the same as Fig. 2 .
| Model | Text | C-Fact | Symbol | World | Phys | Quant | Spatial | Latent | Long | Overall | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| S | R | S | R | S | R | S | R | S | R | S | R | S | R | S | R | S | R | S | R | |
| Coding Models | ||||||||||||||||||||
| GPT-6 Astra | 65.4 | 74.1 | 38.4 | 50.2 | 45.5 | 56.4 | 27.5 | 46.9 | 25.6 | 43.0 | 40.8 | 54.0 | 26.7 | 48.3 | 58.4 | 73.7 | 30.0 | 53.1 | 40.5 | 56.1 |
| GPT-5.6 Sol | 62.5 | 72.4 | 32.2 | 43.3 | 42.3 | 55.5 | 26.0 | 43.5 | 27.7 | 41.6 | 31.6 | 45.8 | 28.2 | 49.4 | 53.3 | 69.1 | 28.4 | 46.7 | 37.5 | 52.4 |
| Prompt Engineering on GPT-Image-2.5 | ||||||||||||||||||||
| GPT 5.6 Sol | 62.2 | 71.7 | 33.0 | 48.7 | 30.4 | 44.7 | 17.8 | 36.2 | 25.0 | 39.4 | 24.3 | 40.8 | 12.5 | 34.3 | 55.8 | 72.3 | 27.2 | 43.2 | 33.0 | 48.9 |
Table 1: GenOlympiad results for closed-source T2I models, coding models, prompt engineering and open-source T2I models across nine capability axes. Best result in each column and each category is highlighted in bold . C-Fact: counterfactual instruction following; Symbol: symbolic systems; World: world-knowledge grounding; Phys: physical consistency; Quant: quantity and precision; Spatial: spatial and 3D reasoning; Latent: latent computation; Long: long-range consistency. Overall is the mean score over all scored tasks of a model. Withheld tasks are excluded from both axis and overall scores (see Fig. 6 ). S and R are strict and relaxed scores.
Figure 6: Grounding failure rate for different models. Lower is better. Closed-source T2I models are shown in blue, coding models in orange, and open-weight T2I models in green. Prompt Engineering in red. The Spearman correlation coefficient between failure rate and Olympiad strict scores is -0.92. It indicates that stronger models generally see lower grounding failures in our evaluation pipeline.
Figure 7: Performance of GPT, Seedream and Nano Banana series across nine axes. And the performance between GPT-Image-2 and Nano Banana 2. The performance improves as stronger models are released. This might not be the case for less strong models (see Appendix Fig. 13 )
Figure 8: Correlation between GenOlympiad relaxed scores and three human-preference-based arena scores. OneIG-Bench is included for comparison. GenOlympiad seems to be a good proxy of Elo scores. Results for other benchmarks are shown in Appendix Fig. 15 .
| VLM pair | Agreement |
|---|---|
| Gemini 2.5 Flash & GPT-5.6 | 96.7% |
| Gemini 2.5 Flash & GPT-4.1 | 96.7% |
| GPT-5.6 & GPT-4.1 | 97.6% |
Table 4: Pairwise agreement between three different VLM judges on the same 365 set of human evaluation questions.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Prompt length distribution, Left: word count distribution, Right: Token distribution using o200k_base . Solid line : median, Dashed line : mean.
| Model | Text | C-Fact. | Symbol. | World. | Phys. | Quant. | Spatial. | Latent. | Long. | Overall | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Str. | Rel. | Str. | Rel. | Str. | Rel. | Str. | Rel. | Str. | Rel. | Str. | Rel. | Str. | Rel. | Str. | Rel. | Str. | Rel. | Str. | Rel. | |
| GPT-Image-2.5 | 30.4 | 41.8 | 23.0 | 36.4 | 20.5 | 34.8 | 6.8 | 24.1 | 8.1 | 23.7 | 13.2 | 25.0 | 18.4 | 34.6 | 6.5 | 21.9 | 9.8 | 25.9 | 14.9 | 29.6 |
| GPT-Image-2.5 | 25.6 | 40.3 | 18.4 | 35.1 | 13.5 | 27.8 | 11.5 | 25.4 | 10.8 | 27.9 | 12.7 | 26.2 | 16.5 | 31.1 | 10.4 | 24.2 | 9.9 | 26.3 | 14.2 | 29.3 |
Table 5: GenOlympiad results for GPT-Image-2.5 Sunburst for its first and second attempts of GenOlympiad across nine capability axes. Best result in each column is highlighted in bold . C-Fact.: counterfactual instruction following; Symbol.: symbolic systems; World.: world-knowledge grounding; Phys.: physical consistency; Quant.: quantity and precision; Spatial.: spatial and 3D reasoning; Latent.: latent computation; Long.: long-range consistency. Overall is the mean score over all scored tasks; Str. and Rel. are strict and relaxed scores.
Figure 12: Different Topics for each axis.
| Auto Pass | Auto Fail | Total | |
|---|---|---|---|
| Human Pass | 174 | 13 | 187 |
| Human Fail | 4 | 174 | 178 |
| Total | 178 | 187 | 365 |
Table 6: Pass/fail distribution between human judgments and our automatic evaluator.
Figure 13: Performance of the FLUX series, Sana series and Stable Diffusion 3.5 series across nine axes. For these less strong open-source models, a newer or larger models’ performance are not always better. Especially in some axes e.g. , World knowledge grounding of FLUX series, Text Fidelity of Sana series, Physical consistency of Stable Diffusion 3.5, newer or larger models’ performance getting worse instead.
| Model | Steps | Guidance | Context length | Precision |
|---|---|---|---|---|
| FLUX.2 dev | 50 | 4.0 | 512 | bf16 |
| FLUX.2 klein 9B | 4 | 1.0 | 1024 | bf16 |
| FLUX.2 klein 4B | 4 | 1.0 | 1024 | bf16 |
| FLUX.1 dev | 28 | 3.5 | 512 | bf16 |
| FLUX.1 schnell | 4 | 0.0 | 512 | bf16 |
| Qwen-Image 20B | 50 | 4.0 | 1024 | bf16 |
Table 7: Generation settings of open-source T2I models. Steps and guidance scale are the default values. The context length is the maximum number of tokens of the primary text encoder; we set it to the largest value the pipeline accepts, so that the full prompt is visible. All images are 1024 1024.
Figure 14: Correlation between withheld rate and human-preference arena scores. Higher arena scores correspond to lower withheld rates on GenOlympiad.
Figure 15: Correlation between arena and some other T2I benchmark GenEval and DPG-Bench.
Figure 16: Sample images of each models with their aesthetic and GenOlympiad relaxed score. Prompts were shorten, not original.
Figure 17: More example images for each capability axis.
More in this topic
Image Generation Evaluation
Image generation evaluation measures the quality, fidelity, diversity, and instruction-following ability of AI-generated images.
See all 108 papers