cs.CVAug 15, 2026
SaveBenchmarking Frontier Text-to-Image Models on Image-Description Prompts
Organizations: Perle
Abstract
Text-to-image models are typically reported on average-case prompts, which understates the gap between systems on compositionally demanding requests involving precise object counts, multi-object attribute binding, legible embedded text, and explicit spatial constraints. We evaluate four production text-to-image systems: Hunyuan 3.0, Gemini 3 Pro Image ("Nano Banana Pro"), Black Forest Labs FLUX.2, and Ideogram 3.0. The evaluation uses the 48 hardest prompts drawn from the DataSeeds.AI Sample Dataset (DSD), selected through an automated complexity-scoring pass over the full corpus. Every generated image is graded using an independent-judge rubric. GPT-5.4-Pro authors an atomic, weighted, mutually exclusive and collectively exhaustive (MECE) evaluation rubric, while Gemini 3.1 Pro Preview independently determines whether each criterion is satisfied. Gemini 3 Pro Image ranks first with a score of 84.8/100, narrowly ahead of FLUX.2 at 82.3/100. Ideogram 3.0 and Hunyuan 3.0 score 65.7/100 and 63.3/100, respectively. Failure analysis shows that the leading systems primarily lose points through object miscounting and geometric artifacts, whereas the trailing systems more frequently produce garbled text. Ideogram 3.0 also frequently omits requested elements. Full per-sample rubrics, scores, and failure annotations are available from the authors upon request.
Figures & tables
| Score range | Definition |
|---|---|
| 1–3 | Simple single-subject scene, no counting, no exact spatial or attribute constraints. |
| 4–6 | Moderate: a few objects, some attribute or spatial relations, mild counting. |
| 7–10 | Hard: precise counts of multiple object classes, many simultaneous attribute bindings, tight spatial/compositional constraints, physics-dependent elements (reflections, shadows, transparency), legible text/script, or many named entities that must each be individually correct. |
Table 1: Complexity/hardness scoring rubric applied to every candidate DSD prompt.
| Property | Value |
|---|---|
| Source dataset | DataSeeds.AI Sample Dataset (DSD) |
| Candidate pool scored for complexity | 7,009 valid rows |
| Selected evaluation prompts | 48 (hardest by complexity score) |
| Complexity score of selected prompts | 9/10 (maximum observed) |
| Models compared | 4 (Hunyuan 3.0, Gemini 3 Pro Image, FLUX.2, Ideogram 3.0) |
| Images graded per model | 48 |
Table 2: Evaluation set summary. Complexity is scored 1–10 (higher = harder); the selected set sits at the top of the observed distribution.
Figure 1: End-to-end benchmarking pipeline, from the raw DSD dataset to the final comparison report. The dashed box (steps 4–6) is repeated independently for each of the 4 candidate systems. Steps 4–6 are sequential , not parallel, within that box: rubric authoring (step 5) depends on the specific image produced in step 4, since the writer is shown that image. The independence that matters for avoiding self-grading bias is not “the writer never sees an image” but “the writer never renders a verdict on its own rubric” – that binary judgment is made independently in step 6 by a different model.
| System | Generation route |
|---|---|
| Hunyuan 3.0 [ 11 ] | hunyuan-image endpoint (Pixazo) |
| Gemini 3 Pro Image [ 4 ] (“Nano Banana Pro”) | Direct via Google GenAI ( gemini-3-pro-image ) |
| Black Forest Labs FLUX.2 [ 1 ] | Direct via Black Forest Labs ( flux-2-pro-preview ) |
| Ideogram 3.0 [ 8 ] | Direct via Ideogram ( ideogram-v3/generate ) |
Table 3: Candidate text-to-image systems and generation route.
| Code | Meaning |
|---|---|
| MISSING_STEP | A requested element is absent from the output. |
| HALLUCINATED_OBSERVATION | An unrequested element was invented. |
| FORMAT_VIOLATION | Wrong size/ratio/output type. |
| CONSTRAINT_BREACH | An explicit non-numeric, non-spatial constraint was broken. |
| INCOMPLETE_TRAJECTORY | Output is cropped, truncated, or partial. |
| MISCOUNT | Wrong number of some countable object/element. |
Table 4: Fixed MECE failure taxonomy used for every criterion verdict.
Figure 2: Reference photograph and all four generated outputs for this prompt. Every system renders the exact address string correctly; the differentiating failure is whether the object reads as “car-shaped” at all (Gemini 3 Pro Image, Hunyuan 3.0) versus a generic wheeled mailbox (FLUX.2, Ideogram 3.0), despite FLUX.2 still outscoring Ideogram 3.0 overall on other criteria.
| Criterion | Weight | Verdict | Failure code |
|---|---|---|---|
| Positive criteria (possible weight: 54) | |||
| The primary subject is a mailbox designed to resemble a car. | +10 | ✗ | MISSING_STEP |
| The mailbox body is green. | +7 | ✓ | – |
| A sign on the mailbox reads exactly “1118 PRINTERY RD.” | +10 | ✓ | – |
| A green metal frame is visible beneath the mailbox. | +5 | ✓ | – |
| Exactly four tire-and-wheel assemblies are attached to the structure. | +8 | ✓ | – |
Table 5: Full rubric and verdicts for FLUX.2 on the “Car-shaped Mailbox” prompt (final score 50.0/100). ✓= met/not triggered; ✗ = not met/triggered (weight then counts against the model).
| Model | Final score | Notable criteria |
|---|---|---|
| Gemini 3 Pro Image | 100.0/100 | All positive criteria met; no negative criteria triggered. |
| Hunyuan 3.0 | 86.8/100 | All positive criteria met; one GEOMETRY_ARTIFACT triggered (front wheel clips the support frame). |
| FLUX.2 | 50.0/100 | Mailbox does not read as car-shaped ( MISSING_STEP , missed); GEOMETRY_ARTIFACT , HALLUCINATED_OBSERVATION , and TEXT_GARBLING all triggered. |
| Ideogram 3.0 | 25.5/100 | Four separate positive misses: not car-shaped ( ), address sign missing a digit ( TEXT_GARBLING , ), only two tires visible ( MISCOUNT , ), no visible frame ( ); plus one GEOMETRY_ARTIFACT . |
Table 6: All four systems on the “Car-shaped Mailbox” prompt.
Figure 3: Reference photograph and all four generated outputs for this prompt. FLUX.2 is the only system that renders a clean 5–6–7–8 block sequence; every other system either miscounts the swimmers, garbles the block numbers, or both.
| Criterion | Weight | Verdict | Failure code |
|---|---|---|---|
| Positive criteria (possible weight: 51) | |||
| Scene is set inside an indoor swimming pool facility. | +8 | ✓ | – |
| Exactly four young boy swimmers are shown. | +10 | ✗ | MISCOUNT |
| Swimmers occupy starting blocks labeled 5, 6, 7, and 8. | +10 | ✗ | TEXT_GARBLING |
| All four swimmers are shown preparing to dive. | +8 | ✗ | MISCOUNT |
| At least one coach or official stands on the pool deck. | +5 | ✓ | – |
Table 7: Full rubric and verdicts for Hunyuan 3.0 on the “Swimming Competition” prompt (final score 13.7/100).
| Model | Final score | Notable criteria |
|---|---|---|
| FLUX.2 | 85.5/100 | All positive criteria met, including the exact 5–6–7–8 block sequence; only one GEOMETRY_ARTIFACT triggered (officials’ hands). |
| Ideogram 3.0 | 57.7/100 | Block numbering garbled ( TEXT_GARBLING , missed); GEOMETRY_ARTIFACT and a second TEXT_GARBLING (poolside signage) triggered. |
| Gemini 3 Pro Image | 43.8/100 | Correct count of four swimmers, but block numbering garbled ( missed); extra hallucinated swimmers on deck, plus GEOMETRY_ARTIFACT and two TEXT_GARBLING triggers. |
| Hunyuan 3.0 | 13.7/100 | Only three of four swimmers rendered ( MISCOUNT on two separate criteria, and ); block numbering garbled ( ); GEOMETRY_ARTIFACT , TEXT_GARBLING , and PHYSICS_VIOLATION all triggered. |
Table 8: All four systems on the “Swimming Competition” prompt.
| Model | Final score | Net score | Avg. unmet criteria | Avg. active errors |
|---|---|---|---|---|
| Gemini 3 Pro Image (Nano Banana Pro) | 84.8/100 | +44.3 | 0.5 | 0.8 |
| Black Forest Labs FLUX.2 | 82.3/100 | +42.6 | 0.5 | 1.0 |
| Ideogram 3.0 | 65.7/100 | +34.1 | 1.4 | 1.5 |
| Hunyuan 3.0 | 63.3/100 | +33.3 | 1.4 | 1.7 |
Table 9: Matched-N ranking: identical 48 prompts, all four models. Net score is the mean unnormalized weighted balance (earned incurred); unmet criteria and active errors are mean counts per prompt.
Figure 4: Reference photograph and all four generated outputs for the prompt “Modern Kitchen”, which produced the widest score spread of any prompt in the evaluation set (89.6-point range between the best- and worst-scoring model). The prompt requires exactly four chairs, two stools, and a specific countertop material – FLUX.2 (93.2/100) satisfies nearly all positive criteria, while Hunyuan 3.0 (3.6/100) miscounts most requested objects and introduces several geometric artifacts.
| Model | Positive-axis (missed requests) | Negative-axis (active errors) |
|---|---|---|
| Gemini 3 Pro Image | MISCOUNT (9), SPATIAL_RELATION_ERROR (7) | TEXT_GARBLING (16), GEOMETRY_ARTIFACT (12), HALLUCINATED_OBSERVATION (9) |
| FLUX.2 | MISCOUNT (8), MISSING_STEP (4) | TEXT_GARBLING (19), GEOMETRY_ARTIFACT (16), PHYSICS_VIOLATION (7) |
| Ideogram 3.0 | MISSING_STEP (24), TEXT_GARBLING (12) | TEXT_GARBLING (27), GEOMETRY_ARTIFACT (24), HALLUCINATED_OBSERVATION (12) |
| Hunyuan 3.0 | TEXT_GARBLING (20), MISCOUNT (12) | TEXT_GARBLING (29), GEOMETRY_ARTIFACT (27), HALLUCINATED_OBSERVATION (15) |
Table 10: Top failure codes per model on the matched-48 set (occurrence counts across all 48 prompts). Codes with fewer than 2 combined occurrences across all models are omitted for brevity.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: “Legacy of Timeless Words” – legible-text and semantic-binding hardness factors (an engraved stone monument with a specific name, date, and quotation). Gemini 3 Pro Image is the only system with a perfect score here; the other three each fail differently – Hunyuan 3.0 renders all engraved text as unreadable glyphs, FLUX.2 engraves the correct name and dates on the pedestal rather than the floor as specified, and Ideogram 3.0 substitutes the requested life dates with an unrelated number.
Figure 6: “3 houses under the clouds” – precise spatial arrangement and multiple distinct object classes (three named buildings, a specific dog breed, specific gate and roof materials). This is one of the few prompts in the set where FLUX.2 and Ideogram 3.0 both reach a perfect score while Gemini 3 Pro Image and Hunyuan 3.0 lose a small amount of credit, illustrating that the leading tier’s advantage is not uniform across every prompt.
Figure 7: “Surreal Spiral Dreams” – a perfect-reflection physics constraint (three snail shells that must appear as six via mirror reflection) combined with precise counting. Ideogram 3.0 scores far below the other three systems here (28.0/100), driven by PHYSICS_VIOLATION and MISCOUNT verdicts on the reflection requirement specifically.
Figure 8: “Colourful Balcony with geranium flowers” – nested attribute binding across three buildings, each with its own wall color, lantern, and balcony detail. Hunyuan 3.0 scores lowest (25.5/100) here, misattributing the right building’s wall color and window-frame color and miscounting the middle balcony’s doors, on top of a duplicated ground-floor geometry artifact.
| System | Endpoint | Explicit parameters |
|---|---|---|
| Hunyuan 3.0 | Pixazo hunyuan-image generate | image_size=square_hd , num_images=1 |
| Gemini 3 Pro Image | Google GenAI generate_content ( gemini-3-pro-image ) | response_modalities=[IMAGE] |
| FLUX.2 | Black Forest Labs flux-2-pro-preview | width=1024 , height=1024 |
| Ideogram 3.0 | Ideogram ideogram-v3/generate | aspect_ratio=1x1 |
Table 11: Explicit generation parameters passed to each system’s API. All other parameters (sampling steps, guidance scale, style, safety filtering, etc.) were left at the provider’s default.