Verifier Choice is a Benchmark Design Variable: Auditing Structural Counting Evaluation in Text-to-Image Models
Organizations: REASONS Lab, Institute for Network Sciences and Cyberspace, Tsinghua University · Virginia Tech
Abstract
Automatic evaluation is now central to text-to-image benchmarking, but the verifier that converts generated images into scores is often treated as an implementation detail. We argue that this is unsafe: an automatic benchmark score is a property of the image generator, prompt distribution, verifier, verifier-use protocol, and scoring rule, not of the generator alone. We study this issue in structural counting, a controlled setting where human-visible counts can be audited. We construct a verifier-centric audit suite over seven counting objects: cell phones, chairs, and traffic lights for object counting, and clover leaflets, dresser drawer fronts, power-strip sockets, and turbine blades for substructure counting. Across 6,240 generated images from Qwen-Image-2512 and FLUX.2-dev, we compare open MLLM verifiers under count-only, reject-aware, zero-shot and few-shot protocols, plus TIFA-like target queries, a detector baseline, and frontier closed-model checks on two hard objects. On a balanced 2,280-image evaluation split, aggregate score matching can coexist with image-level disagreement, and verifier-use protocols and generator distributions change the measured reliability. Every verifier-protocol configuration also shrinks the human score difference between the two generators, by 7.5-25.0 pp, and with a third generator, Z-Image, all four tested protocols reverse the human ordering of Z-Image and FLUX.2-dev on our fixed prompt suite. GPT-5.5 and Claude Opus 4.7 checks do not remove the failure modes: GPT-5.5 still has only 59.92% clover valid-count fidelity, while Claude Opus 4.7 has lower same-slice agreement (). We propose reporting automatic benchmark scores with a factorized verifier reliability profile: aggregate score gap, image-level agreement, generator-comparison distortion, valid-image count fidelity, rejection behavior, output compliance, and directional bias.
Figures & tables
| Object | Type | Targets | Imgs/cell | Imgs/gen. |
| Cell phone | Obj | 60 | 540 | |
| Chair | Obj | 60 | 540 | |
| Traffic light | Obj | 60 | 540 | |
| Clover leaflets | Part | 50 | 300 | |
| Dresser fronts | Part | 50 | 450 | |
| Power-strip sockets | Part | 50 | 450 |
| Subset | Stat. agr. | Stat. | Valid F1 | Reject F1 | Count agr. | Pass agr. | Pass | |
| Broad object –generator sample | 280 | 94.29% | 0.636 | 96.88% | 66.67% | 98.39% | 94.64% | 0.844 |
| Traffic-light test | 360 | 85.56% | 0.707 | 87.32% | 83.23% | 100.00% | 86.39% | 0.724 |
| Protocol | Sees target? | object card? | Can reject? | Per-image output and scoring rule |
| Count-only MLLM | no | yes | no | JSON integer count ; auto pass iff . |
| Reject/count MLLM | no | yes | yes | JSON validity decision and count ; auto pass iff and . |
| TIFA-like target query | yes | optional | no | Yes/no answer to “exactly ?”; image-level diagnostics ask all candidate counts and require exactly one accepted answer. |
| Detector-count | no | no | no | Auto pass iff detector count equals . |
| Detector protocol | Scope | pp | Count valid % | ||
| COCO YOLO11s, fixed | Qwen obj pred | 1561 | -13.58 | 85.47 | 0.294 |
| COCO YOLO11s, oracle object threshold | Qwen obj pred | 1561 | -3.14 | 95.57 | 0.519 |
| YOLO11s object, support-trained | MLLM obj split | 1080 | +4.63 | 87.18 | 0.016 |
| YOLO11s substructure, support-trained | MLLM part split | 1143 | -0.79 | 94.84 | 0.862 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Generator | Object | Valid | Ambiguous | Clear-fail | Valid rate |
| Qwen-Image-2512 | cell phone | 536 | 4 | 0 | 99.26% |
| Qwen-Image-2512 | chair | 539 | 1 | 0 | 99.81% |
| Qwen-Image-2512 | clover leaflets | 288 | 12 | 0 | 96.00% |
| Qwen-Image-2512 | dresser fronts | 441 | 9 | 0 | 98.00% |
| Qwen-Image-2512 | power-strip sockets | 431 | 4 | 15 | 95.78% |
| Qwen-Image-2512 | traffic lights | 474 | 16 | 50 | 87.78% |
| Released component | Contents |
| Generated images | All images by generator, object , prompt template, target count, and seed |
| Human annotations | Human-1 full labels, Human-2 subset plus full traffic-light labels, and Human-3 labels for the same 600 images |
| Verifier outputs | Raw outputs, parsed JSON/counts, and parser status |
| Protocol metadata | Prompts, object cards, support indices, and target counts |
| Evaluation code | Scoring, bootstrap confidence intervals, aggregation, and table generation |
| Detector artifacts | Detector predictions, thresholds, and support/training metadata |
| Model | Protocol | Count out % | A all % | pp | Count valid % | Agree % | |
| Gemma3n-E4B | zero count | 100.00 | 83.46 | +4.17 | 88.88 | 82.85 | 0.436 |
| Gemma3n-E4B | zero reject/count | 100.00 | 84.96 | +5.66 | 90.27 | 84.17 | 0.464 |
| Gemma3n-E4B | few-shot count | 100.00 | 82.46 | +3.16 | 87.97 | 81.67 | 0.408 |
| Gemma3n-E4B | few-shot reject/count | 100.00 | 82.50 | +3.20 | 87.73 | 81.36 | 0.398 |
| Gemma4-E4B | zero count | 100.00 | 70.53 | -8.77 | 73.98 | 71.32 | 0.245 |
| Gemma4-E4B | zero reject/count | 100.00 | 69.34 | -9.96 | 73.69 | 71.01 | 0.250 |
| Model | Protocol | Numeric coverage % | MAE | Signed bias | |
| Gemma3n-E4B | zero count | 100.00 | 0.1140 | -0.0441 | 0.387 |
| Gemma3n-E4B | zero reject/count | 100.00 | 0.0997 | -0.0278 | 0.279 |
| Gemma3n-E4B | few-shot count | 100.00 | 0.1222 | -0.0446 | 0.365 |
| Gemma3n-E4B | few-shot reject/count | 100.00 | 0.1241 | -0.0561 | 0.452 |
| Gemma4-E4B | zero count | 100.00 | 0.2669 | -0.1260 | 0.472 |
| Gemma4-E4B | zero reject/count | 100.00 | 0.2693 | -0.1476 | 0.548 |
| Object | Images | Image acc. | 0-yes rate | -yes rate | Wrong 1-yes rate |
| Cell phone | 180 | 97.78% | 2.22% | 0.00% | 0.00% |
| Chair | 180 | 100.00% | 0.00% | 0.00% | 0.00% |
| Clover leaflets | 120 | 70.83% | 0.83% | 4.17% | 24.17% |
| Dresser fronts | 180 | 63.33% | 13.89% | 6.67% | 16.11% |
| Power-strip sockets | 180 | 98.33% | 0.00% | 1.11% | 0.56% |
| Traffic lights | 180 | 100.00% | 0.00% | 0.00% | 0.00% |
| Model | Object | H all % | A all % | pp | Count valid % | Bias | ||
| GPT-5.5 | Clover | 240 | 79.58 | 52.92 | -26.67 | 59.92 | +0.543 | 0.344 |
| Claude Opus 4.7 | Clover | 240 | 79.58 | 55.83 | -23.75 | 56.96 | +0.414 | 0.203 |
| GPT-5.5 | Traffic | 360 | 53.33 | 63.89 | +10.56 | 94.27 | 0.000 | 0.660 |
| Claude Opus 4.7 | Traffic | 360 | 53.33 | 48.33 | -5.00 | 74.48 | -0.070 | 0.557 |
| GPT-5.5 | Overall | 600 | 63.83 | 59.50 | -4.33 | 75.29 | +0.300 | 0.522 |
| Claude Opus 4.7 | Overall | 600 | 63.83 | 51.33 | -12.50 | 64.80 | +0.212 | 0.432 |
| Detector protocol | Split | H all % | A all % | pp | Count valid % | ||
| COCO YOLO11s, fixed | Qwen obj pred | 1561 | 96.03 | 82.45 | -13.58 | 85.47 | 0.294 |
| COCO YOLO11s, object threshold | Qwen obj pred | 1561 | 96.03 | 92.89 | -3.14 | 95.57 | 0.519 |
| Object detector | Overall | 1080 | 82.69 | 87.31 | +4.63 | 87.18 | 0.016 |
| Object detector | Qwen-Image | 540 | 94.44 | 92.22 | -2.22 | 92.46 | 0.109 |
| Object detector | FLUX.2-dev | 540 | 70.93 | 82.41 | +11.48 | 80.15 | -0.077 |
| Object detector | Cell phones | 360 | 95.00 | 77.50 | -17.50 | 79.34 | 0.175 |
| Field | What to report |
| Benchmark target | Ability, object definitions, prompt templates, target values. |
| Verifier | Model/checkpoint, version, provider, decoding settings, detector weights. |
| Verifier-use protocol | Prompt, object card, few-shot examples, thresholds, output schema, parser. |
| Scoring rule | Per-image pass rule, treatment of invalid/rejected images, aggregation. |
| Human audit | Annotation categories, count rules, split size, inter-annotator or adjudication procedure. |
| Aggregate reliability | Human score, automatic score, , confidence intervals. |
| Pair | Valid/reject agreement (%) | Cohen’s | Exact count / jointly valid (%) |
| H1-H2 | 90.17 [87.67,92.50] | 0.750 [0.688,0.807] | 408/410 (99.51 [98.75,100.00]) |
| H3-H1 | 92.17 [90.00,94.33] | 0.801 [0.745,0.853] | 415/416 (99.76 [99.25,100.00]) |
| H3-H2 | 95.67 [94.00,97.17] | 0.883 [0.837,0.925] | 433/440 (98.41 [97.16,99.53]) |
| Renderer | Support images | zero-shot | few-shot | [95% CI] | Few-shot agree. (%) |
| vLLM 0.20 default (auto string) | before the text | 0.691 | 0.334 | 76.75 | |
| vLLM 0.20 string, interleaved | next to labels | 0.722 | 0.761 | 93.42 | |
| vLLM 0.20 OpenAI content | next to labels | 0.757 | 0.797 | 94.30 | |
| SGLang | next to labels | 0.740 | 0.805 | 94.30 |
| Model | Prot. | (pp) | Agree. (%) | Count fid. (%) | Agree. (pp) | (pp) | ||
| Gemma 3n E4B-it | ZC | +4.17 | 82.85 | 0.436 | 88.88 | -1.23 | -0.045 | -18.33 |
| Gemma 3n E4B-it | ZV | +5.66 | 84.17 | 0.464 | 90.27 | -1.23 | -0.044 | -17.11 |
| Gemma 3n E4B-it | FC | +3.16 | 81.67 | 0.408 | 87.97 | -1.23 | -0.038 | -20.88 |
| Gemma 3n E4B-it | FV | +3.20 | 81.36 | 0.398 | 87.73 | -1.67 | -0.052 | -22.02 |
| Gemma 4 E4B-it | ZC | -8.77 | 71.32 | 0.245 | 73.98 | -1.05 | -0.020 | -20.00 |
| Gemma 4 E4B-it | ZV | -9.96 | 71.01 | 0.250 | 73.69 | -1.75 | -0.041 | -19.04 |
| Verifier | Protocol | Automatic difference (pp) | (pp) [95% CI] |
| Qwen3.5-27B | Zero count | +13.42 | |
| Qwen3.5-27B | Zero reject/count | +16.75 | |
| Qwen3.5-27B | Few-shot count | +13.68 | |
| Qwen3.5-27B | Few-shot reject/count | +16.32 | |
| Qwen3.5-9B | Zero count | +11.05 | |
| Qwen3.5-9B | Zero reject/count | +14.56 |
| Count bin | Exact [95% CI] | Within 1 [95% CI] | Numeric coverage [95% CI] | Rel. MAE [95% CI] | Signed bias [95% CI] |
| 7-10 | 48.11 [43.55,52.54] | 83.98 [80.53,87.24] | 97.53 [96.35,98.57] | 8.24 [6.75,10.46] | 0.11 [-0.05,0.33] |
| 11-20 | 24.87 [20.70,29.10] | 57.49 [53.06,61.78] | 96.81 [95.70,97.85] | 10.27 [9.41,11.18] | -0.46 [-0.72,-0.21] |
| 21-50 | 10.68 [7.29,14.19] | 19.79 [16.15,23.76] | 92.64 [90.62,94.60] | 16.58 [15.15,18.13] | -2.90 [-3.63,-2.21] |
| Protocol | Generator | (%) | (%) | (pp) | Agree. (%) | Count fid. (%) | Reject P/R/F1 (%) | |
| ZC | Qwen | 92.02 | 88.95 | -3.07 | 91.67 | 0.517 | 93.75 | – |
| ZC | Z-Image | 71.32 | 72.72 | +1.40 | 91.05 | 0.778 | 91.64 | – |
| ZC | FLUX | 66.58 | 77.11 | +10.53 | 82.46 | 0.572 | 92.47 | – |
| ZV | Qwen | 92.02 | 88.68 | -3.33 | 91.93 | 0.539 | 93.57 | 57.1/11.1/18.6 |
| ZV | Z-Image | 71.32 | 71.49 | +0.18 | 91.75 | 0.798 | 91.45 | 56.7/22.4/32.1 |
| ZV | FLUX | 66.58 | 74.56 | +7.98 | 86.75 | 0.684 | 94.10 | 100.0/31.8/48.3 |
| Prot. | Pair | Full [95% CI] | Point inversion | No-traffic [95% CI] | ||
| ZC | Q-F | +25.44 | +11.84 | no | ||
| ZC | Q-Z | +20.70 | +16.23 | no | ||
| ZC | Z-F | +4.74 | -4.39 | yes | ||
| ZV | Q-F | +25.44 | +14.12 | no | ||
| ZV | Q-Z | +20.70 | +17.19 | no | ||
| ZV | Z-F | +4.74 | -3.07 | yes |
| Holistic binary | Task-guided binary | Decomposed | Validity-aware | |||||
| Verifier | Agr. | Agr. | Agr. | Agr. | ||||
| Gemma 3n E4B-it | 73.85 | 0.243 | 82.05 | 0.477 | 75.13 | 0.180 | 76.41 | 0.236 |
| Gemma 4 E4B-it | 74.62 | 0.322 | 88.97 | 0.733 | 91.28 | 0.773 | 89.74 | 0.742 |
| Qwen3.5-4B | 94.87 | 0.870 | 95.90 | 0.896 | 95.13 | 0.877 | 95.90 | 0.898 |
| Qwen3.5-9B | 95.90 | 0.896 | 95.64 | 0.891 | 95.38 | 0.884 | 95.90 | 0.898 |
| Qwen3.5-27B | 96.41 | 0.910 | 95.90 | 0.900 | 95.38 | 0.886 | 95.90 | 0.898 |
| Verifier | 1 shot | 2 shots | 3 shots | 6 shots |
| Gemma 3n E4B-it | 93.16 (2.56) | 93.73 (0.85) | 95.16 (1.71) | 97.44 (2.56) |
| Gemma 4 E4B-it | 90.31 (1.71) | 89.46 (2.56) | 91.74 (4.27) | 92.31 (4.27) |
| Qwen3.5-4B | 95.16 (5.13) | 94.87 (4.27) | 94.59 (0.85) | n.e. |
| Qwen3.5-9B | 91.74 (0.85) | 89.74 (1.71) | 92.31 (0.00) | n.e. |
| Slice | (pp) | Count fid. (%) | ||
| Reported test | 1200 | -0.75 | 94.92 | 0.861 |
| Exclude 57 exact seen IDs | 1143 | -0.79 | 94.84 | 0.862 |
Automated Evaluation
Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.
See all 420 papers