DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

Vasco Ramos; Sandra Godinho Silva; Joao Magalhaes; Ricardo Rei; Pedro Henrique Martins

cs.CVOct 2, 2026

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

Authors: Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins

Organizations: Sword Health · NOVALINCS, NOVA University of Lisbon

Abstract

Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations. Recent work addresses this by fine-tuning evaluators on preference data or by prompting a vision-language model, either holistically with the caption or with decomposed verification questions. However, existing approaches fall short: fine-tuned metrics remain bound to one backbone and training distribution; holistic metrics miss fine-grained details; and decomposed metrics rely on a fixed-YES assumption that penalizes faithful images whenever that assumption fails. In contrast, we propose DEPICT, a training-free metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighting questions by how decisively the caption determines them. By replacing fixed references, our agreement rule increases negation accuracy from 19% to 88%. To recover the context lost during decomposition, DEPICT merges this agreement score with a holistic score. We evaluate DEPICT on five benchmarks and eleven backbones from three model families and find that it surpasses all training-free metrics and exceeds fine-tuned evaluators on two out of three human-correlation benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: Text-to-image scoring paradigms. (a) Holistic : directly scores the image-caption pair. (b) Decomposed : answers caption-generated questions on the image. (c) Depict : checks agreement between image- and caption-based answers, then merges it with a holistic score.

Figure 2: Overview of Depict . (1) A VLM converts the caption into yes/no questions $q_{k}$ . (2) A frozen VLM answers each question on the image ( $p^{v}_{k}$ ) and on the caption alone ( $p^{t}_{k}$ ). (3) Questions are scored by answer agreement $a_{k}$ (rewarding matching no answers) and weighted by commitment $w_{k}$ . (4) The weighted decomposed score $S_{\mathrm{dec}}$ is averaged with holistic score $S_{\mathrm{hol}}$ .

Fine-tuned evaluators
Method	Backbone	GenAI-Bench		TIFA160		RichHF
Method	Backbone	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC
ImageReward †	BLIP	0.340	0.379	0.621	0.634	0.275	0.329
PickScore †	CLIP-H	0.354	0.363	0.428	0.434	0.392	0.413
UniGenBench++ †	Qwen2.5-VL-72B	0.461	0.444	0.608	0.590	0.533	0.517
LongT2IBench †	Qwen2.5-VL-7B	0.375	0.376	0.678	0.692	0.517	0.516

Table 1: Human correlation on GenAI-Bench, TIFA160, and RichHF-18K. Bold marks best value per column and underline second best. † : cited from publications under matching protocol; others re-run. Qwen3-VL-4B matches top baseline backbone; final row uses our largest backbone.

Method	Human corr. (SRCC)			Winoground (acc. %)			NegBench
Method	GenAI-B	TIFA160	RichHF	Text	Image	Group	acc. %
DSG	0.549	0.673	0.594	52.0	46.5	44.5	34.7
Soft-TIFA-AM ( Faith )	0.546	0.762	0.636	75.8	71.8	64.8	19.1
VQAScore ( Holistic )	0.698	0.737	0.656	83.2	82.8	75.5	84.7
Agree	0.671	0.763	0.612	74.0	74.0	65.8	88.3
Depict	0.720	0.780	0.667	82.2	82.8	74.5	86.9

Table 2: Fixed-backbone comparison on Qwen3.5-27B. All methods share images, prompts, and decoding. Agree differs from Faith only in scoring rule; Depict combines it with holistic.

\Delta

gives the difference of Depict over the strongest baseline in each column, with its paired BCa 95% CI.

Figure 3: Why agreement and merging work (Qwen3.5-27B). (a) NegBench accuracy by template; VQAScore is unaffected because its single question contains the negation. (b) NegBench accuracy vs. mean SRCC; Agree improves on Faith , and Depict boosts correlation further. (c) GenAI-Bench item pairs grouped by term alignment.

Panel of Figure 4 — Figure 4: Training-free metrics across backbones. Evaluated on five benchmarks (SRCC for correlation benchmarks; accuracy on Winoground and NegBench). (a) Other families in Appendix B.1 . (b) Score differences ( $\Delta$ ) between Depict and the strongest baseline per backbone (paired bootstrap 95% CI). Values $>0$ favor Depict ; solid markers exclude zero, and crosses indicate significant losses. Colors denote the baseline and titles wins count.

Table 3: Component ablations of the decomposed term (Qwen3.5-27B). Wino.: Winoground group accuracy; GenAI: GenAI-Bench SRCC; Neg.: NegBench accuracy; higher is better. (a) Default: questions generated directly from the caption, Agree scoring, commitment weighting

w_{k}

equation 4 ; each row changes one component. (b) Rules without

w_{k}

, then the default. † Identical to Soft-TIFA-AM. ‡ DSG’s hard fixed- yes rule on our questions, without its dependency graph.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Source	Evaluated units	Human signal	Metric	$\bar{N}$ (max)
Correlation with human ratings
GenAI-Bench	GenAI-Bench prompts and generated images	9,600 pairs, 1,600 prompts	1–5 alignment	SRCC, PLCC	5.47 (13)
TIFA160	TIFA160 prompts, 5 images each	800 pairs, 160 prompts	1–5 alignment	SRCC, PLCC	4.89 (11)
RichHF	RichHF-18K test split	955 pairs, 493 captions	misalignment score	SRCC, PLCC	5.52 (32)
Accuracy
Winoground	Winoground	400 examples, 800 matched pairs	correct pairing	text / image / group acc.	4.16 (10)

Appendix

Table 4: Benchmarks. PLCC is computed after a four-parameter logistic fit. NegBench captions follow three templates (affirmative, negation, hybrid); each image has one correct caption and three distractors.

\bar{N}

: mean number of generated questions per unique caption, with the maximum in parentheses.

Panel of Figure 5 — Figure 5: Visual answer prompt , given the image and one question; produces $p^{v}_{k}$ .

Panel of Figure 6 — Figure 6: Text-only answer prompt , given the caption and one question without the image; produces the reference $p^{t}_{k}$ .

Panel of Figure 7 — Figure 7: Question-generation prompt. {prompt} is replaced by the caption; generation runs at temperature 0.

Table 12

Backbone	DSG	Soft-TIFA-AM	VQAScore	Depict	$\bm{\Delta}$
TIFA160
Qwen3.5-4B	0.678	0.715	0.705	0.743	$+0.028^{*}$
Qwen3.5-9B	0.673	0.710	0.702	0.731	$+0.021$
Qwen3.5-27B	0.673	0.762	0.737	0.780	$+0.018$
InternVL3.5-4B	0.592	0.671	0.653	0.696	$+0.026$
InternVL3.5-8B	0.592	0.693	0.693	0.728	$+0.035^{*}$

Appendix

Table 7: Size ladders : Qwen3.5 (4B/9B/27B) and InternVL3.5 (4B/8B/14B). SRCC on the correlation benchmarks; Winoground group and NegBench accuracy as fractions.

\Delta

: Depict minus the strongest baseline in the row; ∗ : paired BCa 95% interval excludes zero (intervals in Fig. 8 ). Bold : best; underline : second.

Backbone	DSG	Soft-TIFA-AM	VQAScore	Depict	$\bm{\Delta}$
TIFA160
Gemma-4-E4B	0.595	0.685	0.694	0.704	$+0.010$
Gemma-4-12B	0.285	0.545	0.496	0.547	$+0.001$
Gemma-4-31B	0.668	0.705	0.697	0.753	$+0.048^{*}$
Qwen3.5-27B	0.673	0.762	0.737	0.780	$+0.018$
Qwen3.6-27B	0.677	0.774	0.752	0.788	$+0.014$

Appendix

Table 8: Architectures and generations. Top of each block: Gemma-4 per-layer-embedding E4B, 12B without a vision encoder, and dense 31B. Bottom: Qwen3.5, 3.6 and 3.8 at a fixed 27B (Qwen3.5-27B repeated from Table 7 as the reference). Columns as in Table 7 .

Panel of Figure 8 — Figure 8: Performance differences on every backbone and benchmark, with paired BCa 95% intervals. Filled points exclude zero.

Affirmative
Backbone	DSG	Soft-TIFA-AM	VQAScore	Agree	Depict
Qwen3.5-4B	0.553	0.485	0.831	0.888	0.873
Qwen3.5-9B	0.467	0.427	0.874	0.911	0.902
Qwen3.5-27B	0.731	0.483	0.885	0.912	0.903
Qwen3-VL-4B	0.410	0.337	0.682	0.819	0.803
Gemma-4-31B	0.815	0.432	0.801	0.796	0.816

Appendix

Table 9: NegBench accuracy by correct-answer template on further backbones (

n{=}5{,}914

, chance

0.25

). Agree : the decomposed term alone; Depict : merged with the holistic term. Bold : best per row. Backbones are a subset of those in Appendix B.1 .

Skill tag	DSG	s-TIFA-AM	VQAScore	Ours	$\Delta$
Basic vs advanced
Basic only (no advanced tag)	0.591	0.661	0.685	0.698	$+0.013$ ∗
Advanced (any advanced tag)	0.427	0.357	0.607	0.646	$+0.040$ ∗
Basic skills
Action Relation	0.584	0.621	0.709	0.733	$+0.024$ ∗
Attribute	0.562	0.594	0.700	0.726	$+0.026$ ∗

Appendix

Table 10: GenAI-Bench SRCC by skill tag (Qwen3.5-27B). A prompt can carry several tags, so item counts overlap. Depict minus the strongest baseline; ∗ : paired BCa 95% interval excludes zero.

Figure 9: Sensitivity to the merge weight

\lambda

(Qwen3.5-27B).

\lambda{=}0

: decomposed term only;

\lambda{=}1

: holistic term only; the dotted line marks

\lambda{=}0.5

, fixed a priori and used throughout. SRCC for the correlation benchmarks, accuracy for NegBench (dashed).

Backbone	Mean $\Delta$ (all items)	Mean $\Delta$ (least committed)	Improved
InternVL3.5-14B	$+1.70$	$+4.06$	5/5
InternVL3.5-4B	$+0.75$	$+1.16$	5/5
InternVL3.5-8B	$+0.30$	$+1.13$	5/5
Qwen3.5-9B	$+0.29$	$+0.99$	5/5
Gemma-4-12B	$+0.06$	$+0.45$	3/5
Qwen3.8-27B	$+0.04$	$+0.45$	4/5

Appendix

Table 12: Gain of Depict from commitment weighting per backbone, weighted minus unweighted, averaged over the five benchmarks, on all items and on the quartile of items with the least committed questions. Units: SRCC points (

\times 100

) for the correlation benchmarks, accuracy points otherwise. Improved : benchmarks with

\Delta>0.01

on all items.

					Depict	Soft-TIFA-AM
Benchmark	Metric	VQAScore	Agree	Soft-TIFA-AM	( Agree $\oplus$ VQA)	$\oplus$ VQA
GenAI-Bench	SRCC	0.698	0.671	0.546	0.720	0.696
TIFA160	SRCC	0.737	0.763	0.762	0.780	0.780
RichHF	SRCC	0.656	0.612	0.636	0.667	0.675
Winoground	group acc	75.5	65.8	64.8	74.5	75.2
NegBench	MCQ acc	84.7	88.3	19.1	86.9	62.2

Appendix

Table 13: Is the merge specific to Agree ? Both merges combine a decomposed term with VQAScore using the same

0.5/0.5

recipe; only the decomposed term differs. Bold : better of the two merges.

	Reasoning on	Winoground	GenAI-Bench †	NegBench †
		group acc.	SRCC	accuracy
Holistic term	none	75.5	0.712	85.8
Holistic term	holistic query	78.2	0.664	84.4
Decomposed term	none	65.7	0.659	89.7
Decomposed term	image and text answers	66.2	0.607	85.2
Merged (final)	none	74.5	0.727	88.6

Appendix

Table 14: Chain-of-thought reasoning in the answering step (Qwen3.5-27B), with questions held fixed so only the answering step changes. Bold : best per column within each block. The remaining combinations (image or text answers alone, or with the holistic query) lead to the same conclusion: none exceeds the reasoning-off row on GenAI-Bench or NegBench. † Run on a 100-prompt (600-image) GenAI-Bench subset and a 400-item NegBench subset; Winoground is the full set (

n{=}400

Benchmark	Unique Q	NO share	Undecided
		% [95% CI]	$w_{k}\!<\!0.2$ , %
GenAI-Bench	8,753	3.0 [2.6, 3.5]	0.3
TIFA160	875	0.1 [0.0, 0.4]	0.0
RichHF	3,019	4.0 [2.7, 5.5]	1.4
Winoground	3,325	2.3 [1.6, 2.9]	1.1
NegBench	11,652	34.7 [33.8, 35.6]	0.2

Appendix

Table 15: Answer distributions per benchmark (Qwen3.5-27B). Shares are over unique (caption, question) pairs: NO share ,

p^{t}<0.5

; undecided , commitment

w_{k}<0.2

. Intervals from a caption-level bootstrap; NegBench is restricted to correct captions, Winoground to matched pairings.

Figure 10: Score distributions on the three human-correlation benchmarks. Fraction of items per score bin for each metric on Qwen3.5-27B (filled) and Qwen3-VL-4B (outline); bars exceeding the axis are labelled with their height.

Panel of Figure 11 — Figure 11: Qualitative Results with scores per questions.

Panel of Figure 12 — Figure 12: Qualitative Results with scores per questions.

Panel of Figure 13 — Figure 13: More examples.

Panel of Figure 14 — Figure 14: GenAI-Bench examples with 6 image, 4 scorers and human evaluation.

Panel of Figure 15 — Figure 15: GenAI-Bench examples with 6 image, 4 scorers and human evaluation.

Figure 16: Negbench example with four candidate captions.

Figure 17: Negbench example with four candidate captions.

Panel of Figure 18 — Figure 18: Question scoring on the failed examples.

Text-Image Alignment

Text-image alignment measures and improves how accurately visual content matches language, from image retrieval and caption checking to text-guided generation.

See all 25 papers

Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation
May 13, 2026Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait +4Human-in-the-Loop Evaluation Automated Evaluation
King Abdullah University of Science and Technology (KAUST) · Thuwal, Saudi Arabia
Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3
TIQA: Human-Aligned Perceptual Text Quality Assessment in Generated Images
Mar 7, 2026Kirill Koltsov, Aleksandr Gushchin, Anastasia Antsiferova +1Image Quality Assessment T2I Generation Evaluation
Lomonosov Moscow State University Moscow, Russia · ISP RAS Research Center for Trusted Artificial Intelligence Lomonosov Moscow State University Moscow, Russia · ISP RAS Research Center for Trusted Artificial Intelligence MSU Institute for Artificial Intelligence Moscow, Russia +1

May 13, 2026J/K move · Enter open · S save

Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation

Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait, Ansar Khangeldin+3

King Abdullah University of Science and Technology (KAUST) · Thuwal, Saudi Arabia

Human-in-the-Loop Evaluation Automated Evaluation T2I Generation Evaluation

Open paper AskPDF arXiv

Text-to-image (T2I) generation has advanced rapidly, making reliable evaluation critical as performance differences between models narrow. Existing evaluation practices typically apply uniform annotation mechanisms, such as Likert-scale or binary question answering (BQA), across heterogeneous evaluation skills, despite fundamental differences in their nature. In this work, we revisit T2I evaluation through the lens of skill-aligned annotation, where annotation strategies reflect the underlying characteristics of each evaluation skill. We systematically compare skill-aligned annotation against uniform baselines and show that it produces more consistent evaluation signals, with higher inter-annotator agreement and improved stability across models. Finally, we present an automated pipeline that instantiates the proposed evaluation protocol, enabling scalable and fine-grained evaluation with spatially grounded feedback. Our work highlights that improving the foundations of image evaluation can increase reliability and efficiency without simply scaling annotation effort. We hope this motivates further research on refining evaluation protocols as a central component of reliable model assessment.

Abstract

Text-Image Alignment

Explore similar work

Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation

Multimodal Language Models as Text-to-Image Model Evaluators

TIQA: Human-Aligned Perceptual Text Quality Assessment in Generated Images