VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

Subhadeep Roy; Christoph Leiter; Steffen Eger

cs.CVOct 12, 2026

VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

Authors: Subhadeep Roy, Christoph Leiter, Steffen Eger

Organizations: Natural Language Learning & Generation Lab, University of Technology Nuremberg

Abstract

Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explainable metrics provide rationales, but the visual evidence supporting those rationales can still be difficult to verify. We introduce VisExMEM, a reference-free evaluator that decomposes captions into atomic claims, localizes visual evidence, verifies each claim, and averages claim-level support into a global score. Explanations use the same evidence and decisions. We train Qwen3.5-based 4B and 9B models on 100K samples with multi-task SFT, counterfactual refinement, and evidence-aware GDPO. Across eight benchmarks, VisExMEM-9B achieves the highest descriptive composite among open-source evaluators in our comparison, exceeding VQAScore by 11.7 points while remaining competitive with GPT-5.6 Sol. We also introduce AGITA-Bench, containing 600 human-annotated image-text pairs and 8.7K atomic claims for global and claim-level evaluation. For untuned Qwen3.5 and GPT-5.6 Sol controls, decomposed prompting raises the same composite by 3.6-5.1 points relative to holistic prompting. Human evaluation shows that VisExMEM explanations outperform prior explainable metrics and approach GPT-5.6 Sol.

Figures & tables

Figure 1: Overview of VisExMEM. VisExMEM decomposes captions into atomic claims, localizes visual evidence for each claim, computes claim support scores, and averages them into a global score. Explanations are generated afterward from the same evidence and fixed claim decisions.

Figure 2: VisExMEM training pipeline. Multi-task SFT learns the structured evaluation tasks, counterfactual refinement targets semantic changes, and evidence-aware GDPO improves verification and evidence dependence. Earlier supervision is replayed in later stages.

Figure 3: AGITA-Bench annotation summary. (a) Global-alignment and (b) atomic-support distributions by direction and difficulty; means are shown above bars. (c) Decomposition quality.

Panel of Figure 4 — Figure 4: Qualitative AGITA-Bench examples with VisExMEM -9B. Easy–Hard cases show the input, localized evidence, caption/prompt, and Human/ VisExMEM -9B global and atomic scores (H/V); human ratings are normalized to $[0,1]$ , with short claim-level explanations.

Method	WG	EqBen	TIFA160	Pick-a-Pic	GenAI	CROC hum	T2I-CB++	CountBench	Avg.
	Group	Group	PAcc	Pref. Acc.	PAcc	Score	SRCC	Acc.
CLIPScore	7.8	25.0	54.1	76.0	52.2	69.8	28.5	30.7	43.0
BLIPv2Score	17.5	35.0	57.5	80.0	55.1	76.1	37.0	47.3	50.7
ALIGNScore	8.3	28.6	59.6	80.0	53.4	71.8	36.1	34.5	46.5
PickScore	6.8	23.6	59.4	70.0	57.7	74.7	30.3	56.5	47.4
ImageReward	12.8	26.4	67.3	75.0	58.7	75.4	50.6	33.8	50.0

Table 1: Main results. Comparison of prior image–text alignment metrics, controlled VLM-as-a-Judge baselines, and VisExMEM across eight established benchmarks. Higher is better. Bold , underlined , and italicized values mark the best, second-best, and third-best results.

Figure 5: T2IScoreScore performance. Ordering (

\mathrm{rank}_{m}

) versus separation (

\mathrm{sep}_{m}

) for (a) all SEGs, (b) Synth, (c) Nat, and (d) Real. Higher is better on both axes.

Table 2: Human Agreement on AGITA-Bench. Global and atomic agreement across Easy, Medium, and Hard difficulty tiers. Bold , underline , and italics denote first, second, and third best, respectively.

Table 3: Training-Stage Ablation. Performance of VisExMEM across successive training stages at 4B and 9B scales. Blue rows show gains relative to the indicated reference stage; bold denotes the best result within each model scale.

Global Atomic Method Easy Med. Hard Avg. Easy Med. Hard Avg. CLIPScore 19.6 -3.6 -8.6 2.5 5.4 -4.8 -0.2 0.1 BLIPv2Score 21.1 3.1 -0.8 7.8 3.3 2.1 4.5 3.3 AlignScore 24.5 3.2 6.2 11.3 5.8 -0.5 0.5 1.9 PickScore 33.5 29.1 34.7 32.4 11.1 -3.2 -2.0 2.0 ImageReward 38.8 21.9 16.8 25.8 – – – – HPSv2 17.5 14.6 13.5 15.2 14.8 5.8 8.0 9.5 EXPERT 50.6 33.8 45.5 43.3 – – – – FLEUR 52.2 31.5 43.7 42.5 – – – – VIEScore 51.2 31.2 42.6 41.7 – – – – VQAScore 54.6 31.7 43.0 43.1 52.7 32.3 41.1 42.0 Qwen3.5-4B (Hol.) 48.5 39.3 40.3 42.7 – – – – Qwen3.5-4B (Dec.) 50.5 37.1 44.1 43.9 41.2 31.7 39.1 37.3 Qwen3.5-9B (Hol.) 53.7 42.2 40.4 45.4 – – – – Qwen3.5-9B (Dec.) 59.0 39.5 41.2 46.6 40.6 30.8 38.2 36.5 GPT-5.6 Sol (Hol.) 72.1 54.6 60.6 62.4 – – – – GPT-5.6 Sol (Dec.) 76.9 60.7 62.5 66.7 70.1 55.6 68.3 64.7 VisExMEM-4B 56.8 43.1 50.6 50.2 46.6 34.1 43.2 41.3 VisExMEM-9B 63.3 45.2 55.6 54.7 56.2 33.0 45.5 44.9

Model Variant WG EqB TIFA Pick GenAI 4B Base (Hol.) 28.2 44.0 64.3 72.5 62.2 SFT 50.8 48.9 56.8 70.0 54.1 vs. Base (+22.6) (+4.9) (-7.5) (-2.5) (-8.1) + CF 61.6 56.2 64.9 77.2 64.9 vs. SFT (+10.8) (+7.3) (+8.1) (+7.2) (+10.8) + Verif.-GRPO 62.5 57.9 66.4 73.9 64.3 + Causal-GRPO 64.1 58.6 67.6 72.8 65.2 + GDPO 65.6 59.1 70.2 80.2 66.3 vs. CF (+4.0) (+2.9) (+5.3) (+3.0) (+1.4) 9B Base (Hol.) 40.4 35.7 50.1 74.2 50.3 SFT 61.9 58.9 59.8 61.8 57.2 vs. Base (+21.5) (+23.2) (+9.7) (-12.4) (+6.9) + CF 66.3 64.9 68.9 77.5 67.2 vs. SFT (+4.4) (+6.0) (+9.1) (+15.7) (+10.0) + Verif.-GRPO 67.4 67.2 71.5 78.0 67.9 + Causal-GRPO 68.5 66.7 71.0 79.1 67.1 + GDPO 68.9 70.1 73.1 85.6 69.4 vs. CF (+2.6) (+5.2) (+4.2) (+8.1) (+2.2)

Table 7

Figure 6: Human evaluation of explanation quality. (a) Correctness, visual grounding, clarity, and diagnostic usefulness. (b) Unsupported-content rate. (c) Localization accuracy and evidence sufficiency. Error bars show 95% confidence intervals.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Representative processed training examples. Real trainer-facing examples from the three VisExMEM training stages. Unused metadata are omitted and coordinates are rounded for readability.

Source	Supervision used	Primary skills
MS COCO	Captions and object boxes	Object, count
Visual Genome	Region descriptions, object boxes, attributes	Object, attribute, relation
VSR	Image–caption pairs with spatial-relation labels	Spatial relation
Flickr30k Entities	Phrase–entity–region correspondences	Object, grounding
Open Images + Localized Narratives	Grounded free-form narratives	Object, attribute, action, relation
DOCCI	Detailed connected and contrasting descriptions	Attribute, relation, negation

Appendix

Table 4: Public data sources used to construct the VisExMEM training corpus.

Task	Target format
Decomposition	{"parts": [...] } ; each part contains part_id , text , type , parent_part_id , head_part_id , and target_part_id .
Verification	{"decision": "yes"} or {"decision": "no"} .
Evidence selection	{"selected_evidence_ids": ["E1", ...]} .
Explanation	{"explanation": "..."} .

Appendix

Table 5: Task-specific target formats used during Stage 1 SFT. Candidate bounding boxes are inputs to evidence selection and are not prediction targets.

	SFT		Counterfactual		Evidence-aware GDPO
Hyperparameter	4B	9B	4B	9B	4B	9B
Training duration (hours)	4.4	4.6	2.7	3.4	40.9	43.3
GPUs	$4\times$ H100	$4\times$ H100	$4\times$ H100	$4\times$ H100	$2\times$ H100	$2\times$ H100
Per-device batch size	$4/2$	$4/2$	1	1	2	2
Gradient accumulation	8	8	8	8	2	2
Learning rate	$10^{-5}/7{\times}10^{-6}$	$10^{-5}/7{\times}10^{-6}$	$5{\times}10^{-7}$	$5{\times}10^{-7}$	$10^{-6}$	$10^{-6}$

Appendix

Table 6: Training hyperparameters for the 4B and 9B VisExMEM models. For SFT,

4/2

denotes per-device batch size

4

for the decomposition curriculum stage and

2

for the remaining stages; the two learning rates correspond to the first three task-specific curriculum stages and the final joint stage, respectively. Replay/update mixtures denote counterfactual:verification:decomposition for Stage 2 and semantic-GDPO:counterfactual:SFT for Stage 3.

	VisPlotBench		CulturalFrames		ProtoBias
Method	PAcc	Group	$\bm{\rho}$	PAcc wp	Animal		Demography		Object
					Acc.	$\bm{\kappa}$	Acc.	$\bm{\kappa}$	Acc.	$\bm{\kappa}$
CLIPScore	82.5	52.6	.145	60.5	25.0	.001	28.0	-.005	32.0	.058
ALIGNScore	79.1	45.3	.149	57.8	51.0	.033	34.0	-.064	32.0	.011
PickScore	77.1	39.5	.206	62.2	32.0	.001	34.0	-.014	31.0	.126
HPSv2	79.7	37.4	.219	61.5	25.0	.024	38.0	-.006	38.0	.125

Appendix

Table 7: Out-of-distribution generalization. Results on VisPlotBench, CulturalFrames, and the 600 human-annotated ProtoBias examples. VisPlotBench reports pairwise accuracy (PAcc) and Group score; CulturalFrames reports Spearman’s

\rho

and within-prompt pairwise accuracy (PAcc wp ); ProtoBias reports category-level accuracy (Acc.) and Cohen’s

\kappa

. Accuracy-based metrics are reported on a 0–100 scale; correlations and

\kappa

are reported on their native scale. Higher is better. Bold, underlined, and italicized values denote the best, second-best, and third-best results in each column, respectively.

Variant	Replay	WG	EqB	TIFA	Pick	GenAI
Base (Hol.)	–	40.4	35.7	50.1	74.2	50.3
+ SFT	–	61.9	58.9	59.8	61.8	57.2
+ CF	Yes	66.3	64.9	68.9	77.5	67.2
+ CF	No	67.9	66.2	71.1	81.1	67.3
$\Delta$	Yes $-$ No	(-1.6)	(-1.3)	(-2.2)	(-3.6)	(-0.1)
+ GDPO	Yes	68.9	70.1	73.1	85.6	69.4

Appendix

Table 8: Replay ablation for VisExMEM -9B. Both paths start from the same SFT checkpoint and either retain or disable replay throughout CF and GDPO.

\Delta

denotes Replay

-

No replay. Higher is better.

Aggregation	WG	EqBen	TIFA160	Pick-a-Pic	GenAI
Arithmetic mean	68.9	70.1	73.1	85.6	69.4
Geometric	68.4	69.6	71.6	84.4	68.7
Harmonic	67.3	60.3	69.6	81.5	64.4
Minimum	62.9	55.4	68.9	80.4	63.7

Appendix

Table 9: Aggregation ablation. Performance of VisExMEM -9B under alternative fixed aggregation rules. Higher is better.

Panel of Figure 8 — Figure 8: Prompt used for the holistic VLM-as-a-Judge setting. The model directly evaluates the complete image–caption pair and returns a single alignment score without atomic decomposition.

Panel of Figure 9 — Figure 9: Prompt used for the decomposed end-to-end VLM-as-a-Judge setting. The model performs atomic decomposition, evidence localization, claim-level scoring, short explanation generation, and final overall scoring in a single response.

Table 18

Panel of Figure 10 — Figure 10: Prompt used for AGITA-Bench atomic decomposition. GPT-5.6 Sol receives only the caption and decomposes it into typed visual semantic units without access to the image or making any image-based correctness judgment.

Panel of Figure 11 — Figure 11: AGITA-Bench annotation interface. Dashboard and sample-level interface used for human annotation.

Panel of Figure 12 — Figure 12: AGITA-Bench annotation guidelines. Complete annotator-facing guidelines.

Evaluation	Mean $\kappa$	Within $\pm 1$ (%)
Task A explanation quality	.59	89.5
Unsupported Content	.42	–
Task B localized evidence	.58	86.9

Appendix

Table 13: Inter-annotator agreement for explanation and evidence evaluation. Values show mean pairwise agreement. Ordinal ratings use quadratic-weighted Cohen’s

\kappa

, while Unsupported Content uses unweighted

\kappa

Panel of Figure 13 — Figure 13: Human explanation and evidence annotation guidelines. Complete instructions provided to annotators.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 988 papers

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
May 20, 2026Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu +5VLM Evaluation Visual Question Answering
University of Waterloo · Stanford University · NVIDIA
CBX-Bench: A Human-Aligned MLLM Council for Benchmarking Concept Bottleneck Model Explanations
Aug 15, 2026Yusuf Meric Karadag, Gulay Oklan, Seref Baris Cagliyan +2Explainability Evaluation Concept Bottleneck Models
Department of Computer Engineering, Middle East Technical University (METU) · Robotics & AI Center (ROMER), METU
Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3

May 20, 2026J/K move · Enter open · S save

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen+4

University of Waterloo · Stanford University · NVIDIA

VLM Evaluation Visual Question Answering Vision-Language Grounding Visually Grounded Reasoning Visual Grounding Multimodal Understanding

Open paper AskPDF arXiv

Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment. Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VISTAQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce GROVE, a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under GROVE, highlighting a substantial gap between answer accuracy and visual evidence alignment.

Abstract

VLM Evaluation

Explore similar work

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

CBX-Bench: A Human-Aligned MLLM Council for Benchmarking Concept Bottleneck Model Explanations

Multimodal Language Models as Text-to-Image Model Evaluators