DiaVLo: Diagnosing Behaviours of Vision-Language Models

Lorenzo Corti; Jie Yang

cs.CLSep 18, 2026

DiaVLo: Diagnosing Behaviours of Vision-Language Models

Authors: Lorenzo Corti, Jie Yang

Organizations: Delft University of Technology

Abstract

Vision-language models (VLMs) rely on storing and transferring appropriate information across their sub-components. Verifying that the VLMs exhibit desired behaviours, while avoiding harmful ones, is central to their reliable deployment. Yet, methods that identify VLM behaviours remain scarce. We present DiaVLo, a diagnostic framework that leverages human curation and VLMs' generation capabilities to construct specifications of desired and observed VLM behaviours, surfacing potential misalignments. Beyond this, DiaVLo also provides causal estimates to identify the most influential concepts steering VLM behaviours. We evaluate DiaVLo on several open-source VLMs under both classification and generation conditions. Our experiments show that DiaVLo produces behaviour labels that correlate with model performance and provide context for measured performance. DiaVLo surfaced behaviours that are clearly aligned and misaligned, alongside patterns in how VLMs perceive, organise, and prioritise concepts.

Figures & tables

Figure 1: With DiaVLo , we frame VLM diagnosis as an addition to benchmarking, enabling the definition and verification of observed model behaviours with respect to specifications of desired behaviours.

Figure 2: In DiaVLo , two parallel processes enable the classification of VLM behaviours. (Top) Scene graph generation and human curation produce SK s that are relevant for the visual and language inputs. (Bottom) VLMs provide RK s by verbalising rationales for their responses. We also compute concept-level causal estimates for deeper insights. (Middle) Finally, DiaVLo classifies model behaviours based on lexical and semantic matching.

	LLaVa-Bench	MMBench	SEED-Bench 2	VQA v2
Initial	580 (-)	1442 (-)	1783 (-)	1798 (-)
Validation	300 (51.7%)	839 (58.2%)	1042 (58.4%)	1051 (58.5%)
Expansion	516 (43.4%)	1024 (42.1%)	1233 (38.7%)	1302 (46.0%)
Expert	546 (11.0%)	1094 (5.9%)	1301 (8.4%)	1391 (7.8%)
M ${}_{\texttt{SK}}{}$ (IQR)	9 (5.75, 11)	7 (5, 10.5)	9 (5, 12)	8.5 (6,12)

Table 1: SK s collected and percentages of relevant SK s (Validation), of new SK s (Expansion), or of expert edits (including median (M

{}_{\texttt{SK}}{}

) and inter-quartile range (IQR)).

	LLaVa-Bench		MMBench		SEED-Bench 2		VQA v2
	N s (N ${}_{\texttt{RK}{}}$ )	M ${}_{\texttt{RK}{}}$ (IQR)	N s (N ${}_{\texttt{RK}{}}$ )	M ${}_{\texttt{RK}{}}$ (IQR)	N s (N ${}_{\texttt{RK}{}}$ )	M ${}_{\texttt{RK}{}}$ (IQR)	N s (N ${}_{\texttt{RK}{}}$ )	M ${}_{\texttt{RK}{}}$ (IQR)
InternVL2	58 (565)	10 (7, 12)	139 (1287)	9 (7, 11)	150 (1254)	8 (6, 10)	150 (1014)	7 (4, 9)
LLaVa-1.6	44 (279)	7 (3, 8.25)	133 (823)	6 (4, 8)	129 (853)	6 (4, 10)	142 (939)	6 (4, 9)
Qwen2.5-VL	56 (504)	9 (6, 12)	139 (601)	4 (3, 5)	150 (554)	3 (2, 4)	150 (336)	2 (2, 2)
ShareGPT4V	49 (279)	5 (3, 8)	135 (645)	4 (2, 7)	109 (241)	1 (1, 2)	21 (46)	1 (1, 1)

Table 2: RK statistics shown across models and datasets: raw number of RK extracted (N), median (M), and inter-quartile range (IQR). We highlight model-dataset pairs based on the number of RK successfully extracted:

>90\%

in green,

70-90\%

in yellow, and

<70\%

in red. We include results from ShareGPT4V on VQA v2 for transparency.

Panel of Figure 3 — Figure 3: Behaviour thresholds’ sensitivity to $\tau_{a}$ (top) and $\tau_{e}$ (bottom). $\tau_{d}$ is set afterwards, based on $\tau_{e}$ . Additional visualisations are included in subsection E.2 .

Figure 4: Identified Model Behaviours.

VLM	LLaVa-Bench		MMBench		SEED-Bench 2		VQA v2
VLM	A	MI	A	MI	A	MI	A	MI
InternVL2	0.673	3.367	0.796	3.200	0.787	0.379	0.667	0.476
LLaVa-1.6	0.657	2.389	0.792	2.512	0.667	0.438	0.585	0.481
Qwen2.5-VL	0.670	3.008	0.743	1.561	0.833	0.188	0.700	0.147
ShareGPT4V	0.687	2.058	0.705	2.116	0.661	0.292	0.810	0.558 †

Table 3: Measured accuracy (A) vs Mutual Information MI

(A,S_{c})

\dagger

: Very small sample size: see Table 2 .

Panel of Figure 5 — Figure 5: Example distributions of causal effects. Full comparisons for all VLMs and dataset combinations can be found in subsection E.6 .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Form to edit triple data with the auto-complete suggestions visible (highlighted).

Panel of Figure 9 — Figure 9: Prompt templates for getting model responses. Model-specific changes are underlined .

Panel of Figure 10 — Figure 10: Prompt templates for generating model rationales. Same for all models.

Panel of Figure 11 — Figure 11: Prompt templates for structuring VLM rationales. Changes are mostly related to how instructions are organised and their clarity.

Dataset	$J_{concepts}$	$J_{relations}$
LLaVa-Bench	0.81	0.82
MMBench – Image Scene	0.41	0.91
MMBench – Image Topic	0.99	0.90
SEED-Bench 2 – Scene Understanding	0.85	0.88
SEED-Bench 2 – Visual Reasoning	0.99	0.94
VQAv2 – How many people are…	0.47	0.89

Appendix

Table 4: Average Jaccard distance, for each dataset, measured between the sets of unique concepts and relationships obtained with and without USE_VISION enabled.

Dataset	Concept Detected		Relations Detected
	w/ USE_VISION	w/o USE_VISION	w/ USE_VISION	w/o USE_VISION
LLaVa-Bench	746	689 ( $\downarrow$ 7.64%)	720	720 (-)
MMBench – Image Scene	7905	6393 ( $\downarrow$ 19.13%)	7650	7650 (-)
MMBench – Image Topic	1980	179 ( $\downarrow$ 90.96%)	1920	1920 (-)
SEED-Bench 2 – Scene Understanding	100903	71883 ( $\downarrow$ 28.76%)	94740	94740 (-)
SEED-Bench 2 – Visual Reasoning	10715	1314 ( $\downarrow$ 87.74%)	9930	9930 (-)

Appendix

Table 5: IETrans concepts and relationships extracted when the vision signal is used (w/ USE_VISION ) compared to when it is not (w/o USE_VISION ).

Panel of Figure 12 — Figure 12: Proportions of samples classified as Aligned, Expanded, or Divergent, and dominant behaviour type as $\tau_{a}$ and $\tau_{e}$ vary.

Figure 13: Results from

\tau_{\textnormal{OWLv2}}

sweep.

Figure 14: Results from

\tau_{\textnormal{nms}}

sweep.

VLM	Dataset	Initial RK	Exact Matching	OWLv2 Matches	Semantic Fallback	Matched	Not Matched
InternVL2	LLaVa-Bench	605	0	385	220	605	0
	MMBench	1341	3	1000	338	1341	0
	SEED-Bench 2	1388	3	1030	355	1388	0
	VQA v2	1186	15	1009	162	1186	0
LLaVa-1.6	LLaVa-Bench	421	0	235	186	421	0
	MMBench	1255	39	828	388	1255	0

Appendix

Table 6: Intermediate RK - SK matching counts and percentages.

Model	LLaVa-Bench		MMBench		SEED-Bench 2		VQA v2
Model	A	MI	A	MI	A	MI	A	MI
InternVL2	0.673	3.367 [3.158, 3.589]	0.796	3.200 [3.053, 3.350]	0.787	0.379 [0.325, 0.437]	0.667	0.476 [0.433, 0.526]
LLaVa-1.6	0.657	2.389 [2.052, 2.711]	0.792	2.512 [2.333, 2.699]	0.667	0.438 [0.385, 0.497]	0.585	0.481 [0.437, 0.531]
Qwen2.5-VL	0.670	3.008 [2.764, 3.270]	0.743	1.561 [1.331, 1.792]	0.833	0.188 [0.115, 0.275]	0.700	0.147 [0.082, 0.230]
ShareGPT4V	0.687	2.058 [1.737, 2.405]	0.705	2.116 [1.932, 2.318]	0.661	0.292 [0.198, 0.404]	0.810	0.558 [0.139, 1.074]

Appendix

Table 7: VLMs accuracy (A) on the four datasets and Mutual Information (MI) measures between accuracy and model behaviour types. In parentheses, we report the bootstrapped 95% confidence intervals for the MI measures.

Panel of Figure 15 — Figure 15: Bootstrapped Mutual Information measures InternVL2 ( Chen et al., 2024 ) .

Panel of Figure 16 — Figure 16: Bootstrapped Mutual Information measures LLaVa-1.6 ( Liu et al., 2024b ) .

Panel of Figure 17 — Figure 17: Bootstrapped Mutual Information measures Qwen2.5-VL ( Bai et al., 2025 ) .

Panel of Figure 18 — Figure 18: Bootstrapped Mutual Information measures ShareGPT4V ( Chen et al., 2023 ) .

Figure 19: Example of aligned model behaviour from LLaVa-1.6 on MMBench.

Figure 20: Example of aligned model behaviour from LLaVa-1.6 on VQA v2.

Figure 21: Example of expanded model behaviour from Qwen2.5-VL on LLaVa-Bench.

Figure 22: Example of expanded model behaviour from LLaVa-1.6 on SEED-Bench-2.

Figure 23: Example of expanded model behaviour from LLaVa-1.6 on MMBench.

Figure 24: Example of expanded model behaviour from InternVL2 on VQA v2.

Figure 25: Example of divergent model behaviour from InternVL2 on VQA v2.

Figure 26: Example of divergent model behaviour from ShareGPT4V on LLaVa-Bench.

Panel of Figure 27 — Figure 27: Model-pairwise distributions of estimated causal effects on LLaVa-Bench ( Liu et al., 2023a ) .

Panel of Figure 28 — Figure 28: Model-pairwise distributions of estimated causal effects on MMBench ( Liu et al., 2024c ) .

Panel of Figure 29 — Figure 29: Model-pairwise distributions of estimated causal effects on SEED-Bench-2 ( Li et al., 2024a ) .

Panel of Figure 30 — Figure 30: Model-pairwise distributions of estimated causal effects on VQA v2 ( Goyal et al., 2017 ) .

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 957 papers

Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models
Mar 16, 2026Lexiang Xiong, Qi Li, Jingwen Ye +1Hallucination Detection in VLMs VLM Interpretability
National University of Singapore · Monash University
VISTA: Auditing Semantic Divergence in Vision-Language Models
Jul 3, 2026Junchi Liao, Jiawen Deng, Fuji RenVLM Evaluation Vision-Language Models
University of Electronic Science and Technology of China
To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
Mar 19, 2026Rui Hong, Shuxue QuanSelective Prediction for VLMs Sycophancy in VLMs
George Mason University, Fairfax, VA, USA · Independent Researcher

Mar 16, 2026J/K move · Enter open · S save

Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models

Lexiang Xiong, Qi Li, Jingwen Ye, Xinchao Wang

National University of Singapore · Monash University

Hallucination Detection in VLMs VLM Interpretability Anomaly Detection

Open paper AskPDF arXiv

Vision-Language Models (VLMs) frequently "hallucinate" - generate plausible yet factually incorrect statements - posing a critical barrier to their trustworthy deployment. In this work, we propose a new paradigm for diagnosing hallucinations, recasting them from static output errors into dynamic pathologies of a model's computational cognition. Our framework is grounded in a normative principle of computational rationality, allowing us to model a VLM's generation as a dynamic cognitive trajectory. We design a suite of information-theoretic probes that project this trajectory onto an interpretable, low-dimensional Cognitive State Space. Our central discovery is a governing principle we term the geometric-information duality: a cognitive trajectory's geometric abnormality within this space is fundamentally equivalent to its high information-theoretic surprisal. Hallucination detection is counts as a geometric anomaly detection problem. Evaluated across diverse settings - from rigorous binary QA (POPE) and comprehensive reasoning (MME) to unconstrained open-ended captioning (MS-COCO) - our framework achieves state-of-the-art performance. Crucially, it operates with high efficiency under weak supervision and remains highly robust even when calibration data is heavily contaminated. This approach enables a causal attribution of failures, mapping observable errors to distinct pathological states: perceptual instability (measured by Perceptual Entropy), logical-causal failure (measured by Inferential Conflict), and decisional ambiguity (measured by Decision Entropy). Ultimately, this opens a path toward building AI systems whose reasoning is transparent, auditable, and diagnosable by design.

Abstract

VLM Evaluation

Explore similar work

Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models

VISTA: Auditing Semantic Divergence in Vision-Language Models

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs