VLTest: Finding Joint Image-Text Failures in Vision-Language Models

Bin Duan; Jintao Lin; Guowei Yang

cs.SEOct 22, 2026

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

Authors: Bin Duan, Jintao Lin, Guowei Yang

Organizations: The University of Queensland, Brisbane, Australia

Abstract

Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.

Figures & tables

Approach	Test inputs	Mutated	Expected answer of generated inputs	Matched four-cell confirmation
Chen et al. ( Chen et al., 2023 )	predefined corruptions of dataset inputs	I or T	assumed unchanged	n/a
NaturalBench ( Li et al., 2024a )	curated natural adversarial pairs	–	human-annotated	n/a
MultiMon ( Tong et al., 2023 )	mined from existing corpora	–	summarized post hoc	n/a
MetaVQA ( Yuan et al., 2021 )	relation-driven transformations of one modality	I or T	defined by the relation	n/a
MetaRA ( Xu et al., 2026 )	relation-driven transformations	I and T	defined by the relation	no
Co-Attack ( Zhang et al., 2022 ) , SGA ( Lu et al., 2023a )	white-box optimization with cross-modal guidance	I+T	assumed under perturbation bounds	no

Table 1. Comparison of VLTest with representative approaches (I = image, T = text). In the last column, n/a means that the approach does not test joint inputs, and no means that it does, without matched single-modality controls.

Method	Image side	Text side	$\phi$	Joint inputs
SINVAD-VQ	latent search (scores)	–	yes	–
VLTest -V	codebook replacement	–	yes	–
TextFooler	–	synonym search (scores)	yes	–
VLTest -T	–	rewrite–expand–prune	yes	–
RND	VLTest -V	VLTest -T	no	random
CRND	VLTest -V	VLTest -T	yes	random

Table 2. Compared methods.

\phi

: relation filters applied. The first four methods test one modality only.

							Confirmation rate (%)
Task	Model	Candidates	Invalid / abstain	Answer changed	Output correct	Confirmed failure	image- only	text- only	joint input
ITM	BLIP	22,776	308	3,399	0	19,069	91.6	86.8	82.0
ITM	ViLT	25,253	356	3,608	0	21,289	90.7	85.4	83.0
VR	BEiT-3	19,164	514	3,686	0	14,964	87.0	77.0	76.7
VR	ViLT	17,179	482	3,524	0	13,173	86.3	77.1	75.0
VQA	PaliGemma	33,698	724	3,953	1,091	27,930	89.3	83.2	81.7

Table 3. RQ1: oracle outcomes of the candidates of VLTest (SF), and the confirmation rate per input type.

	CF			AJF
Method	ITM	VR	VQA	ITM	VR	VQA
SINVAD-VQ	90.0 [68.3,98.8]	85.0 [62.1,96.8]	90.0 [68.3,98.8]	–	–	–
VLTest -V	92.5 [79.6,98.4]	87.5 [73.2,95.8]	92.5 [79.6,98.4]	–	–	–
TextFooler	85.0 [62.1,96.8]	85.0 [62.1,96.8]	90.0 [68.3,98.8]	–	–	–
VLTest -T	90.0 [76.3,97.2]	87.5 [73.2,95.8]	92.5 [79.6,98.4]	–	–	–
RND	–	–	–	85.0 [62.1,96.8]	80.0 [56.3,94.3]	85.0 [62.1,96.8]

Table 4. RQ1: oracle precision in percent with 95% intervals, estimated from samples drawn separately for each method and pooled over the two models of each task. CF: confirmed failures; AJF: attributed joint failures. CF precision is shown for the single-modality methods and VLTest (SF).

	ITM		VR		VQA
Method	BLIP	ViLT	BEiT-3	ViLT	PaliGemma	BEiT-3	SeedCov (%)	Budget used (%)
SINVAD-VQ	24.1 ∗	26.8 ∗	21.6 ∗	18.9 ∗	42.7 ∗	37.5 ∗	63.4	88.9
VLTest -V	50.3	56.4	42.7	37.2	78.5	71.4	75.7	100.0
TextFooler	19.4 ∗	21.8 ∗	15.8 ∗	13.9 ∗	30.6 ∗	27.5 ∗	56.7	81.7
VLTest -T	46.2	50.1	38.4	34.1	74.3	68.0	71.0	100.0

Table 5. RQ2:

\mathrm{CF@1k}

per task–model pair; SeedCov and budget used in percent, averaged over the six pairs. ∗ : significant difference to VLTest -V or VLTest -T (5% level, Holm).

	ITM		VR		VQA
Method	BLIP	ViLT	BEiT-3	ViLT	PaliGemma	BEiT-3	Budget used (%)	Budget per AJF	VarCov (I / T)	SeedCov (%)	Tested stable pairs (%)
RND	65.9	72.3	48.4	41.7	101.6	89.4	100.0	15.8	34.6 / 33.1	84.7	73.5
CRND	116.8	117.9	86.1	75.1	168.4	149.1	100.0	9.1	83.1 / 81.3	92.7	85.2
SR	133.1	146.0	99.2	87.8	192.4	167.1	100.0	7.8	84.6 / 82.2	93.7	100.0
EXT	105.3	118.1	78.6	68.7	155.9	137.2	100.0	9.8	32.1 / 29.8	92.9	99.2
MetaRA-c	–	–	–	–	69.7	61.9	51.4	15.3	18.2 / 16.3	90.5	68.9

Table 6. RQ2:

\mathrm{AJF@1k}

per task–model pair; budget used, budget per AJF, VarCov (image / text), SeedCov, and the share of joint calls on stable pairs, averaged over the pairs a method runs on.

Task	Rejected by the joint input (%)	Further rejected by the controls (%)	Attributed joint failures (%)
ITM	17.5	2.5	80.0
VR	24.1	3.7	72.2
VQA	18.8	2.6	78.6
All	19.7	2.8	77.5

Table 7. RQ3: joint candidates of VLTest (SF) by the first check that fails, starting with the joint input, in percent of the joint candidates, pooled over the two models of each task.

Task	Model	Attributed joint failures	Interaction (%)	Median $\Delta_{\mathrm{int}}$ [IQR]
ITM	BLIP	13,716	16.4	0.121 [0.051, 0.238]
ITM	ViLT	15,163	14.7	0.109 [0.043, 0.219]
VR	BEiT-3	10,368	21.3	0.154 [0.062, 0.298]
VR	ViLT	9,139	22.6	0.163 [0.038, 0.312]
VQA	PaliGemma	19,683	13.8	0.097 [0.038, 0.194]
VQA	BEiT-3	17,321	15.1	0.113 [0.045, 0.226]

Table 8. RQ3: attributed joint failures of VLTest (SF) classified by the margin analysis. Interaction: share of interaction failures;

\Delta_{\mathrm{int}}

: median with interquartile range. Both are computed over the score-classifiable failures, which exclude the 622 failures of PaliGemma with

\eta(x,t)>0

(84,768 in all).

Failure type	ITM	VR	VQA	Total
Attribute or entity binding	24	9	19	52
Spatial or relational composition	17	15	14	46
Quantifier, count, or scope	4	26	4	34
Lexical grounding or co-reference	11	6	12	29
Fine-grained category or part	4	4	11	19
Total	60	60	60	180

Table 9. RQ3: types of attributed joint failures in a stratified random sample of 180, 60 per task.

Figure 2. RQ3: two attributed joint failures, each with the reconstructed seed $(\hat{x}_{0},t_{0})$ , the image-only input, the text-only input, and the joint input. Changed words are in bold. Model outputs from left to right: match, match, match, no match (ITM); brown, brown, brown, yellow (VQA).

Design choice	Setting	Conf. rate (%)	$\mathrm{CF@1k}$	$\mathrm{AJF@1k}$
(a) relation filters	none	71.8	46.2	108.7
	$F_{\mathrm{txt}}$ only	77.6	51.3	123.5
	$F_{\mathrm{img}}$ only	76.4	50.6	120.2
	$F_{\mathrm{pair}}$ only	71.9	52.8	131.4
	all (default)	81.6	55.1	145.9
(b) pool size	$n=250$	80.7	51.8	132.6

Table 10. RQ4: design choices of VLTest (SF), averaged over BLIP (ITM), BEiT-3 (VR), and PaliGemma (VQA).

Task	Model	$\|S\|$	Non-det. floor	$\mathrm{AJF@1k}$	$\Delta$ vs. SR	$\Delta$ vs. CRND
ITM	Qwen3-VL-8B	100	–	169.0	18.2	33.7
ITM	Qwen3-VL-32B	100	–	135.6	14.4	29.8
ITM	GPT-4o mini	43	4.7%	139.5	5.6	33.0
ITM	Gemini 2.5 Flash	41	2.4%	127.3	20.0	27.8
VR	Qwen3-VL-8B	100	–	148.2	21.0	28.2
VR	Qwen3-VL-32B	100	–	111.8	13.2	25.0

Table 11. RQ4: VLTest (SF) on the general-purpose models.

|S|

: admitted seeds.

\Delta

vs. SR and

\Delta

vs. CRND: differences in

\mathrm{AJF@1k}

. Non-det. floor: share of admitted seeds with observed output variation.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Search-Based Testing of Modality Sensitivity in Vision Language Models
Oct 23, 2026Oliver Weißl, Eduardo Breitembach, Andrea StoccoVLM Evaluation VLM Robustness
Technical University of Munich, Garching near Munich, Germany · fortiss GmbH, Munich, Germany
FuzzingRL: Reinforcement Fuzz-Testing for Revealing VLM Failures
Feb 17, 2026Jiajun Xu, Jiageng Mao, Ang Qi +5VLM Robustness Adversarial Attacks on VLMs
University of Southern California · Toyota Research Institute
Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models
Jul 6, 2026Khang Nhat Hoang Vo, Artem Vazhentsev, Artem Shelmanov +2VLM Evaluation Vision-Language Models
1MBZUAI · 2The University of Melbourne

Oct 23, 2026J/K move · Enter open · S save

Search-Based Testing of Modality Sensitivity in Vision Language Models

Oliver Weißl, Eduardo Breitembach, Andrea Stocco

Technical University of Munich, Garching near Munich, Germany · fortiss GmbH, Munich, Germany

VLM Evaluation VLM Robustness

Open paper AskPDF arXiv

Vision-language models (VLMs) process images and text jointly, yet robustness evaluations often perturb these inputs separately or assess only predefined corruption levels. Such evaluations provide limited evidence of which modality is more susceptible to perturbations and whether smaller changes across both modalities can jointly expose failures. We present MoST, a black-box testing framework that characterizes modality sensitivity by comparing image-only, text-only, and joint perturbations. MoST uses multi-objective search to select and combine perturbations and adjust their severity, seeking prediction degradation while limiting changes to images and prompts. We evaluate MoST on four VLMs performing object localization on ImageNet and driving images. The models exhibit distinct sensitivities: textual perturbations dominate for some, whereas others are susceptible across both modalities. In several configurations, joint perturbations match or exceed the effectiveness of single-modality perturbations with smaller modifications to each input. In a human study, 60-76% of assessments across models judge the perturbed inputs sufficiently understandable to perform the task, and human localization is more accurate than that of the corresponding VLMs. These results show that evaluating modalities separately can overlook failure-inducing combinations and that modality sensitivity provides a more detailed account of VLM robustness than aggregate failure rates alone.

Abstract

VLM Evaluation

Explore similar work

Search-Based Testing of Modality Sensitivity in Vision Language Models

FuzzingRL: Reinforcement Fuzz-Testing for Revealing VLM Failures

Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models