Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models

Ravikumar Balakrishnan; Sanket Mendapara; Ankit Garg

cs.CVApr 14, 2026

Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models

Authors: Ravikumar Balakrishnan, Sanket Mendapara, Ankit Garg

Organizations: Cisco Systems

Abstract

We study typographic prompt injection attacks on vision-language models (VLMs), where adversarial text is rendered as images to bypass safety mechanisms, posing a growing threat as VLMs serve as the perceptual backbone of autonomous agents, from browser automation and computer-use systems to camera-equipped embodied agents. In practice, the attack surface is heterogeneous: adversarial text appears at varying font sizes and under diverse visual conditions, while the growing ecosystem of VLMs exhibits substantial variation in vulnerability, complicating defensive approaches. Evaluating 1,000 prompts from SALAD-Bench across four VLMs, namely, GPT-4o, Claude Sonnet 4.5, Mistral-Large-3, and Qwen3-VL-4B-Instruct under varying font sizes (6--28px) and visual transformations (rotation, blur, noise, contrast changes), we find: (1) font size significantly affects attack success rate (ASR), with very small fonts (6px) yielding near-zero ASR while mid-range fonts achieve peak effectiveness; (2) text attacks are more effective than image attacks for GPT-4o (36% vs 8%) and Claude (47% vs 22%), while Qwen3-VL and Mistral show comparable ASR across modalities; (3) text-image embedding distance from two multimodal embedding models (JinaCLIP and Qwen3-VL-Embedding) shows strong negative correlation with ASR across all four models (r = -0.71 to -0.93, p < 0.01); (4) heavy degradations increase embedding distance by 10--12% and reduce ASR by 34--96%, while rotation asymmetrically affects models (Mistral drops 50%, GPT-4o unchanged). These findings highlight that model-specific robustness patterns preclude one-size-fits-all defenses and offer empirical guidance for practitioners selecting VLM backbones for agentic systems operating in adversarial environments.

Figures & tables

Figure 1: Example typographic prompt injection attack using the TAP jailbreak method. (a) Harmful text prompt requesting attack planning guidance. (b) Prompt rendered as a 20px typographic image. (c) 30° rotation transformation increases L2 distance from text embedding.

Font	GPT-4o	Claude	Mistral	Qwen3-VL
6px	0.3	1.2	15.0	23.9
8px	3.5	10.5	67.9	40.8
10px	6.4	21.6	73.5	43.1
12px	7.0	18.8	75.5	42.7
14px	6.1	18.6	72.6	41.4
16px	6.0	17.4	73.2	44.1

Table 1: ASR (%) by font size and modality. Bold = peak image ASR.

n=1{,}000

curated samples.

Font	JinaCLIP		Qwen3-VL
	Mean	Std	Mean	Std
6px	1.265	0.032	0.976	0.097
8px	1.242	0.038	0.839	0.083
10px	1.192	0.042	0.829	0.080
12px	1.155	0.049	0.813	0.078
14px	1.126	0.047	0.813	0.079

Table 2: L2 embedding distance vs. font size for JinaCLIP and Qwen3-VL.

Embedding Model	Target VLM	Pearson $r$	$p$ -value
JinaCLIP	GPT-4o	$-0.795$	$0.002^{***}$
	Claude	$-0.725$	$0.008^{***}$
	Mistral	$-0.714$	$0.009^{***}$
	Qwen3-VL	$-0.799$	$0.002^{***}$
Qwen3-VL-Emb	GPT-4o	$-0.843$	$0.001^{***}$
	Claude	$-0.812$	$0.001^{***}$

Table 3: Correlation between L2 distance and ASR across font sizes.

Figure 2: Font size analysis. (a) JinaCLIP L2 distance decreases by 14% from 6px to 28px. (b) ASR increases with font size, plateauing at 10–12px; dotted lines show text baseline ASR. (c) Scatter plot showing negative correlation between L2 distance and ASR across all models. (d) Pearson correlation coefficients (

{}^{***}p<0.01

			Text ASR (%)				Image ASR (%)
Method	$N$	L2 $d$	GPT	Claude	Mistral	Qwen	GPT	Claude	Mistral	Qwen
GPTFuzz	62	1.07 $\pm$ .04	50.0	24.2	67.7	33.9	0.0	12.9	58.1	38.7
GCG	229	1.09 $\pm$ .04	76.4	59.4	95.6	71.6	4.9	11.4	74.7	72.9
TAP	125	1.12 $\pm$ .04	71.2	51.2	84.0	58.4	38.4	19.2	80.0	56.8
Jailbreak	582	1.13 $\pm$ .04	10.1	42.8	82.8	39.3	2.2	21.0	73.7	37.5

Table 4: Text and image ASR (%) by attack method with JinaCLIP L2 distance (20px font, sorted by L2).

Transformation	JinaCLIP $d$	GPT-4o	Claude	Mistral	Qwen3-VL
Gaussian noise	1.078	5.1	16.6	74.4	41.8
Gray background	1.090	8.1	17.0	76.0	44.0
Low contrast	1.106	7.6	14.9	75.4	41.7
Baseline (none)	1.110	6.0	18.0	74.5	43.4
Inverted colors	1.144	5.4	17.4	74.0	43.5
Rotation 30°	1.150	6.1	8.3	37.7	24.9

Table 5: ASR (%) and JinaCLIP L2 distance under visual transformations (20px font).

Embedding Model	Target VLM	Pearson $r$	$p$ -value
JinaCLIP	GPT-4o	$-0.829$	$0.003^{***}$
	Claude	$-0.893$	$0.001^{***}$
	Mistral	$-0.805$	$0.005^{***}$
	Qwen3-VL	$-0.717$	$0.020^{**}$
Qwen3-VL-Emb	GPT-4o	$-0.628$	$0.052$
	Claude	$-0.880$	$0.001^{***}$

Table 6: Correlation between L2 distance and ASR across visual transformations.

Figure 3: Transformation analysis. (a) JinaCLIP L2 distance by transformation (sorted); dashed line = baseline. (b) ASR by transformation for each VLM. (c) Scatter plot showing strong negative correlation between L2 distance and ASR. (d) Pearson correlation coefficients (

{}^{**}p<0.05

{}^{***}p<0.01

Explore similar work

Apr 28, 2026

One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbations

Typographic prompt injection exploits vision language models' (VLMs) ability to read text rendered in images, posing a growing threat as VLMs power autonomous agents. Prior work typically focus on maximizing attack success rate (ASR) but does not explain why certain renderings bypass safety alignment. We make two contributions. First, an empirical study across four VLMs including GPT-4o and Claude, twelve font sizes, and ten transformations reveals that multimodal embedding distance strongly predicts ASR (

r{=}{-}0.71

{-}0.93

p{<}0.01

), providing an interpretable, model agnostic proxy. Since embedding distance predicts ASR, reducing it should improve attack success, but the relationship is mediated by two factors: perceptual readability (whether the VLM can parse the text) and safety alignment (whether it refuses to comply). Second, we use this as a red teaming tool: we directly maximize image text embedding similarity under bounded

\ell_\infty

perturbations via CWA-SSA across four surrogate embedding models, stress testing both factors without access to the target model. Experiments across five degradation settings on GPT-4o, Claude Sonnet 4.5, Mistral-Large-3, and Qwen3-VL confirm that optimization recovers readability and reduces safety aligned refusals as two co-occurring effects, with the dominant mechanism depending on the model's safety filter strength and the degree of visual degradation.

Ravikumar Balakrishnan, Sanket Mendapara

Cisco Systems

VLM Evaluation VLM Robustness Adversarial Attacks on VLMs Adversarial Attacks

Ask PDF

Aug 13, 2026

QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing

Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to internal model components, limiting their applicability to modern closed-source VLMs. In this paper, we propose QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. QuISE first identifies text regions likely to affect the current query through influence-aware text localization. QuISE then replaces these regions with two semantically distinct replacement texts that are irrelevant to both the query and the image. The final answer is determined by answer consistency across the edited images. Extensive experiments on three typographic-attack benchmarks, four attack settings, and four VLMs show that QuISE consistently improves defended accuracy. QuISE achieves a recovery rate of 67.9-75.0% with a harm rate of 0.5-1.1%.

Shubin Lu, Jiaqi Yin, Yihao Huang

School of Software, Northwestern Polytechnical University, Xi’an, China · Software Engineering Institute, East China Normal University, Shanghai, China

VLM Robustness Adversarial Attacks on VLMs Adversarial Robustness

Ask PDF

Sep 25, 2026

Sorry Robot, Happy Human: Vision-Language Models Read Only One of Two Legible Typographic Layers

Vision-language models (VLMs), despite their success in optical character recognition (OCR) tasks, are vulnerable to typographic attacks and have a fragile structure for images with multiple text layers. In this study, the DecoyBench dataset was created using the Decoy Font method. The dataset consists of 300 images, each containing text with sharp contour lines superimposed on another text with soft shading. Six recent closed-source models from three different model families were evaluated using this dataset under two different prompting conditions (naive and guided) and at two different resolutions (

512\times512

and

64\times64

). A validation study showed that human participants could read both text layers with high accuracy. In contrast, the models, with most variants and both prompting methods, read the contour text with near-human accuracy at high resolution, but almost never fully extracted the shading text. At low resolution, the contour text could not be read by either the models or humans, while the shading text could be extracted with high accuracy. The findings indicate that the evaluated VLMs exhibit a consistent behavioral limitation when processing typographic structures containing multiple spatial frequency layers.

Mert İncidelen, Yamen Kashkash, Asya Berker +1

Department of Artificial Intelligence and Data Engineering, Fırat University, Elazığ, Türkiye · Department of Software Engineering, Fırat University, Elazığ, Türkiye

Vision-Language Models VLM Robustness VLM Evaluation

Ask PDF