When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models

Chengyin Hu; Xuemeng Sun; Jiaju Han; Qike Zhang; Xiang Chen; Xin Wang; Yiwei Wei; Jiahua Long

cs.CVMar 29, 2026

When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models

Authors: Chengyin Hu, Xuemeng Sun, Jiaju Han, Qike Zhang, Xiang Chen, Xin Wang, Yiwei Wei, Jiahua Long

Abstract

Visual-Language Models (VLMs) have demonstrated exceptional cross-modal understanding across various tasks, including zero-shot classification, image captioning, and visual question answering. However, their robustness to physically plausible non-rigid deformations-such as wrinkles on flexible surfaces-remains poorly understood. In this work, we propose a parametric structural perturbation method inspired by the mechanics of three-dimensional fabric wrinkles. Specifically, our method generates photorealistic non-rigid perturbations by constructing multi-scale wrinkle fields and integrating displacement field distortion with surface-consistent appearance variations. To achieve an optimal balance between visual naturalness and adversarial effectiveness, we design a hierarchical fitness function in a low-dimensional parameter space and employ an optimization-based search strategy. We evaluate our approach using a two-stage framework: perturbations are first optimized on a zero-shot classification proxy task and subsequently assessed for transferability on generative tasks. Experimental results demonstrate that our method significantly degrades the performance of various state-of-the-art VLMs, consistently outperforming baselines in both image captioning and visual question-answering tasks.

Figures & tables

Figure 1. Overview of the Proposed Wrinkle-Based Structural Attack Framework. The framework models wrinkle-like deformation and uses a genetic algorithm to optimize parameters for generating adversarial examples across vision-language tasks. Overview of the proposed wrinkle-based structural attack framework.

Method	OpenCLIP ViT-B/16		Meta-CLIP ViT-L/14		EVA-CLIP ViT-G/14		OpenAI CLIP ViT-L/14
Method	ACC (%)	GPT-4o	ACC (%)	GPT-4o	ACC (%)	GPT-4o	ACC (%)	GPT-4o
Clean	97	3.65	98	3.65	98	3.65	93	3.65
Natural Light Attack ( Hsiao et al., 2024 )	94 ( $\downarrow 3$ )	2.32 ( $\downarrow 1.33$ )	97 ( $\downarrow 1$ )	2.14 ( $\downarrow 1.51$ )	97 ( $\downarrow 1$ )	2.16 ( $\downarrow 1.49$ )	93 (–)	2.07 ( $\downarrow 1.58$ )
Shadow Attack ( Zhong et al., 2022 )	84 ( $\downarrow 13$ )	2.11 ( $\downarrow 1.54$ )	82 ( $\downarrow 16$ )	1.91 ( $\downarrow 1.74$ )	95 ( $\downarrow 3$ )	2.03 ( $\downarrow 1.62$ )	79 ( $\downarrow 14$ )	1.78 ( $\downarrow 1.87$ )
ITA ( Liu et al., 2025 )	46 ( $\downarrow 51$ )	2.11 ( $\downarrow 1.54$ )	64 ( $\downarrow 34$ )	2.15 ( $\downarrow 1.50$ )	84 ( $\downarrow 14$ )	2.17 ( $\downarrow 1.48$ )	51 ( $\downarrow 42$ )	2.15 ( $\downarrow 1.50$ )
Ours (random)	52 ( $\downarrow 45$ )	2.71 ( $\downarrow 0.94$ )	68 ( $\downarrow 30$ )	2.47 ( $\downarrow 1.18$ )	75 ( $\downarrow 23$ )	2.89 ( $\downarrow 0.76$ )	53 ( $\downarrow 40$ )	2.64 ( $\downarrow 1.01$ )

Table 1. Performance of VLMs on Zero-shot Classification under Wrinkle-like Structural Perturbations Generated from the COCO Dataset. We report both classification accuracy (ACC) and GPT-4o-based visual naturalness scores. Lower ACC indicates stronger attack effectiveness, while higher GPT-4o scores indicate better visual naturalness. The best results are shown in bold, and the second-best results are underlined.

Panel of Figure 2 — Figure 2. Qualitative Comparison Between Clean and Adversarial Examples. (a) Representative clean and adversarial image pairs, where the proposed perturbation remains visually subtle but causes erroneous predictions on adversarial samples. (b) Comparison of attention visualizations between clean and adversarial images, showing that the perturbation redirects model attention away from semantically meaningful regions.

Image Encoder	Models	Params	Clean	Natural Light Attack ( Hsiao et al., 2024 )	Shadow Attack ( Zhong et al., 2022 )	ITA ( Liu et al., 2025 )	Ours (random)	Ours (best)
OpenAI CLIP ViT-L/14	LLaVA-1.5	7B	78.60	77.00 ( $\downarrow 1.63$ )	74.60 ( $\downarrow 4.00$ )	63.73 ( $\downarrow 14.87$ )	56.58 ( $\downarrow 22.02$ )	54.51 ( $\downarrow 24.09$ )
	LLaVA-1.6	7B	72.10	71.70 ( $\downarrow 0.39$ )	71.17 ( $\downarrow 0.93$ )	61.60 ( $\downarrow 10.51$ )	54.45 ( $\downarrow 17.65$ )	53.15 ( $\downarrow 18.95$ )
	OpenFlamingo	3B	70.20	69.53 ( $\downarrow 0.67$ )	67.80 ( $\downarrow 2.40$ )	53.93 ( $\downarrow 16.27$ )	33.59 ( $\downarrow 36.61$ )	30.18 ( $\downarrow 50.02$ )
	Blip-2 (FlanT5 XL , ViT-L)	3.4B	75.10	70.77 ( $\downarrow 4.33$ )	68.57 ( $\downarrow 6.53$ )	60.93 ( $\downarrow 14.27$ )	61.01 ( $\downarrow 14.09$ )	58.21 ( $\downarrow 16.89$ )
EVA-CLIP ViT-G/14	Blip-2 (FlanT5 XL )	4.1B	74.96	71.27 ( $\downarrow 3.69$ )	68.80 ( $\downarrow 6.17$ )	62.01 ( $\downarrow 11.88$ )	59.48 ( $\downarrow 15.48$ )	57.08 ( $\downarrow 17.88$ )
EVA-CLIP ViT-G/14	InstructBLIP (FlanT5 XL )	4.1B	76.50	72.07 ( $\downarrow 4.43$ )	69.77 ( $\downarrow 6.73$ )	63.20 ( $\downarrow 13.30$ )	63.92 ( $\downarrow 12.58$ )	56.57 ( $\downarrow 19.93$ )

Table 2. Performance Degradation (

\downarrow

) of VLMs on Image Captioning under Wrinkle-like Structural Perturbations Generated from the COCO Dataset. We compare the semantic consistency scores (%) evaluated by GPT-4 for baseline methods and the proposed method. Lower values indicate stronger attack performance. Numbers in bold denote the best results, and underlined numbers denote the second-best results.

Image Encoder	Models	Params	Clean	Natural Light Attack ( Hsiao et al., 2024 )	Shadow Attack ( Zhong et al., 2022 )	ITA ( Liu et al., 2025 )	Ours (random)	Ours (best)
OpenAI CLIP ViT-L/14	LLaVA-1.5	7B	68.00	68.00 ( $\downarrow 0.00$ )	67.00 ( $\downarrow 1.00$ )	48.00 ( $\downarrow 20.00$ )	55.00 ( $\downarrow 13.00$ )	45.00 ( $\downarrow 23.00$ )
	LLaVA-1.6	7B	64.00	63.00 ( $\downarrow 1.00$ )	64.00 ( $\downarrow 0.00$ )	43.00 ( $\downarrow 21.00$ )	47.00 ( $\downarrow 17.00$ )	40.00 ( $\downarrow 24.00$ )
	OpenFlamingo	3B	45.00	39.00 ( $\downarrow 6.00$ )	44.00 ( $\downarrow 1.00$ )	19.00 ( $\downarrow 26.00$ )	20.00 ( $\downarrow 25.00$ )	14.00 ( $\downarrow 31.00$ )
	Blip-2 (FlanT5 XL , ViT-L)	3.4B	63.00	58.00 ( $\downarrow 5.00$ )	50.00 ( $\downarrow 13.00$ )	38.00 ( $\downarrow 25.00$ )	36.00 ( $\downarrow 27.00$ )	32.00 ( $\downarrow 31.00$ )
EVA-CLIP ViT-G/14	Blip-2 (FlanT5 XL )	4.1B	54.00	53.00 ( $\downarrow 1.00$ )	54.00 ( $\downarrow 0.00$ )	33.00 ( $\downarrow 21.00$ )	32.00 ( $\downarrow 22.00$ )	28.00 ( $\downarrow 26.00$ )
EVA-CLIP ViT-G/14	InstructBLIP (FlanT5 XL )	4.1B	68.00	64.00 ( $\downarrow 4.00$ )	62.00 ( $\downarrow 6.00$ )	44.00 ( $\downarrow 24.00$ )	45.00 ( $\downarrow 23.00$ )	41.00 ( $\downarrow 27.00$ )

Table 3. Performance Degradation (

\downarrow

) of VLMs on the VQA task under Wrinkle-like Structural Perturbations Generated from the COCO Dataset. We compare the answer correctness scores (%) evaluated by GPT-4 for baseline methods and the proposed method. Lower values indicate stronger attack performance. Bold numbers denote the best results, and underlined numbers denote the second-best results.

Figure 3. Qualitative Results on Image Captioning and Visual Question-Answering. The figure shows representative outputs for clean and perturbed images, illustrating caption deviations and incorrect answers caused by the proposed perturbation. Qualitative results on image captioning and visual question-answering for clean and perturbed images.

Figure 4. Ablation on Genetic Search Hyperparameters. Attack performance of the proposed wrinkle-like structural perturbation under (a) different numbers of fitness evaluations and (b) different population sizes, showing the trade-off between attack effectiveness and search efficiency across different CLIP models.

Figure 5. Ablation on multi-scale wrinkle components. ASR (%) under different component combinations is reported, including large only, medium only, small only, large+medium, and full.

Panel of Figure 6 — Figure 6. Ablation study of the weighting coefficients $\alpha_{1}$ and $\alpha_{2}$ on EVA-CLIP ViT-G/14.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability
May 21, 2026Xiang Chen, Yuxian Dong, Chao Li +6VLM Robustness Adversarial Patch Attacks
PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
Apr 1, 2026Jingning Xu, Haochen Luo, Chen LiuVLM Robustness Vision-Language Models
Department of Computer Science City University of Hong Kong
VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
Mar 6, 2026Rohit Saxena, Alessandro Suglia, Pasquale MinerviniVLM Robustness VLM Evaluation
University of Edinburgh, UK

May 21, 2026J/K move · Enter open · S save

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability

Xiang Chen, Yuxian Dong, Chao Li, Chengyin Hu+5

VLM Robustness Adversarial Patch Attacks Adversarial Attacks on VLMs Large Vision-Language Models Transfer-Based Adversarial Attacks Physical Adversarial Attacks

Open paper AskPDF arXiv

Vision-language models (VLMs) have achieved strong performance across diverse multimodal tasks, but their adversarial robustness in visible-infrared (VIS-IR) scenarios remains underexplored. This gap is critical because VIS-IR sensing is widely used in real-world perception systems to support reliable understanding under challenging imaging conditions. To address this cross-modal threat setting, we propose CFGPatch, a curved-edge fractal geometric adversarial patch framework for attacking VIS-IR VLMs. CFGPatch builds on triangular fractal geometry and replaces rigid straight-edged primitives with Bezier-curved elements, preserving multi-scale fractal self-similarity while introducing smoother contours, richer directional variation, and more flexible shape deformation. In addition, we design a modality-specific Fraser-spiral rendering mechanism to inject fine-grained texture distortions and misleading perceptual cues into visible and infrared images. By coupling global curved-fractal geometry with local spiral-based appearance interference, CFGPatch disrupts both shape perception and texture interpretation. We further adopt expectation over transformation (EOT) to improve robustness against common image-level transformations. Extensive experiments show that CFGPatch effectively fools VIS-IR VLMs and consistently outperforms standard patch baselines in attack effectiveness and robustness. Moreover, adversarial samples optimized for zero-shot classification transfer well to image captioning and visual question answering, demonstrating strong cross-task transferability and generalizability across downstream tasks.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models