cs.AINov 12, 2026

Scale-Induced Semantic Shift in Visual Prompt Injection Against VLMs and Web Agents

Authors: Yiheng Zhao, Runze Ji, Chenyu Yi, Yi Yu, Bingquan Shen, Alex Kot, Xudong Jiang

Organizations: Nanyang Technological University, Singapore · DSO National Laboratories, Singapore

Abstract

Visual content in multimedia systems may be resized, compressed, or re-rendered before it is interpreted by a vision-language model. We study a security failure mode in which these transformations change the machine-readable semantics of an image: a textual instruction is not recovered at a 512×512512\times512 presentation resolution under our evaluation protocol but becomes recoverable after aggressive downsampling. We formulate this behavior as Scale-Induced Semantic Shift (SISS) and develop a closed-loop black-box framework that combines dual-scale verification, selective self-hardening, shallow-layer low-rank adaptation, and pixel-space noise regularization. On a fixed 300-case benchmark, the third SISS refinement round obtains a 58.75%58.75\% average attack success rate across four closed-source vision-language models, compared with 41.08%41.08\% for the strongest generation baseline. A controlled sweep bounds the effect to aggressive compression: average success decreases to 17.68%17.68\%, 6.85%6.85\%, and 5.60%5.60\% at 128×128128\times128, 224×224224\times224, and 336×336336\times336, respectively. In a downstream web-agent study, the generated images increase hidden-instruction perception, but absolute behavioral compromise remains about 4%4\%. Generic input transformations provide little protection in a separate defense protocol, whereas the proposed ScaleGuard consistency detector reduces non-adaptive success from 61.4%61.4\% to 24.3%24.3\% and one-round defense-aware success from 42.5%42.5\% to 15.0%15.0\%, with no observed rejection among matched clean controls. The results characterize a bounded but reproducible semantic inconsistency in multimedia preprocessing and motivate scale-consistent visual-language inference.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reading Scene States: Uncovering Safety Failures in Vision-Language Models

    Nov 17, 2026Tongxi Wu, Yuyang Cheng, Jiacheng Xian +4VLM SafetyAdversarial Attacks on VLMs