Shape and Substance: Dual-Layer Side-Channel Attacks on Local Vision-Language Models

Eyal Hadad; Mordechai Guri

cs.CRMar 26, 2026

Shape and Substance: Dual-Layer Side-Channel Attacks on Local Vision-Language Models

Authors: Eyal Hadad, Mordechai Guri

Organizations: Ben-Gurion University of the Negev, Israel Faculty of Computer and Information Science

Abstract

On-device Vision-Language Models (VLMs) promise data privacy via local execution. However, we show that the architectural shift toward Dynamic High-Resolution preprocessing (e.g., AnyRes) introduces an inherent algorithmic side-channel. Unlike static models, dynamic preprocessing decomposes images into a variable number of patches based on their aspect ratio, creating workload-dependent inputs. We demonstrate a dual-layer attack framework against local VLMs. In Tier 1, an unprivileged attacker can exploit significant execution-time variations using standard unprivileged OS metrics to reliably fingerprint the input's geometry. In Tier 2, by profiling Last-Level Cache (LLC) contention, the attacker can resolve semantic ambiguity within identical geometries, distinguishing between visually dense (e.g., medical X-rays) and sparse (e.g., text documents) content. By evaluating state-of-the-art models such as LLaVA-NeXT and Qwen2-VL, we show that combining these signals enables reliable inference of privacy-sensitive contexts. Finally, we analyze the security engineering trade-offs of mitigating this vulnerability, reveal substantial performance overhead with constant-work padding, and propose practical design recommendations for secure Edge AI deployments.

Figures & tables

Fig. 1: Dual-layer threat model overview. A co-located, unprivileged process observes shared hardware behavior while a local VLM processes an image with dynamic resolution. The input-dependent workload creates two distinct side-channels: Tier 1 leverages OS-level timing metrics from CPU cores to infer the input’s geometric grid (aspect ratio). In contrast, Tier 2 exploits HPC from the LLC to infer the input’s visual semantic density.

Fig. 2: AnyRes dynamic preprocessing. The model tiles the image into an $(m\times n)$ grid based on aspect ratio and adds a global view, yielding $(m\times n)+1$ patches. Different grids (e.g., $1\times 2$ vs. $2\times 2$ ) produce different amounts of work, creating a timing signal.

Component	Intel Setup	AMD Setup
Experimental Role	Primary Evaluation	Cross-Platform Validation
CPU	Core i7-13700 (16 Cores)	Ryzen 9 7950X (16 Cores)
LLC	30 MB	64 MB
OS Kernel	Linux 6.8 (Ubuntu 24.04)	Linux 6.8 (Ubuntu 24.04)

TABLE I: Hardware Specifications. The Intel setup serves as the primary reference platform, while the AMD setup is used to validate architectural robustness under a significantly larger LLC.

Fig. 3: Deterministic Geometric Leakage. Distribution of inference time across aspect ratios in (a) Cold Cache and (b) Warm Cache states. Note the distinct ”pyramid” structure: Square inputs (1:1) incur a

\sim 2.3\times

latency penalty compared to rectangular inputs (1:2, 2:1), creating non-overlapping clusters that enable 100% classification accuracy regardless of cache state.

Content Class	Time (s)	LLC Misses ( $10^{9}$ )
Crypto-Noise	107.4	$\mathbf{16.9}$ (Lowest)
Document	107.6	$17.6$
Nature	109.0	$17.8$
X-Ray	110.5	$\mathbf{17.9}$ (Highest)

TABLE II: Mean hardware metrics (warm cache). Execution time remains nearly constant across content classes, while LLC misses exhibit a consistent semantic ordering.

Fig. 4: Joint timing and cache leakage. Projection of the Combined Attack vector into a two-dimensional space. The X-axis (execution time) separates inputs by geometry clusters (portrait vs. square) due to varying patch counts. The Y-axis (LLC misses) resolves the semantic ambiguity within each geometric cluster (e.g., separating dense X-ray images from sparse documents).

	Intel (30MB)		AMD (64MB)
Evaluation Phase	Low	High	Low	High
Time (s)	49	111	18	29
(Portrait vs. Square)	2.27x Gap		1.61x Gap
LLC Misses ( $10^{9}$ )	16.9	17.9	4.6	4.6
(Noise vs. X-Ray)	Observable		Not Observable

TABLE III: Cross-Architecture Evaluation. The geometric timing gap remains clear across platforms, reflecting algorithmic control-flow. Conversely, the semantic cache signal is not observable on the AMD architecture due to its larger LLC capacity.

Class	Precision	Recall	F1-Score
Medical Report (Portrait)	0.92	0.73	0.81
Chest X-Ray (Portrait)	0.78	0.93	0.85
Encrypted Data (Square)	0.79	1.00	0.88
Tech Schematic (Square)	1.00	0.67	0.80
Overall Accuracy	84.0%

TABLE IV: Classification Report for the Combined Attack. While achieving an overall accuracy of 84.0%, the attack demonstrates asymmetric effectiveness. It achieves perfect (1.00) and near-perfect (0.93) recall for privacy-critical targets (Encrypted Data and Chest X-Rays). This indicates that the side-channel is highly reliable at identifying sensitive, visually dense inputs, ensuring they are rarely missed despite microarchitectural noise.

Fig. 5: Confusion Matrix of the Combined Attack. The results visually validate the dual-layer threat model: deterministic timing differences eliminate cross-geometry errors (empty off-diagonal quadrants), while cache-miss telemetry resolves semantic ambiguity within identical geometries. Notably, the attack achieves perfect or near-perfect classification for privacy-critical targets, such as encrypted data and X-ray images.

Fig. 6: The Physical Logic of the Attack. The extracted decision tree perfectly mirrors the proposed dual-layer leakage model. Layer 1 (Shape): A fully deterministic root split based on inference time (e.g., $t<80s$ vs. $t>80s$ ) isolates the input’s geometric grid into Portrait or Square. Layer 2 (Substance): Within each geometry cluster, cache contention levels (Low vs. High LLC Misses) differentiate inputs by visual density. This allows the attacker to reliably isolate dense structures (Chest X-Rays, Tech Schematics) from sparse ones (Medical Reports, Encrypted Data).

Fig. 7: Cross-Model Comparison. LLaVA v1.5 (static preprocessing) shows similar latency across aspect ratios, whereas Qwen2-VL (dynamic preprocessing) exhibits a clear step-like timing pattern. Square inputs (

1:1

) incur approximately a 60% latency increase compared to rectangular inputs.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Fig. S1: Visual Density Comparison. Representative samples illustrating differences in spatial frequency and structural density. High-contention classes (Nature, X-Ray) exhibit dense textures across the image plane, while low-contention classes (Document, Noise) contain large homogeneous regions with limited fine-grained structure.

Condition	Noise (Avg)	X-Ray (Avg)	Separation Gap
Idle State	16.9	17.9	1.0
System Load	17.1	18.1	1.0

Appendix

TABLE S1: Leakage Robustness. Mean LLC misses (

\times 10^{9}

) under idle and background-load conditions. While absolute miss counts increase under load, the relative separation remains stable.

Privacy Protection in VLMs

Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.

See all 37 papers

Jun 10, 2026

The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models

A vision encoder compresses image pixels into semantic embeddings, implicitly acting as a privacy boundary by preserving semantic content while attenuating pixel-local detail required for exact text recovery. Encoder-free vision-language models (VLMs) remove this boundary by routing image patches directly into the language-model token stream, thereby exposing an architectural privacy attack surface: intermediate visual tokens become a pre-output side channel. Under a token-access adversary, decoders invert visual-token streams from two encoder-free VLMs, Gemma4 and Fuyu, recovering recognizable image structure and readable held-out access codes, whereas matched encoder-based controls localize target regions but recover no exact strings. Within-model ablations show that the operative factor is spatial sampling fidelity of the visual-token grid, especially character-direction sampling density, rather than token or value count. The leakage is not limited to exported tokens: Gemma4 layer-0 key-value cache tensors are directly invertible, placing the side channel within KV caches commonly persisted by production serving stacks for decoding efficiency. The attack survives clutter, realistic document degradation, and zero-shot transfer to public document images, and it resists value-level defenses such as additive noise and quantization. Effective mitigation must therefore reduce spatial sampling, making removal of the vision encoder a first-class privacy decision in VLM deployment.

Chenyu Zhou, Qiliang Jiang, Shuning Wu +1

School of Engineering, Institute of Science Tokyo Tokyo, Japan · College of Control Science and Engineering, Zhejiang University Hangzhou, China · Department of Electrical and Computer Engineering, National University of Singapore Singapore, Singapore

Vision-Language Models Adversarial Attacks Privacy Leakage in Language Models Privacy Protection in VLMs

Ask PDF

Jul 2, 2026

Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models

Cloud-edge Large Vision-Language Model (LVLM) inference enables efficient deployment by splitting computation between edge devices and cloud servers. In this process, intermediate vision tokens are transmitted from the edge to the cloud over a communication link, thereby exposing a new attack surface. We study vision token manipulation attack (VTM-Attack) under a black-box man-in-the-middle setting, where an adversary intercepts and manipulates a subset of transmitted vision tokens under a budget constraint. We propose four naïve attack strategies and an optimization-based token selection method. Experiments on 6 state-of-the-art LVLMs (3B-72B) across 4 benchmarks show that manipulating only 10% of vision tokens can reduce accuracy by up to 88.31%. These results reveal a critical vulnerability in cloud-edge LVLM inference.

Zikai Zhang, Rui Hu, Olivera Kotevska +1

Department of Computer Science and Engineering, University of Nevada, Reno, Reno, USA · Oak Ridge National Laboratory, Oak Ridge, USA

Adversarial Attacks on VLMs Large Vision-Language Models VLM Inference

Ask PDF

Jun 30, 2026

Steal the Patch Size: Adversarially Manipulate Vision-Language Models

We present a black-box model-stealing attack that recovers private vision-tokenizer configurations of deployed vision-language models (VLMs), including the visual patch size and input preprocessing pipeline. The key idea is a task-level side channel induced by ViT-style patchification: when a synthetic grid image is aligned with the hidden patch grid, boundary cues are erased at tokenization, causing periodic accuracy drop. By sweeping the grid cell size and measuring these collapses, we infer the patch size; by introducing padding and a consistency-check test, we further identify whether preprocessing is dynamic- or fixed-resolution and recover the target resize resolution. Across open-source Qwen-VL variants and proprietary models including GPT and Claude, we reliably recover tokenizer-related parameters. Finally, we show that such leakage enables preprocessing-aware transfer attacks and model-targeted adversarial manipulation.

Kai Hu, Akash Bharadwaj, Weichen Yu +1

Carnegie Mellon University, Pittsburgh, USA.

Adversarial Attacks on VLMs Model Extraction Attacks

Ask PDF

Abstract

Privacy Protection in VLMs

Explore similar work

The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models

Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models

Steal the Patch Size: Adversarially Manipulate Vision-Language Models