Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

Shilinlu Yan; Bowen Chen; Yuechen Zhang; Zhenhong Zhou; Li Sun; Sen Su

cs.CVSep 30, 2026

Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

Authors: Shilinlu Yan, Bowen Chen, Yuechen Zhang, Zhenhong Zhou, Li Sun, Sen Su

Organizations: Beijing University of Posts and Telecommunications · Jiangnan University · Nanyang Technological University · Chongqing University of Posts and Telecommunications

Abstract

Visual-token compression improves the efficiency of large vision-language models, but can expose failures that full-token evaluation misses. We study adversarial images that preserve full-token correctness yet induce errors after compression, even when both inference paths succeed on the clean image. Creating such failures is challenging because perturbing token importance can also damage the visual content needed for full-token inference. We propose Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens. In the primary LLaVA-1.5-7B setting, FATA uses only vision-encoder gradients, without access to the deployed compressor, token budget, or downstream task. Across four visually dependent task subsets and four compressors under a controlled reconstruction protocol, FATA achieves SR = 96.3% full-token accuracy retention and CBR = 22.1% conditional blinding, compared with 89.8% and 15.7% for CAA. Ablations support the role of both objectives in balancing compressed-path failure against full-token preservation. FATA also has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate. These findings motivate assessing adversarial robustness jointly across full-token and compressed inference.

Figures & tables

Figure 1: FATA: attention suppression with feature preservation. A illustrates the compression-triggered failure setting; B presents the two attack objectives; C compares full-token retention and conditional blinding; D shows how answers to the same perturbed TextVQA image vary across compression paths.

Figure 2: FATA overview. Fixed clean targets guide input-only PGD with attention and semantic losses (A–C), followed by full-token and compressed-path evaluation of the same image (D).

	TextVQA-Open					VQAv2-Open					ScienceQA-MC					VQAv2-MC
Method	Clean	Base	CAA	CAGE	FATA	Clean	Base	CAA	CAGE	FATA	Clean	Base	CAA	CAGE	FATA	Clean	Base	CAA	CAGE	FATA
Upper Bound (576 Tokens)
None	92.0	39.9	77.4	19.8	86.6	88.2	66.1	78.9	28.3	86.0	94.5	57.8	89.2	40.6	90.1	98.0	82.5	89.6	48.7	96.2
Retain 192 Tokens ( $K_{model}=192$ )
VisionZIP	89.1	35.7	82.5	16.2	83.3	86.6	61.8	56.2	25.8	84.2	91.3	54.5	86.3	38.2	86.3	96.9	79.1	94.8	45.8	95.1
VisPruner	90.6	36.1	83.9	17.9	83.3	87.4	65.1	84.9	26.6	84.3	91.1	56.5	87.1	40.2	88.7	96.8	79.8	95.1	45.0	95.0

Table 1: Accuracy (%) across four benchmarks and visual-token budgets.

Figure 3: Failure-stage decomposition over 14,316 shared Clean-full-correct cases.

Attack	SR $\uparrow$	CER $\uparrow$	CBR $\uparrow$
Base (Attn.-only)	66.7	54.78	32.9
CAA	89.8	24.65	15.7
CAGE	36.6	75.58	43.1
FATA	96.3	30.38	22.1

Table 2: Overall stealth–lethality trade-off

Variant	Objective	SR $\uparrow$	CBR $\uparrow$
Attention-only	$\mathcal{L}_{\mathrm{attn}}$	66.7	32.9
Semantic-only	$\mathcal{L}_{\mathrm{sem}}$	98.5	14.5
FATA Full	$\mathcal{L}_{\mathrm{attn}}+\lambda\mathcal{L}_{\mathrm{sem}}$	96.3	22.1

Table 3: Dual-objective ablation of FATA.

Figure 4: Qwen3.5 practical-budget CBR by compressor and dataset.

	Feature Squeezing			Mahalanobis-Max			ML-ATD
Attack	AUROC	AUPR	TPR@5FPR	AUROC	AUPR	TPR@5FPR	AUROC	AUPR	TPR@5FPR
FATA	0.560	0.382	0.024	0.695	0.509	0.134	0.828	0.724	0.497
CAGE	0.797	0.655	0.147	0.936	0.882	0.647	0.913	0.854	0.694
CAA	0.564	0.385	0.027	0.833	0.765	0.468	0.802	0.721	0.533
Base	0.665	0.494	0.064	0.813	0.660	0.278	0.952	0.918	0.801

Table 8: Cross-attack detection performance with clean and random negatives. Higher values indicate easier detection; protocol differences and limitations are detailed in Appendix H .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

$K$	.25	.5	.75	1	1.25	1.5	1.75
576	96.0	95.0	94.3	94.5	94.2	95.1	94.3
192	94.1	92.7	93.4	93.8	93.8	92.9	94.0
128	93.0	91.5	92.1	92.1	92.8	91.7	92.7
64	88.9	87.6	88.4	88.5	89.4	88.8	88.7
32	78.8	80.1	80.7	80.6	82.0	81.4	80.9
16	69.4	68.8	67.3	69.4	68.1	69.2	71.0

Appendix

Table 9: FATA accuracy on unfiltered TextVQA-Open across token budgets and step sizes

\alpha

$K$	$\lambda=.5$	$1$	$2$	$4$	$8$
576	94.1	95.0	95.2	95.6	96.0
192	92.0	92.7	93.3	94.2	94.3
128	91.8	91.5	93.1	93.3	92.2
64	87.8	87.6	87.5	89.0	90.1
32	77.4	80.1	81.1	81.4	81.0
16	65.4	68.8	68.8	70.3	70.7

Appendix

Table 10: FATA accuracy on unfiltered TextVQA-Open across token budgets and semantic weights

\lambda

Compressor	TextVQA-Open	VQAv2-Open	ScienceQA-MC	VQAv2-MC
VisionZIP	64	64	32	32
VisPruner	64	64	32	32
PruMerge	64	64	32	32
FlowCut	64	64	32	64

Appendix

Table 11: Clean-only practical token budgets

K_{\mathrm{prac}}^{(d,c)}

for LLaVA-1.5-7B. Each entry is the smallest

K\in\{16,32,64,128,192\}

that retains at least 80% of the corresponding full-token clean accuracy.

Model	Full	Practical Open	Practical MC	Valid images per dataset
LLaVA-1.5-7B	$576$	$64$	$32$ or $64$	$1{,}000$
Qwen3.5-9B	$N_{i}$	$r=1/9$	$r=1/18$	$1{,}000$
InternVL3.5-8B-HF	$N_{i}$	$r=1/9$	$r=1/18$	$998$ or $1{,}000$

Appendix

Table 12: Model-specific budgets and sample counts used in evaluation. Practical budgets are given separately for open-ended and multiple-choice tasks.

Task	Task credit $v_{i}$	Binary correctness $c_{i}$
TextVQA-Open and VQAv2-Open	Normalized VQA credit against the human reference answers, with partial credit allowed.	$1$ for a normalized exact match to an accepted reference answer; $0$ otherwise.
ScienceQA-MC	$1$ if the parsed letter (A–F) matches the annotated answer index converted to a letter; $0$ otherwise.	The same option-letter decision as $v_{i}$ .
VQAv2-MC	$1$ if the parsed letter (A–D) matches the fixed target derived from the modal human answer; $0$ otherwise.	The same option-letter decision as $v_{i}$ .

Appendix

Table 13: Task-level scoring rules for reported accuracy and binary correctness. CBR uses the binary decisions in the final column.

Figure 5: Average task performance versus visual-token budget. The exact per-method values are reported in Table 1 in the main paper.

Figure 9: Loss ablation across 16 dataset–compressor settings, with paired lines and mean diamonds; shading marks full FATA. Error bars show

\pm 1

sample standard deviation across settings

Dataset	$D_{\rm Full}$	$D_{1/3}$	$D_{\rm Prac}$	$A_{1/3}$	$A_{\rm Prac}$
TextVQA-Open	0.70	5.22	8.60	4.52	7.90
VQAv2-Open	1.00	10.00	14.57	9.00	13.57
ScienceQA-MC	-0.20	9.47	23.70	9.67	23.90
VQAv2-MC	1.00	8.05	19.60	7.05	18.60

Appendix

Table 14: Qwen3.5 compression-specific damage and amplification by dataset (percentage points). Positive amplification means that compression increases the attack-induced accuracy gap relative to full-token inference.

Budget	Clean	FATA	Damage (pp)	Amplification (pp)
$2/9$	80.77	68.49	12.28	7.05
$1/9$	70.78	56.20	14.58	9.34
$1/18$	60.07	47.74	12.33	7.09
$1/36$	49.97	41.90	8.06	2.83

Appendix

Table 15: InternVL3.5 accuracy and amplification at additional retention fractions. Each row applies the same fraction to all four datasets and averages over the four compressors.

Compressor	Flip rate	Jaccard
VisionZIP	0.854	0.079
VisPruner	0.925	0.040
FlowCut	0.890	0.059
PruMerge	0.617	0.240

Appendix

Table 16: InternVL3.5 retained-set diagnostics at the practical budget. For each image,

S_{\rm clean}

and

S_{\rm FATA}

contain the selected source-token indices, using representative indices for merging methods. Flip rate is

|S_{\rm clean}\setminus S_{\rm FATA}|/|S_{\rm clean}|

, the fraction of clean-selected tokens removed after attack; Jaccard is

|S_{\rm clean}\cap S_{\rm FATA}|/|S_{\rm clean}\cup S_{\rm FATA}|

. Values are fractions, macro-averaged by Eq. 37 .

Figure 10: ML-ATD reference fitting, feature readout, and stage-wise detection. Visual, Projector, and LLM trajectories are scored and evaluated separately, with stage metrics averaged only for reporting.

Explore similar work

Sep 28, 2026cs.CV

Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Models

Visual token compression reduces the inference cost of Large Vision-Language Models (LVLMs). However, aggregate robustness measures do not reveal whether a particular adversarial failure is induced by compression or inherited from the underlying model. We define a compression-specific failure (CSF) as an adversarial input that remains correct under full-token inference but fails after compression, casting compression-induced risk as a paired failure attribution problem. Within a controlled diagnostic cohort, counterfactuals show that retained-set allocation causally changes compressed correctness and reveal a negative association between recovery and representation drift in displaced evidence. Motivated by these findings, we propose CIRA, a Compression-Induced Risk Attack for Large Vision-Language Models. Under a vision-encoder white-box setting, CIRA optimizes image perturbations through encoder-side objectives that manipulate token priorities across candidate compression budgets while preserving displaced evidence. CIRA uses no downstream questions or labels and requires no access to the language model, deployed compressor, or exact compression budget. Across 12 dataset-compressor settings evaluated at four budgets, CIRA achieves a mean CSFR of 20.35% while limiting full-token attack success to 6.92%, with similar behavior on additional LVLM families. A cross-view selection-stabilization defense substantially suppresses CIRA, although Adaptive CIRA partially restores its effectiveness. These results show that compression-specific failures persist under restricted access and support paired evaluation of full-token and compressed inference for attributing risk to visual-token compression.

Qiankun Li, Yuechen Zhang, Bowen Chen +4

Nanyang Technological University · Beijing University of Posts and Telecommunications

Token Compression Inference Cost Risk

Ask PDF

Sep 27, 2026cs.AI

CoViST: Visual Token Compression via Composable States

Visual token compression lowers the inference cost of vision--language models by representing images with fewer tokens. However, most existing methods compress visual tokens to a reduced set, leaving the amount of visual evidence represented by each token and its original spatial context implicit. Therefore, the compressed representation does not explicitly encode how much visual information each representative carries or where it lies in the original image. This limitation arises even after a single reduction and becomes more pronounced when compression is repeated across decoder layers. To address this issue, we propose CoViST, a training-free framework that represents a compressed image as a composable visual state. Specifically, the state combines representative features with original positions, effective contribution weights, and reusable selection metadata. CoViST constructs this state through coverage-guided selection and conservation-based contribution composition, and explicitly incorporates its contribution and positional information into decoder attention. Each component of the state retains its interpretation under successive reductions, enabling the same formulation to support both fixed compression before prefill and progressive compression within the decoder. Experimental results on seven LLaVA-1.5-7B benchmarks show that CoViST-Fixed retains 99.9%, 99.5%, and 98.1% of uncompressed performance at 192, 128, and 64 tokens, respectively, and CoViST-Pro retains 99.8%, 99.9%, and 99.1% at the corresponding layer-average budgets, outperforming state-of-the-art methods under their respective budget settings. Code will be released publicly.

Qi Zhang, Xiandong Meng, Ronggang Wang +1

Peng Cheng Laboratory Shenzhen, China · Peking University Shenzhen Graduate School Peking University Beijing, China

Token Compression Data Compression Methods Inference Cost

Ask PDF

Jun 30, 2026cs.CV

Steal the Patch Size: Adversarially Manipulate Vision-Language Models

We present a black-box model-stealing attack that recovers private vision-tokenizer configurations of deployed vision-language models (VLMs), including the visual patch size and input preprocessing pipeline. The key idea is a task-level side channel induced by ViT-style patchification: when a synthetic grid image is aligned with the hidden patch grid, boundary cues are erased at tokenization, causing periodic accuracy drop. By sweeping the grid cell size and measuring these collapses, we infer the patch size; by introducing padding and a consistency-check test, we further identify whether preprocessing is dynamic- or fixed-resolution and recover the target resize resolution. Across open-source Qwen-VL variants and proprietary models including GPT and Claude, we reliably recover tokenizer-related parameters. Finally, we show that such leakage enables preprocessing-aware transfer attacks and model-targeted adversarial manipulation.

Kai Hu, Akash Bharadwaj, Weichen Yu +1

Carnegie Mellon University, Pittsburgh, USA.

Visual Tokenizers Preprocessing

Ask PDF