Visual token compression reduces the inference cost of Large Vision-Language Models (LVLMs). However, aggregate robustness measures do not reveal whether a particular adversarial failure is induced by compression or inherited from the underlying model. We define a compression-specific failure (CSF) as an adversarial input that remains correct under full-token inference but fails after compression, casting compression-induced risk as a paired failure attribution problem. Within a controlled diagnostic cohort, counterfactuals show that retained-set allocation causally changes compressed correctness and reveal a negative association between recovery and representation drift in displaced evidence. Motivated by these findings, we propose CIRA, a Compression-Induced Risk Attack for Large Vision-Language Models. Under a vision-encoder white-box setting, CIRA optimizes image perturbations through encoder-side objectives that manipulate token priorities across candidate compression budgets while preserving displaced evidence. CIRA uses no downstream questions or labels and requires no access to the language model, deployed compressor, or exact compression budget. Across 12 dataset-compressor settings evaluated at four budgets, CIRA achieves a mean CSFR of 20.35% while limiting full-token attack success to 6.92%, with similar behavior on additional LVLM families. A cross-view selection-stabilization defense substantially suppresses CIRA, although Adaptive CIRA partially restores its effectiveness. These results show that compression-specific failures persist under restricted access and support paired evaluation of full-token and compressed inference for attributing risk to visual-token compression.
Figures & tables
Figure 1: Overview of the compression-specific failure setting, the CIRA attack framework and the resulting behavior under visual-token compression.
Figure 2: Cumulative CSF recovery under guided and matched-random retained-set exchanges across direct-token exchange fractions.
Figure 3
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Sensitivity of Full ASR and CSFR to the perturbation budget and number of optimization steps on POPE.
Figure 6: Selectivity and cross-layer exclusion stability across scoring-layer configurations.
Figure 7: Cross-view support distributions of clean Top- K tokens and CIRA replacement tokens across compression budgets.
Figure 8: Clean Top- K retention and priority-reallocation profiles under CIRA, CIRA + TCS, and Adaptive CIRA + TCS.
Figure 9: Qualitative examples of compression-specific failures on LLaVA-v1.5-7B across visual-token compressors and retention budgets.
Figure 10: Qualitative examples of compression-specific failures on Qwen3-VL-8B-Instruct across visual-token compressors and retention budgets.
Figure 11: Qualitative examples of compression-specific failures on InternVL3.5-8B across visual-token compressors and retention budgets.