On the Adversarial Robustness of Discrete Image Tokenizers

Rishika Bhagwatkar; Irina Rish; Nicolas Flammarion; Francesco Croce

cs.CVFeb 20, 2026

On the Adversarial Robustness of Discrete Image Tokenizers

Authors: Rishika Bhagwatkar, Irina Rish, Nicolas Flammarion, Francesco Croce

Organizations: Mila - Quebec AI Institute · EPFL · ELLIS Institute Finland - Aalto University

Abstract

Discrete image tokenizers encode visual inputs as sequences of tokens from a finite vocabulary and are gaining popularity in multimodal systems, including encoder-only, encoder-decoder, and decoder-only models. However, unlike CLIP encoders, their vulnerability to adversarial attacks has not been explored. Ours being the first work studying this topic, we first formulate attacks that aim to perturb the features extracted by discrete tokenizers, and thus change the extracted tokens. These attacks are computationally efficient, application-agnostic, and effective across classification, multimodal retrieval, and captioning tasks. Second, to defend against this vulnerability, inspired by recent work on robust CLIP encoders, we fine-tune popular tokenizers with unsupervised adversarial training, keeping all other components frozen. While unsupervised and task-agnostic, our approach significantly improves robustness to both unsupervised and end-to-end supervised attacks and generalizes well to unseen tasks and data. Unlike supervised adversarial training, our approach can leverage unlabeled images, making it more versatile. Overall, our work highlights the critical role of tokenizer robustness in downstream tasks and presents an important step in the development of safe multimodal foundation models.

Figures & tables

Figure 1: Unsupervised vs supervised adversarial attacks for classification. We report the robust accuracy when varying the perturbation radius

\epsilon

(scaled to [0, 255]) for three classifiers on Imagenette: TiTok with ViT probing (left), FlexTok with linear probing (middle), and zero-shot UniTok (right). In most cases, our unsupervised attacks (blue curves), which target only the image tokenizer and do not need label information, perform close to end-to-end supervised attacks (red), which target the entire classifier and use labels. For small

\epsilon

, our unsupervised attacks are slightly worse than the supervised ones. Both attacks are optimized with 100 iterations of APGD on 500 images.

Panel of Figure 2 — Figure 2: Reconstruction of unsupervised attacks. For each tokenizer, we show the reconstruction (given by the corresponding de-tokization models) of the clean images and adversarial images computed by unsupervised attacks at $\epsilon=\nicefrac{{4}}{{255}},\nicefrac{{8}}{{255}}$ with 2500 steps of APGD . The perturbed inputs affect the reconstruction differently depending on the tokenizer, with TiTok yielding the most distorted decoded images while FlexTok being most robust, with still clearly recognizable subjects.

Tokenizer	Imagenette			Caltech101			OI-Crop			OI-Pos			Average
Tokenizer	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$
clean	93.6	2.6	0.0	74.4	0.6	0.0	71.8	7.4	0.8	69.2	5.4	1.4	77.3	4.0	0.6
$\text{AT}^{\nicefrac{{4}}{{255}}}$	91.8	63.6	36.6	73.0	48.2	20.8	66.2	50.6	26.0	67.2	46.0	24.6	74.6	52.1	27.0
$\text{AT}^{\nicefrac{{8}}{{255}}}$	89.6	69.0	48.8	72.4	51.6	32.8	62.0	48.8	35.8	64.8	51.2	35.6	72.2	55.2	38.3
$\text{AT}^{\nicefrac{{12}}{{255}}}$	87.0	71.4	51.0	67.6	51.2	36.8	56.2	49.0	36.8	61.6	49.6	35.2	68.1	55.3	40.0
$\text{AT}^{\nicefrac{{16}}{{255}}}$	83.4	66.6	50.0	61.2	47.6	37.4	50.0	47.2	35.8	59.4	48.8	39.2	63.5	52.6	40.6

Table 1: Evaluation of FuseLIP on image classification and multimodal retrieval. We report the clean and robust accuracy (%) under

\ell_{\infty}

-bounded perturbations with

\epsilon\in\{2/255,4/255\}

for original and robust tokenizers trained on different radii.

Tokenizer	Imagenette			Caltech101			ImageNet			Average
	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$
clean	99.2	0.0	0.0	85.7	0.0	0.0	67.3	0.0	0.0	84.1	0.0	0.0
$\text{AT}^{\nicefrac{{4}}{{255}}}$	99.2	92.1	75.0	81.2	56.9	22.4	66.9	31.9	10.5	82.4	60.3	36.0
$\text{AT}^{\nicefrac{{8}}{{255}}}$	97.8	91.5	82.7	77.4	63.5	43.9	58.3	40.3	23.6	77.8	65.1	50.1
$\text{AT}^{\nicefrac{{12}}{{255}}}$	95.6	88.7	81.4	72.4	60.1	47.6	50.4	36.5	25.6	72.8	61.8	51.5
$\text{AT}^{\nicefrac{{16}}{{255}}}$	92.7	86.3	79.6	65.3	57.5	44.6	42.3	32.1	23.6	66.7	58.7	49.3

Table 2: Evaluation of UniTok on image classification. For each dataset, we report the accuracy (%) under no attack (clean) and

\ell_{\infty}

-bounded perturbations with

\epsilon\in\{2/255,4/255\}

for original and robust tokenizers trained on different radii.

Tokenizer	VQAv2			OK-VQA			GQA			Average
Tokenizer	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$
clean	73.2	14.4	8.5	59.6	2.1	0.8	68.0	13.4	10.0	66.9	10.0	6.4
$\text{AT}^{\nicefrac{{4}}{{255}}}$	67.4	50.5	44.4	53.0	38.8	29.9	67.0	46.2	31.4	62.5	45.2	35.2
$\text{AT}^{\nicefrac{{8}}{{255}}}$	62.7	52.4	46.6	49.2	40.3	32.8	65.4	48.4	38.8	59.1	47.1	39.4
$\text{AT}^{\nicefrac{{12}}{{255}}}$	61.6	52.1	45.7	46.7	39.4	33.9	64.2	49.4	40.2	57.5	47.0	40.0
$\text{AT}^{\nicefrac{{16}}{{255}}}$	57.5	49.7	43.7	45.1	38.8	32.2	61.2	48.4	40.0	54.6	45.6	38.6

Table 3: UniTok-MLLM on VQA. We report the accuracy (%) on VQAv2, OK-VQA, GQA and their average under no attack (clean) and

\ell_{\infty}

-bounded perturbations with

\epsilon\in\{2/255,4/255\}

for original and robust tokenizers trained on different radii.

Figure 3: Unsupervised targeted attack on captioning. We evaluate UniTok-MLLM with the original tokenizer and our robust version trained on ImageNet ( $\epsilon=\nicefrac{{8}}{{255}}$ ). We use our unsupervised attacks ( $\epsilon=\nicefrac{{4}}{{255}}$ , 2,000 iterations) to minimize the distance in embedding space between the features of the perturbed and target images. Under attack, the model with the original UniTok tokenizer generates a caption about the target image, while the model with the robust tokenizer does not.

Figure 4: Supervised targeted attack on captioning. We evaluate the UniTok-MLLM using the original UniTok tokenizer and our robust version trained on ImageNet (

\epsilon=\nicefrac{{8}}{{255}}

). We evaluate using APGD-CE (

\epsilon=\nicefrac{{4}}{{255}}

, 2,000 iterations) for a given target caption. Under attack, the model with the original UniTok tokenizer generates the target caption, while the model with the robust tokenizers does not.

Method	Imagenette		OI-Pos		OI-Crop
Method	clean	$\nicefrac{{2}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$	clean	$\nicefrac{{2}}{{255}}$
clean	75.8	0.0	69.2	5.4	71.8	7.4
end-to-end AT	90.6	79.4	31.8	21.6	15.2	9.6
AT (ImageNet)	84.2	64.4	58.0	48.0	60.4	42.2
AT (CC3M)	87.6	67.0	57.4	49.2	63.6	49.6

Table 4: Analyses of end-to-end fine-tuning and different training data. We compare different approaches for adversarial fine-tuning of TiTok. For Imagenette we use ViT probe and FuseLIP for other datasets (see Sec. 5 ). Our unsupervised training on ImageNet and CC3M provides largely better generalization than end-to-end fine-tuning (robust accuracy by APGD as in AutoAttack).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Targeted attacks on classification for UniTok: We qualitatively evaluate targeted attacks using our unsupervised embedding-space attack and supervised APGD-CE, both for 100 steps with

\epsilon=8/255

. We notice that our unsupervised attack changes the label of the adversarial image as well as its reconstruction, whereas the supervised attack does not change the label of the adversarial image’s reconstruction.

Tokenizer	Imagenette			Caltech101			OI-CROP			OI-POS			Average
	clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$
	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$
clean	86.0	2.6	0.0	69.0	1.0	0.0	65.2	8.2	1.0	60.6	5.6	0.4	70.2	4.4	0.4
$\text{AT}^{\nicefrac{{4}}{{255}}}$	86.2	57.4	28.4	68.2	37.4	12.8	63.8	42.8	24.4	58.4	38.0	17.2	69.2	43.9	20.7
$\text{AT}^{\nicefrac{{8}}{{255}}}$	84.2	64.4	46.8	66.4	44.0	25.8	58.0	48.0	34.4	60.4	42.2	25.8	67.3	49.7	33.2
$\text{AT}^{\nicefrac{{12}}{{255}}}$	82.8	65.6	49.0	62.6	43.8	30.0	56.0	43.4	34.2	60.6	43.6	32.4	65.5	49.1	36.4

Appendix

Table 5: TiTok-S128: We report the clean and robust accuracy (%) on Imagenette, Caltech101, OI-CROP and OI-POS under no attack (clean) and

\ell_{\infty}

-bounded perturbations with

\epsilon\in\{2/255,4/255\}

for original and robust tokenizers trained on different radii. The rightmost block reports averages across datasets.

Tokenizer	Imagenette			Caltech101			OI-CROP			OI-POS			Average
	clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$
	clean	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$
clean	86.0	2.6	0.0	69.0	1.0	0.0	65.2	8.2	1.0	60.6	5.6	0.4	70.2	4.4	0.4
$\text{AT (ImageNet)}^{\nicefrac{{8}}{{255}}}$	84.2	64.4	46.8	66.4	44.0	25.8	58.0	48.0	34.4	60.4	42.2	25.8	67.3	49.7	33.2
$\text{AT (ImageNet)}^{\nicefrac{{16}}{{255}}}$	79.8	62.4	49.6	59.2	42.2	32.0	51.8	42.2	32.6	55.6	44.2	33.2	61.6	47.8	36.9
$\text{AT (CC3M)}^{\nicefrac{{8}}{{255}}}$	82.4	62.0	40.0	63.8	42.0	26.0	53.2	48.2	33.4	59.6	40.4	25.8	64.8	48.2	31.3

Appendix

Table 6: Generalization of FuseLIP-S when TiTok is adversarially finetuned on CC3M: We observe negligible differences in the clean and robust accuracies as compared to fine-tuning on ImageNet-1k, which is

3\times

smaller. Hence, we prefer fine-tuning on the latter.

Tokenizer	Imagenette			Caltech101			OI-CROP			OI-POS			Average
	clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$		clean	$\ell_{\infty}$
		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$		$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$
clean	93.6	2.6	0.0	74.4	0.6	0.0	71.8	7.4	0.8	69.2	5.4	1.4	77.3	4.0	0.6
$\text{AT (ImageNet)}^{\nicefrac{{8}}{{255}}}$	89.6	69.0	48.8	72.4	51.6	32.8	62.0	48.8	35.8	64.8	51.2	35.6	72.2	55.2	38.3
$\text{AT (ImageNet)}^{\nicefrac{{16}}{{255}}}$	83.4	66.6	50.0	61.2	47.6	37.4	50.0	47.2	35.8	59.4	48.8	39.2	63.5	52.6	40.6
$\text{AT (CC3M)}^{\nicefrac{{8}}{{255}}}$	87.6	67.0	46.8	72.4	52.0	32.2	57.4	49.2	37.8	63.6	49.6	34.4	70.3	54.5	37.8

Appendix

Table 7: Generalization of FuseLIP-B when TiTok is adversarially finetuned on CC3M: We observe negligible differences in the clean and robust accuracies as compared to fine-tuning on ImageNet-1k, which is

3\times

smaller. Hence, we prefer fine-tuning on the latter.

	clean	$\epsilon=\nicefrac{{2}}{{255}}$	$\epsilon=\nicefrac{{4}}{{255}}$	$\epsilon=\nicefrac{{8}}{{255}}$	$\epsilon=\nicefrac{{16}}{{255}}$
ImageNet-1k
Clean	67.3	0.0	0.0	0.0	0.0
$\text{AT}^{\nicefrac{{4}}{{255}}}$	66.9	66.3	62.5	17.7	0.8
$\text{AT}^{\nicefrac{{8}}{{255}}}$	58.3	61.7	61.5	42.9	3.8
$\text{AT}^{\nicefrac{{12}}{{255}}}$	50.4	53.0	53.0	49.8	11.1
$\text{AT}^{\nicefrac{{16}}{{255}}}$	42.3	44.2	44.2	43.7	19.6

Appendix

Table 8: UniTok Evaluation under unsupervised attacks. Robustness evaluation of clean and robust UniTok models against unsupervised attacks, 100 steps at different perturbation strengths.

loss version	FuseLIP-S			FuseLIP-B
loss version	$\epsilon=\nicefrac{{2}}{{255}}$	$\epsilon=\nicefrac{{4}}{{255}}$	$\epsilon=\nicefrac{{8}}{{255}}$	$\epsilon=\nicefrac{{2}}{{255}}$	$\epsilon=\nicefrac{{4}}{{255}}$	$\epsilon=\nicefrac{{8}}{{255}}$
Option 1. (default)	70.7	37.8	16.1	70.5	37.6	20.1
Option 2.	70.6	39.7	17.0	71.1	42.1	21.0
Option 3.	76.3	50.5	20.7	78.6	46.9	22.0
Option 4.	76.1	50.2	21.7	77.2	49.0	22.2

Appendix

Table 9: Analysis of the attack objective function. We compare four objective functions, detailed in App. B.3 , which use different combinations of pre- and post-quantization embedding vectors, for our unsupervised attacks. We report the robust accuracy on Imagenette obtained optimizing each loss version with 100 steps of APGD at different perturbation radii for FuseLIP-S and FuseLIP-B. Our default version, indicated as Option 1, achieves the best results (lower robust accuracy) in nearly all cases.

	successful attacks				unsuccessful attacks
tokenizer	FuseLIP-S		FuseLIP-B		FuseLIP-S		FuseLIP-B
tokenizer	$\epsilon=\nicefrac{{2}}{{255}}$	$\epsilon=\nicefrac{{4}}{{255}}$	$\epsilon=\nicefrac{{2}}{{255}}$	$\epsilon=\nicefrac{{4}}{{255}}$	$\epsilon=\nicefrac{{2}}{{255}}$	$\epsilon=\nicefrac{{4}}{{255}}$	$\epsilon=\nicefrac{{2}}{{255}}$	$\epsilon=\nicefrac{{4}}{{255}}$
Clean	124.5	127.4	122.8	127.0	76.8	125.4	118.3	126.0
$\text{AT}^{\nicefrac{{4}}{{255}}}$	0.0	0.0	0.0	105.0	0.0	0.0	0.0	18.1
$\text{AT}^{\nicefrac{{8}}{{255}}}$	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
$\text{AT}^{\nicefrac{{12}}{{255}}}$	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

Appendix

Table 10: How unsupervised attacks change token indices. We report the average number of token indices that change after unsupervised attacks (100 steps of APGD, 500 ImageNet images).

Adversarial Robustness

Adversarial robustness studies how well AI systems withstand deliberately crafted inputs, training-data corruption, and other attacks.

See all 479 papers

Investigating Adversarial Robustness of Multi-modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanAdversarial Training VLM Robustness
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia
Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models
Sep 30, 2026Shilinlu Yan, Bowen Chen, Yuechen Zhang +3VLM Robustness Adversarial Attacks on VLMs
Beijing University of Posts and Telecommunications · Jiangnan University · Nanyang Technological University +1
Towards Robustness against Typographic Attack with Training-free Concept Localization
Jul 2, 2026Bohan Liu, Wenqian Ye, Guangzhi Xiong +3VLM Robustness Adversarial Attacks on VLMs
University of Virginia, Charlottesville VA 22903, USA

Jun 2, 2026J/K move · Enter open · S save

Investigating Adversarial Robustness of Multi-modal Large Language Models

Hashmat Shadab Malik, Muzammal Naseer, Salman Khan

Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia

Adversarial Training VLM Robustness Adversarial Attacks on VLMs Adversarial Robustness Multimodal Large Language Models Multimodal Foundation Models+1

Open paper AskPDF arXiv

Multi-modal Large Language Models (MLLMs) achieve strong performance on vision-language tasks, but incorporating visual inputs through a vision encoder (e.g., CLIP) substantially expands the attack surface, making these models vulnerable to visual adversarial perturbations. Prior defenses typically preserve compatibility with pretrained MLLMs by enforcing strict alignment to CLIP's original embedding space during adversarial fine-tuning; while practical, this constraint fundamentally limits achievable robustness. We present a systematic investigation of adversarial robustness in MLLMs. We first introduce a diagnostic CLIP-alignment protocol that predicts, prior to full MLLM training, which robust vision encoders will transfer effectively to the multimodal setting, revealing that large-scale multimodal adversarial pretraining, rather than unimodal scale alone, is the critical factor for strong robustness transfer. Integrating such encoders into MLLMs via end-to-end multimodal training yields average gains of 28 CIDEr points on captioning and 11.7% VQA accuracy under strong adversarial attacks compared to constrained plug-and-play baselines. We further show that adversarial training applied directly to a standard non-robust MLLM degrades both clean and adversarial performance, establishing robust visual representations as a strict prerequisite, while end-to-end adversarial training from a robust backbone delivers additional gains of 1.9 CIDEr points and 4.3% VQA accuracy. Beyond training-time defenses, lightweight test-time visual stochastic transformations serve as an effective black-box defense for non-robust MLLMs, elevating adversarial performance from near-zero to levels comparable with robust models. Finally, we show that our robust models substantially reduce toxic generation under white-box visual jailbreak attacks. Code and pretrained weights will be released publicly.

Abstract

Adversarial Robustness

Explore similar work

Investigating Adversarial Robustness of Multi-modal Large Language Models

Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

Towards Robustness against Typographic Attack with Training-free Concept Localization