Visually Legible but Adversarial: Watermark-Constrained Attacks on End-to-End VLMs for Document Understanding

Qingzhe Han; Fan Dang; Shengqi Chen; Xinyi Sun; Haoyu Xiong; Lingkun Li; Di Zhang

cs.CVOct 11, 2026

Visually Legible but Adversarial: Watermark-Constrained Attacks on End-to-End VLMs for Document Understanding

Authors: Qingzhe Han, Fan Dang, Shengqi Chen, Xinyi Sun, Haoyu Xiong, Lingkun Li, Di Zhang

Organizations: Beijing Jiaotong University

Abstract

Visible watermarks convey document ownership, confidentiality, and status, but can also carry adversarial signals. We introduce Auto-Regional Adversarial Watermarking (AR-AWM), a model-aware white-box attack on end-to-end visual language models (VLMs) for document understanding. AR-AWM constructs a small, spatially distributed watermark-text support and optimizes only its pixels to disrupt visual representations. The attack is untargeted and question-agnostic: generation uses no questions, answers, parsing prompts, language-model gradients, or language-model outputs. Each attacked page is reused across downstream queries. On Qwen2.5-VL-7B, a watermark support covering 7.98% of page pixels under the reporting threshold reduces DocVQA exact match from 0.899 to 0.150 and average normalized Levenshtein similarity (ANLS) from 0.939 to 0.212. Selected qualitative cases retain visually legible source fields. The same support without optimization and with matched random perturbations retains ANLS of 0.932 and 0.933, respectively, identifying adversarial optimization as the primary cause of degradation. Layout-matched optimized rectangles yield ANLS of 0.276, showing an incremental benefit from text-shaped geometry. AR-AWM also degrades all reported text, formula, reading-order, and table metrics on OmniDocBench. Separate white-box optimization on InternVL2.5-8B reduces ANLS from 0.922 to 0.046, while weak cross-model transfer reveals strong encoder specificity. These results identify conventional watermark-shaped supports as an adversarial surface for document VLMs in the evaluated white-box setting.luated white-box setting.

Figures & tables

Figure 1: Overview of AR-AWM. Given a clean page image $x$ , AR-AWM constructs and fixes a spatially distributed watermark-text support $\Omega_{w}$ , renders the initial watermarked page image $x_{\mathrm{wm}}$ , and optimizes only pixels inside the fixed support. A frozen visual encoder provides the clean and adversarial visual tokens, which the representation objective drives apart without using downstream questions or parsing prompts. The final adversarial page image is reused for question answering and document parsing; in the presented qualitative cases, its underlying content remains visually legible despite downstream errors.

Figure 2: Automatic, model-informed construction of the fixed watermark support $\Omega_{w}$ . A temporary encoder-response placement probe first produces the empirical response map $S$ . The page is then partitioned into adaptive regions, candidate watermark-text bands are generated and ranked within each region, and the union of the selected bands defines $\Omega_{w}$ . The temporary probe trajectory is discarded before support-constrained representation optimization begins.

Variant	EM $\uparrow$	ANLS $\uparrow$	Area $\downarrow$
Clean document	0.899	0.939	–
Prior-work 5-anchor	0.876	0.923	46.54%
Area-matched ordinary	0.887	0.931	8.15%
Same mask, unoptimized	0.887	0.932	7.98%
Same mask, random	0.886	0.933	7.98%
Optimized rectangles	0.202	0.276	7.84%

Table 1: DocVQA results on Qwen2.5-VL-7B. Identical-mask rows hold the 7.98% editable support fixed. Bold in downstream metrics denotes the strongest attack (worst performance).

Variant	Area $\downarrow$	Full PSNR $\uparrow$	OCR Preserve $\uparrow$	Text Overlap $\downarrow$
5-anchor	46.54%	10.45	0.621	44.08%
Area-matched	8.15%	21.47	0.766	7.85%
AR-AWM	7.98%	25.55	0.775	7.28%

Table 2: Automatic page-level visual and support measurements on DocVQA. Full PSNR does not measure in-support imperceptibility, and OCR Preserve is an automated text-recoverability proxy; Figure 4 provides an illustrative visual-legibility case.

Variant	Text edit $\downarrow$	Formula edit $\downarrow$	Order edit $\downarrow$	TEDS $\uparrow$	Overall $\uparrow$
Clean	0.128	0.156	0.205	0.649	53.210
5-anchor	0.139	0.170	0.210	0.616	51.460
AR-AWM	0.199	0.272	0.261	0.571	48.010

Table 3: OmniDocBench results on Qwen2.5-VL-7B. Lower is better for edit distances; higher is better otherwise. Bold in downstream metrics denotes the strongest attack (worst performance).

Panel of Figure 3 — Figure 3: Illustrative OmniDocBench formula failure on a Putnam exam page. The source statement and display formula remain visually recoverable in the attacked crop, but Qwen2.5-VL-7B replaces them with an unrelated claim (display-formula edit $0.0\!\to\!1.0$ ). Parsed excerpts are shortened for layout.

Panel of Figure 4 — Figure 4: Visually legible yet adversarial DocVQA case (doc. 2192). Panels (c,d) show that the underlying page content remains visually legible after the familiar CONFIDENTIAL overlay is optimized. In this illustrative case, the requested fields remain recoverable from the page, but Qwen2.5-VL-7B maps four distinct questions on the same attacked document to one identifier. Clean answers are lightly shortened for layout.

Panel of Figure 5 — Figure 5: Main causal and generalization evidence. (a) Only optimized perturbations collapse ANLS; the text-shaped carrier outperforms matched rectangles. (b) The three regional layouts have similar global drift but different ANLS; global placement has higher Cos@10. (c) Strong degradation begins at 3.92% area and is non-monotonic. (d) White-box effects are large, whereas cross-model transfer is weak.

Placement	EM $\uparrow$	ANLS $\uparrow$	Area $\downarrow$	Cell CV $\downarrow$
Full	0.150	0.212	7.98%	0.010
Global	0.151	0.215	10.11%	1.187
Response only	0.145	0.207	7.98%	0.013
Fixed direction	0.169	0.227	7.86%	0.003

Table 4: DocVQA placement designs. Response ranks bands by response score; coverage adds spatial coverage. Lower Cell CV means more balanced support. Bold in downstream metrics denotes the strongest attack (worst performance).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Watermark text	CONFIDENTIAL
Font / per-cell font size	DejaVu Sans Bold / $\max(18,\operatorname{round}(0.18\min(w_{c},h_{c})))$
Mask rendering / counting threshold	Pillow 8-bit alpha, bicubic rotation / $8/255$
Watermark color / opacity	gray $(80,80,80)$ / $0.32$
Regional grid	automatic, at most $12$ cells
Target cell size / aspect	$224$ / $1.0$

Appendix

Table S1: Numerical rendering, placement-probe, and optimization settings for AR-AWM.

Variant	EM [95% CI] $\uparrow$	ANLS [95% CI] $\uparrow$
Clean document	$0.89942\ [0.88954,0.90901]$	$0.93873\ [0.93114,0.94614]$
Prior-work 5-anchor	$0.87624\ [0.86531,0.88695]$	$0.92289\ [0.91401,0.93155]$
Area-matched ordinary	$0.88727\ [0.87702,0.89730]$	$0.93176\ [0.92381,0.93943]$
Same mask, no optimization	$0.88689\ [0.87676,0.89674]$	$0.93167\ [0.92384,0.93925]$
Same mask, random	$0.88596\ [0.87544,0.89619]$	$0.93253\ [0.92431,0.94046]$
Optimized rectangles	$0.20153\ [0.18781,0.21614]$	$0.27629\ [0.26081,0.29231]$

Appendix

Table S2: DocVQA point estimates and document-clustered bootstrap 95% confidence intervals on Qwen2.5-VL-7B.

Comparison	$\Delta$ EM [95% CI]	$p$	$\Delta$ ANLS [95% CI]	$p$
Full $-$ clean	$-0.74949\ [-0.76515,-0.73361]$	$<10^{-5}$	$-0.72681\ [-0.74349,-0.71008]$	$<10^{-5}$
Full $-$ area-matched	$-0.73733\ [-0.75300,-0.72131]$	$<10^{-5}$	$-0.71984\ [-0.73662,-0.70308]$	$<10^{-5}$
Full $-$ no optimization	$-0.73696\ [-0.75268,-0.72112]$	$<10^{-5}$	$-0.71975\ [-0.73645,-0.70307]$	$<10^{-5}$
Full $-$ random	$-0.73603\ [-0.75203,-0.71970]$	$<10^{-5}$	$-0.72062\ [-0.73751,-0.70388]$	$<10^{-5}$
Full $-$ rectangles	$-0.05160\ [-0.06659,-0.03676]$	$<10^{-5}$	$-0.06437\ [-0.08160,-0.04743]$	$<10^{-5}$
Response regional $-$ fixed regional	$-0.02430\ [-0.03749,-0.01113]$	$0.00029$	$-0.01972\ [-0.03443,-0.00508]$	$0.00865$

Appendix

Table S3: Paired DocVQA differences (first minus second), with document-clustered 95% CIs and two-sided document-level randomization tests.

Metric	Matched units	Clean	AR-AWM	$\Delta$ [95% CI]	Paired $d_{z}$ / $p$
Text edit $\downarrow$	1290 pages	0.12820	0.19897	$+0.07077\ [0.05688,0.08517]$	$+0.272$ / $\leq 10^{-5}$
Formula edit $\downarrow$	200 pages	0.15628	0.27222	$+0.11593\ [0.07575,0.15825]$	$+0.390$ / $\leq 10^{-5}$
Reading-order edit $\downarrow$	1345 pages	0.20470	0.26110	$+0.05640\ [0.04371,0.06958]$	$+0.235$ / $\leq 10^{-5}$
TEDS $\uparrow$	512 tables	0.64948	0.57062	$-0.07887\ [-0.10610,-0.05151]$	$-0.249$ / $\leq 10^{-5}$

Appendix

Table S4: Paired OmniDocBench uncertainty estimates for clean versus AR-AWM (AR-AWM minus clean). Percentile 95% CIs use 20,000 paired bootstrap replicates; two-sided label-swap tests use 100,000 replicates.

d_{z}

is the paired standardized mean difference. Per-metric denominators follow the benchmark annotations, so they are not all the same set of pages.

Layout	Area ratio (%)	Cell CV	Cos@10	Steps to cosine 0.5
Full	$7.978\pm 0.260$	$0.0100\pm 0.0076$	$0.5336\pm 0.0906$	$17.71\pm 13.64$
Global response-guided	$10.112\pm 0.959$	$1.1868\pm 0.1194$	$0.5750\pm 0.0919$	$19.07\pm 10.77$
Response-guided regional	$7.977\pm 0.260$	$0.0126\pm 0.0076$	$0.5337\pm 0.0917$	$17.80\pm 13.83$
Fixed-direction regional	$7.858\pm 0.416$	$0.0027\pm 0.0042$	$0.5348\pm 0.0897$	$17.70\pm 13.08$

Appendix

Table S5: Across-document dispersion of layout and optimization measurements on the same 1286 DocVQA documents. Values are mean

\pm

sample standard deviation.

Target / source	Condition	ANLS [95% CI] $\uparrow$
Qwen2.5-VL-7B / –	Clean	$0.93873\ [0.93114,0.94614]$
Qwen2.5-VL-7B / Qwen2.5-VL-7B	White-box	$0.21192\ [0.19625,0.22779]$
Qwen2.5-VL-7B / InternVL2.5-8B	Transfer	$0.90194\ [0.89323,0.91049]$
InternVL2.5-8B / –	Clean	$0.92151\ [0.91313,0.92964]$
InternVL2.5-8B / InternVL2.5-8B	White-box	$0.04552\ [0.03769,0.05388]$
InternVL2.5-8B / Qwen2.5-VL-7B	Transfer	$0.89706\ [0.88757,0.90623]$

Appendix

Table S6: Model-specificity ANLS with document-clustered bootstrap 95% confidence intervals. “Source” denotes the encoder used to optimize the image.

Variant	Cos@resp $\downarrow$	Cos@all $\downarrow$	Cos-gap $\uparrow$
Response-guided regional	$-0.01723$	$-0.01129$	0.00594
Fixed-direction regional	$-0.01648$	$-0.01105$	0.00543
Full AR-AWM	$-0.01707$	$-0.01132$	0.00575
Global response-guided	$-0.01651$	$-0.01330$	0.00320

Appendix

Table S7: Response-selected visual tokens on DocVQA (1286 documents,

\mathrm{top\text{-}}K

fraction

0.1

). Differences are small at the converged operating point.

Figure S1: Page-level optimization trajectories on 1286 paired DocVQA documents. Lines show means and shaded regions show 95% confidence intervals; values are carried forward after early stopping. The three regional text-shaped variants have nearly identical trajectories despite different downstream ANLS.

Metric	AR-AWM text	Solid rectangles
EM $\uparrow$	0.1499	0.2015
ANLS $\uparrow$	0.2119	0.2763
Cos@10 $\downarrow$	0.5336	0.5771
Steps to cosine 0.5 $\downarrow$	17.71	18.74
Area ratio $\downarrow$	7.98%	7.84%
Full PSNR $\uparrow$	25.55	25.45

Appendix

Table S8: Watermark-text versus non-semantic-rectangle carriers on DocVQA. Both variants use identical regional band layouts and the same representation optimizer.

Variant	OCR Preserve $\uparrow$	Full PSNR $\uparrow$	Area Ratio $\downarrow$	Text Overlap $\downarrow$
Prior-work 5-anchor watermark	0.62103	10.45	46.54%	44.08%
Area-matched ordinary watermark	0.76607	21.47	8.15%	7.85%
Same mask, no optimization	–	–	7.98%	7.28%
Same mask, random perturbation	–	–	7.98%	7.28%
AR-AWM	0.77462	25.55	7.98%	7.28%

Appendix

Table S9: Auxiliary page-level visual and support controls on DocVQA. Dashes indicate unreported OCR or Full PSNR values.

Metric	DocVQA	OmniDocBench
Page images processed	1286	1355
Successful adversarial images	1286	1354
Fallback clean images	0	1
Mean Opt-WM backward passes	88.98	93.68
Mean final $\mathcal{L}_{\mathrm{repr}}$	$-0.01171$	$-0.01194$
Early-stopped images	1284 / 1286	1351 / 1354

Appendix

Table S10: Generation statistics for the main AR-AWM runs.

Target Area	Meas. Area	EM $\uparrow$	ANLS $\uparrow$	Cos@10 $\downarrow$	PSNR(adv, $x_{\mathrm{wm}}$ ) $\uparrow$
3%	3.92%	0.14930	0.20387	0.65499	39.22
6%	7.10%	0.11664	0.18644	0.60710	37.66
10%	10.69%	0.15708	0.22503	0.58076	36.37
15%	14.74%	0.15863	0.23426	0.55129	35.39
20%	18.54%	0.15552	0.22293	0.52407	34.77

Appendix

Table S11: Full-validation DocVQA editable-area sweep. All rows use the same global response-guided layout and evaluation pipeline.

Figure S2: Editable area versus DocVQA ANLS. The connected sweep points use the same global response-guided layout and evaluation pipeline and differ only in measured editable area.

Metric	Qwen2.5-VL-7B	InternVL2.5-8B
Valid attacked images	1286 / 1286	1286 / 1286
Early-stop records	1284 (99.84%)	206 (16.02%)
Records at 1000-step cap	2	1080
Steps, mean $\pm$ std	$88.98\pm 61.06$	$956.69\pm 123.37$
Steps, median [IQR]	$74\ [66,88]$	$1000\ [1000,1000]$
Steps, 95th percentile / maximum	$158/1000$	$1000/1000$

Appendix

Table S12: Per-document DocVQA stopping and wall-clock distributions from saved generation metadata. IQR is the 25th–75th percentile interval.

Panel of Figure 24 — Figure S3: Additional OmniDocBench failure on a three-column Federal Register page. Clean parsing preserves reading order, while AR-AWM emits only the header and an empty table (reading-order edit $0.0\!\to\!1.0$ ).

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 183 papers

May 8, 2026

Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs

In this paper, we investigate the recent state-of-the-art schemes for watermarking large language models (LLMs) outputs. These techniques are claimed to be robust, scalable and production-grade, aimed at promoting responsible usage of LLMs. We analyse the effectiveness of these watermarking techniques against an extensive collection of modified text attacks, which perform targeted semantic changes without altering the general meaning of the text content. Our approach encompasses multiple attack strategies, which include lexical alterations, machine translation, and even neural paraphrasing. The attack efficacy is measured with two target criteria - successful removal of the watermark and preservation of semantic content. We evaluate semantic preservation through BERT scores, text complexity measures, grammatical errors, and Flesch Reading Ease indices. The experimental results reveal varying levels of effectiveness among different watermarking models, with the same underlying result that it is possible to remove the watermark with reasonable effort. This study sheds light on the strengths and weaknesses of existing LLM watermarking systems, suggesting how they should be constructed to improve security of available schemes.

Jonathan Hong Jin Ng, Anh Tu Ngo, Anupam Chattopadhyay

College of Computing and Data Science Nanyang Technological University Singapore

Watermark Removal LLM Watermarking Adversarial Text Attacks

Ask PDF

Oct 10, 2026

ByteMark: A Unified Byte-Level Watermark for Detection Across Language Models

As watermarking of LLM-generated text enters deployment, users increasingly need to check text whose source is unknown. Existing detectors read the watermark through a specific tokenizer or embedding pipeline, and even if all providers adopted the same watermarking method, differences in these representations would leave their watermarks incompatible, and the text alone does not reveal which representation produced it. We propose ByteMark, a unified watermarking scheme that defines its signal on the UTF-8 bytes of published text. Models with different tokenizers embed this signal under one public specification and key, and a single detector verifies the text from the text and key alone. When the tokenizer or embedding model of a baseline detector is replaced, its detection degrades sharply, whereas ByteMark, which uses neither, retains its detection rate; ByteMark also remains detectable when participating models continue or rewrite the text.

Xiaopu Wang, Runze Li

Text Watermarking LLM Watermarking Robustness of Machine-Generated Text Detection

Ask PDF

Dec 4, 2025

Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering

Document Visual Question Answering (DocVQA) enables end-to-end reasoning grounded on information present in a document input. While recent models have shown impressive capabilities, they remain vulnerable to adversarial attacks. In this work, we introduce a novel attack scenario that aims to forge document content in a visually imperceptible yet semantically targeted manner, allowing an adversary to induce specific or generally incorrect answers from a DocVQA model. We develop specialized attack algorithms that can produce adversarially forged documents tailored to different attackers' goals, ranging from targeted misinformation to systematic model failure scenarios. We demonstrate the effectiveness of our approach against two end-to-end state-of-the-art models: Pix2Struct, a vision-language transformer that jointly processes image and text through sequence-to-sequence modeling, and Donut, a transformer-based model that directly extracts text and answers questions from document images. Our findings highlight critical vulnerabilities in current DocVQA systems and call for the development of more robust defenses. We release our open source code at https://github.com/pralab/adv-docVQA.

Marco Pintore, Maura Pintor, Dimosthenis Karatzas +1

University of Cagliari, Italy · CINI, Italy · Computer Vision Center, UAB, Spain

Visual Question Answering VLM Robustness Adversarial Attacks on VLMs Adversarial Perturbations Document Image Understanding

Ask PDF

Abstract

Adversarial Attacks on VLMs

Explore similar work

Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs

ByteMark: A Unified Byte-Level Watermark for Detection Across Language Models

Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering