Visually Legible but Adversarial: Watermark-Constrained Attacks on End-to-End VLMs for Document Understanding

Qingzhe Han; Fan Dang; Shengqi Chen; Xinyi Sun; Haoyu Xiong; Lingkun Li; Di Zhang

cs.CVOct 11, 2026

Visually Legible but Adversarial: Watermark-Constrained Attacks on End-to-End VLMs for Document Understanding

Authors: Qingzhe Han, Fan Dang, Shengqi Chen, Xinyi Sun, Haoyu Xiong, Lingkun Li, Di Zhang

Organizations: Beijing Jiaotong University

Abstract

Visible watermarks convey document ownership, confidentiality, and status, but can also carry adversarial signals. We introduce Auto-Regional Adversarial Watermarking (AR-AWM), a model-aware white-box attack on end-to-end visual language models (VLMs) for document understanding. AR-AWM constructs a small, spatially distributed watermark-text support and optimizes only its pixels to disrupt visual representations. The attack is untargeted and question-agnostic: generation uses no questions, answers, parsing prompts, language-model gradients, or language-model outputs. Each attacked page is reused across downstream queries. On Qwen2.5-VL-7B, a watermark support covering 7.98% of page pixels under the reporting threshold reduces DocVQA exact match from 0.899 to 0.150 and average normalized Levenshtein similarity (ANLS) from 0.939 to 0.212. Selected qualitative cases retain visually legible source fields. The same support without optimization and with matched random perturbations retains ANLS of 0.932 and 0.933, respectively, identifying adversarial optimization as the primary cause of degradation. Layout-matched optimized rectangles yield ANLS of 0.276, showing an incremental benefit from text-shaped geometry. AR-AWM also degrades all reported text, formula, reading-order, and table metrics on OmniDocBench. Separate white-box optimization on InternVL2.5-8B reduces ANLS from 0.922 to 0.046, while weak cross-model transfer reveals strong encoder specificity. These results identify conventional watermark-shaped supports as an adversarial surface for document VLMs in the evaluated white-box setting.luated white-box setting.

Figures & tables

Figure 1: Overview of AR-AWM. Given a clean page image $x$ , AR-AWM constructs and fixes a spatially distributed watermark-text support $\Omega_{w}$ , renders the initial watermarked page image $x_{\mathrm{wm}}$ , and optimizes only pixels inside the fixed support. A frozen visual encoder provides the clean and adversarial visual tokens, which the representation objective drives apart without using downstream questions or parsing prompts. The final adversarial page image is reused for question answering and document parsing; in the presented qualitative cases, its underlying content remains visually legible despite downstream errors.

Figure 2: Automatic, model-informed construction of the fixed watermark support $\Omega_{w}$ . A temporary encoder-response placement probe first produces the empirical response map $S$ . The page is then partitioned into adaptive regions, candidate watermark-text bands are generated and ranked within each region, and the union of the selected bands defines $\Omega_{w}$ . The temporary probe trajectory is discarded before support-constrained representation optimization begins.

Variant	EM $\uparrow$	ANLS $\uparrow$	Area $\downarrow$
Clean document	0.899	0.939	–
Prior-work 5-anchor	0.876	0.923	46.54%
Area-matched ordinary	0.887	0.931	8.15%
Same mask, unoptimized	0.887	0.932	7.98%
Same mask, random	0.886	0.933	7.98%
Optimized rectangles	0.202	0.276	7.84%

Table 1: DocVQA results on Qwen2.5-VL-7B. Identical-mask rows hold the 7.98% editable support fixed. Bold in downstream metrics denotes the strongest attack (worst performance).

Variant	Area $\downarrow$	Full PSNR $\uparrow$	OCR Preserve $\uparrow$	Text Overlap $\downarrow$
5-anchor	46.54%	10.45	0.621	44.08%
Area-matched	8.15%	21.47	0.766	7.85%
AR-AWM	7.98%	25.55	0.775	7.28%

Table 2: Automatic page-level visual and support measurements on DocVQA. Full PSNR does not measure in-support imperceptibility, and OCR Preserve is an automated text-recoverability proxy; Figure 4 provides an illustrative visual-legibility case.

Variant	Text edit $\downarrow$	Formula edit $\downarrow$	Order edit $\downarrow$	TEDS $\uparrow$	Overall $\uparrow$
Clean	0.128	0.156	0.205	0.649	53.210
5-anchor	0.139	0.170	0.210	0.616	51.460
AR-AWM	0.199	0.272	0.261	0.571	48.010

Table 3: OmniDocBench results on Qwen2.5-VL-7B. Lower is better for edit distances; higher is better otherwise. Bold in downstream metrics denotes the strongest attack (worst performance).

Panel of Figure 3 — Figure 3: Illustrative OmniDocBench formula failure on a Putnam exam page. The source statement and display formula remain visually recoverable in the attacked crop, but Qwen2.5-VL-7B replaces them with an unrelated claim (display-formula edit $0.0\!\to\!1.0$ ). Parsed excerpts are shortened for layout.

Panel of Figure 4 — Figure 4: Visually legible yet adversarial DocVQA case (doc. 2192). Panels (c,d) show that the underlying page content remains visually legible after the familiar CONFIDENTIAL overlay is optimized. In this illustrative case, the requested fields remain recoverable from the page, but Qwen2.5-VL-7B maps four distinct questions on the same attacked document to one identifier. Clean answers are lightly shortened for layout.

Panel of Figure 5 — Figure 5: Main causal and generalization evidence. (a) Only optimized perturbations collapse ANLS; the text-shaped carrier outperforms matched rectangles. (b) The three regional layouts have similar global drift but different ANLS; global placement has higher Cos@10. (c) Strong degradation begins at 3.92% area and is non-monotonic. (d) White-box effects are large, whereas cross-model transfer is weak.

Placement	EM $\uparrow$	ANLS $\uparrow$	Area $\downarrow$	Cell CV $\downarrow$
Full	0.150	0.212	7.98%	0.010
Global	0.151	0.215	10.11%	1.187
Response only	0.145	0.207	7.98%	0.013
Fixed direction	0.169	0.227	7.86%	0.003

Table 4: DocVQA placement designs. Response ranks bands by response score; coverage adds spatial coverage. Lower Cell CV means more balanced support. Bold in downstream metrics denotes the strongest attack (worst performance).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Watermark text	CONFIDENTIAL
Font / per-cell font size	DejaVu Sans Bold / $\max(18,\operatorname{round}(0.18\min(w_{c},h_{c})))$
Mask rendering / counting threshold	Pillow 8-bit alpha, bicubic rotation / $8/255$
Watermark color / opacity	gray $(80,80,80)$ / $0.32$
Regional grid	automatic, at most $12$ cells
Target cell size / aspect	$224$ / $1.0$

Appendix

Table S1: Numerical rendering, placement-probe, and optimization settings for AR-AWM.

Variant	EM [95% CI] $\uparrow$	ANLS [95% CI] $\uparrow$
Clean document	$0.89942\ [0.88954,0.90901]$	$0.93873\ [0.93114,0.94614]$
Prior-work 5-anchor	$0.87624\ [0.86531,0.88695]$	$0.92289\ [0.91401,0.93155]$
Area-matched ordinary	$0.88727\ [0.87702,0.89730]$	$0.93176\ [0.92381,0.93943]$
Same mask, no optimization	$0.88689\ [0.87676,0.89674]$	$0.93167\ [0.92384,0.93925]$
Same mask, random	$0.88596\ [0.87544,0.89619]$	$0.93253\ [0.92431,0.94046]$
Optimized rectangles	$0.20153\ [0.18781,0.21614]$	$0.27629\ [0.26081,0.29231]$

Appendix

Table S2: DocVQA point estimates and document-clustered bootstrap 95% confidence intervals on Qwen2.5-VL-7B.

Comparison	$\Delta$ EM [95% CI]	$p$	$\Delta$ ANLS [95% CI]	$p$
Full $-$ clean	$-0.74949\ [-0.76515,-0.73361]$	$<10^{-5}$	$-0.72681\ [-0.74349,-0.71008]$	$<10^{-5}$
Full $-$ area-matched	$-0.73733\ [-0.75300,-0.72131]$	$<10^{-5}$	$-0.71984\ [-0.73662,-0.70308]$	$<10^{-5}$
Full $-$ no optimization	$-0.73696\ [-0.75268,-0.72112]$	$<10^{-5}$	$-0.71975\ [-0.73645,-0.70307]$	$<10^{-5}$
Full $-$ random	$-0.73603\ [-0.75203,-0.71970]$	$<10^{-5}$	$-0.72062\ [-0.73751,-0.70388]$	$<10^{-5}$
Full $-$ rectangles	$-0.05160\ [-0.06659,-0.03676]$	$<10^{-5}$	$-0.06437\ [-0.08160,-0.04743]$	$<10^{-5}$
Response regional $-$ fixed regional	$-0.02430\ [-0.03749,-0.01113]$	$0.00029$	$-0.01972\ [-0.03443,-0.00508]$	$0.00865$

Appendix

Table S3: Paired DocVQA differences (first minus second), with document-clustered 95% CIs and two-sided document-level randomization tests.

Metric	Matched units	Clean	AR-AWM	$\Delta$ [95% CI]	Paired $d_{z}$ / $p$
Text edit $\downarrow$	1290 pages	0.12820	0.19897	$+0.07077\ [0.05688,0.08517]$	$+0.272$ / $\leq 10^{-5}$
Formula edit $\downarrow$	200 pages	0.15628	0.27222	$+0.11593\ [0.07575,0.15825]$	$+0.390$ / $\leq 10^{-5}$
Reading-order edit $\downarrow$	1345 pages	0.20470	0.26110	$+0.05640\ [0.04371,0.06958]$	$+0.235$ / $\leq 10^{-5}$
TEDS $\uparrow$	512 tables	0.64948	0.57062	$-0.07887\ [-0.10610,-0.05151]$	$-0.249$ / $\leq 10^{-5}$

Appendix

Table S4: Paired OmniDocBench uncertainty estimates for clean versus AR-AWM (AR-AWM minus clean). Percentile 95% CIs use 20,000 paired bootstrap replicates; two-sided label-swap tests use 100,000 replicates.

d_{z}

is the paired standardized mean difference. Per-metric denominators follow the benchmark annotations, so they are not all the same set of pages.

Layout	Area ratio (%)	Cell CV	Cos@10	Steps to cosine 0.5
Full	$7.978\pm 0.260$	$0.0100\pm 0.0076$	$0.5336\pm 0.0906$	$17.71\pm 13.64$
Global response-guided	$10.112\pm 0.959$	$1.1868\pm 0.1194$	$0.5750\pm 0.0919$	$19.07\pm 10.77$
Response-guided regional	$7.977\pm 0.260$	$0.0126\pm 0.0076$	$0.5337\pm 0.0917$	$17.80\pm 13.83$
Fixed-direction regional	$7.858\pm 0.416$	$0.0027\pm 0.0042$	$0.5348\pm 0.0897$	$17.70\pm 13.08$

Appendix

Table S5: Across-document dispersion of layout and optimization measurements on the same 1286 DocVQA documents. Values are mean

\pm

sample standard deviation.

Target / source	Condition	ANLS [95% CI] $\uparrow$
Qwen2.5-VL-7B / –	Clean	$0.93873\ [0.93114,0.94614]$
Qwen2.5-VL-7B / Qwen2.5-VL-7B	White-box	$0.21192\ [0.19625,0.22779]$
Qwen2.5-VL-7B / InternVL2.5-8B	Transfer	$0.90194\ [0.89323,0.91049]$
InternVL2.5-8B / –	Clean	$0.92151\ [0.91313,0.92964]$
InternVL2.5-8B / InternVL2.5-8B	White-box	$0.04552\ [0.03769,0.05388]$
InternVL2.5-8B / Qwen2.5-VL-7B	Transfer	$0.89706\ [0.88757,0.90623]$

Appendix

Table S6: Model-specificity ANLS with document-clustered bootstrap 95% confidence intervals. “Source” denotes the encoder used to optimize the image.

Variant	Cos@resp $\downarrow$	Cos@all $\downarrow$	Cos-gap $\uparrow$
Response-guided regional	$-0.01723$	$-0.01129$	0.00594
Fixed-direction regional	$-0.01648$	$-0.01105$	0.00543
Full AR-AWM	$-0.01707$	$-0.01132$	0.00575
Global response-guided	$-0.01651$	$-0.01330$	0.00320

Appendix

Table S7: Response-selected visual tokens on DocVQA (1286 documents,

\mathrm{top\text{-}}K

fraction

0.1

). Differences are small at the converged operating point.

Figure S1: Page-level optimization trajectories on 1286 paired DocVQA documents. Lines show means and shaded regions show 95% confidence intervals; values are carried forward after early stopping. The three regional text-shaped variants have nearly identical trajectories despite different downstream ANLS.

Metric	AR-AWM text	Solid rectangles
EM $\uparrow$	0.1499	0.2015
ANLS $\uparrow$	0.2119	0.2763
Cos@10 $\downarrow$	0.5336	0.5771
Steps to cosine 0.5 $\downarrow$	17.71	18.74
Area ratio $\downarrow$	7.98%	7.84%
Full PSNR $\uparrow$	25.55	25.45

Appendix

Table S8: Watermark-text versus non-semantic-rectangle carriers on DocVQA. Both variants use identical regional band layouts and the same representation optimizer.

Variant	OCR Preserve $\uparrow$	Full PSNR $\uparrow$	Area Ratio $\downarrow$	Text Overlap $\downarrow$
Prior-work 5-anchor watermark	0.62103	10.45	46.54%	44.08%
Area-matched ordinary watermark	0.76607	21.47	8.15%	7.85%
Same mask, no optimization	–	–	7.98%	7.28%
Same mask, random perturbation	–	–	7.98%	7.28%
AR-AWM	0.77462	25.55	7.98%	7.28%

Appendix

Table S9: Auxiliary page-level visual and support controls on DocVQA. Dashes indicate unreported OCR or Full PSNR values.

Metric	DocVQA	OmniDocBench
Page images processed	1286	1355
Successful adversarial images	1286	1354
Fallback clean images	0	1
Mean Opt-WM backward passes	88.98	93.68
Mean final $\mathcal{L}_{\mathrm{repr}}$	$-0.01171$	$-0.01194$
Early-stopped images	1284 / 1286	1351 / 1354

Appendix

Table S10: Generation statistics for the main AR-AWM runs.

Target Area	Meas. Area	EM $\uparrow$	ANLS $\uparrow$	Cos@10 $\downarrow$	PSNR(adv, $x_{\mathrm{wm}}$ ) $\uparrow$
3%	3.92%	0.14930	0.20387	0.65499	39.22
6%	7.10%	0.11664	0.18644	0.60710	37.66
10%	10.69%	0.15708	0.22503	0.58076	36.37
15%	14.74%	0.15863	0.23426	0.55129	35.39
20%	18.54%	0.15552	0.22293	0.52407	34.77

Appendix

Table S11: Full-validation DocVQA editable-area sweep. All rows use the same global response-guided layout and evaluation pipeline.

Figure S2: Editable area versus DocVQA ANLS. The connected sweep points use the same global response-guided layout and evaluation pipeline and differ only in measured editable area.

Metric	Qwen2.5-VL-7B	InternVL2.5-8B
Valid attacked images	1286 / 1286	1286 / 1286
Early-stop records	1284 (99.84%)	206 (16.02%)
Records at 1000-step cap	2	1080
Steps, mean $\pm$ std	$88.98\pm 61.06$	$956.69\pm 123.37$
Steps, median [IQR]	$74\ [66,88]$	$1000\ [1000,1000]$
Steps, 95th percentile / maximum	$158/1000$	$1000/1000$

Appendix

Table S12: Per-document DocVQA stopping and wall-clock distributions from saved generation metadata. IQR is the 25th–75th percentile interval.

Panel of Figure 24 — Figure S3: Additional OmniDocBench failure on a three-column Federal Register page. Clean parsing preserves reading order, while AR-AWM emits only the header and an empty table (reading-order edit $0.0\!\to\!1.0$ ).

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 183 papers

Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs
May 8, 2026Jonathan Hong Jin Ng, Anh Tu Ngo, Anupam ChattopadhyayWatermark Removal LLM Watermarking
College of Computing and Data Science Nanyang Technological University Singapore
ByteMark: A Unified Byte-Level Watermark for Detection Across Language Models
Oct 10, 2026Xiaopu Wang, Runze LiText Watermarking LLM Watermarking
Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering
Dec 4, 2025Marco Pintore, Maura Pintor, Dimosthenis Karatzas +1Visual Question Answering VLM Robustness
University of Cagliari, Italy · CINI, Italy · Computer Vision Center, UAB, Spain

May 8, 2026J/K move · Enter open · S save

Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs

Jonathan Hong Jin Ng, Anh Tu Ngo, Anupam Chattopadhyay

College of Computing and Data Science Nanyang Technological University Singapore

Watermark Removal LLM Watermarking Adversarial Text Attacks

Open paper AskPDF arXiv

In this paper, we investigate the recent state-of-the-art schemes for watermarking large language models (LLMs) outputs. These techniques are claimed to be robust, scalable and production-grade, aimed at promoting responsible usage of LLMs. We analyse the effectiveness of these watermarking techniques against an extensive collection of modified text attacks, which perform targeted semantic changes without altering the general meaning of the text content. Our approach encompasses multiple attack strategies, which include lexical alterations, machine translation, and even neural paraphrasing. The attack efficacy is measured with two target criteria - successful removal of the watermark and preservation of semantic content. We evaluate semantic preservation through BERT scores, text complexity measures, grammatical errors, and Flesch Reading Ease indices. The experimental results reveal varying levels of effectiveness among different watermarking models, with the same underlying result that it is possible to remove the watermark with reasonable effort. This study sheds light on the strengths and weaknesses of existing LLM watermarking systems, suggesting how they should be constructed to improve security of available schemes.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs

ByteMark: A Unified Byte-Level Watermark for Detection Across Language Models

Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering