VizAnchor: Decoding Manipulation Intent from Tampering Visualizations via Dual-Anchor Reasoning

Xiaotian Zhang; Huayuan Ye; Haiyang Zhang; Chenhui Li; Changbo Wang; Sicheng Song

cs.CVAug 25, 2026

VizAnchor: Decoding Manipulation Intent from Tampering Visualizations via Dual-Anchor Reasoning

Authors: Xiaotian Zhang, Huayuan Ye, Haiyang Zhang, Chenhui Li, Changbo Wang, Sicheng Song

Organizations: School of Data Science and Engineering, East China Normal University, Shanghai, China · Division of Emerging Interdisciplinary Areas, The Hong Kong University of Science and Technology, Hong Kong, China · School of Computer Science and Technology, East China Normal University, Shanghai, China

Abstract

Data visualizations are widely used for communicating information, but they are also vulnerable to intentional manipulations that induce misleading interpretations. Existing methods focus on locating tampered regions or recovering hidden information, without explaining how the visualization has been manipulated or why the resulting changes may mislead viewers. We propose VizAnchor, a framework for visualization manipulation understanding through dual-anchor evidence construction and VLM-based reasoning. In the first stage, VizAnchor constructs a semantic anchor to recover authentic chart information and a spatial anchor to localize tampered regions. In the second stage, three specialized agents decode the manipulation. The misleader grounding agent analyzes a four-panel visual prompt to predict the misleader information. The chart narrative reconstruction agent takes the original and tampered charts as inputs and reconstructs their respective visual narratives. Finally, the intent inferring agent integrates the visual evidence and misleader information to infer the misleading intent. We further construct a dataset for tampering localization and a dataset for misleading intent inferring. Evaluation shows that VizAnchor accurately localizes manipulations and produces faithful explanations of their manipulation, misleaders, and misleading intents.

Figures & tables

Figure 1: An example of manipulation intent decoding with VizAnchor. Swapping the category labels of Africa and Oceania reverses the COVID-19 recovery comparison. Guided by semantic and spatial anchors, the multi-agent decoder grounds the manipulation, reconstructs both chart narratives, and infers its misleading intent.

Figure 2: Overview of the proposed VizAnchor framework.

Figure 3: Architecture of the Invertible Watermarking Module. The IWM Encoder embeds metadata into the chart through Transflow Blocks to produce the intermediate watermarked chart

C^{\prime}_{w}

, and then embeds the position map through Position Flow Blocks to obtain

C_{w}

. The IWM Decoder reverses these stages for position and metadata recovery.

Figure 4: Qualitative comparison of tamper localization and misleading-intent inference. Red boxes indicate correctly localized tampered regions, while green boxes highlight noise regions falsely identified as tampering.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
HiNet (81)	29.95	0.8306	0.3692
ISN (81)	34.38	0.9478	0.2340
StampOne (81)	29.44	0.9107	0.1913
StegaStamp (81)	10.32	0.5733	0.6852
WAM (32)	31.55	0.9440	0.1513
EditGuard (64)	33.00	0.8752	0.2416

Table 1: Image quality on VGD. The number in parentheses denotes the payload length in bits. Bold and underlined values indicate the best and second-best results, respectively.

Method	Modified-Pixel Ratio (%)
Method	0	15	30	45	60
HiNet (81)	98.99	97.87	94.68	87.29	74.99
ISN (81)	99.08	98.20	96.73	94.49	91.22
StampOne (81)	91.90	91.07	89.95	88.17	85.21
StegaStamp (81)	98.45	98.07	97.56	96.61	94.72
WAM (32)	92.36	90.87	88.01	83.50	75.98

Table 2: Metadata BitAcc (%) under different modified-pixel ratios on VGD.

Method	Cropped Area Ratio (%)
Method	10	20	30	40	50
WAM (32)	87.47	87.36	87.14	86.62	85.49
VisGuard (81)	97.77	98.65	98.97	98.73	97.63
VizAnchor (81)	99.55	99.61	99.59	99.55	99.36

Table 3: Metadata BitAcc (%) under different cropped-area ratios on VGD.

Method	Cropped Area Ratio (%)
Method	10	20	30	40	50
VisGuard	0.9789	0.9815	0.9824	0.9814	0.9684
VizAnchor	0.9866	0.9875	0.9860	0.9837	0.9790

Table 4: Crop localization IoU on VGD.

Method	IoU $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
VisGuard	0.9710	0.9982	0.9726	0.9851
VizAnchor	0.9789	0.9981	0.9808	0.9893

Table 5: Crop localization on crop subset of VAD-LocEval.

Method	IoU $\uparrow$	Prec. $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
EditGuard	0.5062	0.5969	0.7399	0.6289
VizDefender	0.5324	0.6178	0.7584	0.6556
VisGuard	0.6067	0.7668	0.7622	0.7152
VizAnchor	0.7418	0.8011	0.9105	0.8375

Table 6: Overall local-edit localization results across VDD-LocEval and local-edit subset of VAD-LocEval.

Figure 5: Sample-weighted tamper-type, component recognition, manipulation-process and misleading-intent results across VAD-ReasonEval and VDD-ReasonEval. Exact denotes exact-match accuracy, and M-F1 denotes Macro-F1.

Figure 6: Overall visual-evidence ablation results across VAD-ReasonEval and VDD-ReasonEval.

Figure 7: The results of the user study. Each circle represents the mean score, with error bars indicating the standard deviation. The statistical significance was assessed using the Wilcoxon signed-rank test.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Score	Criterion
0	The prediction is irrelevant or completely incorrect.
1	The prediction contains little relevant information and fails to capture the core reference meaning.
2	The prediction is partially relevant but contains a major error or omits essential information.
3	The prediction is generally correct but remains incomplete, vague, or insufficiently specific.
4	The prediction is correct and specific, with only minor omissions.
5	The prediction is fully correct, specific, complete, and semantically consistent with the reference.

Appendix

Table 7: Scoring rubric for strong-model evaluation.

Method	IoU $\uparrow$	Prec. $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
EditGuard	0.6957	0.7944	0.8130	0.7881
VizDefender	0.7019	0.7994	0.8189	0.7954
VisGuard	0.5572	0.8481	0.6339	0.6756
VizAnchor	0.7775	0.8303	0.9312	0.8609

Appendix

Table 8: Tamper-region detection results on the local-edit subset of VAD-LocEval.

Method	IoU $\uparrow$	Prec. $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
EditGuard	0.4872	0.5772	0.7326	0.6130
VizDefender	0.5154	0.5996	0.7524	0.6416
VisGuard	0.6116	0.7587	0.7750	0.7191
VizAnchor	0.7382	0.7982	0.9084	0.8352

Appendix

Table 9: Tamper-region detection results on VDD-LocEval.

Dataset	Method	Type		Component
Dataset	Method	Acc. $\uparrow$	M-F1 $\uparrow$	Exact $\uparrow$	M-F1 $\uparrow$
VAD	VizDefender	0.8250	0.8475	0.2667	0.4216
VAD	VizAnchor	0.9083	0.9056	0.5667	0.6875
VDD	VizDefender	0.7700	0.7667	0.4200	0.6332
VDD	VizAnchor	0.9000	0.8985	0.7000	0.8075

Appendix

Table 10: Tamper-type and component-recognition results. Exact denotes exact-match accuracy, and M-F1 denotes Macro-F1.

Dataset	Method	Process		Intent
Dataset	Method	Cos. FA $\uparrow$	AI FA $\uparrow$	Cos. FA $\uparrow$	AI FA $\uparrow$
VAD	VizDefender	0.5227	0.3317	0.6636	0.3850
VAD	VizAnchor	0.7300	0.9633	0.7856	0.8867
VDD	VizDefender	0.5422	0.3620	0.6532	0.5020
VDD	VizAnchor	0.6658	0.8340	0.7128	0.8170

Appendix

Table 11: Manipulation-process and misleading-intent evaluation results. AI FA is normalized to

[0,1]

Method	Type		Component
Method	Acc. $\uparrow$	M-F1 $\uparrow$	Exact $\uparrow$	M-F1 $\uparrow$
VizDefender	0.8000	0.8108	0.3364	0.5178
VizAnchor	0.9045	0.9024	0.6273	0.7420

Appendix

Table 12: Sample-weighted tamper-type and component-recognition results across VAD-ReasonEval and VDD-ReasonEval. Exact denotes exact-match accuracy, and M-F1 denotes Macro-F1.

Method	Process		Intent
Method	Cos. FA $\uparrow$	AI FA $\uparrow$	Cos. FA $\uparrow$	AI FA $\uparrow$
VizDefender	0.5316	0.3468	0.6589	0.4382
VizAnchor	0.7008	0.9045	0.7525	0.8550

Appendix

Table 13: Sample-weighted manipulation-process and misleading-intent evaluation results across VAD-ReasonEval and VDD-ReasonEval. AI FA is normalized to

[0,1]

in this aggregate table.

Panel of Figure 8 — Figure 8: Qualitative comparison for VAD sample 00002. Tamper type: modifying data-point values. Normalized AI scores (Process/Intent): VizAnchor 1.00/1.00; VizDefender 0.00/0.80.

Panel of Figure 9 — Figure 9: Qualitative comparison for VAD sample 00018. Tamper type: adding or removing data points. Normalized AI scores (Process/Intent): VizAnchor 1.00/1.00; VizDefender 0.00/0.00.

Panel of Figure 10 — Figure 10: Qualitative comparison for VAD sample 00023. Tamper type: modifying coordinate values. Normalized AI scores (Process/Intent): VizAnchor 1.00/1.00; VizDefender 0.00/0.00.

Panel of Figure 11 — Figure 11: Qualitative comparison for VAD sample 00035. Tamper type: deceptive auxiliary annotations. Normalized AI scores (Process/Intent): VizAnchor 0.80/1.00; VizDefender 0.00/0.00.

Panel of Figure 12 — Figure 12: Qualitative comparison for VAD sample 00041. Tamper type: modifying the legend. Normalized AI scores (Process/Intent): VizAnchor 1.00/1.00; VizDefender 0.00/0.00.

Panel of Figure 13 — Figure 13: Qualitative comparison for VAD sample 00053. Tamper type: hiding labels. Normalized AI scores (Process/Intent): VizAnchor 1.00/1.00; VizDefender 0.00/0.00.

Panel of Figure 14 — Figure 14: Qualitative comparison for VAD sample 00063. Tamper type: adding or removing logos. Normalized AI scores (Process/Intent): VizAnchor 1.00/1.00; VizDefender 0.00/0.00.

Panel of Figure 15 — Figure 15: Qualitative comparison for VAD sample 00073. Tamper type: data–visual disproportion. Normalized AI scores (Process/Intent): VizAnchor 1.00/1.00; VizDefender 0.40/0.00.

Panel of Figure 16 — Figure 16: Qualitative comparison for VAD sample 00088. Tamper type: modifying the colormap. Normalized AI scores (Process/Intent): VizAnchor 1.00/1.00; VizDefender 0.00/0.00.

Setting	Input	Purpose
A1	Tampered chart only	Tests how much the VLM can infer from the final manipulated chart alone.
A2	Tampered chart and mask-overlaid tampered chart	Tests whether localization evidence helps the VLM attend to suspicious regions.
A3	Semantic Anchor and tampered chart	Tests whether authentic–tampered comparison helps identify semantic deviations.
A4	Semantic Anchor, tampered chart, and Tamper Map	Tests whether authentic-chart comparison and localization evidence are complementary.
A5	Complete four-panel visual prompt	Further includes the Localized Comparison to test fine-grained manipulation understanding.

Appendix

Table 14: Visual-input settings used in the evidence ablation.

Dataset	Setting	Tamper Type		Component		Process		Intent
		Acc. $\uparrow$	Macro-F1 $\uparrow$	Exact $\uparrow$	Macro-F1 $\uparrow$	Cosine $\uparrow$	AI $\uparrow$	Cosine $\uparrow$	AI $\uparrow$
VAD	A1	0.403	0.388	0.387	0.460	0.578	0.446	0.716	0.478
VAD	A2	0.600	0.598	0.525	0.608	0.651	0.672	0.763	0.684
VAD	A3	0.767	0.753	0.625	0.706	0.718	0.892	0.779	0.806
VAD	A4	0.775	0.757	0.575	0.683	0.721	0.926	0.796	0.806
VAD	A5	0.775	0.752	0.667	0.694	0.733	0.938	0.794	0.826

Appendix

Table 15: Ablation of the visual evidence used for manipulation understanding on VAD and VDD. AI scores are normalized to

[0,1]

. The best result for each dataset and metric is shown in bold.

Setting	Type		Component		Process		Intent
	Acc. $\uparrow$	Macro-F1 $\uparrow$	Exact $\uparrow$	Macro-F1 $\uparrow$	Cos. FA $\uparrow$	AI FA $\uparrow$	Cos. FA $\uparrow$	AI FA $\uparrow$
A1	0.4816	0.4728	0.4495	0.5417	0.5762	0.4860	0.7001	0.5171
A2	0.6668	0.6675	0.5936	0.6784	0.6369	0.6547	0.7389	0.6904
A3	0.7811	0.7775	0.6575	0.7468	0.6871	0.8393	0.7458	0.7696
A4	0.7864	0.7813	0.6409	0.7274	0.6942	0.8469	0.7605	0.7678
A5	0.8000	0.7953	0.6727	0.7446	0.7035	0.8771	0.7572	0.7905

Appendix

Table 16: Overall visual-evidence ablation results across VAD-ReasonEval and VDD-ReasonEval. AI FA scores are normalized to

[0,1]

. The best result for each metric is shown in bold.

Category	Method	$n$	Type Acc.	Type F1	Comp. Exact	Comp. F1	Proc. Cos.	Proc. AI	Intent Cos.	Intent AI
Modifying data-point values	VizDefender	20	.600	.571	.500	.362	.540	.336	.722	.516
	VizAnchor	20	.900	.878	.800	.447	.674	.840	.788	.830
Adding/removing data points	VizDefender	24	.708	.829	.333	.622	.529	.384	.641	.384
	VizAnchor	24	.958	.958	.292	.894	.734	.950	.753	.750
Modifying coordinate values	VizDefender	20	.900	.878	.400	.788	.496	.250	.602	.320
	VizAnchor	20	.950	.927	1.000	1.000	.672	.940	.761	.850

Appendix

Table 17: Complete VizDefender and VizAnchor pipeline results by annotated tamper type, pooled over both evaluation datasets. Type Acc. is per-class recall and Type F1 is one-vs-rest class F1. AI scores are normalized to

[0,1]

Category	Method	$n$	Type Acc.	Type F1	Comp. Exact	Comp. F1	Proc. Cos.	Proc. AI	Intent Cos.	Intent AI
Bar	VizDefender	99	.778	.778	.313	.458	.578	.370	.708	.472
	VizAnchor	99	.909	.874	.515	.615	.719	.912	.769	.830
Line	VizDefender	32	.812	.758	.344	.453	.511	.290	.656	.394
	VizAnchor	32	.938	.725	.812	.723	.698	.862	.770	.812
Area	VizDefender	10	.800	.829	.500	.667	.579	.356	.645	.444
	VizAnchor	10	.900	.857	.800	.964	.694	.840	.744	.860

Appendix

Table 18: Complete VizDefender and VizAnchor pipeline results by chart type, pooled over both evaluation datasets. Type F1 is the Macro-F1 within each chart subgroup. AI scores are normalized to

[0,1]

Dataset	Setting	Intent Cosine $\uparrow$	Intent AI $\uparrow$
VAD	B1 IIA Only	0.7459	0.844
VAD	B2 w/o CNRA	0.7726	0.878
VAD	B3 w/o MGA	0.7582	0.860
VAD	B4 Full	0.7856	0.886
VDD	B1 IIA Only	0.6906	0.806
VDD	B2 w/o CNRA	0.7083	0.778

Appendix

Table 19: Ablation of the multi-agent reasoning pipeline for misleading-intent inference on VAD and VDD. Intent AI denotes the strong-model evaluation score normalized to

[0,1]

Setting	Intent Cos. FA $\uparrow$	Intent AI FA $\uparrow$
B1 IIA Only	0.7208	0.826
B2 w/o CNRA	0.7434	0.832
B3 w/o MGA	0.7337	0.806
B4 Full	0.7525	0.856

Appendix

Table 20: Sample-weighted overall ablation results for the multi-agent reasoning pipeline across VAD-ReasonEval and VDD-ReasonEval. Intent AI FA is normalized to

[0,1]

Figure 18: Compound-relation error for VAD sample 00100. The title changes from urban to rural population share while the United States and China are also recolored. VizAnchor recognizes the altered topic and both affected countries but does not match the benchmark reference’s intended direction of the cross-country urbanization comparison. Intent cosine remains 0.834, while the normalized AI score is 0.20.

Task	Method	Mean	SD	95% CI
Process	VizDefender	$-0.547$	0.795	$[-0.844,-0.250]$
Process	Vanilla VLM	1.197	0.726	$[0.926,1.468]$
Process	VizAnchor	1.437	0.624	$[1.203,1.670]$
Intent	VizDefender	$-0.010$	0.632	$[-0.246,0.226]$
Intent	Vanilla VLM	1.027	0.717	$[0.759,1.294]$
Intent	VizAnchor	1.370	0.659	$[1.124,1.616]$

Appendix

Table 21: Participant-level descriptive statistics. Scores range from

-2

to 2.

Task	Comparison	$W$	Holm $p$	$r_{\mathrm{rb}}$
Process	VizAnchor vs. VizDefender	1.0	$1.26\times 10^{-5}$	0.995
Process	VizAnchor vs. Vanilla VLM	25.0	0.00472	0.737
Process	Vanilla VLM vs. VizDefender	5.0	$1.29\times 10^{-5}$	0.975
Intent	VizAnchor vs. VizDefender	0.0	$1.12\times 10^{-5}$	1.000
Intent	VizAnchor vs. Vanilla VLM	45.5	0.00477	0.670
Intent	Vanilla VLM vs. VizDefender	11.0	$1.58\times 10^{-5}$	0.949

Appendix

Table 22: Two-sided paired Wilcoxon signed-rank tests. Holm correction is applied across the three pairwise comparisons within each task.

Data Visualization

Data visualization studies how visual representations help people explore, interpret, communicate, and reason about data.

See all 121 papers

When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations
Mar 23, 2026Harsh Nishant Lalai, Raj Sanjay Shah, Hanspeter Pfister +2VLM Evaluation Chart Understanding
Birla Institute of Technology and Science, Pilani · Georgia Institute of Technology · Harvard University
Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering
Mar 30, 2026Yanjie Zhang, Yafei Li, Rui Sheng +5VLM Robustness Chart QA
HKUST, Hong Kong SAR, China · HKUST(GZ), Guangzhou, China
Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents
Apr 22, 2026Yiyang Lu, Woong Shin, Ahmad Maroof Karimi +3Scientific Visualization Data Visualization
William & Mary · Oak Ridge National Laboratory

Mar 23, 2026J/K move · Enter open · S save

When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

Harsh Nishant Lalai, Raj Sanjay Shah, Hanspeter Pfister, Sashank Varma+1

Birla Institute of Technology and Science, Pilani · Georgia Institute of Technology · Harvard University

VLM Evaluation Chart Understanding

Open paper AskPDF arXiv

Visualizations help communicate data insights, but deceptive data representations can distort their interpretation and propagate misinformation. While recent Vision Language Models (VLMs) perform well on many chart understanding tasks, their ability to detect misleading visualizations, especially when deception arises from subtle reasoning errors in captions, remains poorly understood. Here, we evaluate VLMs on misleading visualization-caption pairs grounded in a fine-grained taxonomy of reasoning errors (e.g., Cherry-picking, Causal inference) and visualization design errors (e.g., Truncated axis, Dual axis, inappropriate encodings). To this end, we develop a benchmark that combines real-world visualization with human-authored, curated misleading captions designed to elicit specific reasoning and visualization error types, enabling controlled analysis across error categories and modalities of misleadingness. Evaluating many commercial and open-source VLMs, we find that models detect visual design errors substantially more reliably than reasoning-based misinformation, and frequently misclassify non-misleading visualizations as deceptive. Overall, our work fills a gap between coarse detection of misleading content and the attribution of the specific reasoning or visualization errors that give rise to it.

Abstract

Data Visualization

Explore similar work

When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering

Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents