Misleading charts can lead vision-language models to conclusions that conflict with underlying data. Existing work focuses on inference-time strategies, while training-time supervision remains less explored. We present VizDAGuide, a DAG-structured supervision method for misleading-chart question answering. The method represents intermediate reasoning states and their dependencies as a directed acyclic graph, assigns supervision weights according to their roles, and serializes the graph into a unified training target for open-ended and multiple-choice QA with single-pass inference. Experiments on two benchmarks across Qwen3-VL and InternVL3.5 models show improvements over a content- and weight-matched Linear CoT-SFT baseline on both answer formats. Process-level evaluation shows gains in grounding and operation execution, while ablations indicate that dependency structure and weighted supervision are most effective when combined. These results support DAG-structured weighted supervision for misleading-chart question answering.
Figures & tables
Figure 1: An example multiple-choice question (MCQ) illustrating misleading-chart reasoning. The reversed axis makes the chart visually suggest an increasing trend despite decreasing underlying values.
Figure 2: Overview of VizDAGuide.
Open-ended QA
MCQ
Setting
FAA ↑
RA ↑
Acc . ↑
MCR ↓
Qwen3-VL-4B
Base
34.98
45.03
62.73
17.61
+ Answer-only-SFT
36.02
47.07
72.05
4.77
+ Linear CoT-SFT
56.25
68.28
71.93
15.45
+ U-DAG
55.31
67.89
70.68
15.91
Table 1: Comparison of supervision variants across VLM backbones.
Open-ended QA
Supervision
Gnd. ↑
Op. ↑
Corr. ↑
Pac. ↑
FC ↓
Qwen3-VL-4B
Linear CoT-SFT
57.49
68.21
88.64
48.99
11.10
U-DAG
57.57
67.99
86.66
49.72
2.67
VizDAGuide (Ours)
59.44
70.07
84.90
50.53
1.81
Qwen3-VL-8B
Table 2: Process-level results on open-ended and multiple-choice QA. Gnd., Op., Corr., Pac., FC, and Suff. denote Grounding, Operation, Correction, Process Accuracy, Clean FC, and Sufficiency.
Corr.
Answer Error
Task
Process
4B
8B
4B
8B
OEQ
Grounding
0.447
0.439
9.94
8.68
Operation
0.757
0.756
100.00
100.00
Correction
0.105
0.085
18.44
18.88
MCQ
Grounding
0.294
0.212
57.14
48.57
Operation
0.174
0.129
20.66
22.13
Table 3: Process–answer relation. Corr. measures process–answer correctness correlation; Answer Error is the final-answer error rate when only the corresponding process is incorrect.
Task
Misleader Type
Process
4B
8B
OEQ
Stacked misuse
Operation
0.963
0.960
Axis-range misuse
Operation
0.937
0.946
Dual-axis misuse
Operation
0.930
0.910
3D effect
Operation
0.919
0.909
Log-scale misuse
Operation
0.898
0.904
MCQ
Small size
Sufficiency
1.000
1.000
Table 4: Top-5 misleader types ranked by average process–answer correlation across 4B and 8B. Process denotes the highest-correlated process.
Figure 3: Fine-grained comparison between the Base model and VizDAGuide across question types, chart types, and misleading conditions.
Figure 4: Distribution of the earliest identified error stage in VizDAGuide predictions on open-ended and multiple-choice QA.
(a) Open-ended QA
Model
RA ↑
Gnd. ↑
Op. ↑
Corr. ↑
Pac. ↑
FC ↓
GPT-5.5
64.42
59.34
55.36
63.16
36.54
4.62
Gemini 3 Flash
64.28
59.45
55.86
61.48
37.12
3.76
Kimi-K3
63.83
57.78
55.18
55.98
34.57
1.57
Claude Opus 4.8
63.11
48.47
31.71
52.58
15.50
1.62
Gemini 2.5 Pro
61.63
50.71
51.69
58.18
29.11
6.14
Table 5: Reference comparison with general-purpose models on open-ended and multiple-choice misleading chart QA. External models use a unified JSON evaluation protocol, whereas our models use their native structured-output setting; the results are intended for contextual comparison rather than strict ranking.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
α/β/γ
0.25/0.35/0.40
τ
0.8
wmin/wmax
0.5/2.0
Appendix
Table 6: Hyperparameters used for DAG-guided weight assignment.
Setting
Open-ended
MCQ
LoRA rank / scaling
32 / 64
32 / 64
LoRA dropout
0.05
0.05
Learning rate
5×10−5
3×10−5
Schedule
Constant
Cosine
Warmup ratio
0
0.03
Weight decay
0
0.01
Appendix
Table 7: Training settings for open-ended and multiple-choice QA. The same task-specific configuration is used across model backbones.
Model
Comparator
Metric
Δ [95% CI]
p
Open-ended QA
Qwen3-VL-4B
Linear CoT
FAA
+1.29 [0.45, 2.12]
0.0020
Qwen3-VL-4B
Linear CoT
RA
+1.69 [0.82, 2.55]
0.0001
Qwen3-VL-8B
Linear CoT
FAA
+2.43 [1.63, 3.23]
0.0001
Qwen3-VL-8B
Linear CoT
RA
+2.42 [1.57, 3.25]
0.0001
InternVL3.5-4B
Linear CoT
FAA
+1.14 [0.33, 1.92]
0.0070
Appendix
Table 8: Paired significance analysis for the main comparisons in Table 1 . Differences are reported as VizDAGuide minus the corresponding comparator in percentage points. Confidence intervals use paired chart-group bootstrap, and p -values use two-sided chart-group permutation tests. The reported p -values are not adjusted for multiple comparisons. A value of 0.0001 is the minimum attainable permutation estimate rounded to four decimal places.
Figure 5: Qualitative comparison on a chart with a nonlinear scale. Both methods identify the misleading scale, but VizDAGuide correctly computes the vote ratio (25%), whereas Linear CoT returns an incorrect answer (200%).
Figure 6: Qualitative comparison on a pie chart with misleading slice proportions. Both methods identify the labeled market shares, but VizDAGuide returns the correct ratio (6:1), whereas Linear CoT selects an incorrect answer (4:1).
Figure 7: Qualitative comparison on a bar chart with a truncated axis. Both methods identify the truncated scale and the relevant values, but VizDAGuide returns the correct ratio (50%), whereas Linear CoT selects an incorrect answer (25%).
Figure 8: Qualitative comparison on a truncated-axis chart. VizDAGuide computes the correct ticket-sales ratio (50%) from the numerical values, whereas Linear CoT selects an incorrect answer (20%).
Figure 9: Qualitative comparison on a chart with redundant visual encoding. VizDAGuide identifies Brand B as having the largest market share (40%), whereas Linear CoT incorrectly selects Others.
Figure 10: Qualitative comparison on a chart with a misleading title. Despite the title suggesting a population decline, the plotted values increase. VizDAGuide identifies the correct trend, whereas Linear CoT predicts a decrease.
Figure 11: Qualitative comparison on a revenue chart with a misleading title. VizDAGuide identifies the increasing trend from the plotted values, whereas Linear CoT incorrectly concludes that the available information is insufficient.
Figure 12: Qualitative comparison of overall trend interpretation. VizDAGuide identifies a generally increasing trend despite a decrease in the final interval, whereas Linear CoT selects the insufficient-information option.
Figure 13: Open-ended QA comparison on a chart with an inverted axis. VizDAGuide identifies the correct minimum value (4,000,000), whereas Linear CoT returns 38,000,000.
Figure 14: Open-ended QA comparison on a 3D bar chart. VizDAGuide identifies the correct minimum CPIA rating (3.5), whereas Linear CoT returns 3 after incorrectly reading the chart values.
Figure 15: Open-ended QA comparison on a logarithmic-axis chart. VizDAGuide grounds the relevant values and computes the correct difference (1), whereas Linear CoT returns 2.
Figure 16: Open-ended QA comparison on an inverted-axis line chart. VizDAGuide correctly identifies the increasing numerical trend, whereas Linear CoT predicts the opposite trend.
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.
Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1
University of Waterloo · Vector Institute · Layer 6 AI
Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (perception), locating them and binding them to their labels (grounding), and performing single-step computations such as ranking, totals, and differences (simple reasoning). We introduce ChartProbe, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.
Mahsa Khoshnoodi, Sarah Adel Bargal
Department of Computer Science, Georgetown University
Despite the success of Vision-Language Models (VLMs), misleading charts remain a significant challenge due to their deceptive visual structures and distorted data representations. We present ChartCynics, an agentic dual-path framework designed to unmask visual deception via a "skeptical" reasoning paradigm. Unlike holistic models, ChartCynics decouples perception from verification: a Diagnostic Vision Path captures structural anomalies (e.g., inverted axes) through strategic ROI cropping, while an OCR-Driven Data Path ensures numerical grounding. To resolve cross-modal conflicts, we introduce an Agentic Summarizer optimized via a two-stage protocol: Oracle-Informed SFT for reasoning distillation and Deception-Aware GRPO for adversarial alignment. This pipeline effectively penalizes visual traps and enforces logical consistency. Evaluations on two benchmarks show that ChartCynics achieves 74.43% and 64.55% accuracy, providing an absolute performance boost of ~29% over the Qwen3-VL-8B backbone, outperforming state-of-the-art proprietary models. Our results demonstrate that specialized agentic workflows can grant smaller open-source models superior robustness, establishing a new foundation for trustworthy chart interpretation.
Yanjie Zhang, Yafei Li, Rui Sheng +5
HKUST, Hong Kong SAR, China · HKUST(GZ), Guangzhou, China