Vision-language models (VLMs) are increasingly used to interpret charts and generate natural-language explanations for socially consequential data. However, they may produce different narratives for the same chart when only the referenced social group changes, reinforcing stereotypes and misleading decisions. Despite these risks, no benchmark exists for systematically evaluating bias in chart interpretation across social dimensions. We introduce ChartBias, the first benchmark for auditing bias in VLM-based chart interpretation. ChartBias contains 820 manually curated real-world charts spanning six attributes: race, income, age, religion, immigration status, and gender, yielding 4,319 valid chart, attribute instances and 8,638 paired generations where the chart is fixed and only the group term is swapped. Across 12 proprietary and open-source VLMs, totaling 155,484 model responses, we find three widespread failure modes: narrative shift (same chart, different narratives), group hallucination (assigning a chart to a group without evidence), and preference polarity (favourable trends often linked to one group). We further propose a multi-agent mitigation framework that serves as a strong baseline by separating chart-grounded evidence extraction from group-conditioned generation and using a counterfactual judge to verify that group-driven differences are supported by the chart. The framework substantially reduces narrative shift while preserving chart-grounded reasoning. Our findings show that evaluating chart understanding requires measuring not only accuracy, but also fairness and consistency across social groups. We release ChartBias at https://github.com/vis-nlp/ChartBiasBench.
Figures & tables
Figure 1: Examples of the three ChartBias evaluation settings using GPT-4o outputs. Despite identical chart evidence, the model produces different narratives under race swaps, assigns gender without supporting evidence, and associates negative trends more strongly with immigrants than positive trends.
Figure 2: ChartBias construction pipeline. Charts are collected from four public sources, screened by two annotators, and annotated for attribute eligibility, chart type, topic, and polarity. The resulting benchmark contains 820 charts across six sensitive attributes and supports three evaluation settings: narrative shift, group hallucination, and preference polarity.
Sources
Attributes
Chart Types
VisText
OECD
Statista
Pew
Race
Income
Age
Religion
Immigration
Gender
Line
Bar
Area
Other
325 (40%)
213 (26%)
201 (25%)
81 (10%)
701 (85%)
675 (82%)
606 (74%)
809 (99%)
802 (98%)
726 (89%)
494 (60%)
191 (23%)
127 (15%)
8 (1%)
Table 1: Summary of source composition, attribute eligibility, and chart type distribution in the final dataset.
Figure 3: Overview of the proposed multi-agent counterfactual mitigation framework. The framework masks sensitive attributes, extracts shared chart-grounded evidence, generates paired group-conditioned outputs, and checks their consistency before producing the final responses.
Race
Income
Age
Gender
Religion
Immigration
Model
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
GPT-4o
0.15
73.98
0.15
72.90
0.14
57.10
0.15
63.70
0.17
85.29
0.18
89.15
GPT-5
0.21
89.59
0.20
86.37
0.21
66.50
0.18
66.80
0.22
95.17
0.22
96.59
Gemini-3-flash
0.16
84.97
0.17
87.15
0.08
20.63
✓
0.10
24.10
0.22
92.96
0.19
87.95
Claude-4-Sonnet
0.02
8.77
✓
0.03
9.82
✓
0.14
55.94
0.12
58.54
0.25
86.52
0.21
83.53
Qwen3-VL-2B
0.16
79.60
0.15
71.26
0.07
19.47
✓
0.08
25.76
✓
0.18
82.29
0.17
76.91
Table 2: Narrative shift under counterfactual sensitive-attribute swaps. The first block reports closed/proprietary models and the second block reports open-weight models. Diss. ↓ reports mean semantic dissimilarity between paired generations, and % > 0.1 ↓ reports the share of pairs exceeding the 0.10 threshold. Lower values indicate stronger counterfactual consistency; bold values mark the largest shifts within each attribute. Two One-Sided Tests TOST reports practical equivalence with a 0.10 bound: ✓ denotes equivalence and denotes non-equivalence.
Race
Income
Age
Gender
Religion
Immigration
Model
White
Black
Abs.
Low
High
Abs.
Young
Old
Abs.
Male
Female
Abs.
Christ.
Muslim
Abs.
Citizen
Immig.
Abs.
GPT-4o
71.83
24.89
3.28
63.70
36.30
0.00
65.18
34.82
0.00
41.60
58.40
0.00
75.70
24.30
0.00
79.39
20.61
0.00
GPT-5
63.50
36.50
0.00
69.92
30.08
0.00
47.52
52.34
0.14
61.43
38.57
0.00
66.63
29.15
4.22
80.42
18.08
1.50
Gemini-3-flash
57.77
42.23
0.00
58.07
41.93
0.00
50.17
49.83
0.00
54.41
45.59
0.00
57.11
42.89
0.00
73.57
26.43
0.00
Claude-4-Sonnet
50.90
49.10
0.00
51.60
48.40
0.00
41.91
57.59
0.50
59.64
39.95
0.41
52.66
46.97
0.37
68.58
31.42
0.00
Qwen3-VL-2B
07.00
93.00
0.00
10.00
90.00
0.00
55.78
44.22
0.00
50.96
49.04
0.00
52.52
47.48
0.00
61.45
38.55
0.00
Table 3: Group hallucination under unsupported sensitive-attribute attribution. For each model and attribute, we report the percentage of outputs assigned to each group and the percentage of abstentions ( Abs. ). Because the visual evidence does not support group-level inference, the expected behavior is to abstain rather than assign a sensitive-attribute label; bold values mark the largest attribution or abstention rates within each attribute.
Race (W)
Income (L)
Age (Y)
Gender (M)
Religion (C)
Immigration (Cit.)
Model
Pos.
Neu.
Neg.
Pos.
Neu.
Neg.
Pos.
Neu.
Neg.
Pos.
Neu.
Neg.
Pos.
Neu.
Neg.
Pos.
Neu.
Neg.
GPT-4o
85.00
71.96
50.30
58.74
51.53
87.01
64.80
60.80
70.40
44.30
41.20
37.80
80.95
74.16
71.40
81.10
81.10
74.50
GPT-5
63.00
67.80
58.60
56.30
72.20
87.70
42.60
47.50
55.00
67.30
53.20
61.20
67.60
70.70
60.00
78.70
79.80
84.10
Gemini-3-flash
54.70
61.20
58.20
77.40
49.30
40.10
50.40
47.00
53.30
57.90
51.90
51.70
55.60
63.30
51.90
69.90
73.80
79.30
Claude-4-Sonnet
52.20
49.79
50.28
54.20
50.22
49.42
45.30
36.50
42.60
66.30
55.10
54.20
49.70
59.80
48.60
68.40
70.20
66.80
Qwen3-VL-2B
08.36
06.75
05.08
07.00
10.00
14.60
53.90
53.60
60.90
49.80
51.40
52.20
48.20
52.10
52.40
67.30
52.20
56.70
Table 4: Polarity-conditioned group preference attribution across six sensitive attributes. Values report the percentage of outputs assigned to the first group in each pair across positive, neutral, and negative chart contexts: White (W), low-income (L), young (Y), male (M), Christian (C), and citizen (Cit.).
Race
Income
Age
Gender
Religion
Immigration
Model
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
Diss. ↓
% > 0.1 ↓
TOST
GPT-4o
0.15
73.98
0.15
72.90
0.14
57.10
0.15
63.70
0.17
85.29
0.18
89.15
GPT-4o-Multistep
0.06
16.54
✓
0.05
16.22
✓
0.04
16.67
✓
0.05
17.82
✓
0.06
19.14
✓
0.06
19.47
✓
Gemini-3-flash
0.16
84.97
0.17
87.15
0.08
20.63
0.10
24.10
0.22
92.96
0.19
87.95
Gemini-3-flash-Multistep
0.04
3.26
✓
0.01
0.68
✓
0.04
3.47
✓
0.04
4.08
✓
0.05
5.95
✓
0.04
4.03
✓
Gemini-3 w/o masking
0.08
23.05
0.04
3.93
✓
0.06
26.44
0.05
8.53
✓
0.05
7.17
✓
0.04
5.91
✓
Table 5: Mitigating narrative shift under sensitive-attribute swaps. For each attribute, Diss. ↓ reports mean semantic dissimilarity between paired generations, and % > 0.1 ↓ reports the share of pairs exceeding the 0.10 threshold.
Figure 4: Qualitative analysis of narrative shift across six sensitive attributes. For each example, the chart image and prompt template are held fixed while only the referenced group term is swapped. Highlighted phrases mark the group-conditioned shifts in interpretation.
Figure 5: Heatmap of narrative shift across models and sensitive attributes. Each cell reports the mean semantic dissimilarity between paired responses.
Benchmark
# charts
Real-world
Attrib.
Narr.
Halluc.
Polarity
Chart-to-Text obeid2020chart
44K
Mixed
—
—
—
—
VisText tang2023vistext
12K
✓
—
—
—
—
Mahbub et al. mahbub2025charts
100
✓
1 (geo-econ)
✓
—
—
ChartBias (ours)
820
✓
6
✓
✓
✓
Table 6: Comparison of ChartBias with prior chart-understanding and chart-to-text benchmarks. ChartBias is the only benchmark that combines real-world charts, multi-sensitive attribute (Attrib.) annotation, and supports the analysis of narrative shift (Narr.) support, group hallucination (Halluc.), and preference polarity (Polarity).
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Attribute
Definition in ChartBias
Paired group values
Race
Racial group referenced in the prompt for counterfactual evaluation.
White / Black
Income
Socioeconomic status referenced in the prompt.
High-income / Low-income
Age
Age group referenced in the prompt.
Young / Old
Gender
Gender group referenced in the prompt.
Male / Female
Religion
Religious affiliation referenced in the prompt.
Christian / Muslim
Immigration status
Citizenship or migration status referenced in the prompt.
Citizen / Immigrant
Appendix
Table 7: The six sensitive attributes annotated in ChartBias and their paired binary group values. Pairs are chosen to be common and recognizable rather than exhaustive across all possible identities.
Attribute
Total Samples
Positive
Neutral
Negative
Race
701
287
237
177
Income
675
269
229
177
Age
606
256
181
169
Religion
809
340
259
210
Immigration
802
342
252
208
Gender
726
309
216
201
Appendix
Table 8: Per-attribute polarity distribution in ChartBias. For each sensitive attribute, we report the total number of valid chart–attribute instances and the number of instances labeled as positive, neutral, and negative. Polarity is assigned based on the chart content rather than the sentiment of model-generated text.
Figure 6: Representative examples of polarity annotation from ChartBias. The left chart is labeled positive because it shows an improving or favorable trend, the middle chart is labeled neutral because it shows no clear favorable or unfavorable directional change, and the right chart is labeled negative because it shows a worsening or unfavorable trend. Polarity is assigned based on the chart content itself, not on the sentiment of the generated text.
Figure 7: Attribute eligibility example for the race attribute. The left chart is ineligible because it explicitly mentions racial groups, including White and Black, in the visual content. The right chart is eligible because it contains no race-specific reference, allowing paired counterfactual prompts such as White versus Black while keeping the chart fixed.
Figure 8: Representative examples of ChartBias diversity. The figure shows charts from different sources (OECD, Pew, Statista, and VisText), chart types (line, bar, pie, and area), and topics, including patent applications, refugee resettlement, out-of-school children, school shootings, Zika vaccination, and youth unemployment. The examples also illustrate variation in visual complexity.
Figure 9: Heatmap of preference-polarity gaps across models and sensitive attributes. Each cell reports the absolute difference between the model’s first-group attribution rate on positive versus negative charts. Darker cells indicate stronger polarity-linked group preference.
Figure 10: Qualitative analysis of mitigation outcomes. The mitigation framework reduces narrative shift in some examples, but residual bias remains in others, where paired outputs still assign different severity, framing, or evaluative emphasis to the same chart evidence.
Figure 11: Attribute-wise mitigation gains for narrative shift. Each cell shows the reduction in mean semantic dissimilarity after applying the multi-agent mitigation framework to GPT-4o and Gemini-3-Flash. Darker cells indicate larger improvements.
Figure 12: Prompt template for the narrative-shift setting. The chart image and prompt template are fixed while only the group term [GROUP] is swapped. The analysis instruction is specialized by chart type, and the model is asked to produce a five-sentence chart interpretation with an opinion inside <opinion> tags.
Figure 13: Prompt template for the group-selection setting. The model first produces a factual chart analysis without mentioning either group. The factual analysis is then used in a second prompt that asks the model to select the group.
Figure 14: Main prompt templates for the multi-agent mitigation framework. Attribute masking is deterministic. The chart evidence extraction prompt produces group-neutral chart facts, and the group-conditioned generation prompt uses the same extracted facts to produce paired outputs while treating the group name only as a framing reference.
Figure 15: Prompt template for the consistency checking and rewrite stage. The programmatic judge flags unsupported differences using stance agreement, semantic similarity, and sentiment difference. When a pair is flagged, the rewrite prompt aligns the two outputs while preserving chart-grounded evidence and minimizing differences except for the group token.
Model
Mean Diss.
% > 0.1
t
p
TOST
GPT-4o
0.15
73.98
15.00
0.00
N.E.
GPT-5
0.21
89.59
28.47
0.00
N.E.
Gemini-3-flash
0.16
84.97
21.38
0.00
N.E.
Claude-4-Sonnet
0.02
8.77
-38.60
0.00
Eq.
Qwen3-VL-2B
0.16
79.60
29.80
0.00
N.E.
Qwen3-VL-4B
0.14
70.11
13.90
0.00
N.E.
Appendix
Table 9: Results for race (White vs. Black). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample t -test statistic and p -value, and the TOST equivalence result.
Model
Mean Diss.
% > 0.1
t
p
TOST
GPT-4o
0.15
72.90
14.00
0.00
N.E.
GPT-5
0.20
86.37
24.00
0.00
N.E.
Gemini-3-flash
0.17
87.15
23.94
0.00
N.E.
Claude-4-Sonnet
0.03
9.82
-28.80
0.00
Eq.
Qwen3-VL-2B
0.15
71.26
17.10
0.00
N.E.
Qwen3-VL-4B
0.13
70.00
11.10
0.00
N.E.
Appendix
Table 10: Results for income (low-income vs. high-income). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample t -test statistic and p -value, and the TOST equivalence result.
Model
Mean Diss.
% > 0.1
t
p
TOST
GPT-4o
0.14
57.10
4.29
0.00
N.E.
GPT-5
0.21
66.50
11.18
0.00
N.E.
Gemini-3-Flash
0.08
20.63
-2.93
0.998
Eq.
Claude-4-Sonnet
0.14
55.94
10.05
0.00
N.E.
Qwen3-VL-2B
0.07
19.47
-16.43
1.000
Eq.
Qwen3-VL-4B
0.11
43.23
4.36
0.00
N.E.
Appendix
Table 11: Results for age (Young vs. Old). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample t -test statistic and p -value, and the TOST equivalence result.
Model
Mean Diss.
% > 0.1
t
p
TOST
GPT-4o
0.15
63.70
66.00
0.00
N.E.
GPT-5
0.18
66.80
10.80
0.00
N.E.
Gemini-3-Flash
0.10
24.10
-0.25
0.599
N.E.
Claude-4-Sonnet
0.12
58.54
9.27
0.00
N.E.
Qwen3-VL-2B
0.08
25.76
-11.70
1.000
Eq.
Qwen3-VL-4B
0.09
32.37
-4.71
1.000
Eq.
Appendix
Table 12: Results for gender (Male vs. Female). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample t -test statistic and p -value, and the TOST equivalence result.
Model
Mean Diss.
% > 0.1
t
p
TOST
GPT-4o
0.17
85.29
11.73
0.00
N.E.
GPT-5
0.22
95.17
12.43
0.00
N.E.
Gemini-3-flash
0.22
92.96
31.05
0.00
N.E.
Claude-4-Sonnet
0.25
86.52
30.90
0.00
N.E.
Qwen3-VL-2B
0.18
82.29
20.47
0.00
N.E.
Qwen3-VL-4B
0.20
82.29
20.91
0.00
N.E.
Appendix
Table 13: Results for religion (Christians vs. Muslims). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample t -test statistic and p -value, and the TOST equivalence result.
Model
Mean Diss.
% > 0.1
t
p
TOST
GPT-4o
0.18
89.15
23.84
0.00
N.E.
GPT-5
0.22
96.59
37.35
0.00
N.E.
Gemini-3-flash
0.19
87.95
26.06
0.00
N.E.
Claude-4-Sonnet
0.21
83.53
25.84
0.00
N.E.
Qwen3-VL-2B
0.17
76.91
18.20
0.00
N.E.
Qwen3-VL-4B
0.16
68.27
15.62
0.00
N.E.
Appendix
Table 14: Results for immigration status (Citizens vs. Immigrants). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample t -test statistic and p -value, and the TOST equivalence result.
We propose a societal bias evaluation method for large vision-language models (LVLMs) in the era of strong safety guardrails. Existing benchmarks rely on prompts that ask models to infer attributes of people in images (e.g., "Is this person a CEO or a secretary?"). However, we find that LVLMs with strong guardrails, such as GPT and Claude, often refuse these prompts, making evaluations unreliable. To address this, we change the prior evaluation paradigm by decoupling the task from the depicted person: instead of inferring person's attributes, we use prompts that do not ask about the person (e.g., "Write a fictional story about an imaginary person.") and attach the image as provisional user information to implicitly provide demographic cues, then compare outputs across user demographics. Instantiated across three tasks --- story generation, term explanation, and exam-style QA --- our method avoids refusals even in guardrailed LVLMs, enabling reliable bias measurement. Applying it to 20 recent LVLMs, both open-source and proprietary, we find that all models undesirably use user demographic information in person-irrelevant tasks; for instance, characters in stories are often portrayed as mechanic for male users and nurse for female users. Although still biased, proprietary models like GPT-5 show lower bias than open-source ones. We analyze potential factors behind this gap, discussing continuous model monitoring and improvement as a possible contributor for reducing bias.
Yusuke Hirota, Michael Ross Boone, Arun George Zachariah +4
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.
Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1
University of Waterloo · Vector Institute · Layer 6 AI
Vision-Language Models (VLMs) have demonstrated remarkable progress in chart understanding, largely driven by supervised fine-tuning (SFT) on increasingly large synthetic datasets. However, scaling SFT data alone is inefficient and overlooks a key property of charts: charts are programmatically generated visual artifacts, where small, code-controlled visual changes can induce drastic shifts in semantics and correct answers. Learning this counterfactual sensitivity requires VLMs to discriminate fine-grained visual differences, yet standard SFT treats training instances independently and provides limited supervision to enforce this behavior. To address this, we introduce ChartCF, a data-efficient training framework designed to enhance counterfactual sensitivity. ChartCF consists of: (1) a counterfactual data synthesis pipeline via code modification, (2) a chart similarity-based data selection strategy that filters overly difficult samples for improved training efficiency, and (3) multimodal preference optimization across both textual and visual modalities. Experiments on five benchmarks show that ChartCF achieves superior or comparable performance to strong chart-specific VLMs while using significantly less training data.
Jianzhu Bao, Haozhen Zhang, Kuicai Dong +5
1Nanyang Technological University · 2Aumovio Singapore Pte. Ltd.