Same Chart, Different Story: Bias in Vision-Language Chart Interpretation

Mizanur Rahman; Huan Wu; Arash Asgari; Enamul Hoque Prince; Laleh Seyyed-Kalantari

cs.CLAug 12, 2026

Same Chart, Different Story: Bias in Vision-Language Chart Interpretation

Authors: Mizanur Rahman, Huan Wu, Arash Asgari, Enamul Hoque Prince, Laleh Seyyed-Kalantari

Organizations: York University · Vector Institute · Connected Minds

Abstract

Vision-language models (VLMs) are increasingly used to interpret charts and generate natural-language explanations for socially consequential data. However, they may produce different narratives for the same chart when only the referenced social group changes, reinforcing stereotypes and misleading decisions. Despite these risks, no benchmark exists for systematically evaluating bias in chart interpretation across social dimensions. We introduce ChartBias, the first benchmark for auditing bias in VLM-based chart interpretation. ChartBias contains 820 manually curated real-world charts spanning six attributes: race, income, age, religion, immigration status, and gender, yielding 4,319 valid chart, attribute instances and 8,638 paired generations where the chart is fixed and only the group term is swapped. Across 12 proprietary and open-source VLMs, totaling 155,484 model responses, we find three widespread failure modes: narrative shift (same chart, different narratives), group hallucination (assigning a chart to a group without evidence), and preference polarity (favourable trends often linked to one group). We further propose a multi-agent mitigation framework that serves as a strong baseline by separating chart-grounded evidence extraction from group-conditioned generation and using a counterfactual judge to verify that group-driven differences are supported by the chart. The framework substantially reduces narrative shift while preserving chart-grounded reasoning. Our findings show that evaluating chart understanding requires measuring not only accuracy, but also fairness and consistency across social groups. We release ChartBias at https://github.com/vis-nlp/ChartBiasBench.

Figures & tables

Figure 1: Examples of the three ChartBias evaluation settings using GPT-4o outputs. Despite identical chart evidence, the model produces different narratives under race swaps, assigns gender without supporting evidence, and associates negative trends more strongly with immigrants than positive trends.

Figure 2: ChartBias construction pipeline. Charts are collected from four public sources, screened by two annotators, and annotated for attribute eligibility, chart type, topic, and polarity. The resulting benchmark contains 820 charts across six sensitive attributes and supports three evaluation settings: narrative shift, group hallucination, and preference polarity.

Sources				Attributes						Chart Types
VisText	OECD	Statista	Pew	Race	Income	Age	Religion	Immigration	Gender	Line	Bar	Area	Other
325 (40%)	213 (26%)	201 (25%)	81 (10%)	701 (85%)	675 (82%)	606 (74%)	809 (99%)	802 (98%)	726 (89%)	494 (60%)	191 (23%)	127 (15%)	8 (1%)

Table 1: Summary of source composition, attribute eligibility, and chart type distribution in the final dataset.

Figure 3: Overview of the proposed multi-agent counterfactual mitigation framework. The framework masks sensitive attributes, extracts shared chart-grounded evidence, generates paired group-conditioned outputs, and checks their consistency before producing the final responses.

	Race			Income			Age			Gender			Religion			Immigration
Model	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST
GPT-4o	0.15	73.98		0.15	72.90		0.14	57.10		0.15	63.70		0.17	85.29		0.18	89.15
GPT-5	0.21	89.59		0.20	86.37		0.21	66.50		0.18	66.80		0.22	95.17		0.22	96.59
Gemini-3-flash	0.16	84.97		0.17	87.15		0.08	20.63	$\checkmark$	0.10	24.10		0.22	92.96		0.19	87.95
Claude-4-Sonnet	0.02	8.77	$\checkmark$	0.03	9.82	$\checkmark$	0.14	55.94		0.12	58.54		0.25	86.52		0.21	83.53
Qwen3-VL-2B	0.16	79.60		0.15	71.26		0.07	19.47	$\checkmark$	0.08	25.76	$\checkmark$	0.18	82.29		0.17	76.91

Table 2: Narrative shift under counterfactual sensitive-attribute swaps. The first block reports closed/proprietary models and the second block reports open-weight models. Diss.

\downarrow

reports mean semantic dissimilarity between paired generations, and %

>

0.1

\downarrow

reports the share of pairs exceeding the 0.10 threshold. Lower values indicate stronger counterfactual consistency; bold values mark the largest shifts within each attribute. Two One-Sided Tests TOST reports practical equivalence with a 0.10 bound:

\checkmark

denotes equivalence and denotes non-equivalence.

	Race			Income			Age			Gender			Religion			Immigration
Model	White	Black	Abs.	Low	High	Abs.	Young	Old	Abs.	Male	Female	Abs.	Christ.	Muslim	Abs.	Citizen	Immig.	Abs.
GPT-4o	71.83	24.89	3.28	63.70	36.30	0.00	65.18	34.82	0.00	41.60	58.40	0.00	75.70	24.30	0.00	79.39	20.61	0.00
GPT-5	63.50	36.50	0.00	69.92	30.08	0.00	47.52	52.34	0.14	61.43	38.57	0.00	66.63	29.15	4.22	80.42	18.08	1.50
Gemini-3-flash	57.77	42.23	0.00	58.07	41.93	0.00	50.17	49.83	0.00	54.41	45.59	0.00	57.11	42.89	0.00	73.57	26.43	0.00
Claude-4-Sonnet	50.90	49.10	0.00	51.60	48.40	0.00	41.91	57.59	0.50	59.64	39.95	0.41	52.66	46.97	0.37	68.58	31.42	0.00
Qwen3-VL-2B	07.00	93.00	0.00	10.00	90.00	0.00	55.78	44.22	0.00	50.96	49.04	0.00	52.52	47.48	0.00	61.45	38.55	0.00

Table 3: Group hallucination under unsupported sensitive-attribute attribution. For each model and attribute, we report the percentage of outputs assigned to each group and the percentage of abstentions ( Abs. ). Because the visual evidence does not support group-level inference, the expected behavior is to abstain rather than assign a sensitive-attribute label; bold values mark the largest attribution or abstention rates within each attribute.

	Race (W)			Income (L)			Age (Y)			Gender (M)			Religion (C)			Immigration (Cit.)
Model	Pos.	Neu.	Neg.	Pos.	Neu.	Neg.	Pos.	Neu.	Neg.	Pos.	Neu.	Neg.	Pos.	Neu.	Neg.	Pos.	Neu.	Neg.
GPT-4o	85.00	71.96	50.30	58.74	51.53	87.01	64.80	60.80	70.40	44.30	41.20	37.80	80.95	74.16	71.40	81.10	81.10	74.50
GPT-5	63.00	67.80	58.60	56.30	72.20	87.70	42.60	47.50	55.00	67.30	53.20	61.20	67.60	70.70	60.00	78.70	79.80	84.10
Gemini-3-flash	54.70	61.20	58.20	77.40	49.30	40.10	50.40	47.00	53.30	57.90	51.90	51.70	55.60	63.30	51.90	69.90	73.80	79.30
Claude-4-Sonnet	52.20	49.79	50.28	54.20	50.22	49.42	45.30	36.50	42.60	66.30	55.10	54.20	49.70	59.80	48.60	68.40	70.20	66.80
Qwen3-VL-2B	08.36	06.75	05.08	07.00	10.00	14.60	53.90	53.60	60.90	49.80	51.40	52.20	48.20	52.10	52.40	67.30	52.20	56.70

Table 4: Polarity-conditioned group preference attribution across six sensitive attributes. Values report the percentage of outputs assigned to the first group in each pair across positive, neutral, and negative chart contexts: White (W), low-income (L), young (Y), male (M), Christian (C), and citizen (Cit.).

	Race			Income			Age			Gender			Religion			Immigration
Model	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST	Diss. $\downarrow$	% $>$ 0.1 $\downarrow$	TOST
GPT-4o	0.15	73.98		0.15	72.90		0.14	57.10		0.15	63.70		0.17	85.29		0.18	89.15
GPT-4o-Multistep	0.06	16.54	$\checkmark$	0.05	16.22	$\checkmark$	0.04	16.67	$\checkmark$	0.05	17.82	$\checkmark$	0.06	19.14	$\checkmark$	0.06	19.47	$\checkmark$
Gemini-3-flash	0.16	84.97		0.17	87.15		0.08	20.63		0.10	24.10		0.22	92.96		0.19	87.95
Gemini-3-flash-Multistep	0.04	3.26	$\checkmark$	0.01	0.68	$\checkmark$	0.04	3.47	$\checkmark$	0.04	4.08	$\checkmark$	0.05	5.95	$\checkmark$	0.04	4.03	$\checkmark$
Gemini-3 w/o masking	0.08	23.05		0.04	3.93	$\checkmark$	0.06	26.44		0.05	8.53	$\checkmark$	0.05	7.17	$\checkmark$	0.04	5.91	$\checkmark$

Table 5: Mitigating narrative shift under sensitive-attribute swaps. For each attribute, Diss.

\downarrow

reports mean semantic dissimilarity between paired generations, and %

>

0.1

\downarrow

reports the share of pairs exceeding the 0.10 threshold.

Figure 4: Qualitative analysis of narrative shift across six sensitive attributes. For each example, the chart image and prompt template are held fixed while only the referenced group term is swapped. Highlighted phrases mark the group-conditioned shifts in interpretation.

Figure 5: Heatmap of narrative shift across models and sensitive attributes. Each cell reports the mean semantic dissimilarity between paired responses.

Benchmark	# charts	Real-world	Attrib.	Narr.	Halluc.	Polarity
Chart-to-Text obeid2020chart	44K	Mixed	—	—	—	—
VisText tang2023vistext	12K	$\checkmark$	—	—	—	—
Mahbub et al. mahbub2025charts	100	$\checkmark$	1 (geo-econ)	$\checkmark$	—	—
ChartBias (ours)	820	$\checkmark$	6	$\checkmark$	$\checkmark$	$\checkmark$

Table 6: Comparison of ChartBias with prior chart-understanding and chart-to-text benchmarks. ChartBias is the only benchmark that combines real-world charts, multi-sensitive attribute (Attrib.) annotation, and supports the analysis of narrative shift (Narr.) support, group hallucination (Halluc.), and preference polarity (Polarity).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Attribute	Definition in ChartBias	Paired group values
Race	Racial group referenced in the prompt for counterfactual evaluation.	White / Black
Income	Socioeconomic status referenced in the prompt.	High-income / Low-income
Age	Age group referenced in the prompt.	Young / Old
Gender	Gender group referenced in the prompt.	Male / Female
Religion	Religious affiliation referenced in the prompt.	Christian / Muslim
Immigration status	Citizenship or migration status referenced in the prompt.	Citizen / Immigrant

Appendix

Table 7: The six sensitive attributes annotated in ChartBias and their paired binary group values. Pairs are chosen to be common and recognizable rather than exhaustive across all possible identities.

Attribute	Total Samples	Positive	Neutral	Negative
Race	701	287	237	177
Income	675	269	229	177
Age	606	256	181	169
Religion	809	340	259	210
Immigration	802	342	252	208
Gender	726	309	216	201

Appendix

Table 8: Per-attribute polarity distribution in ChartBias. For each sensitive attribute, we report the total number of valid chart–attribute instances and the number of instances labeled as positive, neutral, and negative. Polarity is assigned based on the chart content rather than the sentiment of model-generated text.

Figure 6: Representative examples of polarity annotation from ChartBias. The left chart is labeled positive because it shows an improving or favorable trend, the middle chart is labeled neutral because it shows no clear favorable or unfavorable directional change, and the right chart is labeled negative because it shows a worsening or unfavorable trend. Polarity is assigned based on the chart content itself, not on the sentiment of the generated text.

Figure 7: Attribute eligibility example for the race attribute. The left chart is ineligible because it explicitly mentions racial groups, including White and Black, in the visual content. The right chart is eligible because it contains no race-specific reference, allowing paired counterfactual prompts such as White versus Black while keeping the chart fixed.

Figure 8: Representative examples of ChartBias diversity. The figure shows charts from different sources (OECD, Pew, Statista, and VisText), chart types (line, bar, pie, and area), and topics, including patent applications, refugee resettlement, out-of-school children, school shootings, Zika vaccination, and youth unemployment. The examples also illustrate variation in visual complexity.

Figure 9: Heatmap of preference-polarity gaps across models and sensitive attributes. Each cell reports the absolute difference between the model’s first-group attribution rate on positive versus negative charts. Darker cells indicate stronger polarity-linked group preference.

Figure 10: Qualitative analysis of mitigation outcomes. The mitigation framework reduces narrative shift in some examples, but residual bias remains in others, where paired outputs still assign different severity, framing, or evaluative emphasis to the same chart evidence.

Figure 11: Attribute-wise mitigation gains for narrative shift. Each cell shows the reduction in mean semantic dissimilarity after applying the multi-agent mitigation framework to GPT-4o and Gemini-3-Flash. Darker cells indicate larger improvements.

Panel of Figure 12 — Figure 12: Prompt template for the narrative-shift setting. The chart image and prompt template are fixed while only the group term [GROUP] is swapped. The analysis instruction is specialized by chart type, and the model is asked to produce a five-sentence chart interpretation with an opinion inside <opinion> tags.

Panel of Figure 13 — Figure 13: Prompt template for the group-selection setting. The model first produces a factual chart analysis without mentioning either group. The factual analysis is then used in a second prompt that asks the model to select the group.

Panel of Figure 14 — Figure 14: Main prompt templates for the multi-agent mitigation framework. Attribute masking is deterministic. The chart evidence extraction prompt produces group-neutral chart facts, and the group-conditioned generation prompt uses the same extracted facts to produce paired outputs while treating the group name only as a framing reference.

Panel of Figure 15 — Figure 15: Prompt template for the consistency checking and rewrite stage. The programmatic judge flags unsupported differences using stance agreement, semantic similarity, and sentiment difference. When a pair is flagged, the rewrite prompt aligns the two outputs while preserving chart-grounded evidence and minimizing differences except for the group token.

Model	Mean Diss.	% $>$ 0.1	$t$	$p$	TOST
GPT-4o	0.15	73.98	15.00	0.00	N.E.
GPT-5	0.21	89.59	28.47	0.00	N.E.
Gemini-3-flash	0.16	84.97	21.38	0.00	N.E.
Claude-4-Sonnet	0.02	8.77	-38.60	0.00	Eq.
Qwen3-VL-2B	0.16	79.60	29.80	0.00	N.E.
Qwen3-VL-4B	0.14	70.11	13.90	0.00	N.E.

Appendix

Table 9: Results for race (White vs. Black). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample

t

-test statistic and

p

-value, and the TOST equivalence result.

Model	Mean Diss.	% $>$ 0.1	$t$	$p$	TOST
GPT-4o	0.15	72.90	14.00	0.00	N.E.
GPT-5	0.20	86.37	24.00	0.00	N.E.
Gemini-3-flash	0.17	87.15	23.94	0.00	N.E.
Claude-4-Sonnet	0.03	9.82	-28.80	0.00	Eq.
Qwen3-VL-2B	0.15	71.26	17.10	0.00	N.E.
Qwen3-VL-4B	0.13	70.00	11.10	0.00	N.E.

Appendix

Table 10: Results for income (low-income vs. high-income). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample

t

-test statistic and

p

-value, and the TOST equivalence result.

Model	Mean Diss.	% $>$ 0.1	$t$	$p$	TOST
GPT-4o	0.14	57.10	4.29	0.00	N.E.
GPT-5	0.21	66.50	11.18	0.00	N.E.
Gemini-3-Flash	0.08	20.63	-2.93	0.998	Eq.
Claude-4-Sonnet	0.14	55.94	10.05	0.00	N.E.
Qwen3-VL-2B	0.07	19.47	-16.43	1.000	Eq.
Qwen3-VL-4B	0.11	43.23	4.36	0.00	N.E.

Appendix

Table 11: Results for age (Young vs. Old). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample

t

-test statistic and

p

-value, and the TOST equivalence result.

Model	Mean Diss.	% $>$ 0.1	$t$	$p$	TOST
GPT-4o	0.15	63.70	66.00	0.00	N.E.
GPT-5	0.18	66.80	10.80	0.00	N.E.
Gemini-3-Flash	0.10	24.10	-0.25	0.599	N.E.
Claude-4-Sonnet	0.12	58.54	9.27	0.00	N.E.
Qwen3-VL-2B	0.08	25.76	-11.70	1.000	Eq.
Qwen3-VL-4B	0.09	32.37	-4.71	1.000	Eq.

Appendix

Table 12: Results for gender (Male vs. Female). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample

t

-test statistic and

p

-value, and the TOST equivalence result.

Model	Mean Diss.	% $>$ 0.1	$t$	$p$	TOST
GPT-4o	0.17	85.29	11.73	0.00	N.E.
GPT-5	0.22	95.17	12.43	0.00	N.E.
Gemini-3-flash	0.22	92.96	31.05	0.00	N.E.
Claude-4-Sonnet	0.25	86.52	30.90	0.00	N.E.
Qwen3-VL-2B	0.18	82.29	20.47	0.00	N.E.
Qwen3-VL-4B	0.20	82.29	20.91	0.00	N.E.

Appendix

Table 13: Results for religion (Christians vs. Muslims). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample

t

-test statistic and

p

-value, and the TOST equivalence result.

Model	Mean Diss.	% $>$ 0.1	$t$	$p$	TOST
GPT-4o	0.18	89.15	23.84	0.00	N.E.
GPT-5	0.22	96.59	37.35	0.00	N.E.
Gemini-3-flash	0.19	87.95	26.06	0.00	N.E.
Claude-4-Sonnet	0.21	83.53	25.84	0.00	N.E.
Qwen3-VL-2B	0.17	76.91	18.20	0.00	N.E.
Qwen3-VL-4B	0.16	68.27	15.62	0.00	N.E.

Appendix

Table 14: Results for immigration status (Citizens vs. Immigrants). We report mean dissimilarity, the percentage of cases with dissimilarity greater than 0.10, the one-sided one-sample

t

-test statistic and

p

-value, and the TOST equivalence result.

Cultural Bias in VLMs

See all 14 papers

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models
Aug 30, 2026Yusuke Hirota, Michael Ross Boone, Arun George Zachariah +4Language Model Bias Evaluation VLM Evaluation
NVIDIA
Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1VLM Evaluation Data Visualization
University of Waterloo · Vector Institute · Layer 6 AI
Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding
May 11, 2026Jianzhu Bao, Haozhen Zhang, Kuicai Dong +5Data Visualization VLM Adaptation
1Nanyang Technological University · 2Aumovio Singapore Pte. Ltd.

Aug 30, 2026J/K move · Enter open · S save

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models

Yusuke Hirota, Michael Ross Boone, Arun George Zachariah, Jibin Rajan Varghese+3

NVIDIA

Language Model Bias Evaluation VLM Evaluation

Open paper AskPDF arXiv

We propose a societal bias evaluation method for large vision-language models (LVLMs) in the era of strong safety guardrails. Existing benchmarks rely on prompts that ask models to infer attributes of people in images (e.g., "Is this person a CEO or a secretary?"). However, we find that LVLMs with strong guardrails, such as GPT and Claude, often refuse these prompts, making evaluations unreliable. To address this, we change the prior evaluation paradigm by decoupling the task from the depicted person: instead of inferring person's attributes, we use prompts that do not ask about the person (e.g., "Write a fictional story about an imaginary person.") and attach the image as provisional user information to implicitly provide demographic cues, then compare outputs across user demographics. Instantiated across three tasks --- story generation, term explanation, and exam-style QA --- our method avoids refusals even in guardrailed LVLMs, enabling reliable bias measurement. Applying it to 20 recent LVLMs, both open-source and proprietary, we find that all models undesirably use user demographic information in person-irrelevant tasks; for instance, characters in stories are often portrayed as mechanic for male users and nurse for female users. Although still biased, proprietary models like GPT-5 show lower bias than open-source ones. We analyze potential factors behind this gap, discussing continuous model monitoring and improvement as a possible contributor for reducing bias.

Abstract

Cultural Bias in VLMs

Explore similar work

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding