ChartHal: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understanding
Organizations: Department of Computer Science and Technology, Tsinghua University · State Key Laboratory of Cognitive Intelligence, iFLYTEK, Beijing, China.
Abstract
Large Vision-Language Models (LVLMs) have recently demonstrated remarkable progress, yet hallucination remains a critical barrier, particularly in chart understanding, which requires sophisticated perceptual and cognitive abilities as well as rigorous factual accuracy. While prior work has investigated hallucinations and chart comprehension independently, their intersection remains largely unexplored. To address this gap, we present ChartHal, a benchmark that features a fine-grained taxonomy of hallucination scenarios in chart understanding, along with a human-validated dataset of 1,062 samples. Our evaluation shows that state-of-the-art LVLMs suffer from severe hallucinations on ChartHal, including proprietary models such as GPT-5 and o4-mini, which achieve only 34.46% and 22.79% accuracy, respectively. Further analysis reveals that questions involving information absent from or contradictory to charts are especially likely to trigger hallucinations, underscoring the urgent need for more robust mitigation strategies. Code and data are available at https://github.com/ymcui/ChartHal
Figures & tables
| Desc. | Reason | Open | Total | |
|---|---|---|---|---|
| Irrel. | 56 | 99 | 114 | 269 |
| Inexist. | 151 | 105 | 88 | 344 |
| Contra. | 76 | 54 | 80 | 210 |
| Normal | 100 | 64 | 75 | 239 |
| Total | 383 | 322 | 357 | 1062 |
| Model | Question Type | Chart-Question Relation | Overall | |||||
| Desc. | Reason | Open | Irrel. | Inexist. | Contra. | Normal | ||
| Gemini-2.5-Pro | 60.31 | 50.31 | 36.69 | 36.80 | 52.62 | 49.52 | 58.58 | 49.34 |
| GPT-5 (high) | 43.34 | 36.65 | 22.97 | 20.82 | 29.07 | 19.52 | 70.71 | 34.46 |
| GPT-5 | 42.82 | 40.06 | 20.17 | 23.05 | 30.81 | 16.67 | 67.78 | 34.37 |
| GPT-5-mini (high) | 33.42 | 25.78 | 20.45 | 8.55 | 15.99 | 11.90 | 75.73 | 26.74 |
| GPT-5-mini | 34.20 | 24.53 | 21.57 | 8.18 | 15.99 | 9.05 | 79.92 | 27.02 |
| Mitigation Strategies | Overall Score |
|---|---|
| LLaVA-1.5-7B | 4.14 |
| + HA-DPO | 4.05 |
| + VCD | 6.78 |
| + WoodPecker | 11.21 |
| Qwen2.5-VL-72B-Instruct | 54.24 |
| + WoodPecker | 51.41 |
| Benchmark | Unique Unanswerable Questions | Free-form | Has Auxiliary Instructions |
|---|---|---|---|
| HallusionBench Guan et al. (2024) | N/A | No | No |
| CharXiv Wang et al. (2024c) | 19 | Yes | Yes |
| ChartQA Pro Masry et al. (2025) | 368 | Yes | Yes |
| ChartHal (Ours) | 823 | Yes | No |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | w/ Aux. Instr. | w/o Aux. Instr. |
|---|---|---|
| Qwen2.5-VL-3B | 76.0 | 1.4 |
| Qwen2.5-VL-7B | 56.6 | 18.0 |
| Model | Question Type | Chart-Question Relation | Overall | |||||
| Desc. | Reason | Open | Irrel. | Inexist. | Contra. | Normal | ||
| LLaVA-1.5-7B | 3.39 | 2.80 | 6.16 | 1.86 | 2.91 | 0.00 | 12.13 | 4.14 |
| + HA-DPO | 2.61 | 2.17 | 7.28 | 1.12 | 2.62 | 0.00 | 12.97 | 4.05 |
| + VCD | 6.53 | 3.73 | 9.80 | 0.74 | 8.43 | 0.00 | 17.15 | 6.78 |
| + WoodPecker | 12.01 | 11.49 | 10.08 | 8.55 | 16.57 | 0.48 | 15.90 | 11.21 |
| Qwen2.5-VL-72B-Instruct | 58.49 | 59.63 | 44.82 | 65.06 | 69.77 | 19.05 | 50.63 | 54.24 |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate a question that appears to be a simple, factual query about the chart, as if the answer could be directly extracted from it. However, make sure that the question is actually unrelated to the chart content (i.e. the question should refer to a topic that does not exist in the chart at all), so that no correct answer can be found in the image. The question should still sound plausible and specific, and it should be a closed question that expects a concrete answer. Answer: Please generate a short response to the question, pointing out that the topic or element asked about is not present in the chart, and therefore the question cannot be answered based on the chart. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate a question that appears to be a simple, factual question related to the chart. The question should be consistent with the chart’s overall topic or domain, but it must ask about some specific detail—such as a category, label, year, or variable—that is not actually present in the chart. The question should seem plausible and answerable from the chart at first glance, but in reality the necessary information is missing. Answer: Please generate a short response that explains the question refers to a detail not shown in the chart (e.g., a year, category, or variable), and therefore it cannot be answered based on the chart data. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate a question that begins with a factual statement that appears to be based on the chart but is actually contradicted by the chart. Then, ask a simple, descriptive question based on that incorrect premise. The question should sound plausible, as if it could be answered by referring to the chart, but it should be logically invalid due to the contradiction in the premise. Make sure the contradiction is in the premise statement (not in the actual question). Do not directly ask a false question. Instead, provide a false statement first, then follow it up with a concrete, factual question based on it. Answer: Please generate a short response that points out the contradiction in the premise compared to the actual chart data, clarifies what the correct chart information is, and provide the correct answer ignoring the false premise. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate a simple, factual question that can be directly answered using the information shown in the chart. The question should be clearly related to the chart, refer to elements (such as labels, data points, categories, or values) that are actually present, and involve no inference or assumptions beyond what is visually provided. Make sure the question is correct and logically consistent with the chart. Answer: Please generate a short, accurate answer that directly uses the correct data from the chart to respond to the question. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate a reasoning-based question that appears to require simple inference from the chart (such as comparing values, calculating differences, or identifying trends). However, the question must actually be unrelated to the content of the chart. That is, the question should refer to a topic that does not exist in the chart at all. Make it sound as if the chart could support reasoning to answer the question, but in reality, the necessary information is completely absent.Ensure that the reasoning required is straightforward and answerable in principle, but the problem is unanswerable here because the chart has nothing to do with the topic. Answer: Please generate a short response pointing out that the question refers to a topic that is not present in the chart, and that no inference can be made from the given data. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate a reasoning-based question that is clearly related to the chart’s topic or domain and requires simple inference (such as computing a difference or trend). However, make sure the question asks about specific information—such as a year, group, variable, or category—that is not actually present in the chart. The question should sound plausible and as if it could be answered using the chart, but in reality, the key data needed for inference is missing. Answer: Please generate a response that explains the question cannot be answered because the required data (year, group, variable, etc.) is missing from the chart. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate a question that begins with a statement about the chart, where the statement contradicts the actual data shown. Then, based on that false statement, ask a reasoning-based question that would normally require a simple inference (e.g., computing a difference or identifying a trend). The question should sound logical and answerable, but the initial premise must be factually incorrect according to the chart. Do not contradict the chart in the actual question; instead, embed the contradiction in the premise statement. Ensure that the reasoning expected is straightforward, even though the basis is flawed. Answer: Please generate a short response that points out the contradiction in the premise compared to the chart, and give the correct answer to the question if the question is answerable ignoring the false premise. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate a reasoning-based question that is logically consistent with the chart and can be answered by using the information provided. The question should involve simple inference, such as comparing values, identifying trends, computing differences or ratios, or summarizing grouped data. Make sure the question only refers to elements that are actually present in the chart, and that it is objectively answerable. Answer: Please generate a short, accurate answer that performs the correct inference or calculation using data from the chart, and presents the result clearly. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate an open-ended question that seems to ask for an explanation, reasoning, or opinion based on a chart, but in fact, the topic of the question is entirely unrelated to the content of the chart. The question should sound like it could be asked in response to a chart, but the actual theme (e.g., domain, topic, entities) should not appear in the chart at all. Avoid asking about specific information. Instead, ask for explanations, causes, interpretations, or predictions related to a topic that the chart does not cover. Answer: Please generate a short response pointing out that the topic mentioned in the question is not covered in the chart, so no meaningful explanation can be provided based on the chart data. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate an open-ended question that appears to ask for an explanation or interpretation of the chart, but it must be based on information—such as a year, category, variable, or region—that is not actually present in the chart. The question should be relevant to the general topic of the chart and sound plausible, but should include some missing or fabricated element that prevents it from being truly answerable. Do not include specific numerical queries or factual comparisons. Focus on asking for explanations, causes, interpretations, or predictions, but make sure they hinge on something not found in the chart. Answer: Please generate a response pointing out the specific element referred to is not shown in the chart, so any explanation would be purely hypothetical. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate an open-ended question that begins with a statement about the chart, where the statement clearly contradicts the actual information shown. Then, based on this false premise, ask an analytical or interpretive question. The question should sound plausible and related to the chart’s topic, but the initial assumption must be incorrect according to the chart. The question should not request a specific fact, but rather an explanation, a cause, an interpretation, or a prediction (e.g., why something happened, what it implies, etc.). The contradiction must be in the premise, not the question itself. Answer: Please generate an answer pointing out the false assumption in the question’s premise, and suggests that any analysis based on it may be invalid or misleading. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| Given an image of a chart, your task is to generate a question and a corresponding reference answer, following the specific instructions below. Instruction for Generation: Question: Please generate an open-ended question that is logically consistent with the chart and invites explanation, interpretation, or speculation based on the data shown. The question should reference trends, patterns, comparisons, or anomalies that are truly present in the chart, and should encourage divergent thinking rather than factual retrieval and calculation. Avoid factual questions and focus on asking for explanations, causes, interpretations, or predictions. Answer: Please generate a thoughtful answer that responds to the question using evidence from the chart. The response should be tentative and interpretive, not definitive. Output Format: Question: <Insert the generated question here> Answer: <Insert the generated answer here> |
| Prompt |
| You are given a question about a chart and a corresponding response. The question is completely unrelated to the chart. Your task is to evaluate whether the response correctly identifies that the question cannot be answered from the chart. Guidelines: - The response should explicitly indicate that the question is unrelated to the chart. - It should not attempt to fabricate an answer. - If the response fails to point out the irrelevance or tries to answer the question, it should be considered incorrect. Question: <|QUESTION|> Response: <|RESPONSE|> Score the response above as follows: - 1 = Correctly identifies irrelevance and avoids answering. - 0 = Does not identify irrelevance OR tries to answer the question. DO NOT provide any explanation or additional text. Just return the score as a single integer. If the response is empty, give a score of 0. Output Format: Score: <Insert your score here, either 0 or 1> |
| Prompt |
| You are given a question about a chart and a corresponding response. The question is related to the chart but asks about information that does not exist in the chart. Your task is to evaluate whether the response correctly identifies that the information is missing and that the question cannot be answered from the chart. Guidelines: - The response should clearly state that the requested information does not exist in the chart. - It should not attempt to guess or fabricate an answer. - If the response ignores the missing information issue or provides an answer, it should be considered incorrect. Question: <|QUESTION|> Response: <|RESPONSE|> Score the response above as follows: - 1 = Correctly identifies that the information is not in the chart and avoids guessing. - 0 = Does not identify missing information OR gives a fabricated answer. DO NOT provide any explanation or additional text. Just return the score as a single integer. If the response is empty, give a score of 0. Output Format: Score: <Insert your score here, either 0 or 1> |
| Prompt |
| You are given a question about a chart and a corresponding response. The question is related to the chart but asks about information that does not exist in the chart. Your task is to evaluate whether the response correctly identifies that the information is missing and and either states the question is unanswerable or provides a speculative answer. Guidelines: - The response should clearly state that the requested information does not exist in the chart. - It should either state that the question is unanswerable OR provide a speculative answer with an speculative tone. - If the response ignores the missing information issue and directly provides an definitive answer, it should be considered incorrect. Question: <|QUESTION|> Response: <|RESPONSE|> Score the response above as follows: - 1 = Correctly identifies that the information is not in the chart and avoids guessing. - 0 = Does not identify missing information OR gives a fabricated answer. DO NOT provide any explanation or additional text. Just return the score as a single integer. If the response is empty, give a score of 0. Output Format: Score: <Insert your score here, either 0 or 1> |
| Prompt |
| You are given a question about a chart, a corresponding response, and a ground truth answer. The question contains a statement that contradicts the chart, and then asks a follow-up based on that contradiction. Your task is to evaluate whether the response correctly identifies the contradiction and either states the question is unanswerable or provides a correct answer ignoring the contradiction in the question. Guidelines: - The response should explicitly point out that the question contains incorrect or contradictory information relative to the chart. - It should either state that the question is unanswerable OR provide a correct answer that aligns with the given Ground Truth. - If the ground truth does not provide an answer, the response should also not provide an answer. - When the response tries to provide an answer, for questions expecting numerical answers, if the ground truth specifies an acceptable range (e.g., ‘‘Answers in the range of [35%, 40%] are ACCEPTABLE’’ or similar phrasing), then the answer is considered correct if it falls within this range. Otherwise, the semantic meaning of the answer should match the ground truth without omissions or additional unrelated information. - If the response fails to mention the contradiction OR answers the question based on the contradictory statement, it is incorrect. Question: <|QUESTION|> Ground Truth: <|REF_ANS|> Response: <|RESPONSE|> Score the response above as follows: - 1 = Correctly identifies the contradiction AND either states the question is unanswerable or provides a correct answer that matches the given Ground Truth. - 0 = Does not mention the contradiction OR answers the question as if it were valid. DO NOT provide any explanation or additional text. Just return the score as a single integer. If the response is empty, give a score of 0. Output Format: Score: <Insert your score here, either 0 or 1> |
| Prompt |
| You are given a question about a chart and a corresponding response. The question contains a statement that contradicts the chart, and then asks a follow-up based on that contradiction. Your task is to evaluate whether the response correctly identifies the contradiction and either states the question is unanswerable or provides a tentative answer ignoring the contradiction in the question. Guidelines: - The response should explicitly point out that the question contains incorrect or contradictory information relative to the chart. - It should either state that the question is unanswerable or provide a speculative answer with an appropriately uncertain tone. - If the response fails to mention the contradiction, answers the question based on the contradictory statement, sounds overconfident OR states an absolute answer, it is incorrect. Question: <|QUESTION|> Response: <|RESPONSE|> Score the response above as follows: - 1 = Correctly identifies the contradiction AND either states the question is unanswerable or provides a tentative answer with an appropriately uncertain tone. - 0 = Does not mention the contradiction OR answers the question as if it were valid OR sounds overconfident. DO NOT provide any explanation or additional text. Just return the score as a single integer. If the response is empty, give a score of 0. Output Format: Score: <Insert your score here, either 0 or 1> |
| Prompt |
| You are given a question about a chart, a corresponding response, and a ground truth answer. The question is answerable from the chart and has a definitive correct answer. Your task is to evaluate how accurate the model response is compared to the ground truth. 1. If the expected answer is related to numeric values: - If the ground truth specifies an acceptable range (e.g., ‘‘Answers in the range of [35%, 40%] are ACCEPTABLE’’ or similar phrasing), then: - Give a score of 1 if the response contains a numeric value within this range. - Otherwise, give a score of 0. - If no range is specified: - Give a score of 1 if and only if the two numbers are exactly equal in value. Different notations (e.g., ‘0.01’ and ‘10 ˆ -2’, ‘1500’ and ‘1.5e3’) are acceptable. - Give a score of 0 if the numbers differ in value. 2. If the expected answer is related to predefined terms: - Give a score of 1 if and only if the final answer and the ground truth refer to the same term(s). Different forms or representations (e.g., and alpha; and R ˆ 2_t,h,v,m) are acceptable. - If the question requires multiple terms, different order is acceptable as long as all terms match exactly. - Give a score of 0 if any required term is incorrect, missing, or replaced with a different term (e.g., ACC+ vs ACC; P-101 vs P=101). 3. For all other cases: - Give a score of 1 only if the semantic meaning of the model response matches the ground truth without omissions or additional unrelated information. - Any deviation in meaning should result in a score of 0. Question: <|QUESTION|> Ground Truth: <|REF_ANS|> Response: <|RESPONSE|> DO NOT provide any explanation or additional text. Just return the score as a single integer. If the response is empty, give a score of 0. Output Format: Score: <Insert your score here, either 0 or 1> |
| Prompt |
| You are given a question and a model response. The question is open-ended and does not have a definitive answer. Your task is to evaluate whether the response uses an appropriately uncertain tone and avoids giving a misleadingly definitive answer. Guidelines: - The response should acknowledge uncertainty, use hedging language (e.g., "possibly," "likely," "one reason could be"). - It should not present opinions as facts or give definitive causes. - If the response sounds overconfident or states an absolute answer, it is incorrect. Question: <|QUESTION|> Response: <|RESPONSE|> Score the response above as follows: - 1 = Maintains uncertainty and provides plausible reasoning. - 0 = Presents a definitive or overconfident answer without acknowledging uncertainty. DO NOT provide any explanation or additional text. Just return the score as a single integer. If the response is empty, give a score of 0. Output Format: Score: <Insert your score here, either 0 or 1> |
VLM Hallucination
Vision-language model hallucination is the production of claims unsupported by or in conflict with visual evidence.
See all 74 papers