cs.CVSep 22, 2025

ChartHal: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understanding

Authors: Xingqi Wang, Yiming Cui, Xin Yao, Shijin Wang, Guoping Hu, Xiaoyu Qin

Organizations: Department of Computer Science and Technology, Tsinghua University · State Key Laboratory of Cognitive Intelligence, iFLYTEK, Beijing, China.

Abstract

Large Vision-Language Models (LVLMs) have recently demonstrated remarkable progress, yet hallucination remains a critical barrier, particularly in chart understanding, which requires sophisticated perceptual and cognitive abilities as well as rigorous factual accuracy. While prior work has investigated hallucinations and chart comprehension independently, their intersection remains largely unexplored. To address this gap, we present ChartHal, a benchmark that features a fine-grained taxonomy of hallucination scenarios in chart understanding, along with a human-validated dataset of 1,062 samples. Our evaluation shows that state-of-the-art LVLMs suffer from severe hallucinations on ChartHal, including proprietary models such as GPT-5 and o4-mini, which achieve only 34.46% and 22.79% accuracy, respectively. Further analysis reveals that questions involving information absent from or contradictory to charts are especially likely to trigger hallucinations, underscoring the urgent need for more robust mitigation strategies. Code and data are available at https://github.com/ymcui/ChartHal

Figures & tables

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

    May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1VLM EvaluationData Visualization

  2. When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs

    Apr 23, 2026Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny +3VLM EvaluationLLM Hallucination Mitigation

  3. UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space

    Aug 4, 2026Amir Mohammad Ezzati, Kiyan Rezaee, Bardiya Kariminia +4Hallucination Detection in VLMsVision-Language Models