VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought

Eunsoo Lee; Jeongwoo Lee; Minki Hong; Jangho Choi; Jihie Kim

cs.AIMar 12, 2026

VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought

Authors: Eunsoo Lee, Jeongwoo Lee, Minki Hong, Jangho Choi, Jihie Kim

Organizations: Department of Computer Science and Artificial Intelligence, Dongguk University · Department of Electronics and Electrical Engineering, Dongguk University

Abstract

Large vision-language models (LVLMs) struggle to reliably detect visual primitives in charts and align them with semantic representations, which severely limits their performance on complex visual reasoning. This lack of perceptual grounding constitutes a major bottleneck for chart-based reasoning. We propose VisDoT, a framework that enhances visual reasoning through human-like interpretation grounding. We formalize four perceptual tasks based on the theory of graphical perception, including position and length. Building on this foundation, we introduce Decomposition-of-Thought (DoT) prompting, which sequentially separates questions into visual perception sub-questions and logic sub-questions. Fine-tuning InternVL with VisDoT achieves a +11.2% improvement on ChartQA and surpasses GPT-4o on the more challenging ChartQAPro benchmark. On the newly introduced VisDoTQA benchmark, the model improves by +33.2%. Furthermore, consistent zero-shot gains on diverse open-domain VQA benchmarks confirm the generalizability of the perception-logic separation strategy for visual question answering. VisDoT leverages human-like perception to enhance visual grounding, achieving state-of-the-art chart understanding and interpretable visual reasoning.

Figures & tables

Figure 1: Comparison of visual reasoning between existing LVLMs and VisDoT. While LVLMs fail to visual perception in spatial structure, VisDoT leverages decomposition-of-thought (DoT) to accurately infer answers through sequential visual analysis.

Task	Description
Position	Compares object positions along a common scale (e.g., x- or y-axis) to determine relative order. As the most accurate perceptual channel for conveying quantitative information, it plays a critical role in effective visual communication.
Length	A distortion-free visual attribute across visualized data types Stevens (2017) , used as a secondary cue to position.
Pattern	Links pattern cues to legends and data to distinguish categories. Assesses visual label mapping ability.
Extract	Reads explicitly shown values. Evaluates numerical recognition similar to standard QA tasks.

Table 1: Core perceptual tasks derived from graphical perception theory, designed to align LVLM attention with structured visual features for visualized data reasoning. A detailed description of each task is provided in Appendix A.1

Figure 3: Perception-following QA examples for the four task types in Table 1 . Each question is decomposed into perception and logic sub-questions using the DoT prompt, enabling structured and interpretable chart reasoning.

Figure 4: The model is guided to decompose a complex visual question into perception and logic sub-questions (Question Decomposition) and generate intermediate reasoning steps sequentially (Problem Solving), enabling structured and interpretable visual inference.

Task Type	# QA Pairs
Position	66,617
Length	31,614
Pattern	61,268
Extract	174,411
Total	331,969

Table 2: Distribution of QA pairs across the four perceptual reasoning tasks in the VisDoTQA dataset.

Model	Param	ChartQA			ChartQAPro						VisDoTQA
		Mac.	Hum.	Avg.	Factoid	Multi.	Conv.	Check.	Hypo.	Avg.	Pos.	Len.	Pat.	Ext.	Avg.
GPT-4o	–	–	–	85.7	35.76	46.72	34.75	45.49	28.91	37.67	74.57	55.00	36.33	57.03	57.14
Gemini-Flash-2.0	–	90.00	80.24	85.12	43.43	60.28	40.25	67.62	24.47	46.85	77.71	73.75	41.57	50.95	61.96
ChartGemma	3B	90.80	69.52	80.16	6.86	0.00	16.00	1.22	6.53	24.86	6.84	24.86	47.92	27.72	30.8
InternVL	2B	83.52	64.80	74.16	13.86	10.74	14.02	45.90	18.92	17.81	44.29	54.17	19.10	17.87	34.20
InternVL	4B	84.00	66.16	75.08	25.81	10.98	15.85	47.13	25.07	32.11	58.00	59.17	25.47	27.38	43.30

Table 3: Comparison of baseline and VisDoT-tuned models across ChartQA, ChartQAPro, and VisDoTQA. VisDoT models leverage perception-following datasets and DoT-based supervision. For entries reported as default / short-answer † , the value after “/” is obtained under the short-answer (final-answer-only) setting. Abbreviations: Mac. = Machine, Hum. = Human, Avg. = Average, Multi. = Multi-choice, Conv. = Conversational, Check. = Fact-checking, Hypo. = Hypothetical, Pos. = Position, Len. = Length, Pat. = Pattern, Ext. = Extract.

Method	# QA	ChartQA	ChartQAPro	VisDoTQA
ChartGemma	163K	83.96	36.61	26.79
ECD	320K	75.24	17.24	58.30
VisDoT (Ours)	7.4K	84.08	34.02	70.00

Table 4: Comparison of chart distillation/data-synthesis pipelines using publicly released instruction data: ChartGemma Masry et al. (2025b) and ECD Yang et al. (2025) . All methods are fine-tuned on the same backbone (InternVL2.5-4B) and evaluated on ChartQA, ChartQAPro, and VisDoTQA. # QA denotes the number of QA pairs used for fine-tuning; for VisDoT, we randomly sample 7.4K QA pairs for this comparison, while Table 3 reports results using the full VisDoTQA training set (331,969 QA pairs).

Method	ChartQA			VisDoTQA
	Machine	Human	Avg.	Extract	Position	Length	Pattern	Avg.
Base	84.00	66.16	75.08	58.00	59.17	25.47	27.38	43.30
CoT-trained model	73.96 / 87.68 †	63.92	68.94 / 75.8 †	57.14	52.50	30.34	56.27	49.55
Perceptual-trained model	91.20	72.56	81.88	65.43	62.92	22.10	28.14	45.80
DoT-trained model	89.68 / 91.28 †	76.24	82.96 / 83.76 †	68.86	66.25	51.31	66.54	63.57
Perceptual + DoT (Ours)	83.44 / 91.20 †	76.96	80.20 / 84.08 †	78.57	65.83	61.42	71.10	70.00

Table 5: Performance comparison of training strategies on ChartQA and VisDoTQA using InternVL2.5-4B as the base model. † denotes short-answer evaluation (final-answer-only).

Model	Pos.	Len.	Pat.	Ext.	Avg.
All	83.43%	63.33%	63.30%	74.90%	72.32%
w/o Position	73.43%	66.25%	63.30%	72.24%	69.20%
w/o Length	85.14%	65.83%	64.04%	70.34%	72.50%
w/o Pattern	84.00%	64.58%	57.68%	69.96%	70.27%

Table 6: Performance across four perceptual tasks under ablation settings. For brevity, we denote Position as Pos. , Length as Len. , Pattern as Pat. , Extract as Ext. , and Average as Avg. .

Dataset	Direct	CoT	Ours (DoT)
POPE	84.29%	84.64%	86.07%
MMMU	17.7%	35.5%	37.7%

Table 7: Zero-shot performance comparison on POPE and MMMU benchmarks using Direct prompting, CoT, and our proposed DoT strategy.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Epoch	Learning Rate	Batch Size	Hours
InternVL-2B	1	4e-5	16	10
InternVL-4B	1	4e-5	10	12
Gemma3-4B	1	2e-4	4	20
Qwen2.5-VL-3B	1	2e-4	64	29

Appendix

Table 8: Training configurations used for different LVLM backbones, including number of epochs, learning rate, batch size, and training time.

Model	Param	ChartQA			ChartQAPro						VisDoTQA
		Mac.	Hum.	Avg.	Factoid	Multi.	Conv.	Check.	Hypo.	Avg.	Pos.	Len.	Pat.	Ext.	Avg.
Gemma (CoT)	4B	–	58.24	–	11.01	24.77	–	47.13	12.24	–	46.86	38.75	17.98	43.35	37.41
Qwen (CoT)	3B	–	67.12	–	24.46	32.24	5.40	45.49	33.97	–	67.43	54.58	48.69	65.78	59.82
VisDoT_Gemma	4B	84.16	77.04	80.60	23.40	28.04	–	54.10	38.78	–	79.14	58.75	59.18	70.72	67.32
VisDoT_Qwen	3B	86.96	79.60	83.28	26.46	37.38	–	43.44	40.82	–	77.14	55.42	59.55	73.00	68.04

Appendix

Table 9: Comparison of CoT and VisDoT-tuned models across ChartQA, ChartQAPro, and VisDoTQA. VisDoT models leverage perception-following datasets and DoT-based supervision. Gray cells (if any) denote short-answer performance. Abbreviations: Mac. = Machine, Hum. = Human, Avg. = Average, Multi. = Multi-choice, Conv. = Conversational, Check. = Fact-checking, Hypo. = Hypothetical, Pos. = Position, Len. = Length, Pat. = Pattern, Ext. = Extract.

Figure 10: ChartQAPro Fact Checking task

System Prompt: Question Generation - Position Prompt
You are an AI assistant that generates location-based analytical questions from structured visual data, such as charts and tables. Your task is to analyze the image and generate questions that extract values based on their spatial position .
How to Generate Questions
- Use the relative position of elements instead of direct category names as the primary reference.
- However, also include the actual name/label in parentheses after each positional reference for clarity.
- Example:
- Rows (Vertical Order) : "third region from the top (South America)", "second from the bottom (Europe)"

Appendix

Table 10: Perception-following Question Generation - Position Prompt

System Prompt: Perception-following Question Generation - Length Prompt
System Prompt
You are an expert in analyzing data visualizations and generating insightful questions.
Based on a given chart image, generate appropriate questions.
Required Operations
- extract : Ask for the value of a specific category.
- ranking : Identify the item, year, or group with the highest, 2nd, 3rd,… lowest value.

Appendix

Table 11: Perception-following Question Generation - Length Prompt

System Prompt: Perception-following Question Generation - Pattern Prompt
You are an AI assistant that generates analytical questions based on recurring visual patterns in charts. Your task is to create complex, meaningful questions that require pattern recognition and reasoning over visual groupings such as colors, legends, markers, or labeled categories, rather than simple label-based lookup.
Required Operations (use each at least twice across 20 questions):
- Pattern Counting : Identifying how many elements share a common visual pattern (e.g., “How many wedges are in shades of blue?”).
- Color/Legend Matching : Comparing values between categories based on shared legend color or visual symbol.
- Pattern-based Addition & Subtraction : Summing or comparing values across elements sharing the same visual style (e.g., line types, bar textures).
- Ratio : Determining the proportional relationship between visually grouped categories.

Appendix

Table 12: Perception-following Question Generation - Pattern Prompt

System Prompt: Perception-following Question Generation - Extract Prompt
You are an AI assistant that generates analytical questions based on numerical data in charts. Your task is to create complex, meaningful questions that require reasoning and analysis rather than simple data lookup.
Required Operations (use each at least twice across 20 questions):
- Counting : Identifying how many items meet a specific numerical condition (e.g., "How many regions exceed 20%?").
- Addition & Subtraction : Summing or finding differences between multiple values.
- Multiplication & Division : Calculating proportional changes or percentage comparisons.
- Average & Median : Comparing means and middle values across multiple categories or regions.

Appendix

Table 13: Perception-following Question Generation - Extract Prompt

System Prompt: DoT Prompt
You are an AI assistant that analyzes chart images and constructs the reasoning process to derive answers when given question–answer pairs.
You must follow a structured approach by decomposing the question into sub-questions and solving them systematically.
Response Format:
—
Let’s break down this problem.
Question Phase:

Appendix

Table 14: DoT Prompt

Category	Prompt Template
ChartQAPRO (fact checking)	You are a meticulous chart-analysis assistant. Based on the chart image, decide whether the following statement is true or false. Your final answer must be either ‘True’ or ‘False’.
ChartQAPRO (multi choice)	You are a meticulous chart-analysis assistant. Read the chart image and select the most appropriate answer from the multiple-choice options in the question, returning only the final answer as a single letter: A, B, C, or D.
ChartQAPRO (hypothetical, factoid)	You are a meticulous chart-analysis assistant. Analyze the chart image and answer the following question.
ChartQAPRO (conversational)	You are a meticulous chart-analysis assistant. Your goal is to read the multi-turn conversation carefully and provide the answer to the final question. Conversation: {conversation_text} Question: {final_question}
ChartQA, VisDoT	You are a meticulous chart-analysis assistant. Analyze the chart image and answer the following question.

Appendix

Table 15: Model Inference Prompt

Figure 18: Example of position-based reasoning.

Figure 19: Example of length-based reasoning.

Figure 20: Example of pattern-based reasoning.

Figure 21: Example of extractive reasoning.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 846 papers

ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning
Aug 13, 2026Mahsa Khoshnoodi, Sarah Adel BargalChart Understanding Chart QA
Department of Computer Science, Georgetown University
Chart-RL: Policy Optimization Reinforcement Learning for Enhanced Visual Reasoning in Chart Question Answering with Vision Language Models
Apr 3, 2026Yunfei Bai, Amit Dhanda, Shekhar JainChart QA VLM Reasoning
Amazon
Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1VLM Evaluation Data Visualization
University of Waterloo · Vector Institute · Layer 6 AI

Aug 13, 2026J/K move · Enter open · S save

ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

Mahsa Khoshnoodi, Sarah Adel Bargal

Department of Computer Science, Georgetown University

Chart Understanding Chart QA Visually Grounded Reasoning VLM Evaluation Visual Grounding

Open paper AskPDF arXiv

Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (perception), locating them and binding them to their labels (grounding), and performing single-step computations such as ranking, totals, and differences (simple reasoning). We introduce ChartProbe, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.

Abstract

VLM Reasoning

Explore similar work

ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

Chart-RL: Policy Optimization Reinforcement Learning for Enhanced Visual Reasoning in Chart Question Answering with Vision Language Models

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models