Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

Changjiang Jiang; Qiannian Zhao; Lei Xin; Jinxiang Xie; Preslav Nakov; Zhuohan Xie

cs.AIAug 23, 2026

Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

Authors: Changjiang Jiang, Qiannian Zhao, Lei Xin, Jinxiang Xie, Preslav Nakov, Zhuohan Xie

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Nanjing University

Abstract

Multimodal Large Language Models (MLLMs) capable of thinking with images often rely on external tools for fine-grained perception. However, this reliance introduces significant inference latency and fails to effectively resolve the spatial-structural gap-a fundamental challenge in text-dense and structurally relational visuals (e.g., charts and visual tables) where strict relative spatial arrangements bind textual elements. Without external tools, standard MLLMs struggle with such fine-grained visual reasoning tasks. To address these issues, we propose Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model. TwSG distills the benefits of multi-step reasoning and micro-cropping into a single efficient forward pass during inference. Specifically, we use an MLLM to identify key regions guided by ground-truth answers, and then prompt a teacher model to generate high-quality visual question-answering (VQA) data. These fine-grained, region-based supervisory signals are subsequently distilled back into the full-image representation. Our training pipeline consists of two stages: (1) a cold-start supervised fine-tuning (SFT) phase using multi-turn data with focused area descriptions to foster complex reasoning and error recovery; and (2) a reinforcement fine-tuning (RFT) phase driven by a novel process reward mechanism, TL-GRPO, which encourages strategic reasoning. Extensive experiments across various MLLM architectures demonstrate that TwSG reduces inference latency while substantially improving accuracy and robustness, endowing models with native fine-grained region description and flexible reasoning capabilities.

Figures & tables

Figure 1: Pipeline for cold-start SFT data generation.

Figure 2: Comparison between Tag-level Importance Sampling and token-level or sequence-level importance sampling.

Method	Size	Chart							Visual Tabular				Avg.
		ChartQA		ChartQAPro					TableVQA-Bench
		M	H	Factoid	MCQ	Convers.	FactChk.	Hypoth.	VWTQ	VWTQ-Syn	VTabFact	FinTabNetQA
General MLLMs
	4B	95.44	78.80	37.34	43.46	31.48	52.46	43.86	60.11	65.55	83.20	77.58	60.84
LLaVA-OneVision-1.5	8B	94.64	78.88	38.88	35.05	34.72	53.28	46.38	62.20	71.33	84.00	76.82	61.47
	3B	94.24	74.24	29.24	38.32	33.86	49.59	38.91	56.27	65.59	75.60	77.76	57.60

Table 1: Performance comparison on chart and visual-tabular reasoning benchmarks. Accuracy (%) is reported for the machine-generated (M) and human-generated (H) subsets of ChartQA, five question types of ChartQAPro, and four data sources of TableVQA-Bench. Avg. denotes the macro-average over all 11 subsets.

Figure 3: Samples per second.

Effect of TL-GRPO
Benchmark	GRPO	GRPO + TL-IS	GRPO + CGS	TL-GRPO (Full)
ChartQAPro	49.40	52.34	53.65	55.60
TableVQA-Bench	77.80	83.33	82.53	86.79
Reward Component Ablation
Benchmark	w/o $R$	$R_{\mathrm{ans}}$	$R_{\mathrm{ans}}+R_{\mathrm{fmt}}$	$R_{\mathrm{ans}}+R_{\mathrm{fmt}}+R_{\mathrm{verify}}$
ChartQAPro	48.22	47.30	52.30	55.60

Table 2: Ablation studies of TL-GRPO on ChartQAPro and TableVQA-Bench. w/o

R

denotes the SFT checkpoint before reinforcement learning. Best results are shown in bold.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Table 3: Performance comparison on chart reasoning benchmarks. Left: results on ChartQA with open-source MLLMs. Right: comparison with proprietary models on the CharXiv-R subset. “Tools” indicates whether external tool use is enabled. Bold denotes the best, and underline the second best. ChartMOE performs inference with an external Python tool.

(a) Judge Model
Method	ChartQAPro	TableVQA-Bench
GPT-4o	54.44	85.32
Qwen3.6-plus	56.32	85.90
Qwen3-VL-72B-Instruct (Default)	55.60	86.79

Appendix

Table 4: Ablation studies on reward verification and reward weighting.

(a) Seed
Seed	ChartQAPro	TableVQA
42	55.60	86.79
1024	55.47	86.62
8192	55.71	86.84

Appendix

Table 5: Impact of decoding hyperparameters. Left: different random seeds. Right: different sampling temperatures.

Panel of Figure 4 — Figure 4: Prompt for Verify Reward.

Figure 5: Illustrative examples of the 45 chart and table categories covered in our benchmark.

Panel of Figure 6 — Figure 6: Prompt for ROI extraction.

Panel of Figure 7 — Figure 7: Prompt for region description and text extraction.

Panel of Figure 8 — Figure 8: Prompt for CoT refinement and structuring.

Figure 9: The Static of TwSG-12K.

Figure 10: Reasoning comparisons between TwSG and existing CvTR-domain MLLMs.

Visually Grounded Reasoning

Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.

See all 342 papers

iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning
May 29, 2026Chang-Bin Zhang, Yujie Zhong, Qiang Zhang +1Large Vision-Language Models Vision-Language Grounding
Visual AI Lab, The University of Hong Kong · Independent Researcher · University of Science and Technology of China
LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
Feb 26, 2026Yuxiang Shen, Hailong Huang, Zhenkun Gao +6VLM Reasoning Large Vision-Language Models
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
Jul 7, 2026Zhiwei Yang, Yuanchen Wu, Nan Zhang +3Multimodal Large Language Models Multimodal CoT Reasoning
Tencent Youtu Lab, Shanghai, China · Fudan University, Shanghai, China

May 29, 2026J/K move · Enter open · S save

iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning

Chang-Bin Zhang, Yujie Zhong, Qiang Zhang, Kai Han

Visual AI Lab, The University of Hong Kong · Independent Researcher · University of Science and Technology of China

Large Vision-Language Models Vision-Language Grounding Visually Grounded Reasoning Fine-Grained Visual Perception RL for VLMs

Open paper AskPDF arXiv

While visually grounded Chain-of-Thought (CoT) has emerged as a promising paradigm to enhance fine-grained perception in multimodal large language models (MLLMs), its efficacy during the inference phase remains underexplored. In this work, we empirically find that mandating explicit object boxes in visually grounded CoT during inference often degrades performance compared to standard textual CoT, which reasons without explicit visual grounding. We hypothesize that the visual localization capability can be internalized into the textual CoT and that the mandatory explicit grounding introduces unnecessary interference with the model's primary objective of answer prediction. To address this problem, we propose Internalizing Visually Grounded Reasoning (iVGR), a novel reinforcement learning framework that transfers localization capabilities into the textual reasoning process. We employ a dual-stream training strategy, where a textual stream is aligned with a high-quality visually grounded stream via a proposed consistency reward, enabling the model to localize accurately without explicit grounding during inference. Extensive experiments demonstrate that our method significantly outperforms existing baselines on fine-grained benchmarks, while maintaining the flexibility to support tool-assisted inference workflows.

Abstract

Visually Grounded Reasoning

Explore similar work

iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models