Multimodal Large Language Models (MLLMs) capable of thinking with images often rely on external tools for fine-grained perception. However, this reliance introduces significant inference latency and fails to effectively resolve the spatial-structural gap-a fundamental challenge in text-dense and structurally relational visuals (e.g., charts and visual tables) where strict relative spatial arrangements bind textual elements. Without external tools, standard MLLMs struggle with such fine-grained visual reasoning tasks. To address these issues, we propose Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model. TwSG distills the benefits of multi-step reasoning and micro-cropping into a single efficient forward pass during inference. Specifically, we use an MLLM to identify key regions guided by ground-truth answers, and then prompt a teacher model to generate high-quality visual question-answering (VQA) data. These fine-grained, region-based supervisory signals are subsequently distilled back into the full-image representation. Our training pipeline consists of two stages: (1) a cold-start supervised fine-tuning (SFT) phase using multi-turn data with focused area descriptions to foster complex reasoning and error recovery; and (2) a reinforcement fine-tuning (RFT) phase driven by a novel process reward mechanism, TL-GRPO, which encourages strategic reasoning. Extensive experiments across various MLLM architectures demonstrate that TwSG reduces inference latency while substantially improving accuracy and robustness, endowing models with native fine-grained region description and flexible reasoning capabilities.
Figures & tables
Figure 1: Pipeline for cold-start SFT data generation.
Figure 2: Comparison between Tag-level Importance Sampling and token-level or sequence-level importance sampling.
Method
Size
Chart
Visual Tabular
Avg.
ChartQA
ChartQAPro
TableVQA-Bench
M
H
Factoid
MCQ
Convers.
FactChk.
Hypoth.
VWTQ
VWTQ-Syn
VTabFact
FinTabNetQA
General MLLMs
4B
95.44
78.80
37.34
43.46
31.48
52.46
43.86
60.11
65.55
83.20
77.58
60.84
LLaVA-OneVision-1.5
8B
94.64
78.88
38.88
35.05
34.72
53.28
46.38
62.20
71.33
84.00
76.82
61.47
3B
94.24
74.24
29.24
38.32
33.86
49.59
38.91
56.27
65.59
75.60
77.76
57.60
Table 1: Performance comparison on chart and visual-tabular reasoning benchmarks. Accuracy (%) is reported for the machine-generated (M) and human-generated (H) subsets of ChartQA, five question types of ChartQAPro, and four data sources of TableVQA-Bench. Avg. denotes the macro-average over all 11 subsets.
Figure 3: Samples per second.
Effect of TL-GRPO
Benchmark
GRPO
GRPO + TL-IS
GRPO + CGS
TL-GRPO (Full)
ChartQAPro
49.40
52.34
53.65
55.60
TableVQA-Bench
77.80
83.33
82.53
86.79
Reward Component Ablation
Benchmark
w/o R
Rans
Rans+Rfmt
Rans+Rfmt+Rverify
ChartQAPro
48.22
47.30
52.30
55.60
Table 2: Ablation studies of TL-GRPO on ChartQAPro and TableVQA-Bench. w/o R denotes the SFT checkpoint before reinforcement learning. Best results are shown in bold.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Table 3: Performance comparison on chart reasoning benchmarks. Left: results on ChartQA with open-source MLLMs. Right: comparison with proprietary models on the CharXiv-R subset. “Tools” indicates whether external tool use is enabled. Bold denotes the best, and underline the second best. ChartMOE performs inference with an external Python tool.
(a) Judge Model
Method
ChartQAPro
TableVQA-Bench
GPT-4o
54.44
85.32
Qwen3.6-plus
56.32
85.90
Qwen3-VL-72B-Instruct (Default)
55.60
86.79
Appendix
Table 4: Ablation studies on reward verification and reward weighting.
(a) Seed
Seed
ChartQAPro
TableVQA
42
55.60
86.79
1024
55.47
86.62
8192
55.71
86.84
Appendix
Table 5: Impact of decoding hyperparameters. Left: different random seeds. Right: different sampling temperatures.
Figure 4: Prompt for Verify Reward.
Figure 5: Illustrative examples of the 45 chart and table categories covered in our benchmark.
Figure 6: Prompt for ROI extraction.
Figure 7: Prompt for region description and text extraction.
Figure 8: Prompt for CoT refinement and structuring.
Figure 9: The Static of TwSG-12K.
Figure 10: Reasoning comparisons between TwSG and existing CvTR-domain MLLMs.
Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.
While visually grounded Chain-of-Thought (CoT) has emerged as a promising paradigm to enhance fine-grained perception in multimodal large language models (MLLMs), its efficacy during the inference phase remains underexplored. In this work, we empirically find that mandating explicit object boxes in visually grounded CoT during inference often degrades performance compared to standard textual CoT, which reasons without explicit visual grounding. We hypothesize that the visual localization capability can be internalized into the textual CoT and that the mandatory explicit grounding introduces unnecessary interference with the model's primary objective of answer prediction. To address this problem, we propose Internalizing Visually Grounded Reasoning (iVGR), a novel reinforcement learning framework that transfers localization capabilities into the textual reasoning process. We employ a dual-stream training strategy, where a textual stream is aligned with a high-quality visually grounded stream via a proposed consistency reward, enabling the model to localize accurately without explicit grounding during inference. Extensive experiments demonstrate that our method significantly outperforms existing baselines on fine-grained benchmarks, while maintaining the flexibility to support tool-assisted inference workflows.
Chang-Bin Zhang, Yujie Zhong, Qiang Zhang +1
Visual AI Lab, The University of Hong Kong · Independent Researcher · University of Science and Technology of China
Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately 4.0× inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.
Yuxiang Shen, Hailong Huang, Zhenkun Gao +6
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
Multimodal Large Language Models (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.
Zhiwei Yang, Yuanchen Wu, Nan Zhang +3
Tencent Youtu Lab, Shanghai, China · Fudan University, Shanghai, China