Vision-Language Models (VLMs) have achieved remarkable progress on a wide range of challenging multimodal understanding and reasoning tasks. However, existing reasoning paradigms, such as the classical Chain-of-Thought (CoT), rely solely on textual information and often underutilize important visual cues. While prior work has incorporated pixel-level visual cues, these representations require precise spatial localization, introducing additional learning complexity. To address this, we propose PatchCue, a novel patch-based visual cue paradigm designed to significantly enhance the visual reasoning capabilities of VLMs. By partitioning images into patches and representing cues at the patch level, PatchCue aligns better with human perceptual habits and leverages the patch-tokenized input of modern VLMs. We train VLMs using a two-stage approach: cold-start supervised fine-tuning to output patch-level cues, followed by reinforcement learning with a process-supervised cue reward that guides intermediate visual reasoning steps. Extensive experiments on multiple VLMs and diverse benchmarks, including general visual question answering, complex reasoning, and document understanding, demonstrate that PatchCue consistently improves overall model performance. Our results show that patch-level cues outperform both pixel-level bounding boxes and point-based cues, providing a more effective and cognitively aligned visual reasoning paradigm.
Figures & tables
Figure 1: Comparison of reasoning with different cue types: (a) Text-only: reasoning based solely on textual information; (b) Pixel-bbox: cues represented as precise pixel-level bounding boxes; (c) Pixel-point: cues indicated by single pixel points highlighting key regions; (d) Patch-bbox: cues represented as patch-level regions to capture localized visual information; (e) SFT training comparison shows that patch-based cues improve model performance more effectively than pixel-bbox or pixel-point cues.
Figure 2: Overview of PatchCue. We divide images into fixed-size patches in order to represent important regions as visual cues. During the model’s reasoning process, it is essential not only to identify which patches are relevant to the given question but also to accurately reference and integrate these cues throughout each reasoning step. This structured use of patch-level cues helps the model ground its intermediate reasoning in the visual content, improving both interpretability and overall performance.
Figure 3: Data Pipeline. Starting from the collected original data, we filter to obtain challenging samples. Then extract and ground the key visual cues in the images, and finally construct new reasoning sequences based on these cues.
Figure 4: Data Distribution. In the left figure, we show the distribution of the number of cues per sample, where most cue data are concentrated between 2 and 5 cues; in the right figure, we show the distribution of the proportion of cue regions, with the majority of samples having cue regions occupying less than 40% of the image.
Category
Benchmark
Qwen2.5-VL-3B
+PC
Qwen2.5-VL-7B
+PC
MiMo-VL-7B
+PC
General visual question answering
HallusionBench
46.3
47.5
52.9
53.5
52.3
53.7
MMVet
60.0
63.2
69.7
74.2
72.2
75.8
MMBench
79.1
79.1
82.2
82.5
83.2
83.8
MMStar
55.9
56.8
63.9
66.2
67.2
67.4
MMVP
70.7
70.3
77.7
79.3
72.7
74.3
RealWorldQA
67.5
68.0
68.5
69.3
73.3
77.5
Table 1: Main results across multiple benchmarks. We evaluate the performance of various VLMs trained with our PatchCue paradigm, demonstrating consistent improvements over baseline models across diverse datasets. The notation “+PC" indicates models trained with our patch-bbox visual cue data.
Category
Benchmark
Baseline
Pixel-Bbox
Pixel-Point
Patch-Bbox
Patch-Point
Labels
General visual question answering
HallusionBench
52.9
51.5
53.0
52.5
53.1
52.9
MMVet
69.7
66.0
65.3
70.4
65.3
63.3
MMBench
82.2
81.2
80.0
82.1
80.1
78.6
MMStar
63.9
64.9
64.8
65.6
64.7
64.6
MMVP
77.7
78.1
77.3
78.7
78.0
77.6
RealWorldQA
68.5
69.3
69.3
69.3
69.9
69.9
Table 2: Performance comparison across different forms of visual cues. We compare the impact of different visual cue formats on model performance under the same data scale and reasoning paradigm, where “Baseline” denotes the original results of Qwen2.5-VL 7B. The other columns show the results after SFT training using the cue data corresponding to each representation type.
Training Setup
AI2D [ 22 ]
ChartQA [ 31 ]
MMStar [ 4 ]
MMVP [ 47 ]
Baseline
83.9
87.3
63.9
77.7
Gen Only (1:0)
83.5
87.5
65.3
76.7
Hybrid (2:1)
84.6
87.5
64.6
78.0
Hybrid (1:1)
84.6
87.9
65.6
78.7
Hybrid (1:2)
85.1
87.4
64.9
71.0
Cue Only (0:1)
80.8
86.6
60.3
68.0
Table 3: Performance comparison under different training data setups. We evaluate the impact of training data composition on model performance. “Baseline” denotes the original Qwen2.5-VL-7B results. Other rows indicate training on different combinations of general (Gen) and patch-bbox cue (Cue) data, with ratios specified as Gen:Cue.
Model
AI2D [ 22 ]
ChartQA [ 31 ]
MMStar [ 4 ]
MMVP [ 47 ]
Baseline
83.9
87.3
65.3
77.7
+SFT
84.6
87.9
65.6
78.7
+RL(w/o Rcue )
83.9
87.5
65.7
78.7
+RL
84.7
88.1
66.2
79.3
Table 4: Ablation of cue reward. We compare the performance differences between RL training with and without the use of Rcue . “Baseline” denotes the original Qwen2.5-VL-7B results, “+SFT” represents the results after SFT training, “+RL (w/o Rcue )” indicates RL training based on the SFT model without applying Rcue , and “+RL” denotes RL training with the incorporation of Rcue .
Method
AI2D [ 22 ]
ChartQA [ 31 ]
MMStar [ 4 ]
MMVP [ 47 ]
VisualCoT [ 40 ]
83.9
85.0
63.8
75.2
CogCom [ 35 ]
84.6
85.6
64.0
76.0
MINI-CoT [ 6 ]
84.1
87.9
64.8
77.7
PatchCue
84.7
88.1
66.2
79.3
Table 5: Performance comparison of different methods on multimodal benchmarks. We sample ∼ 12K instances from VisualCoT, CogCom, and MINI-CoT, and fine-tune the same backbone (Qwen2.5-VL-7B) with an identical SFT protocol for a fair comparison across methods with different original settings.
Figure 5: Case Study. We compare the model’s outputs before and after PatchCue training. After training, the model can generate visual cues during reasoning, improving both its perception and the interpretability of its reasoning process.
Hyperparameter
Settings
DeepSpeed Stage
2
Warmup Ratio
0.05
Trainable Module
LLM
Epoch
1
LR Schedule
cosine
Learning Rate
1e-5
Table 6: Hyperparameters for SFT Training.
Hyperparameter
Settings
Beta
0.001
Torch Dtype
bfloat16
Learning Rate
1e-6
Warmup Ratio
0.05
Num Generations
8
Epoch
3
Table 7: Hyperparameters for GRPO Training
Figure 6: Successful examples where visual cues effectively guide the model to conduct accurate and interpretable reasoning.
Figure 7: Common failure cases of cue-based reasoning, including flawed reasoning despite reasonable cues, incorrect cue localization, and localization deviations that mislead the final prediction.