Vision-Language Models (VLMs) have achieved remarkable progress on a wide range of challenging multimodal understanding and reasoning tasks. However, existing reasoning paradigms, such as the classical Chain-of-Thought (CoT), rely solely on textual information and often underutilize important visual cues. While prior work has incorporated pixel-level visual cues, these representations require precise spatial localization, introducing additional learning complexity. To address this, we propose PatchCue, a novel patch-based visual cue paradigm designed to significantly enhance the visual reasoning capabilities of VLMs. By partitioning images into patches and representing cues at the patch level, PatchCue aligns better with human perceptual habits and leverages the patch-tokenized input of modern VLMs. We train VLMs using a two-stage approach: cold-start supervised fine-tuning to output patch-level cues, followed by reinforcement learning with a process-supervised cue reward that guides intermediate visual reasoning steps. Extensive experiments on multiple VLMs and diverse benchmarks, including general visual question answering, complex reasoning, and document understanding, demonstrate that PatchCue consistently improves overall model performance. Our results show that patch-level cues outperform both pixel-level bounding boxes and point-based cues, providing a more effective and cognitively aligned visual reasoning paradigm.
Figures & tables
Figure 1: Comparison of reasoning with different cue types: (a) Text-only: reasoning based solely on textual information; (b) Pixel-bbox: cues represented as precise pixel-level bounding boxes; (c) Pixel-point: cues indicated by single pixel points highlighting key regions; (d) Patch-bbox: cues represented as patch-level regions to capture localized visual information; (e) SFT training comparison shows that patch-based cues improve model performance more effectively than pixel-bbox or pixel-point cues.
Figure 2: Overview of PatchCue. We divide images into fixed-size patches in order to represent important regions as visual cues. During the model’s reasoning process, it is essential not only to identify which patches are relevant to the given question but also to accurately reference and integrate these cues throughout each reasoning step. This structured use of patch-level cues helps the model ground its intermediate reasoning in the visual content, improving both interpretability and overall performance.
Figure 3: Data Pipeline. Starting from the collected original data, we filter to obtain challenging samples. Then extract and ground the key visual cues in the images, and finally construct new reasoning sequences based on these cues.
Figure 4: Data Distribution. In the left figure, we show the distribution of the number of cues per sample, where most cue data are concentrated between 2 and 5 cues; in the right figure, we show the distribution of the proportion of cue regions, with the majority of samples having cue regions occupying less than 40% of the image.
Category
Benchmark
Qwen2.5-VL-3B
+PC
Qwen2.5-VL-7B
+PC
MiMo-VL-7B
+PC
General visual question answering
HallusionBench
46.3
47.5
52.9
53.5
52.3
53.7
MMVet
60.0
63.2
69.7
74.2
72.2
75.8
MMBench
79.1
79.1
82.2
82.5
83.2
83.8
MMStar
55.9
56.8
63.9
66.2
67.2
67.4
MMVP
70.7
70.3
77.7
79.3
72.7
74.3
RealWorldQA
67.5
68.0
68.5
69.3
73.3
77.5
Table 1: Main results across multiple benchmarks. We evaluate the performance of various VLMs trained with our PatchCue paradigm, demonstrating consistent improvements over baseline models across diverse datasets. The notation “+PC" indicates models trained with our patch-bbox visual cue data.
Category
Benchmark
Baseline
Pixel-Bbox
Pixel-Point
Patch-Bbox
Patch-Point
Labels
General visual question answering
HallusionBench
52.9
51.5
53.0
52.5
53.1
52.9
MMVet
69.7
66.0
65.3
70.4
65.3
63.3
MMBench
82.2
81.2
80.0
82.1
80.1
78.6
MMStar
63.9
64.9
64.8
65.6
64.7
64.6
MMVP
77.7
78.1
77.3
78.7
78.0
77.6
RealWorldQA
68.5
69.3
69.3
69.3
69.9
69.9
Table 2: Performance comparison across different forms of visual cues. We compare the impact of different visual cue formats on model performance under the same data scale and reasoning paradigm, where “Baseline” denotes the original results of Qwen2.5-VL 7B. The other columns show the results after SFT training using the cue data corresponding to each representation type.
Training Setup
AI2D [ 22 ]
ChartQA [ 31 ]
MMStar [ 4 ]
MMVP [ 47 ]
Baseline
83.9
87.3
63.9
77.7
Gen Only (1:0)
83.5
87.5
65.3
76.7
Hybrid (2:1)
84.6
87.5
64.6
78.0
Hybrid (1:1)
84.6
87.9
65.6
78.7
Hybrid (1:2)
85.1
87.4
64.9
71.0
Cue Only (0:1)
80.8
86.6
60.3
68.0
Table 3: Performance comparison under different training data setups. We evaluate the impact of training data composition on model performance. “Baseline” denotes the original Qwen2.5-VL-7B results. Other rows indicate training on different combinations of general (Gen) and patch-bbox cue (Cue) data, with ratios specified as Gen:Cue.
Model
AI2D [ 22 ]
ChartQA [ 31 ]
MMStar [ 4 ]
MMVP [ 47 ]
Baseline
83.9
87.3
65.3
77.7
+SFT
84.6
87.9
65.6
78.7
+RL(w/o Rcue )
83.9
87.5
65.7
78.7
+RL
84.7
88.1
66.2
79.3
Table 4: Ablation of cue reward. We compare the performance differences between RL training with and without the use of Rcue . “Baseline” denotes the original Qwen2.5-VL-7B results, “+SFT” represents the results after SFT training, “+RL (w/o Rcue )” indicates RL training based on the SFT model without applying Rcue , and “+RL” denotes RL training with the incorporation of Rcue .
Method
AI2D [ 22 ]
ChartQA [ 31 ]
MMStar [ 4 ]
MMVP [ 47 ]
VisualCoT [ 40 ]
83.9
85.0
63.8
75.2
CogCom [ 35 ]
84.6
85.6
64.0
76.0
MINI-CoT [ 6 ]
84.1
87.9
64.8
77.7
PatchCue
84.7
88.1
66.2
79.3
Table 5: Performance comparison of different methods on multimodal benchmarks. We sample ∼ 12K instances from VisualCoT, CogCom, and MINI-CoT, and fine-tune the same backbone (Qwen2.5-VL-7B) with an identical SFT protocol for a fair comparison across methods with different original settings.
Figure 5: Case Study. We compare the model’s outputs before and after PatchCue training. After training, the model can generate visual cues during reasoning, improving both its perception and the interpretability of its reasoning process.
Hyperparameter
Settings
DeepSpeed Stage
2
Warmup Ratio
0.05
Trainable Module
LLM
Epoch
1
LR Schedule
cosine
Learning Rate
1e-5
Table 6: Hyperparameters for SFT Training.
Hyperparameter
Settings
Beta
0.001
Torch Dtype
bfloat16
Learning Rate
1e-6
Warmup Ratio
0.05
Num Generations
8
Epoch
3
Table 7: Hyperparameters for GRPO Training
Figure 6: Successful examples where visual cues effectively guide the model to conduct accurate and interpretable reasoning.
Figure 7: Common failure cases of cue-based reasoning, including flawed reasoning despite reasonable cues, incorrect cue localization, and localization deviations that mislead the final prediction.
Recent advances in Vision-Language Models (VLMs) have benefited from Reinforcement Learning (RL) for enhanced reasoning. However, existing methods still face critical limitations, including the lack of low-level visual information and effective visual feedback. To address these problems, this paper proposes a unified multimodal interleaved reasoning framework ForeSight, which enables VLMs to See Further with low-level visual cues and Think Deeper with effective visual feedback. First, it introduces a set of low-level visual tools to integrate essential visual information into the reasoning chain, mitigating the neglect of fine-grained visual features. Second, a mask-based visual feedback mechanism is elaborated to incorporate visual reflection into the thinking process, enabling the model to dynamically re-examine and update its answers. Driven by RL, ForeSight learns to autonomously decide on tool invocation and answer verification, with the final answer accuracy as the reward signal. To evaluate the performance of the proposed framework, we construct a new dataset, Character and Grounding SalBench (CG-SalBench), based on the SalBench dataset. Experimental results demonstrate that the ForeSight-7B model significantly outperforms other models with the same parameter scale, and even surpasses the current SOTA closed-source models on certain metrics.
Zhiheng Wu, Tong Wang, Shuning Wang +2
1Baidu Inc. · †Independent researcher. · 2Zhejiang University +1
Large Vision-Language Models (LVLMs) have shown strong promise for multimodal reasoning, yet often struggle with tasks requiring concepts beyond what is directly observable in the input image. Existing methods generate intermediate images or latent visual tokens to guide reasoning, but these representations can introduce errors and increasingly interfere with textual reasoning as reasoning progresses. We propose Visual-to-Text Chain-of-Thought Distillation (V2T), a framework that enables LVLMs to internalize visual reasoning without generating intermediate visual representations at inference time. V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher's logits and cross-entropy supervision from ground-truth textual reasoning. When reasoning images can be mapped to the original image, V2T can additionally distill the teacher's attention to corresponding regions, while ground-truth bounding boxes can further guide a subsequent reinforcement learning stage. Experiments across multiple multimodal reasoning benchmarks show that V2T consistently outperforms the teacher and existing baselines, improving average accuracy by 14.3% on a held-out set and 2.7% on the broader visual evaluation suite. Moreover, lightweight SFT and substantially reduced RL make V2T up to 42x faster to train than state-of-the-art baselines.
Wenhan Yang, Nilay Naharas, Ali Payani +1
University of California, Los Angeles · Cisco Research
Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet they remain limited on tasks requiring fine-grained understanding of small or easily overlooked image regions. Recent thinking with images approaches mitigate this limitation by allowing models to inspect images through operations such as zooming, cropping, or code execution. While effective, their multi-turn interactions with VLMs introduce redundant token computation and slow inference, and separate region inspection from the reasoning step that depends on it. To address these issues, we propose Look Before You Reason (LoRe), an implicit visual thinking framework for efficient visual reasoning within a single-turn VLM interaction. Given an image and a question, LoRe predicts the most relevant image region, aligns it with the corresponding image patch features, and strengthens the associated key-value (KV) cache entries, enabling subsequent decoding to directly exploit the selected visual cues without additional visual inputs or explicit tool calls. We further employ reinforcement learning to guide the VLM toward identifying informative image regions and leveraging them during reasoning. To support fine-grained evaluation of multimodal reasoning, we introduce COCO-ObjQA, a large-scale benchmark for object-centric multimodal reasoning. Experiments on three challenging multimodal benchmarks demonstrate that LoRe consistently improves VLM reasoning performance with low additional inference overhead.