Vision-language models (VLMs) aim to reason by jointly leveraging visual and textual modalities. While allocating additional inference-time computation has proven effective for large language models (LLMs), achieving similar scaling in VLMs remains challenging. A key obstacle is that visual inputs are typically provided only once at the start of generation, while textual reasoning (e.g., early visual summaries) is generated autoregressively, causing reasoning to become increasingly text-dominated and allowing early visual grounding errors to accumulate. Moreover, vanilla guidance for visual grounding during inference is often coarse and noisy, making it difficult to steer reasoning over long texts. To address these challenges, we propose Saliency-Aware Principle (SAP) selection. SAP operates on high-level reasoning principles rather than token-level trajectories, which enable stable control over discrete generation under noisy feedback while allowing later reasoning steps to re-consult visual evidence when renewed grounding is required. In addition, SAP supports multi-route inference, enabling parallel exploration of diverse reasoning behaviors. SAP is model-agnostic and data-free, requiring no additional training. Empirical results show that SAP achieves competitive performance, especially in reducing object hallucination, under comparable token-generation budgets while yielding more stable reasoning and lower response latency than CoT-style long sequential reasoning.
Figures & tables
Figure 1: Text reliance in vision-language reasoning. Our SAP, a multi-route approach, employs visual grounding as guidance, alleviating object hallucination in text-dominated LongCoT.
Figure 2: Better perception (Right) is obtained by SAP (Middle) evaluated by perception-intensive benchmarks, OCRVQA and POPE-recall, about object hallucination. Left: Text reliance in reasoning from Qwen3-VL-8B-Thinking on MS-COCO. Right: Saliency-Aware Principle (SAP) selection pipeline and empirical comparison between SAP implementation with Qwen3-VL-8B serie on benchmarks.
LLM, utility and fitness metric on ( r , s ) [those given]
Table 1: Notation: S aliency- A ware P rinciple Selection (SAP).
Reproduced Results
Reported Results (References)
benchmarks ( ↑ )
instruct
thinking
ins.+SAP (ours)
Grok-1.5V
GPT-4V
Claude-S
Claude-O
Gemini-1.5
avg. except MME
75.4
75.9
76.6
Set I. Vision and Thinking: Perception / Reasoning / General Knowledge
POPE-recall ( Li et al., 2023b )
83.9
79.6 ↓
89.9 ↑
–
–
–
–
–
MME-p ( Fu et al., 2025 )
1744.7
1663.2 ↓
1755.5 ↑
–
–
–
–
–
MME-r ( Fu et al., 2025 )
656.7
722.5 ↑
689.9 ↑
–
–
–
–
–
Table 2: Comparison between longer and wider reasoning strategies across 16 vision-language benchmarks. SAP yields better perception and higher average performance with stable visual grounding. Benchmarks are grouped into two sets according to their evaluation focus. All methods use the same backbone and inference-time decoding budget unless otherwise specified. Baselines: Qwen3-VL-8B-Instruct; Qwen3-VL-8B-Thinking: LongCoT, sequential inference-time scaling; SAP (ours): parallel inference-time scaling on routes.
settings
#gen. evo.
#routes τ
decision
avg. perf.
instruct
0
1
–
75.4
w/o evolution
0
τ
elite
73.4
w/o multi-route
T
1
elite
73.0
full SAP
T
τ
elite
76.6
w/ summarizer
T
τ
agg.
74.8
Table 3: Ablation of SAP modules. Harmful principles should not be aggregated. Evolution and filtering are verified important.
λ #new born
2
2
4
4
6
μ #elites
2
4
2
4
6
avg. perf.
76.6
74.8
76.8
76.5
76.8
Table 6
Figure 3: Ablation on weights of rewarding terms in fitness (loss design): aggregation of routes for final answer reduces the reliance on consensus in multimodal reasoning. Diversity, consensus, evidence refer to weights ratio of wd,wc,we where ∑w=1 and uncertainty penalty is always 1, wu=1 .
Figure 4: SAP supports inference-time speedup via parallelism.
Vision-language models (VLMs) have achieved remarkable success across diverse tasks. However, concerns about their trustworthiness persist, particularly regarding tendencies to lean more on textual cues than visual evidence and the risk of producing ungrounded or fabricated responses. To address these issues, we propose Saliency-R1, a framework for improving the interpretability and faithfulness of VLMs reasoning. Specifically, we introduce a novel saliency map technique that efficiently highlights critical image regions contributing to generated tokens without additional computational overhead. This can further be extended to trace how visual information flows through the reasoning process to the final answers, revealing the alignment between the thinking process and the visual context. We use the overlap between the saliency maps and human-annotated bounding boxes as the reward function, and apply Group Relative Policy Optimization (GRPO) to align the salient parts and critical regions, encouraging models to focus on relevant areas when conduct reasoning. Experiments show Saliency-R1 improves reasoning faithfulness, interpretability, and overall task performance.
Shizhan Gong, Minda Hu, Qiyuan Zhang +2
The Chinese University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
We introduce Self-Saliency, a method for training Vision-Language Models (VLMs) to increase the alignment between their visual attention and the image regions mentioned in their reasoning. Self-Saliency uses a grounding model to localize the objects mentioned in each reasoning step and treats the resulting areas as supervision for the model's visual attention. Previous work on steering visual attention determines target image regions based solely on the image and question. In contrast, we show that conditioning the target regions on the model's generated reasoning improves downstream performance. For proper evaluation, we build a unified, broad suite of 25 visual reasoning benchmarks, where we reproduce the results of previous methods. We find that Self-Saliency significantly outperforms both prior attention-steering methods and baselines that ground image-level text, achieving both a better average rank and a better mean score. Post-training analysis shows that the model primarily adapts its reasoning text to existing attention patterns, producing shorter steps that refer to larger regions. Nevertheless, when controlling for generated text, attention to grounded regions increases significantly across the relevant layer. Finally, we identify a consistent geometric bias in VLM visual attention toward the image border. However, our ablations show that Self-Saliency's gains cannot be explained by simply aligning attention with the center of the image, highlighting the importance of aligning visual attention with the regions mentioned in the model's reasoning.
Uri Berger, Gal Chechik, Gal Dalal
NVIDIA Research · The Hebrew University of Jerusalem · University of Melbourne +1
Vision-Language Models often struggle with complex visual reasoning due to the visual information loss in textual CoT. Existing methods either add the cost of tool calls or rely on localized patch-based embeddings that are insufficient to extract semantics in multi-step reasoning. We propose "Decompose, Look, and Reason" (DLR), a reinforced latent reasoning framework that dynamically decomposes queries into textual premises, extracts premise-conditioned continuous visual latents, and deduces answers through grounded rationales. We introduce a three-stage training pipeline and propose a novel Spherical Gaussian Latent Policy, to enable effective exploration in the latent space. Extensive experiments on vision-centric benchmarks show that DLR consistently outperforms strong baselines, including text-only, interleaved multimodal CoT, and latent reasoning methods, while providing superior stepwise interpretability.
Mengdan Zhu, Senhao Cheng, Liang Zhao
Emory University · University of Michigan, Ann Arbor