Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs

Zhiyu Pan; Yizheng Wu; Jiashen Hua; Junyi Feng; Shaotian Yan; Bing Deng; Zhiguo Cao; Jieping Ye

cs.CVMar 3, 2026

Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs

Authors: Zhiyu Pan, Yizheng Wu, Jiashen Hua, Junyi Feng, Shaotian Yan, Bing Deng, Zhiguo Cao, Jieping Ye

Organizations: Huazhong University of Science and Technology · Alibaba Cloud

Abstract

Reasoning has emerged as a key capability of large language models. In linguistic tasks, this capability can be enhanced by self-improving techniques that refine reasoning paths for subsequent finetuning. However, extending these language-based self-improving approaches to vision language models (VLMs) presents a unique challenge:~visual hallucinations in reasoning paths cannot be effectively verified or rectified. Our solution starts with a key observation about visual contrast: when presented with a contrastive VQA pair, i.e., two visually similar images with synonymous questions, VLMs identify relevant visual cues more precisely. Motivated by this observation, we propose Visual Contrastive Self-Taught Reasoner (VC-STaR), a novel self-improving framework that leverages visual contrast to mitigate hallucinations in model-generated rationales. We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modal similarity, and generate rationales using VC-STaR. Consequently, we obtain a new visual reasoning dataset, VisCoR-55K, which is then used to boost the reasoning capability of various VLMs through supervised finetuning. Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also surpasses models finetuned on the SoTA visual reasoning datasets, demonstrating that the inherent contrastive ability of VLMs can bootstrap their own visual reasoning. Project at: https://github.com/zhiyupan42/VC-STaR.

Figures & tables

Panel of Figure 1 — Figure 1: Contrasting makes the VLM see better. (a) Contrastive VQA pairs compels a more accurate response. (b) Compared with a previous self-improving method STaR ( Zelikman et al., 2022 ) that enhances the quality of reasoning with hints (ground-truth answers), contrasting with hints can rectify more cases. The blocks along the $x$ -axis mark initial VLM failures. The color of each block indicates the outcome of rectifying: green for success and gray for failure. Tested VLM is Qwen $2.5$ VL- $7$ B ( Bai et al., 2025 ) .

Figure 2: VisCoR- $55$ K. We introduce the Vis ual Co ntrastive R easoning dataset (VisCoR- $55$ K), a new collection of $55$ K high-quality visual reasoning samples. Spanning the domains of general VQA, reasoning, math, graph/chart, and OCR, each sample is created by leveraging a contrastive counterpart to generate a faithful rationale. Rationales are shown in the Sec. A.4 .

Figure 3: Contrastive VQA pair curation pipeline. To facilitate effective contrastive analysis, we curate corresponding challenging counterparts for VQA samples from a pool of diverse datasets. Each curated pair consists of two samples that share a synonymous question but feature distinct yet semantically similar images. Collected pairs are filtered by a difficulty-based sampling procedure.

Figure 4: Faithful rationale generation pipeline. A contrastive analysis can be obtained based on the curated contrastive VQA pair. Leveraging the property of VLMs illustrated in Fig. 1 , the contrastive analysis is then used to trigger a rethinking procedure, which refines the naive rationale into a more faithful one. This pipeline is designed to generate rationales for supervised finetuning.

Panel of Table 1 — Table 1: Performance comparison with self-improving baselines and the models trained on off-the-shelf visual reasoning datasets on hallucination, math, and general benchmarks. We adopt the Qwen $2.5$ VL- $7$ B as our base model, and report its reasoning performance as a baseline. MME-RW is short for MME-RealWorld Zhang et al. (2025b) ; R1-OV is short for R1-Onevision ( Yang et al., 2025b ) . Blue ( red ) numbers in parentheses represent performance gains ( drops ) relative to the baseline. The best performance is in boldface , and the second best is underlined .

Figure 5: Qualitative Comparison with base model. The second row shows the directly response from the base model, the third row shows the response when the base model is prompted to “think stey by step”, the last row shows the model improved with our VC-STaR. We highlight the key visual evidences with red boxes for clarity of visualization. More results are in Sec. A.4 .

Figure 6: Performance comparison with other contrastive VQA pair construction strategies. Rationales in all settings are generated from the proposed VC-STaR. The red dashed line represents the base model (Qwen

2.5

VL-

7

B) performance.

Table 8

Setting	Pos.	Neg.	QR	QA	QG	QC	CA	CC	CAt	Total
Base Model	-	-	43.8	51.4	31.5	44.4	30.2	60.6	44.4	45.4
VC-STaR	✔	✘	48.3 (+4.5)	52.8 (+1.4)	46.8 (+15.3)	57.1 (+12.7)	42.9 (+12.7)	60.1 (-0.5)	44.4 (+0.0)	50.6 (+5.2)
VC-STaR	✘	✔	51.6 (+7.8)	56.8 (+5.4)	33.9 (+2.4)	59.2 (+14.8)	46.0 (+15.8)	70.8 (+10.2)	66.7 (+22.3)	53.7 (+8.3)
VC-STaR	✔	✔	53.5 (+9.7)	57.3 (+5.9)	46.3 (+14.8)	55.6 (+11.2)	36.5 (+6.3)	72.7 (+12.1)	55.6 (+11.2)	54.7 (+9.3)

Table 4: Analysis about the effect of positive and negative contrastive VQA counterparts on GQA benchmark. We adopt the Qwen

2.5

VL-

7

B as our base model, and report its reasoning performance as a baseline. QR: query for relationships; QA: query for attributes; QG: query for global information; QC: query for category; CA: comparing of attribute; CC: choosing the object of one certain category; CAt: choosing the object of one certain attribute. Blue ( red ) numbers in parentheses represent performance gains ( drops ) relative to the baseline.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Statistics of the contrastive VQA pair curation. The bar chart (left y-axis) displays the total number of contrastive VQA pairs in each dataset, with colors indicating the data category. The line graph (right y-axis) plots the ratio of median samples identified within those pairs for each dataset. In the upper right, the pie charts provide a categorical breakdown of the sample.

Figure 8: Examples of rationales in VisCoR- $55$ K.

Figure 9: Additional qualitative comparison.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Improving Reasoning in Vision-Language Models via Perception Verified Self-Training
Jun 20, 2026Sourabh Sharma, Sonam Gupta, SadbhawnaVLM Reasoning VLM Hallucination Mitigation
Malaviya National Institute of Technology Jaipur, India · IBM Software Innovation Lab, India
See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection
Apr 27, 2026Zhiheng Wu, Tong Wang, Shuning Wang +2Tool Use in VLMs VLM Reasoning
1Baidu Inc. · †Independent researcher. · 2Zhejiang University +1
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
Apr 17, 2026Yige Xu, Yongjie Wang, Zizhuo Wu +3Vision-Language Models VLM Adaptation
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China

Jun 20, 2026J/K move · Enter open · S save

Improving Reasoning in Vision-Language Models via Perception Verified Self-Training

Sourabh Sharma, Sonam Gupta, Sadbhawna

Malaviya National Institute of Technology Jaipur, India · IBM Software Innovation Lab, India

VLM Reasoning VLM Hallucination Mitigation

Open paper AskPDF arXiv

Achieving human-like reasoning in Vision-Language Models (VLMs) remains a long-standing challenge. Recent approaches leverage Chain-of-Thought (CoT) rationales generated by human annotators or proprietary models, which are costly and difficult to scale. Self-training offers a promising alternative but often suffers from visual hallucinations and language shortcuts because rationales are filtered only by answer correctness without verifying visual perception. We propose a perception-verified self-training framework that enforces visually grounded reasoning. Our method employs a CoT template (caption-reasoning-conclusion) that disentangles perception from reasoning, enabling independent verification of visual understanding. To compensate for the absence of ground-truth captions, we introduce PerceptEval, an unsupervised method that evaluates caption quality based on its alignment with visual and textual elements in the image. Using caption verification together with answer correctness, we partition the data into easy, medium, and hard subsets and design a two-stage curriculum learning strategy. Stage 1 trains on easy samples, while Stage 2 enhances medium samples by regenerating reasoning conditioned on verified captions and retaining only those with correct conclusions. This ensures training is performed exclusively on perceptually grounded reasoning, reducing hallucinations and language shortcuts. Extensive experiments across diverse domains and models demonstrate improvements of up to 16% over standard self-training baselines, showing that our framework provides a scalable and cost-effective solution for advancing multimodal reasoning without manually annotated CoT rationales.

Abstract

VLM Reasoning

Explore similar work

Improving Reasoning in Vision-Language Models via Perception Verified Self-Training

See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap