Latent Visual Contrastive Reasoning with Multi-Branch Exploration

Kegeng Tang; Jingbo Wang; Yangming Zhang; Huanshu Zhang; Shaogang Ren; Zihao Wang

cs.CVOct 27, 2026

Latent Visual Contrastive Reasoning with Multi-Branch Exploration

Authors: Kegeng Tang, Jingbo Wang, Yangming Zhang, Huanshu Zhang, Shaogang Ren, Zihao Wang

Organizations: Department of Computer Science & Engineering, University of Tennessee at Chattanooga, Chattanooga, TN, USA · Department of Computer Science, University of Texas at Arlington, Arlington, TX, USA · Department of Electrical Engineering, The Pennsylvania State University, University Park, PA, USA

Abstract

Multimodal large language models have made substantial progress in visual question answering, but complex visual reasoning often requires locating and comparing multiple pieces of question-relevant visual evidence. Existing methods that move intermediate reasoning into the visual latent space usually follow a single linear reasoning trajectory, which can be limiting when multiple visual hypotheses or competing evidence regions are relevant. We propose Latent Visual Contrastive Reasoning, a branch-aware latent reasoning framework that generates multiple question-conditioned latent visual branches as soft hypotheses over image tokens. To train these branches, our method uses bounding-box evidence annotations to build target evidence representations and introduces a soft contrastive branch-evidence alignment objective. This objective encourages evidence-relevant branches to align with annotated visual regions while contrasting them against negative evidence. We instantiate our method on Qwen2.5-VL backbones and train it with standard supervised fine-tuning. Experiments on perception-intensive visual reasoning benchmarks show that the proposed method achieves overall improvements over the corresponding backbone and single-path latent reasoning baselines at both 3B and 7B scales.

Figures & tables

Fig. 1: Overall framework of LVCR. Given an image-question pair, LVCR encodes the image and question into visual and textual tokens, and proposes multiple question-conditioned latent branches as candidate visual hypotheses over the image. These branches are contrastively aligned with answer-relevant visual evidence and separated from irrelevant regions. LVCR then ranks the branches using relevance scores computed by cosine similarity with the question representation, and injects the selected branch tokens into the MLLM decoder, which combines them with the original multimodal context to generate the final answer.

Model	$V^{*}$	$V^{*}_{\mathrm{D.A.}}$	$V^{*}_{\mathrm{R.P.}}$	MMVP	Counting	IQ-Test	Jigsaw	Relative Reflect	Spatial Relation
Qwen2.5-VL-3B	73.8	80.9	63.2	60.0	65.8	24.7	53.3	42.5	81.1
Vision-R1	71.7	72.2	71.1	66.0	60.0	20.0	44.7	32.8	76.2
Vanilla SFT	70.7	80.0	56.6	59.0	67.5	22.0	52.0	42.5	84.6
LVR	70.7	79.1	57.9	60.7	64.2	28.7	52.7	40.3	81.1
LVCR (Ours)	75.4	82.6	64.5	61.0	62.5	28.0	54.0	42.5	84.6

TABLE I: Main results for 3B-scale models. All numbers are accuracy (%). Vision-R1 and LVR are reproduced on the 3B backbone using their official implementations. The best result in each column is shown in bold, and the second-best result is underlined.

Model	$V^{*}$	$V^{*}_{\mathrm{D.A.}}$	$V^{*}_{\mathrm{R.P.}}$	MMVP	Counting	IQ-Test	Jigsaw	Relative Reflect	Spatial Relation
Qwen2.5-VL-7B	78.5	81.7	73.7	66.7	66.7	26.0	52.0	38.8	87.4
PAPO	36.1	25.2	52.6	54.3	66.7	29.3	52.0	39.6	88.8
Vision-R1	70.2	70.4	69.7	46.7	51.7	26.7	27.3	44.8	66.4
PixelReasoner	80.1	81.7	77.6	67.0	65.0	23.3	52.7	40.3	88.1
Vanilla SFT	79.1	82.6	73.7	65.7	67.5	26.7	45.3	33.6	88.8
LVR	79.8	82.3	76.1	70.5	67.3	27.7	52.7	42.5	86.2

TABLE II: Main results for 7B-scale models. All numbers are accuracy (%). For PAPO, Vision-R1, and PixelReasoner, we evaluate their publicly released 7B checkpoints. For LVR, we retrain the model following its official training recipe. The best result in each column is shown in bold, and the second-best result is underlined.

Fig. 2: Ablation study on the NCE loss coefficient using LVCR-3B with

K{=}4

. We compare the default setting with

\lambda_{\mathrm{NCE}}=0.1

against a variant trained without the NCE loss, i.e.,

\lambda_{\mathrm{NCE}}=0

. Removing the NCE loss leads to consistent performance drops on

V^{*}

, MMVP, and BLINK, indicating that contrastive branch-evidence alignment helps latent branches capture question-relevant visual cues.

Fig. 3: Accuracy (%) of three branch injection strategies on

V^{*}

, MMVP, and BLINK (aggregated). Ranked denotes our default LVCR strategy; Top-1 injects only the highest-ranked branch; Unranked injects all branches without ordering. Results are reported using LVCR-3B with

K{=}4

MLLM Reasoning

See all 31 papers

SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows
Oct 1, 2026Albert Gao, Bing Xue, Andrea ZanetteVisual Reasoning Latent Visual Reasoning
Carnegie Mellon University · Meta
GLaQ: Grounding Latent Queries in Visual Evidence for Multimodal Reasoning
Aug 16, 2026Zesheng Yang, Lingling Zhang, Xinyu Zhang +4Multimodal Reasoning Visually Grounded Reasoning
Xi’an Jiaotong University · Xidian University
Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence
Sep 28, 2026Xi Xiao, Tianchen Zhao, Youngeun Kim +10Large Vision-Language Models Latent Visual Reasoning
University of Alabama at Birmingham · Amazon AGI · Work done during an internship at Amazon AGI.

Oct 1, 2026J/K move · Enter open · S save

SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows

Albert Gao, Bing Xue, Andrea Zanette

Carnegie Mellon University · Meta

Visual Reasoning Latent Visual Reasoning VLM Reasoning Visually Grounded Reasoning Latent Reasoning

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) often answer visual reasoning questions by relying on linguistic priors rather than task-relevant visual evidence. Textual chain-of-thought reasoning can partially mitigate this issue by encouraging models to decompose visual questions into intermediate evidence-seeking steps, but generating these steps autoregressively increases inference cost. Latent reasoning avoids explicit rationale generation, but existing approaches provide limited control over what intermediate states encode, making it difficult to impose separate supervision for planning, grounding, and evidence selection. We propose Structured Latent Visual Reasoning (SLVR), a training framework that bridges explicit chain-of-thought and latent reasoning by organizing multimodal reasoning into typed latent stages for planning, grounding, evidence selection, and reasoning integration. SLVR first trains the model to rely on the image by masking answer-revealing text and contrasting the correct answer with visually plausible distractors. It then organizes reasoning into latent stages for planning, grounding, evidence selection, and integration, supervising each stage with the corresponding signal: plans, boxes, visual evidence, and final rationales. This gives latent reasoning an explicit functional structure while avoiding generated textual chains at inference time. Built on Qwen2.5-VL-7B, SLVR improves consistently across multimodal reasoning benchmarks, with absolute gains of +9.4 on MMVP and +14.2 on BLINK Relation, as well as improvements on V*, MathVista, and ChartQA. These results suggest that structured latent supervision can improve fine-grained visual reasoning without the decoding overhead of textual CoT. Project page is available here.

Abstract

MLLM Reasoning

Explore similar work

SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows

GLaQ: Grounding Latent Queries in Visual Evidence for Multimodal Reasoning

Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence