Organizations: Department of Computer Science & Engineering, University of Tennessee at Chattanooga, Chattanooga, TN, USA · Department of Computer Science, University of Texas at Arlington, Arlington, TX, USA · Department of Electrical Engineering, The Pennsylvania State University, University Park, PA, USA
Multimodal large language models have made substantial progress in visual question answering, but complex visual reasoning often requires locating and comparing multiple pieces of question-relevant visual evidence. Existing methods that move intermediate reasoning into the visual latent space usually follow a single linear reasoning trajectory, which can be limiting when multiple visual hypotheses or competing evidence regions are relevant. We propose Latent Visual Contrastive Reasoning, a branch-aware latent reasoning framework that generates multiple question-conditioned latent visual branches as soft hypotheses over image tokens. To train these branches, our method uses bounding-box evidence annotations to build target evidence representations and introduces a soft contrastive branch-evidence alignment objective. This objective encourages evidence-relevant branches to align with annotated visual regions while contrasting them against negative evidence. We instantiate our method on Qwen2.5-VL backbones and train it with standard supervised fine-tuning. Experiments on perception-intensive visual reasoning benchmarks show that the proposed method achieves overall improvements over the corresponding backbone and single-path latent reasoning baselines at both 3B and 7B scales.
Figures & tables
Fig. 1: Overall framework of LVCR. Given an image-question pair, LVCR encodes the image and question into visual and textual tokens, and proposes multiple question-conditioned latent branches as candidate visual hypotheses over the image. These branches are contrastively aligned with answer-relevant visual evidence and separated from irrelevant regions. LVCR then ranks the branches using relevance scores computed by cosine similarity with the question representation, and injects the selected branch tokens into the MLLM decoder, which combines them with the original multimodal context to generate the final answer.
Model
V∗
VD.A.∗
VR.P.∗
MMVP
Counting
IQ-Test
Jigsaw
Relative Reflect
Spatial Relation
Qwen2.5-VL-3B
73.8
80.9
63.2
60.0
65.8
24.7
53.3
42.5
81.1
Vision-R1
71.7
72.2
71.1
66.0
60.0
20.0
44.7
32.8
76.2
Vanilla SFT
70.7
80.0
56.6
59.0
67.5
22.0
52.0
42.5
84.6
LVR
70.7
79.1
57.9
60.7
64.2
28.7
52.7
40.3
81.1
LVCR (Ours)
75.4
82.6
64.5
61.0
62.5
28.0
54.0
42.5
84.6
TABLE I: Main results for 3B-scale models. All numbers are accuracy (%). Vision-R1 and LVR are reproduced on the 3B backbone using their official implementations. The best result in each column is shown in bold, and the second-best result is underlined.
Model
V∗
VD.A.∗
VR.P.∗
MMVP
Counting
IQ-Test
Jigsaw
Relative Reflect
Spatial Relation
Qwen2.5-VL-7B
78.5
81.7
73.7
66.7
66.7
26.0
52.0
38.8
87.4
PAPO
36.1
25.2
52.6
54.3
66.7
29.3
52.0
39.6
88.8
Vision-R1
70.2
70.4
69.7
46.7
51.7
26.7
27.3
44.8
66.4
PixelReasoner
80.1
81.7
77.6
67.0
65.0
23.3
52.7
40.3
88.1
Vanilla SFT
79.1
82.6
73.7
65.7
67.5
26.7
45.3
33.6
88.8
LVR
79.8
82.3
76.1
70.5
67.3
27.7
52.7
42.5
86.2
TABLE II: Main results for 7B-scale models. All numbers are accuracy (%). For PAPO, Vision-R1, and PixelReasoner, we evaluate their publicly released 7B checkpoints. For LVR, we retrain the model following its official training recipe. The best result in each column is shown in bold, and the second-best result is underlined.
Fig. 2: Ablation study on the NCE loss coefficient using LVCR-3B with K=4 . We compare the default setting with λNCE=0.1 against a variant trained without the NCE loss, i.e., λNCE=0 . Removing the NCE loss leads to consistent performance drops on V∗ , MMVP, and BLINK, indicating that contrastive branch-evidence alignment helps latent branches capture question-relevant visual cues.
Fig. 3: Accuracy (%) of three branch injection strategies on V∗ , MMVP, and BLINK (aggregated). Ranked denotes our default LVCR strategy; Top-1 injects only the highest-ranked branch; Unranked injects all branches without ordering. Results are reported using LVCR-3B with K=4 .
Multimodal large language models (MLLMs) often answer visual reasoning questions by relying on linguistic priors rather than task-relevant visual evidence. Textual chain-of-thought reasoning can partially mitigate this issue by encouraging models to decompose visual questions into intermediate evidence-seeking steps, but generating these steps autoregressively increases inference cost. Latent reasoning avoids explicit rationale generation, but existing approaches provide limited control over what intermediate states encode, making it difficult to impose separate supervision for planning, grounding, and evidence selection. We propose Structured Latent Visual Reasoning (SLVR), a training framework that bridges explicit chain-of-thought and latent reasoning by organizing multimodal reasoning into typed latent stages for planning, grounding, evidence selection, and reasoning integration. SLVR first trains the model to rely on the image by masking answer-revealing text and contrasting the correct answer with visually plausible distractors. It then organizes reasoning into latent stages for planning, grounding, evidence selection, and integration, supervising each stage with the corresponding signal: plans, boxes, visual evidence, and final rationales. This gives latent reasoning an explicit functional structure while avoiding generated textual chains at inference time. Built on Qwen2.5-VL-7B, SLVR improves consistently across multimodal reasoning benchmarks, with absolute gains of +9.4 on MMVP and +14.2 on BLINK Relation, as well as improvements on V*, MathVista, and ChartQA. These results suggest that structured latent supervision can improve fine-grained visual reasoning without the decoding overhead of textual CoT. Project page is available here.
Chain-of-thought reasoning has substantially improved the problem-solving capabilities of multimodal large language models. Fine-grained visual evidence, however, remains difficult to preserve and reuse across text-based reasoning steps. To address this limitation, tool-augmented thinking-with-images methods maintain visual access externally by revisiting or manipulating the image, but require predefined tools and additional inference-time processing. As an internal alternative, continuous visual latent reasoning retains intermediate computation in hidden states. However, its prevailing autoregressive construction makes each latent state depend on its predecessors, so later states may repeat information already present in the latent sequence rather than capture complementary visual details. We introduce GLaQ, a grounded latent-query framework that replaces sequential latent rollout with a fixed set of context-conditioned queries grounded in the original visual tokens. The grounded queries are reinjected for answer generation, providing direct and coordinated access to source visual evidence. We train GLaQ with localized-view supervision followed by reinforcement learning under task-level rewards. Across five benchmarks for fine-grained visual understanding and perception, GLaQ-7B gains 5.99--9.66% over its base model and leads all compared visual latent methods, suggesting that direct query-to-image grounding can recover localized evidence from the full image without external visual operations or autoregressive latent rollouts.
Latent visual reasoning (LVR) enables multimodal large language models (MLLMs) to perform intermediate computation in continuous latent tokens rather than expressing every reasoning step in words. However, unlike textual CoT, latent reasoning is not directly observable, making it difficult to supervise what latent tokens learn. In this work, we first conduct a thorough analysis of latent-token behavior and identify a latent evidence-credit gap: latent tokens respond only weakly to image perturbations that alter the correct answer. We hypothesize that this issue stems from the lack of explicit supervision during GRPO training. These findings suggest that a final-answer reward provides too little guidance on what visual evidence to preserve or how credit should be assigned across latent tokens. To bridge this gap, we propose ReaLVR, which brings visual-evidence supervision to the model's own free-running latent trajectories. ReaLVR contrasts correct and model-generated wrong answers to determine where stronger supervision is needed, and relevant and mismatched visual evidence to specify what to preserve. Across three model families, ReaLVR consistently outperforms evaluated LVR baselines, achieving the highest five-task average of 63.7% on Qwen2.5-VL-7B. Crucially, we are the first to scale visual reasoning in latent space, showing that our framework continues to deliver robust improvements at frontier model scales up to 235B. Further analyses show more question-sensitive latent-token positions, stronger alignment with relevant visual regions, and greater fixed-context dependence on the most attended latent tokens.
Xi Xiao, Tianchen Zhao, Youngeun Kim +10
University of Alabama at Birmingham · Amazon AGI · Work done during an internship at Amazon AGI.