Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization

Sai Srinivas Kancheti; Aditya Kanade; Rohit Sinha; Vineeth N Balasubramanian; Tanuja Ganu

cs.CVApr 9, 2026

Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization

Authors: Sai Srinivas Kancheti, Aditya Kanade, Rohit Sinha, Vineeth N Balasubramanian, Tanuja Ganu

Organizations: CSE Dept., IIT Hyderabad, Hyderabad, India · Microsoft Research, Bengaluru, India

Abstract

Multimodal reasoning models (MRMs) trained with reinforcement learning with verifiable rewards (RLVR) show improved accuracy on visual reasoning benchmarks. However, we observe that accuracy gains often come at the cost of reasoning quality: generated Chain-of-Thought (CoT) traces are frequently inconsistent with the final answer and poorly grounded in the visual evidence. We systematically study this phenomenon across seven challenging real-world spatial reasoning benchmarks and find that it affects contemporary MRMs such as ViGoRL-Spatial, TreeVGR as well as our own models trained with standard Group Relative Policy Optimization (GRPO). We characterize CoT reasoning quality along two complementary axes: "logical consistency" (does the CoT entail the final answer?) and "visual grounding" (does each reasoning step accurately describe objects, attributes, and spatial relationships in the image?). To address this, we propose Faithful GRPO (FGRPO), a variant of GRPO that enforces consistency and grounding as constraints via Lagrangian dual ascent. FGRPO incorporates batch-level consistency and grounding constraints into the advantage computation within a group, adaptively adjusting the relative importance of constraints during optimization. We evaluate FGRPO on Qwen2.5-VL-7B and 3B backbones across seven spatial datasets. Our results show that FGRPO substantially improves reasoning quality, reducing the inconsistency rate from 24.5% to 1.7% and improving visual grounding scores by +13%. It also improves final answer accuracy over simple GRPO, demonstrating that faithful reasoning enables better answers.

Figures & tables

Panel of Figure 1 — Figure 1: Unfaithful reasoning masked by correct answers. Both models answer correctly, but only FGRPO reasons faithfully. Left: The GRPO-Task model incorrectly claims there are no visible paths, contradicting its own answer of “1.0” (50% faithfulness, inconsistent). Right: The GRPO-Task model reasons toward “lamp” but answers “box” (33% faithfulness, inconsistent). FGRPO produces visually grounded reasoning in both cases (100% faithfulness, consistent). Sentences are color-coded: blue are grounded and orange are ungrounded.

Figure 2: Overview of the FGRPO training pipeline. We show advantage computation for a training batch with 3 samples, each with 2 rollouts. For each prompt-image pair

s_{i}

, the policy samples

G=2

rollouts and we compute the task reward

R_{T}

, the consistency reward

R_{C}

, and grounding rewards

R_{S}

and

R_{G}

(only

R_{G}

is shown for clarity). The consistency and semantic-grounding rewards are provided by an online VLM judge. Each signal is independently normalized within the rollout group to obtain decoupled advantages, which are then combined into the final FGRPO advantage using Lagrange multipliers. Batch-level constraint statistics are used to update the multipliers via dual ascent, increasing the weight of violated constraints and decreasing the weight of satisfied ones.

Method	CVB 2D	CVB 3D	Mind Cube	MMVP	Omni Spatial	Real WorldQA	SAT Real	Avg.
Base model (7B): Qwen2.5-VL-7B-Instruct
Non-reasoning	77.17	83.78	35.11	75.78	45.23	69.02	63.11	64.17
CoT prompting	75.92	76.09	30.83	72.44	40.40	63.05	59.22	59.71
Base model (3B): Qwen2.5-VL-3B-Instruct
Non-reasoning	70.58	74.00	43.71	64.67	45.92	65.88	59.00	60.54
CoT prompting	71.21	68.00	40.38	63.67	40.77	62.35	55.00	57.34

Table 1: Main results. Pass@1 accuracy on seven spatial reasoning benchmarks. All 7B models use Qwen2.5-VL-7B-Instruct as backbone; 3B models use Qwen2.5-VL-3B-Instruct. Bold = best, underline = second best among open-weight 7B models. FGRPO achieves the highest average accuracy at both scales, outperforming all MRM baselines and the unconstrained GRPO-T baseline while simultaneously improving reasoning quality.

Figure 3: Per-dataset reasoning quality breakdown . (a) Semantic grounding (

S

): FGRPO achieves uniformly higher semantic grounding than GRPO-T across all seven benchmarks (86.0% vs. 72.7%), with the largest gains on MindCube (+22.8 pp) and OmniSpatial (+21.1 pp). TreeVGR also outperforms GRPO-T (81.9%). (b) Inconsistency rate: FGRPO reduces inconsistency to 1.7% on average, compared to 26.1% (GRPO-T), 26.0% (TreeVGR), and 16.2% (ViGoRL). MindCube and OmniSpatial are the most challenging for all baselines; FGRPO virtually eliminates inconsistency on every benchmark.

Figure 4: FGRPO (squares) vs GRPO-T (circles); larger shape size indicates higher inconsistency.

Method	Acc (%) $\uparrow$	IR (%) $\downarrow$
GRPO (coupled advantage)
Task only (GRPO-T)	65.16	26.12
+ Additive $R_{C}$ , $R_{G}$	64.97	4.22
+ Multiplicative $R_{C}$ , $R_{G}$	63.47	19.61
FGRPO (decoupled advantage)
Consistency only ( $R_{C}$ )	66.16	0.54

Table 2: First three rows are GRPO.

Method	Acc (%) $\uparrow$	IR (%) $\downarrow$
GRPO-T	65.17	26.12
FGRPO (fixed $\lambda$ )	66.32	1.11
FGRPO (adaptive $\lambda$ )	67.16	1.73

Table 3: Adaptive

\lambda

s yield best accuracy

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: An overview of the two-stage training pipeline. We curate CoT training data using MCTS with a strong visual teacher, covering bounding-box grounded visual reasoning. We perform SFT on the curated CoT data using data from SAT ( Ray et al., 2025 ) , VGR ( Wang et al., 2025c ) and VisCoT ( Shao et al., 2024a ) , followed by RL training. We add a subset of TreeVGR-RL-37K ( Wang et al., 2025a ) to construct diverse RL data with bounding-box supervision.

Table 9

Hyperparameter	Value
Advantage estimator	CGRPO
Constraint keys	[consistency, bbox_reward, faithfulness]
Consistency constraint
Threshold ( $\tau_{C}$ )	0.95
Initial $\lambda_{C}$	1.0
Masking	Only correct predictions ( $R_{\mathrm{acc}}>0$ )

Appendix

Table 6: FGRPO Constraint Hyperparameters

Hyperparameter	Value
Teacher Model	Qwen2.5-VL-72B-Instruct
$N_{\mathrm{sim}}$	8
$c_{\mathrm{puct}}$	2.0
Branching Factor ( $k$ )	3
Rollouts per Node ( $n$ )	2
Max Depth	10

Appendix

Table 7: Hyperparameters for MCTS-based data synthesis.

Dataset	#Questions	Subtasks
CV-Bench2D ( Tong et al., 2024a )	1.4K	2D, spatial relations
CV-Bench3D ( Tong et al., 2024a )	1.2K	3D, depth
MindCube ( Yin et al., 2025 )	1K	3D, multi-image
MMVP ( Tong et al., 2024b )	300	2D, spatial relations
OmniSpatial ( Jia et al., 2025 )	1.5K	3D, dynamics
RealWorldQA ( xAI, 2025 )	765	2D, real-world

Appendix

Table 8: Summary of evaluation datasets. Star (*) indicates circular evaluation is employed.

Panel of Figure 6 — Figure 6: Prompt used for the consistency constraint evaluation. The judge receives only the question, reasoning trace, and final answer—no image is provided. It outputs YES (consistent) or NO (inconsistent).

Panel of Figure 7 — Figure 7: Prompt used for the semantic grounding (faithfulness) constraint evaluation. The judge receives the image, question, accumulated reasoning context, and the latest sentence to evaluate. It outputs CORRECT , INCORRECT , or SKIP . The per-sample semantic grounding score $S$ is the fraction of visual sentences (non- SKIP ) that are CORRECT .

Panel of Figure 8 — Figure 8: Training dynamics (7B FGRPO). Top: Lagrange multiplier trajectories. Bottom: constraint satisfaction over training steps.

Figure 9: Perspective estimation. GRPO-Task claims the photo was taken from above despite the upward-looking perspective of the wine glass, contradicting its own correct answer. FGRPO correctly identifies the low vantage point.

Figure 10: Navigation and signage. GRPO-Task misreads the “Entering” sign as an “Exiting” sign and contradicts its own answer. FGRPO correctly interprets the signage and produces consistent reasoning.

Figure 11: Relative distance. GRPO-T incorrectly concludes the stop sign is closer, contradicting its answer of “school zone sign.” FGRPO correctly identifies relative depth from the scene layout.

Figure 12: Navigation and signage. GRPO-Task claims that the “Entering” sign does not lead to terminal B, yet selects option A (yes) as the final answer contradicting its reasoning. Whereas, FGRPO correctly identifies the entry for Terminal B and answers yes.

Figure 13: Object counting. GRPO-Task claims two giraffes are present despite the image showing three, contradicting its correct answer. FGRPO accurately counts three giraffes with consistent reasoning.

Figure 14: Depth with bounding boxes. GRPO-Task claims the lamp (red box) is closer, contradicting its answer of “pillow” (blue box). FGRPO correctly reasons about relative depth using the bounding-box annotations.

Figure 15: Directional reasoning. GRPO-Task concludes the left lane is correct for reaching Porte de Vaise, contradicting its answer of “Yes” (right lane). FGRPO correctly reads the road sign and produces consistent reasoning.

Figure 16: Egocentric spatial reasoning. GRPO-Task reasons that turning right would move parallel to the bus rather than toward the door, contradicting its answer. FGRPO correctly reasons about the egocentric perspective change.

Figure 17: FGRPO responses on the eval set. We observe that FGRPO responds faithfully and exhibits both spatial groundedness and consistency.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 251 papers

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
May 21, 2026Changyuan Tian, Zhicong Lu, Huaxing Liu +7Multimodal Large Language Models Multimodal Reasoning
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1
Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs
Apr 17, 2026Sai Srinivas Kancheti, Aditya Sanjiv Kanade, Vineeth N. Balasubramanian +1Multimodal Model Evaluation Visual Spatial Reasoning
IIT, Hyderabad · Microsoft Research India
The Art of Interrogation: Consistency Amplifies Factuality in Spatial Reasoning
Jun 10, 2026Theo Uscidda, Marta Tintore Gazulla, Maks Ovsjanikov +2Reinforcement Learning 3D Spatial Reasoning
CREST, ENSAE, Institut Polytechnique de Paris · Google Zurich · Google DeepMind

May 21, 2026J/K move · Enter open · S save

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention

Changyuan Tian, Zhicong Lu, Huaxing Liu, Xiang Wang+6

AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University+1

Multimodal Large Language Models Multimodal Reasoning Visually Grounded Reasoning Visual Grounding Faithfulness in VLMs RL for VLMs+1

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for advancing complex reasoning in large language models, and recent work extends RLVR to multimodal large language models (MLLMs). This transfer, however, surfaces a faithfulness challenge: faithful perception of task-relevant visual evidence and faithful use of that evidence during reasoning, leading to unsatisfactory gains on multimodal benchmarks. Specifically, existing perception supervision often operates on textual descriptions rather than natively on image regions, and faithful use is largely overlooked, exposing the perception-reasoning disconnect where correctly perceived evidence is dropped or contradicted during reasoning. To close these gaps, we propose Faithful-MR1, a training framework that anchors and reinforces visual attention to address both halves of faithful multimodal reasoning. The Anchoring stage turns perception into an explicit pre-reasoning subtask, supervising a dedicated <Focus> token's attention directly against image regions rather than through textual descriptions. The Reinforcing stage exposes faithful use through counterfactual image intervention, rewarding answer-correct trajectories that concentrate visual attention where vision causally matters. Extensive experiments demonstrate that Faithful-MR1 outperforms recent multimodal reasoning baselines on both Qwen2.5-VL-Instruct 3B and 7B backbones while using substantially less training data.

Abstract

Visual Spatial Reasoning

Explore similar work

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention

Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs

The Art of Interrogation: Consistency Amplifies Factuality in Spatial Reasoning