Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design

Omar Sharif; Eftekhar Hossain; Nikhil Singh; Patrick Ng

cs.CVJan 1, 2026

Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design

Authors: Omar Sharif, Eftekhar Hossain, Nikhil Singh, Patrick Ng

Organizations: Department of Computer Science, Dartmouth College · Department of Computer Science, University of Central Florida · Independent Researcher

Abstract

Reinforcement learning with verifiable rewards has driven major gains in LLM reasoning, and it is intuitive to assume this recipe will transfer well to multimodal models. However, multimodal models do two things: first, perceive what is in an image, then reason about what it implies. Because these stages are graded jointly, it is hard to tell how much room reasoning alone has to grow. We study this on algorithmic visual puzzles, where both components are necessary and show that perception, not reasoning, is the binding constraint. Replacing images with simple textual descriptions raises performance by over 20 points on average for Claude models. We then evaluate six reward designs aimed at inducing visual grounding during reasoning without chain-of-thought supervision. Training Qwen-2.5-VL-7B with GRPO, reward design induces long, structured reasoning with self-reflection and visual references, yielding a 5.56-point gain over the base model. These gains are, however, uneven; no single reward improves all categories, and rewards with verifiable accuracy signals trade cross-domain transfer for in-domain accuracy. These results point to perception-aware reward design as a path forward, so that signals correct perception at its source rather than the reasoning that inherits its errors.

Figures & tables

Figure 1: Overview of the GRPO-based training pipeline for MLLMs. For each sample, the policy model generates a completion conditioned on the prompt associated with a specific reward type ({ $\mathcal{R}_{\textsc{format}}$ , $\mathcal{R}_{\textsc{outcome}}$ , $\mathcal{R}_{\textsc{structured}}$ , $\mathcal{R}_{\textsc{graded}}$ , $\mathcal{R}_{\textsc{grounded}}$ , $\mathcal{R}_{\textsc{process}}$ }). Each completion is then evaluated using the reward calculation framework (§ 2 ). Completions with different rewards are shown in Figure 4 . The resulting reward scores are passed into the GRPO trainer, which updates the policy model accordingly.

Reward Function	Reward Content	Reward Calculation	Design Motivation
$\mathcal{R}_{\textsc{format}}$	<think, answer>	$\alpha_{t}\text{think}+\alpha_{a}\text{accuracy}$	Baseline reward for RL training for formatted reasoning and accuracy.
$\mathcal{R}_{\textsc{outcome}}$	<answer>	$\alpha_{a}*\text{accuracy}$	Is accuracy reward enough to encourage reasoning? Can a model learn the format from the prompt alone, without any separate reward for the format?
$\mathcal{R}_{\textsc{structured}}$	<image description, think, rethink, answer>	$\alpha_{i}\text{image}+\alpha_{t}\text{think}+\alpha_{r}\text{rethink}+\alpha_{a}\text{accuracy}$	Does distribution of reward across intuitive tags incentivize better reasoning?
$\mathcal{R}_{\textsc{graded}}$	<image description, think, rethink, answer (partial)>	$\alpha_{i}\text{image}+\alpha_{t}\text{think}+\alpha_{r}\text{rethink}+\alpha_{a}\text{accuracy}(partial)$	Does giving partial reward to the model when the final answer is close to the ground-truth encourage correct reasoning and answer?
$\mathcal{R}_{\textsc{grounded}}$	<think, visual, answer>	$\alpha_{t}\text{think}+\alpha_{v}\text{visual}+\alpha_{a}*\text{accuracy}$	Focus on integrating visual grounding within reasoning.
$\mathcal{R}_{\textsc{process}}$	<image description, think, rethink, answer-tag>	$\alpha_{i}\text{image}+\alpha_{t}\text{think}+\alpha_{r}\text{rethink}+\alpha_{a}\text{answer-tag}$	Can the model generalize or improve its performance without any verifiable reward?

Table 1: Summary of reward functions for incentivizing visual reasoning in MLLMs. Reward Content specifies which output tags are evaluated, and Reward Calculation indicates the weighting of each reward component. Each non-answer tag’s score is based on the amount of unique content generated within that tag, with counts passed through a

tanh()

smoothing function to prevent disproportionately large rewards. Parameter values (

\alpha_{t},\alpha_{r},\alpha_{i},\alpha_{v},\alpha_{a}

) are provided in Appendix Table 7 .

Figure 2: Generic prompt structure used across all categories in both multimodal and text-only settings.

	Claude-3.5		Claude-3.7 (thinking)
Category	MM	TO	MM	TO
Clock	3	84	7	83
Checker move	16	9	39	44
Maze	10	15	14	26
Move box	33	58	34	70
N-queens	25	95	25	97

Table 2: Comparison of accuracy in multimodal (MM) and text-only (TO) settings in different categories.

Category		Clock-3K Data						Diverse-8k Data
	Base	$\mathcal{R}_{F}$	$\mathcal{R}_{O}$	$\mathcal{R}_{S}$	$\mathcal{R}_{G}$	$\mathcal{R}_{V}$	$\mathcal{R}_{P}$	$\mathcal{R}_{F}$	$\mathcal{R}_{O}$	$\mathcal{R}_{S}$	$\mathcal{R}_{G}$	$\mathcal{R}_{V}$	$\mathcal{R}_{P}$
Clock	3	37	8	13	10	4	1	6	6	4	1	2	3
Checker move	6	1	3	4	5	5	10	4	10	2	10	6	1
Maze	6	0	5	10	7	0	11	26	10	12	9	30	9
Move box	25	13	24	21	17	17	19	30	43	54	30	20	21
N-queens	8	2	6	7	6	6	5	17	11	15	16	13	9

Table 3: Evaluation results for RL trained Qwen-2.5-VL-7B model across different reward functions.

\mathcal{R}_{F}

\mathcal{R}_{O}

\mathcal{R}_{S}

\mathcal{R}_{G}

\mathcal{R}_{V}

, and

\mathcal{R}_{P}

denote {

\mathcal{R}_{\textsc{format}}

\mathcal{R}_{\textsc{outcome}}

\mathcal{R}_{\textsc{structured}}

\mathcal{R}_{\textsc{graded}}

\mathcal{R}_{\textsc{grounded}}

\mathcal{R}_{\textsc{process}}

} respectively. The base score indicate the performance with Chain-of-Thought approach. For each row, the best value is highlighted in a dark shade and the second-best in a light shade. All tied values within a class are highlighted in a light shade.

	Clock-3K		Diverse-8k
	ID	CD	ID	CD
Baseline	3	10.75	10.5	9.4
$\mathcal{R}_{\textsc{format}}$	37	4.25	19.75	8
$\mathcal{R}_{\textsc{outcome}}$	8	9.37	17.5	13.2
$\mathcal{R}_{\textsc{structured}}$	13	12.12	21.25	11
$\mathcal{R}_{\textsc{graded}}$	10	10.12	14.00	15.20

Table 4: In-domain (ID) and cross-domain (CD) performance of the RL-trained Qwen-2.5-VL-7B model under different reward functions. For Clock-3K, the Clock category is treated as in-domain, while the remaining eight categories are considered cross-domain. For Diverse-8K, the aggregate performance over four categories—Clock, Maze, Move-Box, and N-Queens is reported as in-domain, while the remaining five categories are considered cross-domain.

Figure 3: Performance comparison across categories showing baseline vs. maximum scores and improvement margins. Reward functions that achieved the maximum score are indicated below each category. For the number slide category no reward improve over the baseline. Results are reported for models trained on the Diverse-8K dataset.

Category	CoT	$\mathcal{R}_{O}$	$\max$ ( $\mathcal{R}_{S}$ , $\mathcal{R}_{G}$ , $\mathcal{R}_{V}$ )
Maze	6	10	30
Move box	25	43	54
Rotting kiwi	24	30	27
Water jugs	3	14	27

Table 5: Head-to-head comparison of the outcome-only reward (

\mathcal{R}_{\textsc{outcome}}

) and reasoning-oriented rewards (

\mathcal{R}_{\textsc{structured}}

\mathcal{R}_{\textsc{graded}}

\mathcal{R}_{\textsc{grounded}}

) on visually dependent categories. The best reasoning-oriented reward improves performance by up to 29 points over the CoT baseline and consistently outperforms the outcome-only reward illustrating the impact of longer reasoning.

Figure 4: An example comparing reasoning between the base (Qwen-2.5-VL-7B) with chain-of-thought approach and RL-trained versions with different reward strategies. With the $\mathcal{R}_{\textsc{structured}}$ reward, the model generates grounded, multi-step visual reasoning–accurately describing fine-grained image details such as the distinct colors of the hour and minute hands. Notably, this structured long-form reasoning emerges naturally from RL training (see Figures 14 and 15 for additional examples). The model also performs step-by-step self-evaluation, revisiting and verifying each reasoning step before concluding with its final answer.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Training Steps	$500$
Batch Size	$128$
Max Prompt Length	$2048$
Max Response Length	$2048$
Max Image Pixels	$262144$
Min Image Pixels	$65536$

Appendix

Table 6: Parameters for the GRPO training

Reward Function	Parameter Weights
$\mathcal{R}_{\textsc{format}}$	$\alpha_{t}$ = $0.10$ , $\alpha_{a}$ = $0.90$
$\mathcal{R}_{\textsc{outcome}}$	$\alpha_{a}$ = $1.0$
$\mathcal{R}_{\textsc{structured}}$ , $\mathcal{R}_{\textsc{graded}}$	$\alpha_{i}=0.06$ , $\alpha_{t}$ = $0.06$ , $\alpha_{r}$ = $0.08$ , $\alpha_{a}$ = $0.80$
$\mathcal{R}_{\textsc{grounded}}$	$\alpha_{t}$ = $0.10$ , $\alpha_{v}$ = $0.10$ , $\alpha_{a}$ = $0.80$
$\mathcal{R}_{\textsc{process}}$	$\alpha_{i}=0.30$ , $\alpha_{t}$ = $0.30$ , $\alpha_{r}$ = $0.30$ , $\alpha_{a}$ = $0.10$

Appendix

Table 7: Weight configurations used across different reward functions. To ensure factual accuracy, a high weight is consistently assigned to answer correctness (

\alpha_{a}

). The other components are weighted based on empirical observations.

Figure 5: Question: Emily’s event is going to start in 1 hour 10 minutes. The current time is shown on the clock. The clock is a standard analog clock without the second hand. What time will the event start?

Figure 6: Question: A checker game is being played on a grid of 4 squares with 1 green and 2 red checkers. Initially, the checkers are arranged as shown in the starting configuration, with the 3 checkers occupying 3 squares and one unoccupied square. Green checkers only move rightward and red checkers only move leftward. Every move is either i) a slide to the adjacent empty square, or ii) a jump over one position to an empty square, provided the checker being jumped over is of a different color. Each square can accommodate a maximum of one checker at any time. How many moves are required to reach the ending configuration from the starting configuration following the specified rules?

Figure 7: Question: This is maze having 13 * 11 cells. The empty cells are coloured white and the obstacle cells are coloured black. From an empty cell, you can only move up, down, left, or right to another adjacent empty cell. You cannot move diagonally between two empty cells and cannot step into a cell with an obstacle. The entry cell of the maze is shown with the green arrow. The exit cell of the maze is shown with the blue arrow. Suppose you have found the most optimal path in the maze between the entrance and exit, where you need to go through the least number of empty cells and you need to make the least number of left and right turns. What is the total number of left turns do you need to make in this optimal path?

Figure 8: Question: A storekeeper is a puzzle in which the player pushes boxes around in a warehouse trying to get them to target locations. The game is represented by an 6 x 6 grid of characters grid where each element is a wall, floor, or box. Your task is to move the box to the end flag under the following rules: 1. The box can be moved to an adjacent free cell by standing next to the box and then moving in the direction of the box by 1 grid. This is a push. 2. The player cannot walk through the box. What is the minimum number of pushes to move the box to the end flag.

Figure 9: Question: You are given a 9 * 9 chessboard. The Manhattan distance between two squares in a chessboard is equal to the minimal number of orthogonal King moves between these squares on the otherwise empty board. The objective is to place 9 chess queens on this board so that no two queens threaten each other; i.e. no two queens share the same row, column, or diagonal. 7 queens have already been placed in some of the squares of the board, as shown in the image. Suppose you pick two squares to place the two remaining queen pieces in a way that fulfills the objective. What is the Manhattan distance between these two squares?

Figure 10: Question: The board shown in the image is a sliding puzzle of 3 * 3 tile dimensions. It has 8 numbered tiles and one unoccupied (open) position. Tiles in the same row or column of the open position can be moved by sliding them horizontally or vertically, respectively. All tiles always stay and move inside the red boundary wall, as shown in the image. A move is defined as moving the open position by one tile unit in any available direction. You start from the board position shown in the image and perform exactly 4 moves such that the open position is seen moving in the following sequence: up, up, right, left. What is the maximum number in the row that now has the open position?

Figure 11: Question: You are given a 3 x 3 grid in which each cell can contain either no kiwi, one fresh kiwi, or one rotten kiwi. Every minute, any fresh kiwi that is 4-directionally adjacent to a rotten kiwi also becomes rotten. What is the minimum number of minutes that must elapse until no cell has a fresh kiwi?

Figure 12: Question: You are playing a Tower of Hanoi game with 3 rods and 6 disks of various diameters, which can slide onto any rod. You are given the starting and ending configuration of the game as shown in the top and the bottom of the image, respectively. The game has the following rules: i) Only one disk may be moved at a time; ii) Each move consists of taking the upper disk from one of the stacks and placing it on top of another stack or on an empty rod; and iii) No disk can be placed on top of a disk that is smaller than it. What is the minimum number of moves required to go from the starting to the ending configuration?

Figure 13: Question: You are given 3 jugs of capacities 11, 5, 4 litres. Initially, the amount of water that is contained in each jar is shown in the image. A single step of water pouring from one jug to another is constrained by the following rules: i) take a non-empty jug and pour water from it to another non-full jug until the first one becomes empty or the second one becomes full; and ii) no water can be split while pouring. The objective is to reach the amounts of 11, 1, 1 litres of water in the jugs from left to right, respectively. What is the minimum number of water pouring steps required to achieve the objective?

Image	Textual Representation	Image	Textual Representation
	Current Time: 2:43		Starting Configuration: G-RR
Ending Configuration: RR-G
	The maze is illustrated in the following grid, where white empty cells are denoted by 0 and black obstacle cells are denoted by 1 .
1 1 1 1 1 1 1 1 1 1 1
0 0 0 0 0 0 1 0 0 0 1
1 1 1 1 1 0 1 1 1 0 1

Appendix

Table 8: Examples of how images from different classes are converted into textual representations. In the multimodal setting, images are provided to the model, while in the text-only setting, their textual descriptions are used.

Image	Question	Reasoning, Answer, and Comment
	Question: Daniel’s event is going to start in 1 hour 40 minutes. The current time is shown on the clock. The clock is a standard analog clock without the second hand. What will be the time when the event starts?
Ground-Truth: 4:23	Reasoning : The clock shows an analog face with hour and minute hands. The minute hand (brown) is pointing to 3, which indicates 15 minutes past the hour. The hour hand (green) is between 8 and 9, but closer to 8, confirming it’s 8:15 . To find when the event starts, I need to add 1 hour and 40 minutes to 8:15. Adding 1 hour gives 9:15, and adding 40 more minutes results in 9:55.
Answer : 9:55
Comment : Incorrectly interpret the positions of the minute and hour hands, leading to interpreting the wrong time. But the calculation of time is correct.
	Question: A storekeeper is a puzzle in which the player pushes boxes around in a warehouse trying to get them to target locations. The game is represented by an 6 x 6 grid of characters grid where each element is a wall, floor, or box. Your task is to move the box to the end flag under the following rules: 1. The box can be moved to an adjacent free cell by standing next to the box and then moving in the direction of the box by 1 grid. This is a push. 2. The player cannot walk through the box. What is the minimum number of pushes to move the box to the end flag.
Ground-Truth: 4	Reasoning : The puzzle shows a maze with brick walls, a box in the center, and a checkered flag on the right. The player starts at the bottom center .

Appendix

Table 9: Qualitative example illustrating failure in visual perception leading to incorrect model predictions. Incorrect visual generations are highlighted in red , with our comments on the visual errors provided.

Figure 14: The RL-trained model generates detailed and well-structured reasoning for solving the puzzle, thoroughly describing all relevant visual and algorithmic elements without unnecessary repetition. This example illustrates how the $\mathcal{R}_{\textsc{structured}}$ reward function guides the model toward clear, precise, and long multistep reasoning. The key part of the reasoning is bold or highlighted.

Figure 15: The RL-trained model demonstrates structured multistep visual reasoning without any supervised training samples. The reward incentivizes the model to elicit long reasoning and to explain the image in detail, like Kiwi’s representation, grid structure, color, and markings. The designed reward encourages long, detailed chains of thought, yielding clear, stepwise, and self-checked explanations.

Prompt for

\mathcal{R}_{\textsc{format}}

reward A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>.

Appendix

Table 25

Reasoning for VLMs

See all 78 papers

Mar 30, 2026

Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning

Reinforcement learning with verifiable rewards (RLVR) has substantially enhanced the reasoning capabilities of multimodal large language models (MLLMs). However, existing RLVR approaches typically rely on outcome-driven optimization that updates both perception and reasoning using a shared reward based solely on the final answer. This shared reward blurs credit assignment, frequently improving reasoning patterns while failing to reliably enhance the accuracy of upstream visual evidence extraction. To address this perception bottleneck, we introduce PRCO (Perception-Reasoning Coevolution), a dual-role RLVR framework with a shared policy. PRCO consists of two cooperative roles: an Observer that generates an evidence caption tailored to the question and a Solver that predicts the final answer based on this caption. Crucially, PRCO employs role-specific reward signals: the Solver is optimized using verifiable outcome rewards on the final answer, while the Observer receives a utility reward derived from the Solver's downstream success. Extensive experiments across eight challenging multimodal reasoning benchmarks demonstrate that PRCO yields consistent improvements across model scales by over 7 points on average accuracy compared to the base model, outperforming prior open-source RL-tuned baselines.

Ziqi Miao, Haonan Jia, Lijun Li +4

Multimodal Reasoning Multimodal Large Language Models RL for VLMs Reinforcement Learning with Verifiable Rewards

Ask PDF

May 13, 2026

Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning

Achieving robust perception-reasoning synergy is a central goal for advanced Vision-Language Models (VLMs). Recent advancements have pursued this goal via architectural designs or agentic workflows. However, these approaches are often limited by static textual reasoning or complicated by the significant compute and engineering burden of external agentic complexity. Worse, this heavy investment does not yield proportional gains, often witnessing a "seesaw effect" on perception and reasoning. This motivates a fundamental rethinking of the true bottleneck. In this paper, we argue that the root cause of this trade-off is an ambiguity in modality credit assignment: when a VLM fails, is it due to flawed perception ("bad seeing") or flawed logic ("bad thinking")? To resolve this, we introduce a reinforcement learning framework that improves perception-reasoning synergy by reliably rewarding the perception fidelity. We explicitly decompose the generation process into interleaved perception and reasoning steps. This decoupling enables targeted supervision on perception. Crucially, we introduce Perception Verification (PV), leveraging a "blindfolded reasoning" proxy to reward perceptual fidelity independently of reasoning outcomes. Furthermore, to scale training across free-form VL tasks, we propose Structured Verbal Verification, which replaces high-variance LLM judging with structured algorithmic execution. These techniques are integrated into a Modality-Aware Credit Assignment (MoCA) mechanism, which routes rewards to the specific source of error -- either bad seeing or bad thinking -- enabling a single VLM to achieve simultaneous performance gains across a wide task spectrum.

Haozhe Wang, Qixin Xu, Changpeng Wang +4

1Hong Kong University of Science and Technology (HKUST) · 2Tsinghua University · 3Zhejiang University +2

Multimodal Reasoning VLM Reasoning Visually Grounded Reasoning RL for VLMs

Ask PDF

Feb 9, 2026

Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMs

While chain-of-thought (CoT) reasoning has substantially improved multimodal large language models (MLLMs) on complex reasoning tasks, existing approaches largely rely on long textual reasoning trajectories and provide limited mechanisms for learning stable visual attention policies. Our analysis shows that current MLLMs exhibit weak visual focus: early-stage visual misalignment is rarely corrected during subsequent reasoning, leading to error propagation and failed inferences. We argue that this limitation stems from inadequate credit assignment for visual attention during training. To address this issue, we propose SAYO, a visual reasoning model trained with a reinforcement learning (RL) framework that introduces a region-level visual attention-based reward. This reward explicitly aligns optimization signals with visually grounded reasoning steps, enabling the model to learn more reliable attention behaviors. Extensive experiments across multiple multimodal benchmarks demonstrate that SAYO consistently improves performance on diverse reasoning and perception tasks. Project is open at https://github.com/Cratileo/Sayo.

Siqu Ou, Tianrui Wan, Zhiyuan Zhao +2

TeleAI · Shanghai Jiao Tong University · Northwestern Polytechnical University

Reasoning for VLMs Attention Intervention in VLMs Visual Evidence Sensitivity in VLMs RL for VLMs Vision-Language Grounding

Ask PDF

Abstract

Reasoning for VLMs

Explore similar work

Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning

Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning

Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMs