cs.CVJan 1, 2026
SaveDisentangling Perception and Reasoning in Multimodal LLMs via Reward Design
Organizations: Department of Computer Science, Dartmouth College · Department of Computer Science, University of Central Florida · Independent Researcher
Abstract
Reinforcement learning with verifiable rewards has driven major gains in LLM reasoning, and it is intuitive to assume this recipe will transfer well to multimodal models. However, multimodal models do two things: first, perceive what is in an image, then reason about what it implies. Because these stages are graded jointly, it is hard to tell how much room reasoning alone has to grow. We study this on algorithmic visual puzzles, where both components are necessary and show that perception, not reasoning, is the binding constraint. Replacing images with simple textual descriptions raises performance by over 20 points on average for Claude models. We then evaluate six reward designs aimed at inducing visual grounding during reasoning without chain-of-thought supervision. Training Qwen-2.5-VL-7B with GRPO, reward design induces long, structured reasoning with self-reflection and visual references, yielding a 5.56-point gain over the base model. These gains are, however, uneven; no single reward improves all categories, and rewards with verifiable accuracy signals trade cross-domain transfer for in-domain accuracy. These results point to perception-aware reward design as a path forward, so that signals correct perception at its source rather than the reasoning that inherits its errors.
Figures & tables
Figure 1: Overview of the GRPO-based training pipeline for MLLMs. For each sample, the policy model generates a completion conditioned on the prompt associated with a specific reward type ({ , , , , , }). Each completion is then evaluated using the reward calculation framework (§ 2 ). Completions with different rewards are shown in Figure 4 . The resulting reward scores are passed into the GRPO trainer, which updates the policy model accordingly.
| Reward Function | Reward Content | Reward Calculation | Design Motivation |
| <think, answer> | Baseline reward for RL training for formatted reasoning and accuracy. | ||
| <answer> | Is accuracy reward enough to encourage reasoning? Can a model learn the format from the prompt alone, without any separate reward for the format? | ||
| <image description, think, rethink, answer> | Does distribution of reward across intuitive tags incentivize better reasoning? | ||
| <image description, think, rethink, answer (partial)> | Does giving partial reward to the model when the final answer is close to the ground-truth encourage correct reasoning and answer? | ||
| <think, visual, answer> | Focus on integrating visual grounding within reasoning. | ||
| <image description, think, rethink, answer-tag> | Can the model generalize or improve its performance without any verifiable reward? |
Table 1: Summary of reward functions for incentivizing visual reasoning in MLLMs. Reward Content specifies which output tags are evaluated, and Reward Calculation indicates the weighting of each reward component. Each non-answer tag’s score is based on the amount of unique content generated within that tag, with counts passed through a smoothing function to prevent disproportionately large rewards. Parameter values ( ) are provided in Appendix Table 7 .
Figure 2: Generic prompt structure used across all categories in both multimodal and text-only settings.
| Claude-3.5 | Claude-3.7 (thinking) | |||
| Category | MM | TO | MM | TO |
| Clock | 3 | 84 | 7 | 83 |
| Checker move | 16 | 9 | 39 | 44 |
| Maze | 10 | 15 | 14 | 26 |
| Move box | 33 | 58 | 34 | 70 |
| N-queens | 25 | 95 | 25 | 97 |
Table 2: Comparison of accuracy in multimodal (MM) and text-only (TO) settings in different categories.
| Category | Clock-3K Data | Diverse-8k Data | |||||||||||
| Base | |||||||||||||
| Clock | 3 | 37 | 8 | 13 | 10 | 4 | 1 | 6 | 6 | 4 | 1 | 2 | 3 |
| Checker move | 6 | 1 | 3 | 4 | 5 | 5 | 10 | 4 | 10 | 2 | 10 | 6 | 1 |
| Maze | 6 | 0 | 5 | 10 | 7 | 0 | 11 | 26 | 10 | 12 | 9 | 30 | 9 |
| Move box | 25 | 13 | 24 | 21 | 17 | 17 | 19 | 30 | 43 | 54 | 30 | 20 | 21 |
| N-queens | 8 | 2 | 6 | 7 | 6 | 6 | 5 | 17 | 11 | 15 | 16 | 13 | 9 |
Table 3: Evaluation results for RL trained Qwen-2.5-VL-7B model across different reward functions. , , , , , and denote { , , , , , } respectively. The base score indicate the performance with Chain-of-Thought approach. For each row, the best value is highlighted in a dark shade and the second-best in a light shade. All tied values within a class are highlighted in a light shade.
| Clock-3K | Diverse-8k | |||
| ID | CD | ID | CD | |
| Baseline | 3 | 10.75 | 10.5 | 9.4 |
| 37 | 4.25 | 19.75 | 8 | |
| 8 | 9.37 | 17.5 | 13.2 | |
| 13 | 12.12 | 21.25 | 11 | |
| 10 | 10.12 | 14.00 | 15.20 | |
Table 4: In-domain (ID) and cross-domain (CD) performance of the RL-trained Qwen-2.5-VL-7B model under different reward functions. For Clock-3K, the Clock category is treated as in-domain, while the remaining eight categories are considered cross-domain. For Diverse-8K, the aggregate performance over four categories—Clock, Maze, Move-Box, and N-Queens is reported as in-domain, while the remaining five categories are considered cross-domain.
Figure 3: Performance comparison across categories showing baseline vs. maximum scores and improvement margins. Reward functions that achieved the maximum score are indicated below each category. For the number slide category no reward improve over the baseline. Results are reported for models trained on the Diverse-8K dataset.
| Category | CoT | ( , , ) | |
| Maze | 6 | 10 | 30 |
| Move box | 25 | 43 | 54 |
| Rotting kiwi | 24 | 30 | 27 |
| Water jugs | 3 | 14 | 27 |
Table 5: Head-to-head comparison of the outcome-only reward ( ) and reasoning-oriented rewards ( , , ) on visually dependent categories. The best reasoning-oriented reward improves performance by up to 29 points over the CoT baseline and consistently outperforms the outcome-only reward illustrating the impact of longer reasoning.
Figure 4: An example comparing reasoning between the base (Qwen-2.5-VL-7B) with chain-of-thought approach and RL-trained versions with different reward strategies. With the reward, the model generates grounded, multi-step visual reasoning–accurately describing fine-grained image details such as the distinct colors of the hour and minute hands. Notably, this structured long-form reasoning emerges naturally from RL training (see Figures 14 and 15 for additional examples). The model also performs step-by-step self-evaluation, revisiting and verifying each reasoning step before concluding with its final answer.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
| Training Steps | |
| Batch Size | |
| Max Prompt Length | |
| Max Response Length | |
| Max Image Pixels | |
| Min Image Pixels |
Table 6: Parameters for the GRPO training
| Reward Function | Parameter Weights |
| = , = | |
| = | |
| , | , = , = , = |
| = , = , = | |
| , = , = , = |
Table 7: Weight configurations used across different reward functions. To ensure factual accuracy, a high weight is consistently assigned to answer correctness ( ). The other components are weighted based on empirical observations.
Figure 5: Question: Emily’s event is going to start in 1 hour 10 minutes. The current time is shown on the clock. The clock is a standard analog clock without the second hand. What time will the event start?
Figure 6: Question: A checker game is being played on a grid of 4 squares with 1 green and 2 red checkers. Initially, the checkers are arranged as shown in the starting configuration, with the 3 checkers occupying 3 squares and one unoccupied square. Green checkers only move rightward and red checkers only move leftward. Every move is either i) a slide to the adjacent empty square, or ii) a jump over one position to an empty square, provided the checker being jumped over is of a different color. Each square can accommodate a maximum of one checker at any time. How many moves are required to reach the ending configuration from the starting configuration following the specified rules?
Figure 7: Question: This is maze having 13 * 11 cells. The empty cells are coloured white and the obstacle cells are coloured black. From an empty cell, you can only move up, down, left, or right to another adjacent empty cell. You cannot move diagonally between two empty cells and cannot step into a cell with an obstacle. The entry cell of the maze is shown with the green arrow. The exit cell of the maze is shown with the blue arrow. Suppose you have found the most optimal path in the maze between the entrance and exit, where you need to go through the least number of empty cells and you need to make the least number of left and right turns. What is the total number of left turns do you need to make in this optimal path?
Figure 8: Question: A storekeeper is a puzzle in which the player pushes boxes around in a warehouse trying to get them to target locations. The game is represented by an 6 x 6 grid of characters grid where each element is a wall, floor, or box. Your task is to move the box to the end flag under the following rules: 1. The box can be moved to an adjacent free cell by standing next to the box and then moving in the direction of the box by 1 grid. This is a push. 2. The player cannot walk through the box. What is the minimum number of pushes to move the box to the end flag.
Figure 9: Question: You are given a 9 * 9 chessboard. The Manhattan distance between two squares in a chessboard is equal to the minimal number of orthogonal King moves between these squares on the otherwise empty board. The objective is to place 9 chess queens on this board so that no two queens threaten each other; i.e. no two queens share the same row, column, or diagonal. 7 queens have already been placed in some of the squares of the board, as shown in the image. Suppose you pick two squares to place the two remaining queen pieces in a way that fulfills the objective. What is the Manhattan distance between these two squares?
Figure 10: Question: The board shown in the image is a sliding puzzle of 3 * 3 tile dimensions. It has 8 numbered tiles and one unoccupied (open) position. Tiles in the same row or column of the open position can be moved by sliding them horizontally or vertically, respectively. All tiles always stay and move inside the red boundary wall, as shown in the image. A move is defined as moving the open position by one tile unit in any available direction. You start from the board position shown in the image and perform exactly 4 moves such that the open position is seen moving in the following sequence: up, up, right, left. What is the maximum number in the row that now has the open position?
Figure 11: Question: You are given a 3 x 3 grid in which each cell can contain either no kiwi, one fresh kiwi, or one rotten kiwi. Every minute, any fresh kiwi that is 4-directionally adjacent to a rotten kiwi also becomes rotten. What is the minimum number of minutes that must elapse until no cell has a fresh kiwi?
Figure 12: Question: You are playing a Tower of Hanoi game with 3 rods and 6 disks of various diameters, which can slide onto any rod. You are given the starting and ending configuration of the game as shown in the top and the bottom of the image, respectively. The game has the following rules: i) Only one disk may be moved at a time; ii) Each move consists of taking the upper disk from one of the stacks and placing it on top of another stack or on an empty rod; and iii) No disk can be placed on top of a disk that is smaller than it. What is the minimum number of moves required to go from the starting to the ending configuration?
Figure 13: Question: You are given 3 jugs of capacities 11, 5, 4 litres. Initially, the amount of water that is contained in each jar is shown in the image. A single step of water pouring from one jug to another is constrained by the following rules: i) take a non-empty jug and pour water from it to another non-full jug until the first one becomes empty or the second one becomes full; and ii) no water can be split while pouring. The objective is to reach the amounts of 11, 1, 1 litres of water in the jugs from left to right, respectively. What is the minimum number of water pouring steps required to achieve the objective?
| Image | Textual Representation | Image | Textual Representation |
| Current Time: 2:43 | Starting Configuration: G-RR | ||
| Ending Configuration: RR-G | |||
| The maze is illustrated in the following grid, where white empty cells are denoted by 0 and black obstacle cells are denoted by 1 . | |||
| 1 1 1 1 1 1 1 1 1 1 1 | |||
| 0 0 0 0 0 0 1 0 0 0 1 | |||
| 1 1 1 1 1 0 1 1 1 0 1 |
Table 8: Examples of how images from different classes are converted into textual representations. In the multimodal setting, images are provided to the model, while in the text-only setting, their textual descriptions are used.
| Image | Question | Reasoning, Answer, and Comment |
| Question: Daniel’s event is going to start in 1 hour 40 minutes. The current time is shown on the clock. The clock is a standard analog clock without the second hand. What will be the time when the event starts? | ||
| Ground-Truth: 4:23 | Reasoning : The clock shows an analog face with hour and minute hands. The minute hand (brown) is pointing to 3, which indicates 15 minutes past the hour. The hour hand (green) is between 8 and 9, but closer to 8, confirming it’s 8:15 . To find when the event starts, I need to add 1 hour and 40 minutes to 8:15. Adding 1 hour gives 9:15, and adding 40 more minutes results in 9:55. | |
| Answer : 9:55 | ||
| Comment : Incorrectly interpret the positions of the minute and hour hands, leading to interpreting the wrong time. But the calculation of time is correct. | ||
| Question: A storekeeper is a puzzle in which the player pushes boxes around in a warehouse trying to get them to target locations. The game is represented by an 6 x 6 grid of characters grid where each element is a wall, floor, or box. Your task is to move the box to the end flag under the following rules: 1. The box can be moved to an adjacent free cell by standing next to the box and then moving in the direction of the box by 1 grid. This is a push. 2. The player cannot walk through the box. What is the minimum number of pushes to move the box to the end flag. | ||
| Ground-Truth: 4 | Reasoning : The puzzle shows a maze with brick walls, a box in the center, and a checkered flag on the right. The player starts at the bottom center . |
Table 9: Qualitative example illustrating failure in visual perception leading to incorrect model predictions. Incorrect visual generations are highlighted in red , with our comments on the visual errors provided.
Figure 14: The RL-trained model generates detailed and well-structured reasoning for solving the puzzle, thoroughly describing all relevant visual and algorithmic elements without unnecessary repetition. This example illustrates how the reward function guides the model toward clear, precise, and long multistep reasoning. The key part of the reasoning is bold or highlighted.
Figure 15: The RL-trained model demonstrates structured multistep visual reasoning without any supervised training samples. The reward incentivizes the model to elicit long reasoning and to explain the image in detail, like Kiwi’s representation, grid structure, color, and markings. The designed reward encourages long, detailed chains of thought, yielding clear, stepwise, and self-checked explanations.
| Prompt for reward A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>. |