cs.CLOct 2, 2025
SaveWhat MLLMs Learn about When they Learn about Multimodal Reasoning
Organizations: Microsoft Research AI Frontiers · Yonsei University · Seoul National University
Abstract
Evaluation of multimodal reasoning models is typically reduced to a single accuracy score, implicitly treating reasoning as a unitary capability. We introduce MathLens, a benchmark of textbook-style geometry problems that exposes this assumption by operationally decomposing performance into perception, reasoning, and multimodal-specific components. Each problem is derived from a symbolic specification and accompanied by visual diagrams, text-only variants, multimodal questions, and targeted perceptual probes, enabling controlled measurement of each component. Using this decomposition, we show that common training strategies are associated with systematically different capability profiles that are invisible under aggregate accuracy. Across the models we study, reinforcement learning primarily improves perceptual grounding and robustness to diagram variation, while our analysis suggests that textual SFT yields gains through reflective reasoning. In contrast, as perception and reasoning improve, a growing fraction of remaining errors fall outside these components and are categorized as multimodal-specific. These results suggest that apparent progress in multimodal reasoning reflects shifting balances among subskills rather than uniform advancement, motivating evaluation beyond scalar accuracy.
Figures & tables
Figure 1: MathLens decomposes multimodal reasoning errors into perception, reasoning, and a residual multimodal-specific category, revealing component-specific shifts after finetuning that are hidden by aggregate accuracy. Each training strategy affects components differently; e.g., text SFT yields a minor gain ( ) in reasoning but a drop ( ) on the multimodal-specific residual (model details in Appendix E ).
Figure 2: Based on MathLens annotations, the joint Multimodal Reasoning Test (1) is decomposed into a Perception Test and a textual Reasoning Test. The Perception Test evaluates questions answerable directly from the diagram, such as reading an annotated angle (e.g., in (2)). The Reasoning Test (3) replaces the diagram with a complete textual description (e.g., “There are triangle … ”), such that the question can be solved without visual access. Finally, the Multimodal-specific category (4) highlights cases where multimodal reasoning fails even though perception and reasoning, when tested independently, succeed.
Figure 3: Sample data generation process in MathLens . From a semantic state representation, we build controlled text descriptions , perception probes , and questions with no overlap with visual content. Also, new diagrams are rendered from the semantic state to avoid visual familiarity effects 2 2 2 Here, an obtuse angle is drawn acute to prevent visual estimation and enforce geometric deduction, consistent with Olympiad practice (e.g., “not drawn to scale” in AMC). .
Figure 4: Impact of multimodal reasoning training on MathLens . We evaluate pretrained backbones alongside models finetuned for multimodal reasoning, reporting accuracy (%). MathLens is sensitive to gains from multimodal reasoning-oriented finetuning. Inference settings for API models are given in Appendix E .
Figure 9: Robustness to semantic-level visual modifications. Box plots show accuracy on modified diagrams. Points report accuracy on the unmodified base diagrams and the overall consistency score. Multimodal RL improves consistency, whereas multimodal SFT reduces it.
Figure 10: Finetuning shifts by diagram type. Bars show percentage point changes for existing versus OOD rendered diagrams. Only statistically significant changes are shown.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Example of MathVerse text descriptions. The textual description fails to fully encode geometric relations, requiring external information to be inferred from the diagram. This incompleteness makes it unsuitable for evaluating pure reasoning capacity.
Figure 12: Inconsistencies in manual error analyses across benchmarks. Pie charts show variation in error categories across multimodal reasoning datasets, underscoring the lack of standardized criteria. Analyses are typically restricted to a single model. Figures are taken from the respective papers ( Hao et al., 2025 ; Wang et al., 2024b ; Zhang et al., 2024 ) .
| Category | Rate (%) |
|---|---|
| Diagram – Invalid | 23.3 |
| Diagram – Poor visual | 18.2 |
| Intervention – Poor visual | 1.3 |
| Question – Invalid text | 0.5 |
| Question – Invalid visual | 0.3 |
| Question – Invalid perception | 0.3 |
Table 1: Outcome of manual screening, by the first criterion each candidate fails.
| Model | Type | Source |
| 7–9B Open-Weight | ||
| VL-Rethinker | backbone | Qwen/Qwen2.5-VL-7B-Instruct |
| mm-rl | TIGER-Lab/VL-Rethinker-7B | |
| ShareVL-R1 | backbone | Qwen/Qwen2.5-VL-7B-Instruct |
| mm-rl | HuanjinYao/R1-ShareVL-7B | |
| R1-OneVision | backbone | Qwen/Qwen2.5-VL-7B-Instruct |
Table 2: Model configurations studied in this work.
Figure 13: Perception probe results by question type. Improvements differ across skills: angle-related and cocircularity tasks improve with multimodal RL, line-length tasks show little change, and polygon detection improves only with textual SFT. A likely factor is whether notations are colocated with the geometric element they describe or spatially separated.
| Model | MathVista | MathVerse | MathVision | MathLens -E | MathLens |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 68.2 | 46.3 | 25.1 | 34.7 | 33.2 |
| R1-Onevision-7B | 64.1 | 46.4 | 29.9 | 35.4 | 29.8 |
| MM-Eureka-Qwen-7B | 73.0 | 50.3 | 26.9 | 34.0 | 31.3 |
| ThinkLite-VL-7B | 74.3 | 52.2 | 29.9 | 34.3 | 32.9 |
| R1-ShareVL-7B | 75.4 | 52.8 | 29.5 | 36.7 | 35.4 |
| VL-Rethinker-7B | 74.9 | 54.2 | 32.3 | 37.6 | 35.7 |
Table 3: Full results of all models used to produce the correlation plot in Figure 6 .
| Model | Variant | Text | Raw | Base | Add | Add | Flip | Merge | Rename | Rotate | Consistency |
| lines | shapes | ||||||||||
| VL-Rethinker-7B | Backbone | 38.9 | 34.7 | 33.2 | 26.1 | 28.1 | 30.6 | 26.9 | 29.4 | 31.4 | 31.0 |
| MM-RL | 41.1 | 37.6 | 35.7 | 31.4 | 29.4 | 34.7 | 30.6 | 33.5 | 35.5 | 37.3 | |
| ShareVL-R1-7B | Backbone | 38.9 | 34.7 | 33.2 | 26.1 | 28.1 | 30.6 | 26.9 | 29.4 | 31.4 | 31.0 |
| MM-RL | 42.1 | 36.7 | 35.4 | 31.1 | 32.7 | 33.9 | 33.7 | 35.1 | 35.3 | 41.3 | |
| R1-OneVision-7B | Backbone | 38.9 | 34.7 | 33.2 | 26.1 | 28.1 | 30.6 | 26.9 | 29.4 | 31.4 | 31.0 |
Table 4: Downstream accuracy across diagram modifications with output consistency under these conditions. Text indicates performance from textual descriptions instead of diagrams, Raw denotes original human-generated diagrams, Base the newly rendered diagrams, and the other cases semantic-space modifications followed by rendering.
| Model | Variant | Tri- | Quad- | Parallel | Perpen- | Collinear | Co- | Same len. | Val. len. | Same | Val. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| -angle | -rilateral | -dicular | -circular | ||||||||
| VL-Rethinker-7B | Backbone | 69.1 | 52.3 | 49.0 | 78.2 | 59.5 | 65.6 | 57.4 | 86.6 | 50.0 | 68.1 |
| MM-RL | 67.6 | 55.2 | 60.6 | 76.6 | 67.4 | 81.7 | 50.0 | 86.9 | 50.0 | 75.8 | |
| ShareVL-R1-7B | Backbone | 69.1 | 52.3 | 49.0 | 78.2 | 59.5 | 65.6 | 57.4 | 86.6 | 50.0 | 68.1 |
| MM-RL | 71.4 | 57.7 | 62.5 | 78.2 | 66.0 | 88.9 | 57.4 | 86.4 | 69.2 | 72.4 | |
| R1-OneVision-7B | Backbone | 69.1 | 52.3 | 49.0 | 78.2 | 59.5 | 65.6 | 57.4 | 86.6 | 50.0 | 68.1 |
Table 5: Perception probe accuracy by question types.
Figure 14: Distribution of error types per model. Most gains come from cases failing in both perception and reasoning. Multimodal-specific residual errors become more prominent as the other categories improve.
Figure 15: Data samples from MathLens-NS . We curate problem instances from a prior dataset ( Fu et al. (2024) ) and annotate the text description and perception-centric question-answers.
Figure 16: Data samples from MathLens-NS . We curate problem instances from a prior dataset ( Hao et al. (2025) ) and annotate the text description and perception-centric question-answers.
Figure 17: Error type distribution across models on MathLens-NS . Most shifts are associated with perception-related cases.
Figure 18: Model responses for a downstream geometry problem in MathLens . The backbone model fails in perception, leading to an incorrect answer. The textual SFT model initially makes the same mistake but recovers the perceptual detail and produces the correct answer.
Figure 19: Model responses for a downstream geometry problem in MathLens . The backbone model exhibits a reasoning error, whereas the multimodal RL model provides the correct solution.
Figure 20: Model responses for a downstream geometry problem in MathLens . The multimodal RL model inherits the backbone model’s flawed reasoning, leading to perception errors and downstream failure.
Figure 21: Model responses for a downstream geometry problem in MathLens . The multimodal RL model corrects the perceptual error of the backbone model but ultimately fails to produce the correct answer due to a reasoning error.
Figure 22: Model responses for a perception probe in MathLens . The textual SFT model corrects the perceptual error present in the backbone model.
Figure 23: Model responses for a perception probe in MathLens . The textual SFT model corrects the perceptual error present in the backbone model.
Figure 24: Model responses for a perception probe in MathLens . The textual SFT model corrects the perceptual error present in the backbone model.
More in this topic
Multimodal Reasoning
Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.
See all 508 papers