Evaluation of multimodal reasoning models is typically reduced to a single accuracy score, implicitly treating reasoning as a unitary capability. We introduce MathLens, a benchmark of textbook-style geometry problems that exposes this assumption by operationally decomposing performance into perception, reasoning, and multimodal-specific components. Each problem is derived from a symbolic specification and accompanied by visual diagrams, text-only variants, multimodal questions, and targeted perceptual probes, enabling controlled measurement of each component. Using this decomposition, we show that common training strategies are associated with systematically different capability profiles that are invisible under aggregate accuracy. Across the models we study, reinforcement learning primarily improves perceptual grounding and robustness to diagram variation, while our analysis suggests that textual SFT yields gains through reflective reasoning. In contrast, as perception and reasoning improve, a growing fraction of remaining errors fall outside these components and are categorized as multimodal-specific. These results suggest that apparent progress in multimodal reasoning reflects shifting balances among subskills rather than uniform advancement, motivating evaluation beyond scalar accuracy.
Figures & tables
Figure 1: MathLens decomposes multimodal reasoning errors into perception, reasoning, and a residual multimodal-specific category, revealing component-specific shifts after finetuning that are hidden by aggregate accuracy. Each training strategy affects components differently; e.g., text SFT yields a minor gain ( ↗ ) in reasoning but a drop ( ↓ ) on the multimodal-specific residual (model details in Appendix E ).
Figure 2: Based on MathLens annotations, the joint Multimodal Reasoning Test (1) is decomposed into a Perception Test and a textual Reasoning Test. The Perception Test evaluates questions answerable directly from the diagram, such as reading an annotated angle (e.g., ∠DFB in (2)). The Reasoning Test (3) replaces the diagram with a complete textual description (e.g., “There are triangle BDF … ∠DFB=65∘ ”), such that the question can be solved without visual access. Finally, the Multimodal-specific category (4) highlights cases where multimodal reasoning fails even though perception and reasoning, when tested independently, succeed.
Figure 3: Sample data generation process in MathLens . From a semantic state representation, we build controlled text descriptions , perception probes , and questions with no overlap with visual content. Also, new diagrams are rendered from the semantic state to avoid visual familiarity effects 2 2 2 Here, an obtuse angle is drawn acute to prevent visual estimation and enforce geometric deduction, consistent with Olympiad practice (e.g., “not drawn to scale” in AMC). .
Figure 4: Impact of multimodal reasoning training on MathLens . We evaluate pretrained backbones alongside models finetuned for multimodal reasoning, reporting accuracy (%). MathLens is sensitive to gains from multimodal reasoning-oriented finetuning. Inference settings for API models are given in Appendix E .
Figure 5Figure 6
Figure 9: Robustness to semantic-level visual modifications. Box plots show accuracy on modified diagrams. Points report accuracy on the unmodified base diagrams and the overall consistency score. Multimodal RL improves consistency, whereas multimodal SFT reduces it.
Figure 10: Finetuning shifts by diagram type. Bars show percentage point changes for existing versus OOD rendered diagrams. Only statistically significant changes are shown.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Example of MathVerse text descriptions. The textual description fails to fully encode geometric relations, requiring external information to be inferred from the diagram. This incompleteness makes it unsuitable for evaluating pure reasoning capacity.
Figure 12: Inconsistencies in manual error analyses across benchmarks. Pie charts show variation in error categories across multimodal reasoning datasets, underscoring the lack of standardized criteria. Analyses are typically restricted to a single model. Figures are taken from the respective papers ( Hao et al., 2025 ; Wang et al., 2024b ; Zhang et al., 2024 ) .
Category
Rate (%)
Diagram – Invalid
23.3
Diagram – Poor visual
18.2
Intervention – Poor visual
1.3
Question – Invalid text
0.5
Question – Invalid visual
0.3
Question – Invalid perception
0.3
Appendix
Table 1: Outcome of manual screening, by the first criterion each candidate fails.
Model
Type
Source
7–9B Open-Weight
VL-Rethinker
backbone
Qwen/Qwen2.5-VL-7B-Instruct
mm-rl
TIGER-Lab/VL-Rethinker-7B
ShareVL-R1
backbone
Qwen/Qwen2.5-VL-7B-Instruct
mm-rl
HuanjinYao/R1-ShareVL-7B
R1-OneVision
backbone
Qwen/Qwen2.5-VL-7B-Instruct
Appendix
Table 2: Model configurations studied in this work.
Figure 13: Perception probe results by question type. Improvements differ across skills: angle-related and cocircularity tasks improve with multimodal RL, line-length tasks show little change, and polygon detection improves only with textual SFT. A likely factor is whether notations are colocated with the geometric element they describe or spatially separated.
Model
MathVista
MathVerse
MathVision
MathLens -E
MathLens
Qwen2.5-VL-7B
68.2
46.3
25.1
34.7
33.2
R1-Onevision-7B
64.1
46.4
29.9
35.4
29.8
MM-Eureka-Qwen-7B
73.0
50.3
26.9
34.0
31.3
ThinkLite-VL-7B
74.3
52.2
29.9
34.3
32.9
R1-ShareVL-7B
75.4
52.8
29.5
36.7
35.4
VL-Rethinker-7B
74.9
54.2
32.3
37.6
35.7
Appendix
Table 3: Full results of all models used to produce the correlation plot in Figure 6 .
Model
Variant
Text
Raw
Base
Add
Add
Flip
Merge
Rename
Rotate
Consistency
lines
shapes
VL-Rethinker-7B
Backbone
38.9
34.7
33.2
26.1
28.1
30.6
26.9
29.4
31.4
31.0
MM-RL
41.1
37.6
35.7
31.4
29.4
34.7
30.6
33.5
35.5
37.3
ShareVL-R1-7B
Backbone
38.9
34.7
33.2
26.1
28.1
30.6
26.9
29.4
31.4
31.0
MM-RL
42.1
36.7
35.4
31.1
32.7
33.9
33.7
35.1
35.3
41.3
R1-OneVision-7B
Backbone
38.9
34.7
33.2
26.1
28.1
30.6
26.9
29.4
31.4
31.0
Appendix
Table 4: Downstream accuracy across diagram modifications with output consistency under these conditions. Text indicates performance from textual descriptions instead of diagrams, Raw denotes original human-generated diagrams, Base the newly rendered diagrams, and the other cases semantic-space modifications followed by rendering.
Model
Variant
Tri-
Quad-
Parallel
Perpen-
Collinear
Co-
Same len.
Val. len.
Same ∠
Val. ∠
-angle
-rilateral
-dicular
-circular
VL-Rethinker-7B
Backbone
69.1
52.3
49.0
78.2
59.5
65.6
57.4
86.6
50.0
68.1
MM-RL
67.6
55.2
60.6
76.6
67.4
81.7
50.0
86.9
50.0
75.8
ShareVL-R1-7B
Backbone
69.1
52.3
49.0
78.2
59.5
65.6
57.4
86.6
50.0
68.1
MM-RL
71.4
57.7
62.5
78.2
66.0
88.9
57.4
86.4
69.2
72.4
R1-OneVision-7B
Backbone
69.1
52.3
49.0
78.2
59.5
65.6
57.4
86.6
50.0
68.1
Appendix
Table 5: Perception probe accuracy by question types.
Figure 14: Distribution of error types per model. Most gains come from cases failing in both perception and reasoning. Multimodal-specific residual errors become more prominent as the other categories improve.
Figure 15: Data samples from MathLens-NS . We curate problem instances from a prior dataset ( Fu et al. (2024) ) and annotate the text description and perception-centric question-answers.
Figure 16: Data samples from MathLens-NS . We curate problem instances from a prior dataset ( Hao et al. (2025) ) and annotate the text description and perception-centric question-answers.
Figure 17: Error type distribution across models on MathLens-NS . Most shifts are associated with perception-related cases.
Figure 18: Model responses for a downstream geometry problem in MathLens . The backbone model fails in perception, leading to an incorrect answer. The textual SFT model initially makes the same mistake but recovers the perceptual detail and produces the correct answer.
Figure 19: Model responses for a downstream geometry problem in MathLens . The backbone model exhibits a reasoning error, whereas the multimodal RL model provides the correct solution.
Figure 20: Model responses for a downstream geometry problem in MathLens . The multimodal RL model inherits the backbone model’s flawed reasoning, leading to perception errors and downstream failure.
Figure 21: Model responses for a downstream geometry problem in MathLens . The multimodal RL model corrects the perceptual error of the backbone model but ultimately fails to produce the correct answer due to a reasoning error.
Figure 22: Model responses for a perception probe in MathLens . The textual SFT model corrects the perceptual error present in the backbone model.
Figure 23: Model responses for a perception probe in MathLens . The textual SFT model corrects the perceptual error present in the backbone model.
Figure 24: Model responses for a perception probe in MathLens . The textual SFT model corrects the perceptual error present in the backbone model.
Vision-language models (VLMs) can behave very differently depending on whether the same underlying problem is presented textually or visually. This inconsistency suggests that different views expose different reasoning paths and failure modes that standard multimodal post-training does not fully exploit. To study and exploit this phenomenon, we construct ODA-Data, a high-quality paired multimodal geometry dataset with text-dominant, image-dominant, and combined image+text views of the same problems, together with splits for training and evaluating view dependent reasoning behaviors. We then develop Modality-Informed Reciprocal Reasoning Optimization (MIRROR), a reinforcement learning approach that derives a cross-view self-distillation signal from the model's own modality-dependent capabilities. For each problem, MIRROR evaluates the model under all views, selects the best-performing view as a teacher, and trains the remaining views using a GRPO task-reward objective augmented with a reverse-KL regularizer toward that teacher. Across reasoning benchmarks that evaluate on multimodal mathematics problems, MIRROR improves over standard RL and yields more accurate and consistent behavior across different input modality representations.
Wen Ye, Yuxiao Qu, Aviral Kumar +1
University of Southern California · Carnegie Mellon University
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Multimodal large language models (MLLMs) have shown promising reasoning abilities, yet evaluating their performance in specialized domains remains challenging. STEM reasoning is a particularly valuable testbed because it provides highly verifiable feedback, but existing benchmarks often permit unimodal shortcuts due to modality redundancy and focus mainly on final-answer accuracy, overlooking the reasoning process itself. To address this challenge, we introduce StepSTEM: a graduate-level benchmark of 283 problems across mathematics, physics, chemistry, biology, and engineering for fine-grained evaluation of cross-modal reasoning in MLLMs. StepSTEM is constructed through a rigorous curation pipeline that enforces strict complementarity between textual and visual inputs. We further propose a general step-level evaluation framework for both text-only chain-of-thought and interleaved image-text reasoning, using dynamic programming to align predicted reasoning steps with multiple reference solutions. Experiments across a wide range of models show that current MLLMs still rely heavily on textual reasoning, with even Gemini 3.1 Pro and Claude Opus 4.6 achieving only 38.29% accuracy. These results highlight substantial headroom for genuine cross-modal STEM reasoning and position StepSTEM as a benchmark for fine-grained evaluation of multimodal reasoning. Source code is available at https://github.com/lll-hhh/STEPSTEM.
Jing Jin, Hao Liu, Yan Bai +9
1Tsinghua University · 3Central South University · 2Meituan Inc +3