Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Qwen Team · Beijing Institute of Technology · Tsinghua University · Zhejiang University
When MLLMs fail at Science, Technology, Engineering, and Mathematics (STEM) visual reasoning, a fundamental question arises: is it due to perceptual deficiencies or reasoning limitations? Through systematic scaling analysis that independently scales perception and reasoning components, we uncover a critical insight: scaling perception consistently outperforms scaling reasoning. This reveals perception as the true lever limiting current STEM visual reasoning. Motivated by this insight, our work focuses on systematically enhancing the perception capabilities of MLLMs by establishing code as a powerful perceptual medium--executable code provides precise semantics that naturally align with the structured nature of STEM visuals. Specifically, we construct ICC-1M, a large-scale dataset comprising 1M Image-Caption-Code triplets that materializes this code-as-perception paradigm through two complementary approaches: (1) Code-Grounded Caption Generation treats executable code as ground truth for image captions, eliminating the hallucinations inherent in existing knowledge distillation methods; (2) STEM Image-to-Code Translation prompts models to generate reconstruction code, mitigating the ambiguity of natural language for perception enhancement. To validate this paradigm, we further introduce STEM2Code-Eval, a novel benchmark that directly evaluates visual perception in STEM domains. Unlike existing work relying on problem-solving accuracy as a proxy that only measures problem-relevant understanding, our benchmark requires comprehensive visual comprehension through executable code generation for image reconstruction, providing deterministic and verifiable assessment. Code is available at https://github.com/TongkunGuan/Qwen-CodePercept.
Figures & tables
Figure 1: The scaling analysis reveals perception as the bottleneck in STEM. We decouple visual STEM reasoning into perception (image-to-caption) and reasoning (caption-to-answer) stages, then independently scale each component while keeping the other constant. All components use Qwen3-VL-Thinking [ 36 ] models and evaluation in the representative MathVision Dataset [ 47 ] . Both experiments demonstrate that scaling perception consistently outperforms scaling reasoning. This finding motivates our focus on systematically enhancing MLLMs’ perception capabilities in STEM.
Figure 2: The overview pipeline of CodePercept that enhances MLLMs’ visual perception in STEM domains through code-grounded learning. (Part 01) Starting from public STEM data, we construct high-quality image-code pairs via three complementary pipelines: (1) Image Reproduce converts existing STEM images into executable Python codes, (2) Image Diversity extracts concepts from seed images and generates diverse instantiations while preserving semantic validity, and (3) Solid Geometry employs parametric templates to generate complex solid geometry images with corresponding codes, addressing MLLMs’ solid geometry limitations. (Part 02) The synthesized data enables two novel training tasks (Code-Grounded Caption Generation and STEM Image-to-Code Translation) that fundamentally shift how we approach visual perception. (Part 03) These processes culminate in ICC-1M, a dataset of over 1M curated image-caption-code triplets. We employ both Supervised Finetuning and Reinforcement Learning to train models that achieve robust visual perception capabilities.
Image Captioner
Benchmark Datasets (%)
Average
MathVision
MathVista
MathVerse
DynaMath
WeMath
LogicVista
LLM Solver: Qwen3-30A3-Thinking [ 55 ]
Claude-Opus 4.1-Thinking [ 1 ]
59.61
71.10
56.19
73.25
44.86
59.28
60.72
GPT5-Thinking [ 33 ]
60.03
65.20
69.56
71.00
54.57
53.02
62.23
Gemini2.5-Pro
66.80
74.80
73.47
81.42
60.29
66.44
70.53
KeyeVL1.5-8B [ 43 ]
54.11
64.90
49.95
62.37
33.62
45.19
51.69
Table 1: Evaluating the perception abilities of various MLLMs using a captioner-solver setup. Each MLLM (captioner) generates an image description, and a fixed LLM solver then performs problem-solving based solely on that description. More accurate descriptions are expected to yield better reasoning results.
Model
Image Score
Code Score
Avg.
Exec Rate
Gemini2.5-Flash-Thinking
57.25
60.87
59.06
85.20
Claude-Opus 4.1-Thinking
55.90
56.19
56.05
97.10
GPT5-Thinking
64.97
64.98
64.98
96.60
Gemini2.5-Pro-Thinking
68.89
75.41
72.15
91.70
Intern-S1-8B [ 3 ]
6.02
15.87
10.95
26.60
InternVL3.5-8B [ 50 ]
13.50
18.16
15.83
56.50
Table 2: Performance evaluation on our STEM2Code-Eval with 1k samples. We employ three metrics to assess their performances : (1) Image Scoring measures the visual similarity between the generated and original images; (2) Code Scoring assesses the quality, structure, and correctness of the generated Python code itself; (3) Exec Rate reports the execution success rate.
Group
Image Captioner
Benchmark Datasets (%)
Average
MathVision
MathVista
MathVerse
DynaMath
WeMath
LogicVista
LLM Solver: Qwen3-30A3-Thinking [ 55 ]
1
Qwen3-VL-8B-Instruct [ 36 ]
54.37
69.60
63.75
72.19
45.43
56.82
60.36
+ IR-CodeCap
55.86
69.50
63.50
72.59
46.95
57.05
60.91
+ ID-CodeCap
58.32
71.20
64.52
71.70
46.76
60.40
62.15
+ SG-CodeCap
59.93
68.81
66.54
72.73
47.05
62.64
62.75
Table 3: Ablation study on perception ability, evaluated using the captioner-solver setup. More accurate descriptions are expected to yield better reasoning results. “IM”, “ID”, “SG” refer to Image Reproduce, Image Diversity, and Solid Geometry pipeline, respectively. “CodeCap” and “ImCode” refer to code-grounded captions and explanatory image-code pairs, respectively.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The training curves of our proposed models. (a) the curves of CodePercept-S1 models. (b) the curves of CodePercept-R1 models.
Figure 6: The pipeline of our proposed STEM2Code-Eval Benchmark, which includes 1000 image-code pairs that have been checked and revised by human annotators.
Image Captioner
Benchmark Datasets (%)
Average
MathVision
MathVista
MathVerse
DynaMath
WeMath
LogicVista
LLM Solver: Qwen3-30A3-Thinking [ 55 ]
KeyeVL1.5-8B [ 43 ]
54.11
64.90
49.95
62.37
33.62
45.19
51.69
Intern-S1-8B [ 3 ]
51.67
65.70
51.90
63.61
33.43
51.23
52.92
GLM-4.1V-9B [ 20 ]
53.75
64.60
54.47
66.17
40.76
51.00
55.13
InternVL3.5-8B [ 50 ]
53.32
67.70
53.40
68.12
41.05
51.68
55.88
Appendix
Table 4: Performance comparison of various MLLMs across six STEM reasoning benchmarks.
Model
Image Score
Code Score
Exec Rate
Avg
Intern-S1-8B [ 3 ]
6.02
15.87
26.60
16.16
InternVL3.5-8B [ 50 ]
13.50
18.16
56.50
29.38
MiniCPM-V-4.5 [ 44 ]
13.91
23.69
50.80
29.47
MiMo-VL-7B-RL [ 54 ]
14.54
22.41
60.30
32.42
Ovis2.5-9B [ 29 ]
9.76
11.26
89.40
36.81
KeyeVL1.5-8B [ 43 ]
20.33
22.47
73.40
38.73
Appendix
Table 5: Performance evaluation on our STEM2Code-Eval with 1k samples. We employ three metrics to assess their performances : (1) Image Scoring measures the visual similarity between the generated and original images; (2) Code Scoring assesses the quality, structure, and correctness of the generated Python code itself; (3) Exec Rate reports the execution success rate.
We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.
While humans develop core visual skills long before acquiring language, contemporary Multimodal LLMs (MLLMs) still rely heavily on linguistic priors to compensate for their fragile visual understanding. We uncovered a crucial fact: state-of-the-art MLLMs consistently fail on basic visual tasks that humans, even 3-year-olds, can solve effortlessly. To systematically investigate this gap, we introduce BabyVision, a benchmark designed to assess core visual abilities independent of linguistic knowledge for MLLMs. BabyVision spans a wide range of tasks, with 388 items divided into 22 subclasses across four key categories. Empirical results and human evaluation reveal that leading MLLMs perform significantly below human baselines. Gemini3-Pro-Preview scores 49.7, lagging behind 6-year-old humans and falling well behind the average adult score of 94.1. These results show despite excelling in knowledge-heavy evaluations, current MLLMs still lack fundamental visual primitives. Progress in BabyVision represents a step toward human-level visual perception and reasoning capabilities. We also explore solving visual reasoning with generation models by proposing BabyVision-Gen and automatic evaluation toolkit. Our code and benchmark data are released at https://github.com/UniPat-AI/BabyVision for reproduction.
Liang Chen, Weichu Xie, Yiyan Liang +27
1UniPat AI · 6Peking University · 3Alibaba Group +8
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences