When asked to describe a molecular diagram, a Vision-Language Model correctly identifies ``a benzene ring with an -OH group.'' When asked to reason about the same image, it answers incorrectly. The model can see but it cannot think about what it sees. We term this the perception-integration gap: a failure where visual information is successfully extracted but lost during downstream reasoning, invisible to single-configuration benchmarks that conflate perception with integration under one accuracy number. To systematically expose such failures, we introduce DISSECT, a 12,000-question diagnostic benchmark spanning Chemistry (7,000) and Biology (5,000). Every question is evaluated under five input modes -- Vision+Text, Text-Only, Vision-Only, Human Oracle, and a novel Model Oracle in which the VLM first verbalizes the image and then reasons from its own description -- yielding diagnostic gaps that decompose performance into language-prior exploitation, visual extraction, perception fidelity, and integration effectiveness. Evaluating 18~VLMs, we find that: (1) Chemistry exhibits substantially lower language-prior exploitability than Biology, confirming molecular visual content as a harder test of genuine visual reasoning; (2) Open-source models consistently score higher when reasoning from their own verbalized descriptions than from raw images, exposing a systematic integration bottleneck; and (3) Closed-source models show no such gap, indicating that bridging perception and integration is the frontier separating open-source from closed-source multimodal capability. The Model Oracle protocol is both model and benchmark agnostic, applicable post-hoc to any VLM evaluation to diagnose integration failures.
Figures & tables
Figure 1. The DISSECT five-mode evaluation protocol. Each question is evaluated under Vision+Text, Text-Only, Vision-Only, Human Oracle, and Model Oracle configurations. Five boxes showing the same chemistry question rendered under five different input configurations.
Chemistry (7,000 questions)
Biology (5,000 questions)
Model
V+T
T
V
OH
OM
LPG ↓
IG
V+T
T
V
OH
OM
LPG ↓
IG
Closed-Source
GPT-5
71.2
24.3
62.4
79.1
72.0
0.34
+0.8
78.3
51.2
71.1
84.2
78.9
0.65
+0.6
Gemini 2.5 Flash
68.7
22.8
59.6
76.4
69.5
0.33
+0.8
75.8
49.3
68.4
82.1
76.5
0.65
+0.7
Claude Sonnet 4
70.4
23.6
61.2
78.3
71.2
0.34
+0.8
77.1
50.4
69.8
83.6
77.8
0.65
+0.7
Open-Source Large ( ≥ 7B)
Table 1. Performance (%) on DISSECT across five evaluation modes. V+T : Vision+Text; T : Text-Only; V : Vision-Only; OH : Human Oracle; OM : Model Oracle; LPG : Language Prior Gap ( T/V+T , lower = more visually dependent); IG : Integration Gap ( OM−V+T , positive = integration bottleneck). Best open-source per column in bold ; best overall underlined .
Model
V+T
OM
CoT
Think
2P-Img
R-IG
Closed-Source
GPT-5
71.2
72.0
74.6
76.1
72.4
-2.6
Gemini 2.5 Flash
68.7
69.5
72.1
73.6
70.1
-2.6
Claude Sonnet 4
70.4
71.2
73.8
75.2
71.7
-2.6
Open-Source (selected)
InternVL3-78B
62.4
68.9
64.7
64.7 †
63.6
+4.2
Table 2. Control experiments on Chemistry (7,000 questions).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Mode
Name
Image Input
Text Input
1
V+T
Diagram only (Fig. 2 )
Question text
2
T
None
Question text only
3
V
Full image with questions (Fig. 3 )
None
4
OH
Diagram only (Fig. 2 )
Human annotation + question text
5
OM
Diagram only (Fig. 2 )
Pass 2: model description + question
Appendix
Table 3. Biology sample: input construction across all five modes.
Figure 2. Biology sample, Modes 1/4/5 — Diagram of the human female reproductive system with labeled structures (A = Fallopian tube / Oviduct, B = Uterus, C = Ovary) and the ovarian follicle development cycle (primordial follicle → primary → secondary → ovulation → corpus luteum → corpus albicans). Question text is provided as a separate text channel.
Figure 3. Biology sample, Mode 3 (V) — The complete image including the diagram and the three sub-questions (a), (b), (c) rendered within the image. This single image is the only input; no separate text channel is provided. The model must OCR the questions from the image while simultaneously interpreting the anatomical diagram and follicle development cycle.
Mode
Name
Image Input
Text Input
1
V+T
Cell diagram (Fig. 4 )
Question text
2
T
None
Question text only
3
V
Composite image (Fig. 5 )
None
4
OH
Cell diagram (Fig. 4 )
Human annotation + question text
5
OM
Cell diagram (Fig. 4 )
Pass 2: model description + question
Appendix
Table 4. Chem Q30 input construction across all five modes.
Figure 4. Chem Q30 — Electrolysis cell diagram showing 0.60 M SnSO 4 solution (0.50 L), inert anode and cathode, Sn 2+ and SO 4 2 − ions, electrode reactions, Sn deposit on cathode, and 4.60 A current source. This image is used in Modes 1 (V+T), 4 ( OH ), and 5 ( OM Pass 1).
Figure 5. Chem Q30 — Mode 3 (V) composite image: the electrolysis cell diagram with the complete question text and all four options rendered below. This single image is the only input; no separate text channel is provided.
Mode
Biology (Reproductive System)
Chemistry (Chem Q30: Electrolysis)
1 (V+T)
Requires identifying labeled structures (A, B, C) + interpreting follicle development cycle + reproductive biology reasoning
Requires reading cell setup + Faraday’s law calculation
2 (T)
Mixed dependency: (a) and (c) answerable from knowledge; (b) unanswerable—“part B” is meaningless without the diagram
Largely answerable: all numerical data is in the text
3 (V)
Dense diagram labels (“Ovary,” “Uterus,” “Corpus Luteum”) overlap with question text; model must parse which is which
Numerical OCR errors (misreading 0.60 as 0.80) cascade into incorrect calculations
4 ( OH )
Annotations map B = Uterus, resolving the critical label; remaining task is physiological reasoning about menstruation
Annotations provide n-factor and pre-computed moles; remaining task is arithmetic
5 ( OM )
Perceptual risk: correctly reading labels “Ovary” and “Uterus” but failing to associate them with letter labels A, B, C
Perceptual risk: misreading charges or concentrations; small errors cause large calculation errors
Appendix
Table 5. Diagnostic signals for each mode across Biology and Chemistry examples. The same five-mode framework reveals different failure patterns depending on the subject and question type.
Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.
Yige Xu, Yongjie Wang, Zizhuo Wu +3
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China
Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.
Rong Yu Xu, Prayag Tiwari, Shaolei Zhang
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China
While Vision-Language Models (VLMs) have demonstrated significant potential in chemical visual understanding, current models are predominantly optimized for direct visual question-answering tasks. This paradigm often results in "black-box" systems that fail to utilize the inherent capability of Large Language Models (LLMs) to infer underlying reaction mechanisms. In this work, we introduce ChemVLR, a chemical VLM designed to prioritize reasoning within the perception process. Unlike conventional chemical VLMs, ChemVLR analyzes visual inputs in a fine-grained manner by explicitly identifying granular chemical descriptors, such as functional groups, prior to generating answers. This approach ensures the production of explicit and interpretable reasoning paths for complex visual chemical problems. To facilitate this methodology, we implement a cross-modality reverse-engineering strategy, combined with a rigorous filtering pipeline, to curate a large-scale reasoning-and-captioning dataset comprising 760k high-quality samples across molecular and reaction tasks. Furthermore, we adopt a three-stage training framework that systemically builds model perception and reasoning capacity. Experiments demonstrate that ChemVLR achieves state-of-the-art (SOTA) performance, surpassing both leading proprietary models and domain-specific open-source baselines. We also provide comprehensive ablation studies to validate our training strategy and data generation designs. Code and model weights will be available at https://github.com/xxlllz/ChemVLR.
Xuanle Zhao, Xinyuan Cai, Xiang Cheng +2
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences