DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs

Dikshant Kukreja; Kshitij Sah; Karan Goyal; Mukesh Mohania; Vikram Goyal

cs.CVApr 6, 2026

DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs

Authors: Dikshant Kukreja, Kshitij Sah, Karan Goyal, Mukesh Mohania, Vikram Goyal

Organizations: IIIT Delhi, India

Abstract

When asked to describe a molecular diagram, a Vision-Language Model correctly identifies ``a benzene ring with an -OH group.'' When asked to reason about the same image, it answers incorrectly. The model can see but it cannot think about what it sees. We term this the perception-integration gap: a failure where visual information is successfully extracted but lost during downstream reasoning, invisible to single-configuration benchmarks that conflate perception with integration under one accuracy number. To systematically expose such failures, we introduce DISSECT, a 12,000-question diagnostic benchmark spanning Chemistry (7,000) and Biology (5,000). Every question is evaluated under five input modes -- Vision+Text, Text-Only, Vision-Only, Human Oracle, and a novel Model Oracle in which the VLM first verbalizes the image and then reasons from its own description -- yielding diagnostic gaps that decompose performance into language-prior exploitation, visual extraction, perception fidelity, and integration effectiveness. Evaluating 18~VLMs, we find that: (1) Chemistry exhibits substantially lower language-prior exploitability than Biology, confirming molecular visual content as a harder test of genuine visual reasoning; (2) Open-source models consistently score higher when reasoning from their own verbalized descriptions than from raw images, exposing a systematic integration bottleneck; and (3) Closed-source models show no such gap, indicating that bridging perception and integration is the frontier separating open-source from closed-source multimodal capability. The Model Oracle protocol is both model and benchmark agnostic, applicable post-hoc to any VLM evaluation to diagnose integration failures.

Figures & tables

Figure 1. The DISSECT five-mode evaluation protocol. Each question is evaluated under Vision+Text, Text-Only, Vision-Only, Human Oracle, and Model Oracle configurations. Five boxes showing the same chemistry question rendered under five different input configurations.

Closed-Source
	Chemistry (7,000 questions)							Biology (5,000 questions)
Model	V+T	T	V	$O_{H}$	$O_{M}$	LPG $\downarrow$	IG	V+T	T	V	$O_{H}$	$O_{M}$	LPG $\downarrow$	IG
GPT-5	71.2	24.3	62.4	79.1	72.0	0.34	+0.8	78.3	51.2	71.1	84.2	78.9	0.65	+0.6
Gemini 2.5 Flash	68.7	22.8	59.6	76.4	69.5	0.33	+0.8	75.8	49.3	68.4	82.1	76.5	0.65	+0.7
Claude Sonnet 4	70.4	23.6	61.2	78.3	71.2	0.34	+0.8	77.1	50.4	69.8	83.6	77.8	0.65	+0.7
Open-Source Large ( $\geq$ 7B)

Table 1. Performance (%) on DISSECT across five evaluation modes. V+T : Vision+Text; T : Text-Only; V : Vision-Only;

\mathbf{O_{H}}

: Human Oracle;

\mathbf{O_{M}}

: Model Oracle; LPG : Language Prior Gap (

T/\text{V+T}

, lower = more visually dependent); IG : Integration Gap (

O_{M}-\text{V+T}

, positive = integration bottleneck). Best open-source per column in bold ; best overall underlined .

Closed-Source
Model	V+T	$O_{M}$	CoT	Think	2P-Img	R-IG
GPT-5	71.2	72.0	74.6	76.1	72.4	-2.6
Gemini 2.5 Flash	68.7	69.5	72.1	73.6	70.1	-2.6
Claude Sonnet 4	70.4	71.2	73.8	75.2	71.7	-2.6
Open-Source (selected)
InternVL3-78B	62.4	68.9	64.7	64.7 $\dagger$	63.6	+4.2

Table 2. Control experiments on Chemistry (7,000 questions).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Mode	Name	Image Input	Text Input
1	V+T	Diagram only (Fig. 2 )	Question text
2	T	None	Question text only
3	V	Full image with questions (Fig. 3 )	None
4	$O_{H}$	Diagram only (Fig. 2 )	Human annotation + question text
5	$O_{M}$	Diagram only (Fig. 2 )	Pass 2: model description + question

Appendix

Table 3. Biology sample: input construction across all five modes.

Figure 2. Biology sample, Modes 1/4/5 — Diagram of the human female reproductive system with labeled structures (A = Fallopian tube / Oviduct, B = Uterus, C = Ovary) and the ovarian follicle development cycle (primordial follicle $\to$ primary $\to$ secondary $\to$ ovulation $\to$ corpus luteum $\to$ corpus albicans). Question text is provided as a separate text channel.

Figure 3. Biology sample, Mode 3 (V) — The complete image including the diagram and the three sub-questions (a), (b), (c) rendered within the image. This single image is the only input; no separate text channel is provided. The model must OCR the questions from the image while simultaneously interpreting the anatomical diagram and follicle development cycle.

Mode	Name	Image Input	Text Input
1	V+T	Cell diagram (Fig. 4 )	Question text
2	T	None	Question text only
3	V	Composite image (Fig. 5 )	None
4	$O_{H}$	Cell diagram (Fig. 4 )	Human annotation + question text
5	$O_{M}$	Cell diagram (Fig. 4 )	Pass 2: model description + question

Appendix

Table 4. Chem Q30 input construction across all five modes.

Figure 4. Chem Q30 — Electrolysis cell diagram showing 0.60 M SnSO 4 solution (0.50 L), inert anode and cathode, Sn 2+ and SO 4 2 $-$ ions, electrode reactions, Sn deposit on cathode, and 4.60 A current source. This image is used in Modes 1 (V+T), 4 ( $O_{H}$ ), and 5 ( $O_{M}$ Pass 1).

Figure 5. Chem Q30 — Mode 3 (V) composite image: the electrolysis cell diagram with the complete question text and all four options rendered below. This single image is the only input; no separate text channel is provided.

Mode	Biology (Reproductive System)	Chemistry (Chem Q30: Electrolysis)
1 (V+T)	Requires identifying labeled structures (A, B, C) + interpreting follicle development cycle + reproductive biology reasoning	Requires reading cell setup + Faraday’s law calculation
2 (T)	Mixed dependency: (a) and (c) answerable from knowledge; (b) unanswerable—“part B” is meaningless without the diagram	Largely answerable: all numerical data is in the text
3 (V)	Dense diagram labels (“Ovary,” “Uterus,” “Corpus Luteum”) overlap with question text; model must parse which is which	Numerical OCR errors (misreading 0.60 as 0.80) cascade into incorrect calculations
4 ( $O_{H}$ )	Annotations map B = Uterus, resolving the critical label; remaining task is physiological reasoning about menstruation	Annotations provide n-factor and pre-computed moles; remaining task is arithmetic
5 ( $O_{M}$ )	Perceptual risk: correctly reading labels “Ovary” and “Uterus” but failing to associate them with letter labels A, B, C	Perceptual risk: misreading charges or concentrations; small errors cause large calculation errors

Appendix

Table 5. Diagnostic signals for each mode across Biology and Chemistry examples. The same five-mode framework reveals different failure patterns depending on the subject and question type.

Reasoning for VLMs

See all 36 papers

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
Apr 17, 2026Yige Xu, Yongjie Wang, Zizhuo Wu +3Vision-Language Models VLM Adaptation
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China
From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Sep 28, 2026Rong Yu Xu, Prayag Tiwari, Shaolei ZhangEfficient VLM Inference VLM Interpretability
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China
ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding
Apr 8, 2026Xuanle Zhao, Xinyuan Cai, Xiang Cheng +2Vision-Language Models VLM Reasoning
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Apr 17, 2026J/K move · Enter open · S save

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap

Yige Xu, Yongjie Wang, Zizhuo Wu, Kaisong Song+2

College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China

Vision-Language Models VLM Adaptation Visual Reasoning VLM Reasoning Modality Gap in VLMs Multimodal Model Evaluation

Open paper AskPDF arXiv

Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.

Abstract

Reasoning for VLMs

Explore similar work

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding