SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

Ye Shen; Yuting Zheng; Dun Pei; Zijian Chen; Wenlong Zhang; Qi Jia; Guangtao Zhai

cs.AIAug 26, 2026

SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

Authors: Ye Shen, Yuting Zheng, Dun Pei, Zijian Chen, Wenlong Zhang, Qi Jia, Guangtao Zhai

Organizations: Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University

Abstract

Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs) to advance the development of scientific fields. In this work, we introduce SciMIF, a novel benchmark designed to evaluate the capability of MLLMs in following complex scientific instructions. Specifically, based on an extensive analysis of 22 distinct tasks across 5 representative scientific disciplines, we propose a comprehensive taxonomy comprising 10 constraint groups that captures both general functional requirements and discipline-specific characteristics. Guided by this taxonomy, we develop a high-fidelity instruction injection pipeline to systematically augment existing scientific datasets. We conduct comprehensive experiments on multiple state-of-the-art closed-source and open-source MLLMs. Our findings reveal significant performance disparities across different scientific disciplines, with chemistry posing greater challenges for current MLLMs. Furthermore, we observe that increasing the model scale does not yield corresponding improvements in constraint adherence, and current models still struggle severely with fine-grained constraints and instructions requiring the deep application of disciplinary knowledge. SciMIF fills the current void in evaluating multimodal instruction adherence within scientific domains, laying a crucial foundation for future enhancements of MLLMs in rigorous scientific applications. Data and code will be released at https://github.com/shenye7436/SciMIF .

Figures & tables

Figure 1: An overview of SciMIF, illustrating its coverage of five scientific disciplines, one general constraint domain, and ten functional constraint categories for constructing diverse scientific instruction-following constraints.

Benchmark	Scope	# Grp.	# Const.	# Disc.	# Samp.	Multi- modal
General-purpose instruction-following benchmarks
IFEval ( Zhou et al. 2023 )	General	9	25	–	541	✗
FollowBench ( Jiang et al. 2024 )	General	5	16	–	820	✗
MM-IFEngine ( Ding et al. 2025 )	General	6	32	–	400	✓
Scientific instruction-following benchmarks
SciIF ( Su et al. 2026 )	Scientific	3	10	4	1,244	✗

Table 1: Comparison of general-purpose and scientific instruction-following benchmarks. Scope specifies whether they evaluate general, scientific, or both types of constraints. # Grp. and # Const. denote the numbers of functional constraint groups and individual constraints; # Disc. and # Samp. denote covered scientific disciplines and evaluation samples. “–” indicates scientific discipline coverage is not applicable.

Figure 2: An overview of the data construction pipeline.

Statistic	Chem.	Geo.	Bio.	Mat.	Phy.
Total	518	523	493	497	496
Scientific Constraints	7	5	5	5	3
Task Types	8	4	3	4	3
Avg. Question (/tokens)	231.46	168.17	147.29	215.60	293.68
Multimodal Samples	$\times$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$

Table 2: Overall statistics of the constructed dataset.

Model	Chemistry			Geography			Biology			Material			Physics			Overall
Model	CSR	ISR	DRFR	CSR	ISR	DRFR	CSR	ISR	DRFR	CSR	ISR	DRFR	CSR	ISR	DRFR	Overall
Closed-Source MLLMs
GPT-5.2	72.87	46.33	73.99	82.30	61.57	82.40	88.87	72.82	86.58	87.79	78.07	88.93	87.97	69.56	86.94	65.67
Grok-4-Fast	65.56	33.01	67.37	83.96	53.15	83.25	80.18	45.53	79.06	89.74	75.25	89.01	82.06	58.27	81.16	53.04
Gemini-3.1-Pro-Preview	65.79	32.24	66.37	79.53	47.80	80.01	75.30	44.02	72.01	89.94	75.45	89.00	77.46	50.60	77.78	50.02
Claude-Sonnet-4.6	62.02	26.31	63.42	71.31	31.72	72.75	76.56	46.32	73.75	81.54	67.40	84.06	86.48	65.12	85.87	47.37

Table 3: Performance (%) of evaluated MLLMs. Overall denotes the average ISR across the five scientific disciplines. The best score in each column is bolded separately for closed-source and open-source MLLMs.

Figure 3: DRFR scores (%) across different scientific constraint domains for the evaluated MLLMs.

Figure 4: DRFR scores (%) of representative MLLMs between general and scientific constraint domains.

Model	Format	Precision	Selection	Structure	Number	Letter	Procedure	Method	Terminology	Unit
Closed-Source MLLMs
GPT-5.2	72.18	84.69	82.24	83.41	68.62	51.30	94.19	95.91	73.59	85.61
Grok-4-Fast	82.31	77.99	82.89	77.35	58.50	53.90	78.14	94.55	72.23	80.00
Gemini-3.1-Pro-Preview	74.38	75.60	80.92	63.90	55.76	50.00	78.29	96.68	72.23	77.09
Claude-Sonnet-4.6	50.67	85.37	83.44	76.91	58.89	35.33	76.74	98.28	74.60	88.86
Open-Source MLLMs

Table 4: DRFR scores (%) of different constraint groups from evaluated MLLMs. The best score in each column is bolded separately for closed-source and open-source MLLMs.

Figure 5: Proportions of Correct Followed (CF), Correct Violated (CV), Incorrect Followed (IF), and Incorrect Violated (IV) samples across representative models.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Group	Instruction Name	Description
General	Format	json_constraint	Format the output answer strictly as a JSON object.
	Selection	choose_from	Select and output exactly one option from choices.
	Precision	decimal_number	Ensure that all numerical quantities in the output retaining the required number of decimal places.
	Letter	all_uppercase	Output format must be in all uppercase.
Chemistry	Terminology	molecular_validity	Generate valid chemical nomenclature adhering to specified systems.
Chemistry	Procedure	reaction_steps	Output an orderly, multi-step chemical reaction sequence.

Appendix

Table 5: Representative Constraints Selected from Each Domain.

Figure 6: Distribution of the ten functional constraint groups across the five scientific disciplines in SciMIF. The proportions represent the frequency with which each group is instantiated within the corresponding discipline.

Group	Constraint Name	Description	Metric
Unit	chemistry: unit_consistency	Output numerical results using specified chemical stoichiometric units.	Script-Based
Terminology	chemistry: molecular_validity	Generate valid chemical nomenclature adhering to specified systems.	Script-Based
Number	chemistry: atom_count_constraint	Ensure generated molecules contain a specified number of atoms.	Script-Based
	chemistry: atom_bond_constraint	Ensure generated molecules contain a specified number of chemical bonds.	Script-Based
	chemistry: atom_group_constraint	Ensure generated molecules contain a specified number of functional groups.	Script-Based
Method	chemistry: method_constraint	Apply specified chemical formulas or laws for calculation and reasoning.	Script-Based

Appendix

Table 6: Constraint Specifications in the Chemistry Domain.

Group	Constraint Name	Description	Metric
Unit	geography: unit_consistency	Output numerical results with specified geographical units.	Script-Based
Terminology	geography: address_hierarchy	Generate valid geographical addresses following spatial hierarchies.	Script-Based
Format	geography: scene_option_constraint	Select the answer strictly from a provided list of geographical scenes.	Script-Based
Method	geography: method_constraint	Apply specified geographical formulas or laws for calculation and reasoning.	Script-Based
Procedure	geography: analysis_steps_constraint	Conduct reasoning according to specified geographical analysis steps.	LLM-as-a-Judge

Appendix

Table 7: Constraint Specifications in the Geography Domain.

Group	Constraint Name	Description	Metric
Unit	biology: unit_consistency	Output numerical results using specified biological units.	Script-Based
Terminology	biology: entity_relationship_format_validity	Extract and format specified biological entity relationships accurately.	Script-Based
Number	biology: protein_length	Output the required length of a specified biological sequence.	Script-Based
Method	biology: method_constraint	Use specified biological formulas or laws for calculation and reasoning.	Script-Based
Procedure	biology: analysis_steps_constraint	Analyze biological molecules according to a specified sequential bio-analysis flow.	LLM-as-a-Judge

Appendix

Table 8: Constraint Specifications in the Biology Domain.

Group	Constraint Name	Description	Metric
Unit	material: unit_consistency	Output numerical results using specified materials science units.	Script-Based
Terminology	material: characterization_technique	Identify valid materials characterization techniques.	Script-Based
Format	material: property_type	Output different formats based on the material properties.	Script-Based
Method	material: method_constraint	Apply specified materials science formulas or laws for reasoning.	Script-Based
Procedure	material: analysis_steps_constraint	Conduct reasoning according to specified materials analysis steps.	LLM-as-a-Judge

Appendix

Table 9: Constraint Specifications in the Material Domain.

Group	Constraint Name	Description	Metric
Unit	physics: unit_consistency	Output numerical results using specified physical units.	Script-Based
Method	physics: method_constraint	Apply specified physical formulas or laws for calculation and reasoning.	Script-Based
Procedure	physics: analysis_steps	Solve problems step-by-step with physical principles.	LLM-as-a-Judge

Appendix

Table 10: Constraint Specifications in the Physics Domain.

Group	Constraint Name	Description	Metric
Precision	general: decimal_number	Ensure that all numerical quantities in the output retaining the required number of decimal places.	Script-Based
	general: scientific_annotation	Express all numerical quantities strictly in scientific notation.	Script-Based
Letter	general: all_uppercase	Output format must be in all uppercase.	Script-Based
	general: all_lowercase	Format the output answer strictly in all lowercase letters.	Script-Based
Format	general: wrap_up	Enclose the final output answer within a specified box format.	Script-Based
	general: json_constraint	Format the output answer strictly as a JSON object.	Script-Based

Appendix

Table 11: Constraint Specifications in the General Domain.

Source	Task	Num.	Source	Task	Num.
ChemEval ( Huang et al. 2024 )	chemistry_numerical _task	70	IMAGEO-Bench ( LI et al. 2025 )	earth_scene_option	100
ChemEval ( Huang et al. 2024 )	chemistry_entity_option	70	LAB-Bench ( Laurent et al. 2024 )	biology_sequence_qa	143
ChemEval ( Huang et al. 2024 )	chemistry_reaction_steps	20	Mol-Instructions ( Fang et al. 2024 )	biology_entity_ relationship	100
ChemEval ( Huang et al. 2024 )	chemistry_molecular _format	70	Mol-Instructions ( Fang et al. 2024 )	biology_analysis_steps	250
ChemEval ( Huang et al. 2024 )	chemistry_open_ended	198	MaScQA ( Zaki et al. 2023 )	material_numerical _problem	100
S2-TOMG-Bench-mini ( Li et al. 2025 )	chemistry_molcustom_ bondnum	30	MatCha ( Lai et al. 2025 )	material_characterization _technique	100

Appendix

Table 12: Overview of task types, source datasets, and sample counts.

Model	Chemistry	Geography	Biology	Material	Physics	General	Scientific
Closed-Source MLLMs
GPT-5.2	72.65	91.28	92.69	92.06	93.66	74.65	88.74
Grok-4-Fast	68.43	83.50	82.00	91.92	88.40	74.46	82.93
Gemini-3.1-Pro-Preview	63.51	89.97	80.24	93.48	88.84	66.04	83.45
Claude-Sonnet-4.6	71.38	86.27	80.11	91.41	93.65	61.14	84.74
Open-Source MLLMs

Appendix

Table 13: DRFR scores (%) of different constraint domains from evaluated MLLMs. The best score in each column is bolded separately for closed-source and open-source MLLMs.

Model	$\phi$	Jaccard	$p$ -value
GPT-5.2	0.1178	0.3585	$3.21\times 10^{-9}$
Grok-4-Fast	0.0081	0.2930	0.68
Gemini-3.1-Pro-Preview	0.1548	0.3538	$7.35\times 10^{-15}$
Claude-Sonnet-4-6	0.2069	0.3640	$4.31\times 10^{-25}$
Qwen3.5-397B-A17B	0.0672	0.3300	$7.68\times 10^{-4}$
InternVL3.5-14B	0.1459	0.2704	$2.33\times 10^{-13}$

Appendix

Table 14: Within-model associations between scientific correctness and instruction following.

\phi

coefficient measures the direction and strength of the binary association, Jaccard measures the overlap between the two outcomes, and the p-values are obtained from two-sided Pearson

\chi^{2}

tests of independence.

	MaScQA	Matcha	LLM4Mat-Bench	MatSciBench
DRFR	93.19	86.47	63.92	96.28

Appendix

Table 15: Per-source DRFR (%) of GPT-5.2 on materials science.

Modality	Geography	Material	Physics
Multimodal	64.67	85.50	74.49
Text-only	87.92	90.16	83.76

Appendix

Table 16: DRFR scores (%) of Gemini-3.1-Pro-Preview on multimodal and text-only subsets.

Panel of Figure 7 — Figure 7: Scientific answer correctness before and after constraint augmentation for two representative models across the five disciplines. Before denotes evaluation on the original scientific questions, while After denotes evaluation after compatible scientific and general constraints are added. The comparison shows how additional instruction-following requirements are associated with changes in scientific answer correctness.

Figure 8: GPT-5.2 performance across individual scientific constraints.

Figure 9: A representative physics case illustrating partial constraint satisfaction. The response satisfies the required analysis procedure and response structure but violates the lowercase constraint.

Explore similar work

SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
May 11, 2026Longteng Guo, Xuanxu Lin, Dongze Hao +5Multimodal Large Language Models Scientific QA
Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) · OPPO AI Center
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Jun 11, 2026Pierre Beckmann, Marco Valentino, Andre FreitasLLM Evaluation Scientific Reasoning
Idiap Research Institute, Switzerland · EPFL, Switzerland · School of Computer Science, University of Sheffield, UK +2
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models
May 19, 2026Yiyang Gu, Junwei Yang, Junyu Luo +15AI for Science LLM Evaluation
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5

May 11, 2026J/K move · Enter open · S save

SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation

Longteng Guo, Xuanxu Lin, Dongze Hao, Tongtian Yue+4

Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) · OPPO AI Center

Multimodal Large Language Models Scientific QA Multimodal Reasoning Multimodal Model Evaluation

Open paper AskPDF arXiv

Scientific reasoning is a key aspect of human intelligence, requiring the integration of multimodal inputs, domain expertise, and multi-step inference across various subjects. Existing benchmarks for multimodal large language models (MLLMs) often fail to capture the complexity and traceability of reasoning processes necessary for rigorous evaluation. To fill this gap, we introduce SciVQR, a multimodal benchmark covering 54 subfields in mathematics, physics, chemistry, geography, astronomy, and biology. SciVQR includes domain-specific visuals, such as equations, charts, and diagrams, and challenges models to combine visual comprehension with reasoning. The tasks range from basic factual recall to complex, multi-step inferences, with 46% including expert-authored solutions. SciVQR not only evaluates final answers but also examines the reasoning process, providing insights into how models reach their conclusions. Our evaluation of leading MLLMs, including both proprietary and open-source models, reveals significant limitations in handling complex multimodal reasoning tasks, underscoring the need for improved multi-step reasoning and better integration of interdisciplinary knowledge in advancing MLLMs toward true scientific intelligence. The dataset and evaluation code are publicly available at https://github.com/CASIA-IVA-Lab/SciVQR.

Abstract

Explore similar work

SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models