SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

Ye Shen; Yuting Zheng; Dun Pei; Zijian Chen; Wenlong Zhang; Qi Jia; Guangtao Zhai

cs.AIAug 26, 2026

SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

Authors: Ye Shen, Yuting Zheng, Dun Pei, Zijian Chen, Wenlong Zhang, Qi Jia, Guangtao Zhai

Organizations: Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University

Abstract

Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs) to advance the development of scientific fields. In this work, we introduce SciMIF, a novel benchmark designed to evaluate the capability of MLLMs in following complex scientific instructions. Specifically, based on an extensive analysis of 22 distinct tasks across 5 representative scientific disciplines, we propose a comprehensive taxonomy comprising 10 constraint groups that captures both general functional requirements and discipline-specific characteristics. Guided by this taxonomy, we develop a high-fidelity instruction injection pipeline to systematically augment existing scientific datasets. We conduct comprehensive experiments on multiple state-of-the-art closed-source and open-source MLLMs. Our findings reveal significant performance disparities across different scientific disciplines, with chemistry posing greater challenges for current MLLMs. Furthermore, we observe that increasing the model scale does not yield corresponding improvements in constraint adherence, and current models still struggle severely with fine-grained constraints and instructions requiring the deep application of disciplinary knowledge. SciMIF fills the current void in evaluating multimodal instruction adherence within scientific domains, laying a crucial foundation for future enhancements of MLLMs in rigorous scientific applications. Data and code will be released at https://github.com/shenye7436/SciMIF .

Figures & tables

Figure 1: An overview of SciMIF, illustrating its coverage of five scientific disciplines, one general constraint domain, and ten functional constraint categories for constructing diverse scientific instruction-following constraints.

Benchmark	Scope	# Grp.	# Const.	# Disc.	# Samp.	Multi- modal
General-purpose instruction-following benchmarks
IFEval ( Zhou et al. 2023 )	General	9	25	–	541	✗
FollowBench ( Jiang et al. 2024 )	General	5	16	–	820	✗
MM-IFEngine ( Ding et al. 2025 )	General	6	32	–	400	✓
Scientific instruction-following benchmarks
SciIF ( Su et al. 2026 )	Scientific	3	10	4	1,244	✗

Table 1: Comparison of general-purpose and scientific instruction-following benchmarks. Scope specifies whether they evaluate general, scientific, or both types of constraints. # Grp. and # Const. denote the numbers of functional constraint groups and individual constraints; # Disc. and # Samp. denote covered scientific disciplines and evaluation samples. “–” indicates scientific discipline coverage is not applicable.

Figure 2: An overview of the data construction pipeline.

Statistic	Chem.	Geo.	Bio.	Mat.	Phy.
Total	518	523	493	497	496
Scientific Constraints	7	5	5	5	3
Task Types	8	4	3	4	3
Avg. Question (/tokens)	231.46	168.17	147.29	215.60	293.68
Multimodal Samples	$\times$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$

Table 2: Overall statistics of the constructed dataset.

Model	Chemistry			Geography			Biology			Material			Physics			Overall
Model	CSR	ISR	DRFR	CSR	ISR	DRFR	CSR	ISR	DRFR	CSR	ISR	DRFR	CSR	ISR	DRFR	Overall
Closed-Source MLLMs
GPT-5.2	72.87	46.33	73.99	82.30	61.57	82.40	88.87	72.82	86.58	87.79	78.07	88.93	87.97	69.56	86.94	65.67
Grok-4-Fast	65.56	33.01	67.37	83.96	53.15	83.25	80.18	45.53	79.06	89.74	75.25	89.01	82.06	58.27	81.16	53.04
Gemini-3.1-Pro-Preview	65.79	32.24	66.37	79.53	47.80	80.01	75.30	44.02	72.01	89.94	75.45	89.00	77.46	50.60	77.78	50.02
Claude-Sonnet-4.6	62.02	26.31	63.42	71.31	31.72	72.75	76.56	46.32	73.75	81.54	67.40	84.06	86.48	65.12	85.87	47.37

Table 3: Performance (%) of evaluated MLLMs. Overall denotes the average ISR across the five scientific disciplines. The best score in each column is bolded separately for closed-source and open-source MLLMs.

Figure 3: DRFR scores (%) across different scientific constraint domains for the evaluated MLLMs.

Figure 4: DRFR scores (%) of representative MLLMs between general and scientific constraint domains.

Model	Format	Precision	Selection	Structure	Number	Letter	Procedure	Method	Terminology	Unit
Closed-Source MLLMs
GPT-5.2	72.18	84.69	82.24	83.41	68.62	51.30	94.19	95.91	73.59	85.61
Grok-4-Fast	82.31	77.99	82.89	77.35	58.50	53.90	78.14	94.55	72.23	80.00
Gemini-3.1-Pro-Preview	74.38	75.60	80.92	63.90	55.76	50.00	78.29	96.68	72.23	77.09
Claude-Sonnet-4.6	50.67	85.37	83.44	76.91	58.89	35.33	76.74	98.28	74.60	88.86
Open-Source MLLMs

Table 4: DRFR scores (%) of different constraint groups from evaluated MLLMs. The best score in each column is bolded separately for closed-source and open-source MLLMs.

Figure 5: Proportions of Correct Followed (CF), Correct Violated (CV), Incorrect Followed (IF), and Incorrect Violated (IV) samples across representative models.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Group	Instruction Name	Description
General	Format	json_constraint	Format the output answer strictly as a JSON object.
	Selection	choose_from	Select and output exactly one option from choices.
	Precision	decimal_number	Ensure that all numerical quantities in the output retaining the required number of decimal places.
	Letter	all_uppercase	Output format must be in all uppercase.
Chemistry	Terminology	molecular_validity	Generate valid chemical nomenclature adhering to specified systems.
Chemistry	Procedure	reaction_steps	Output an orderly, multi-step chemical reaction sequence.

Appendix

Table 5: Representative Constraints Selected from Each Domain.

Figure 6: Distribution of the ten functional constraint groups across the five scientific disciplines in SciMIF. The proportions represent the frequency with which each group is instantiated within the corresponding discipline.

Group	Constraint Name	Description	Metric
Unit	chemistry: unit_consistency	Output numerical results using specified chemical stoichiometric units.	Script-Based
Terminology	chemistry: molecular_validity	Generate valid chemical nomenclature adhering to specified systems.	Script-Based
Number	chemistry: atom_count_constraint	Ensure generated molecules contain a specified number of atoms.	Script-Based
	chemistry: atom_bond_constraint	Ensure generated molecules contain a specified number of chemical bonds.	Script-Based
	chemistry: atom_group_constraint	Ensure generated molecules contain a specified number of functional groups.	Script-Based
Method	chemistry: method_constraint	Apply specified chemical formulas or laws for calculation and reasoning.	Script-Based

Appendix

Table 6: Constraint Specifications in the Chemistry Domain.

Group	Constraint Name	Description	Metric
Unit	geography: unit_consistency	Output numerical results with specified geographical units.	Script-Based
Terminology	geography: address_hierarchy	Generate valid geographical addresses following spatial hierarchies.	Script-Based
Format	geography: scene_option_constraint	Select the answer strictly from a provided list of geographical scenes.	Script-Based
Method	geography: method_constraint	Apply specified geographical formulas or laws for calculation and reasoning.	Script-Based
Procedure	geography: analysis_steps_constraint	Conduct reasoning according to specified geographical analysis steps.	LLM-as-a-Judge

Appendix

Table 7: Constraint Specifications in the Geography Domain.

Group	Constraint Name	Description	Metric
Unit	biology: unit_consistency	Output numerical results using specified biological units.	Script-Based
Terminology	biology: entity_relationship_format_validity	Extract and format specified biological entity relationships accurately.	Script-Based
Number	biology: protein_length	Output the required length of a specified biological sequence.	Script-Based
Method	biology: method_constraint	Use specified biological formulas or laws for calculation and reasoning.	Script-Based
Procedure	biology: analysis_steps_constraint	Analyze biological molecules according to a specified sequential bio-analysis flow.	LLM-as-a-Judge

Appendix

Table 8: Constraint Specifications in the Biology Domain.

Group	Constraint Name	Description	Metric
Unit	material: unit_consistency	Output numerical results using specified materials science units.	Script-Based
Terminology	material: characterization_technique	Identify valid materials characterization techniques.	Script-Based
Format	material: property_type	Output different formats based on the material properties.	Script-Based
Method	material: method_constraint	Apply specified materials science formulas or laws for reasoning.	Script-Based
Procedure	material: analysis_steps_constraint	Conduct reasoning according to specified materials analysis steps.	LLM-as-a-Judge

Appendix

Table 9: Constraint Specifications in the Material Domain.

Group	Constraint Name	Description	Metric
Unit	physics: unit_consistency	Output numerical results using specified physical units.	Script-Based
Method	physics: method_constraint	Apply specified physical formulas or laws for calculation and reasoning.	Script-Based
Procedure	physics: analysis_steps	Solve problems step-by-step with physical principles.	LLM-as-a-Judge

Appendix

Table 10: Constraint Specifications in the Physics Domain.

Group	Constraint Name	Description	Metric
Precision	general: decimal_number	Ensure that all numerical quantities in the output retaining the required number of decimal places.	Script-Based
	general: scientific_annotation	Express all numerical quantities strictly in scientific notation.	Script-Based
Letter	general: all_uppercase	Output format must be in all uppercase.	Script-Based
	general: all_lowercase	Format the output answer strictly in all lowercase letters.	Script-Based
Format	general: wrap_up	Enclose the final output answer within a specified box format.	Script-Based
	general: json_constraint	Format the output answer strictly as a JSON object.	Script-Based

Appendix

Table 11: Constraint Specifications in the General Domain.

Source	Task	Num.	Source	Task	Num.
ChemEval ( Huang et al. 2024 )	chemistry_numerical _task	70	IMAGEO-Bench ( LI et al. 2025 )	earth_scene_option	100
ChemEval ( Huang et al. 2024 )	chemistry_entity_option	70	LAB-Bench ( Laurent et al. 2024 )	biology_sequence_qa	143
ChemEval ( Huang et al. 2024 )	chemistry_reaction_steps	20	Mol-Instructions ( Fang et al. 2024 )	biology_entity_ relationship	100
ChemEval ( Huang et al. 2024 )	chemistry_molecular _format	70	Mol-Instructions ( Fang et al. 2024 )	biology_analysis_steps	250
ChemEval ( Huang et al. 2024 )	chemistry_open_ended	198	MaScQA ( Zaki et al. 2023 )	material_numerical _problem	100
S2-TOMG-Bench-mini ( Li et al. 2025 )	chemistry_molcustom_ bondnum	30	MatCha ( Lai et al. 2025 )	material_characterization _technique	100

Appendix

Table 12: Overview of task types, source datasets, and sample counts.

Model	Chemistry	Geography	Biology	Material	Physics	General	Scientific
Closed-Source MLLMs
GPT-5.2	72.65	91.28	92.69	92.06	93.66	74.65	88.74
Grok-4-Fast	68.43	83.50	82.00	91.92	88.40	74.46	82.93
Gemini-3.1-Pro-Preview	63.51	89.97	80.24	93.48	88.84	66.04	83.45
Claude-Sonnet-4.6	71.38	86.27	80.11	91.41	93.65	61.14	84.74
Open-Source MLLMs

Appendix

Table 13: DRFR scores (%) of different constraint domains from evaluated MLLMs. The best score in each column is bolded separately for closed-source and open-source MLLMs.

Model	$\phi$	Jaccard	$p$ -value
GPT-5.2	0.1178	0.3585	$3.21\times 10^{-9}$
Grok-4-Fast	0.0081	0.2930	0.68
Gemini-3.1-Pro-Preview	0.1548	0.3538	$7.35\times 10^{-15}$
Claude-Sonnet-4-6	0.2069	0.3640	$4.31\times 10^{-25}$
Qwen3.5-397B-A17B	0.0672	0.3300	$7.68\times 10^{-4}$
InternVL3.5-14B	0.1459	0.2704	$2.33\times 10^{-13}$

Appendix

Table 14: Within-model associations between scientific correctness and instruction following.

\phi

coefficient measures the direction and strength of the binary association, Jaccard measures the overlap between the two outcomes, and the p-values are obtained from two-sided Pearson

\chi^{2}

tests of independence.

	MaScQA	Matcha	LLM4Mat-Bench	MatSciBench
DRFR	93.19	86.47	63.92	96.28

Appendix

Table 15: Per-source DRFR (%) of GPT-5.2 on materials science.

Modality	Geography	Material	Physics
Multimodal	64.67	85.50	74.49
Text-only	87.92	90.16	83.76

Appendix

Table 16: DRFR scores (%) of Gemini-3.1-Pro-Preview on multimodal and text-only subsets.

Panel of Figure 7 — Figure 7: Scientific answer correctness before and after constraint augmentation for two representative models across the five disciplines. Before denotes evaluation on the original scientific questions, while After denotes evaluation after compatible scientific and general constraints are added. The comparison shows how additional instruction-following requirements are associated with changes in scientific answer correctness.

Figure 8: GPT-5.2 performance across individual scientific constraints.

Figure 9: A representative physics case illustrating partial constraint satisfaction. The response satisfies the required analysis procedure and response structure but violates the lowercase constraint.

Explore similar work

May 11, 2026

SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation

Scientific reasoning is a key aspect of human intelligence, requiring the integration of multimodal inputs, domain expertise, and multi-step inference across various subjects. Existing benchmarks for multimodal large language models (MLLMs) often fail to capture the complexity and traceability of reasoning processes necessary for rigorous evaluation. To fill this gap, we introduce SciVQR, a multimodal benchmark covering 54 subfields in mathematics, physics, chemistry, geography, astronomy, and biology. SciVQR includes domain-specific visuals, such as equations, charts, and diagrams, and challenges models to combine visual comprehension with reasoning. The tasks range from basic factual recall to complex, multi-step inferences, with 46% including expert-authored solutions. SciVQR not only evaluates final answers but also examines the reasoning process, providing insights into how models reach their conclusions. Our evaluation of leading MLLMs, including both proprietary and open-source models, reveals significant limitations in handling complex multimodal reasoning tasks, underscoring the need for improved multi-step reasoning and better integration of interdisciplinary knowledge in advancing MLLMs toward true scientific intelligence. The dataset and evaluation code are publicly available at https://github.com/CASIA-IVA-Lab/SciVQR.

Longteng Guo, Xuanxu Lin, Dongze Hao +5

Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) · OPPO AI Center

Multimodal Large Language Models Scientific QA Multimodal Reasoning Multimodal Model Evaluation

Ask PDF

Jun 11, 2026

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

Three paradigmatic forms of inference recur across scientific reasoning: deduction, induction, and causal abduction. Reliably evaluating LLMs on these in scientific settings is currently out of reach: scientific benchmarks built on human annotations are costly and lack mechanistic ground truth, while synthetic logical-reasoning benchmarks do not resemble real scientific documents. We introduce SciR, a benchmark that combines multi-paradigm reasoning with controllable scientific rendering, anchored on three paradigmatic scientific problems. Tasks are generated from formal objects (deduction tree, inductive rule hypothesis, causal graph) to guarantee verifiable answers, then rendered into multi-document scientific discourse via per-track domain-tuned genres. The construction lets us independently vary two difficulty axes: how hard it is to extract the key information needed for inference, and how hard the principled inference itself is. We test six models. Both axes hurt every model, and their effects compound. The rendering even hurts neurosymbolic pipelines, which hand inference to a verified solver. The two axes yield a per-model extraction-vs-inference profile: for instance, reasoning models like deepseek-r1 mostly surpass non-reasoning instruct models on the inference axis. To our knowledge, SciR is the first multi-paradigm scientific-reasoning benchmark with parametric control on both extraction and inference difficulty.

Pierre Beckmann, Marco Valentino, Andre Freitas

Idiap Research Institute, Switzerland · EPFL, Switzerland · School of Computer Science, University of Sheffield, UK +2

LLM Evaluation Scientific Reasoning Synthetic Benchmark Generation Structured Reasoning Scientific Reasoning in Language Models

Ask PDF

May 19, 2026

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models

Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice. Most benchmarks are manually curated or domain-generic, limiting scalability and alignment with real scientific use cases. In this paper, we propose a new framework named SciCustom to address the problem. It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. SciCustom first organizes scientific knowledge into ontology-grounded knowledge units with controlled granularity and trains a tagger to map large-scale data instances into this knowledge space. Given a custom requirement, relevant knowledge units are identified via voting-based multi-model consensus. These units enable relevance-aware benchmark retrieval via binary search, followed by proxy subset selection and data-grounded benchmark generation for efficient evaluation. Experiments in chemistry and healthcare demonstrate that SciCustom reveals fine-grained differences in LLM scientific capabilities that standard benchmarks overlook, while requiring neither expert annotation nor synthetic question generation. This work provides a scalable and application-aware foundation for benchmarking scientific capabilities in LLMs. The source code is available at https://github.com/yjwtheonly/SciCustom.

Yiyang Gu, Junwei Yang, Junyu Luo +15

State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5

AI for Science LLM Evaluation

Ask PDF