PolyReal: A Benchmark for Real-World Polymer Science Workflows

Wanhao Liu; Weida Wang; Jiaqing Xie; Suorong Yang; Jue Wang; Benteng Chen; Guangtao Mei; Zonglin Yang

cs.CVApr 3, 2026

PolyReal: A Benchmark for Real-World Polymer Science Workflows

Authors: Wanhao Liu, Weida Wang, Jiaqing Xie, Suorong Yang, Jue Wang, Benteng Chen, Guangtao Mei, Zonglin Yang,

Organizations: University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Fudan University · National University of Singapore ·

Abstract

Multimodal Large Language Models (MLLMs) excel in general domains but struggle with complex, real-world science. We posit that polymer science, an interdisciplinary field spanning chemistry, physics, biology, and engineering, is an ideal high-stakes testbed due to its diverse multimodal data. Yet, existing benchmarks related to polymer science largely overlook real-world workflows, limiting their practical utility and failing to systematically evaluate MLLMs across the full, practice-grounded lifecycle of experimentation. We introduce PolyReal, a novel multimodal benchmark grounded in real-world scientific practices to evaluate MLLMs on the full lifecycle of polymer experimentation. It covers five critical capabilities: (1) foundational knowledge application; (2) lab safety analysis; (3) experiment mechanism reasoning; (4) raw data extraction; and (5) performance & application exploration. Our evaluation of leading MLLMs on PolyReal reveals a capability imbalance. While models perform well on knowledge-intensive reasoning (e.g., Experiment Mechanism Reasoning), they drop sharply on practice-based tasks (e.g., Lab Safety Analysis and Raw Data Extraction). This exposes a severe gap between abstract scientific knowledge and its practical, context-dependent application, showing that these real-world tasks remain challenging for MLLMs. Thus, PolyReal helps address this evaluation gap and provides a practical benchmark for assessing AI systems in real-world scientific workflows.

Figures & tables

Figure 1: Overview of the PolyReal benchmark and its five-stage workflow.

Benchmark	Domain	Size	Modality	Workflow	Type
MacBench	Organic Chem.	80	Multi	$\times$	MCQ, Num.
SFE	General Chem.	285	Multi	$\times$	MCQ, EM, OQ
MatSciBench	Polymer Chem.	12	Multi	$\times$	Num., OQ
MSQA	Polymer Chem.	257	Single	$\times$	T/F, OQ
MatCha	Polymer Chem.	98	Multi	$\times$	MCQ
PolyReal	Polymer Chem.	545	Multi	✓	OQ, Rank

Table 1: Comparison with representative chemistry and materials benchmarks. MCQ: multiple-choice; Num.: numeric; EM: exact match; OQ: open question; T/F: true/false; Rank: ranking.

Figure 2: Overview of the PolyReal benchmark’s composition, comprising 545 high-quality question-answer pairs: (a) Distribution of questions across the five core workflow modules covering the full research lifecycle; and (b) Comprehensive topic coverage, detailing the distribution across eight key sub-fields of polymer science.

Figure 3: Comparison of model performance across PolyReal , evaluated by F1 score.

Metric-Specific Score Correlation	Spearman’s $\rho$
Recall ( $R$ ) Scores (Completeness)	0.95
Precision ( $P$ ) Scores (Correctness)	0.91
F1-Scores	0.94 *

Table 2: Spearman’s (

\rho

) correlation between Human Expert and Automated Evaluator scores for P, R, and F1 metrics. *

Figure 4: Qualitative examples of MLLM failure modes: (Case 1) misapplying known principles and (Case 2) misinterpreting raw spectral data.

Closed-source Models
	FKA			LSA			EMR			RDA			PAE			Overall
Model	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
O3	0.608	0.775	0.641	0.366	0.541	0.412	0.565	0.745	0.611	0.424	0.553	0.422	0.579	0.733	0.609	0.567	0.733	0.601
GPT-5	0.544	0.799	0.594	0.317	0.563	0.373	0.543	0.801	0.608	0.418	0.523	0.426	0.581	0.742	0.623	0.529	0.760	0.578
Claude-Sonnet-4.5-Thinking	0.519	0.752	0.580	0.282	0.411	0.317	0.539	0.700	0.578	0.354	0.489	0.356	0.526	0.663	0.539	0.503	0.692	0.546
Gemini-2.5-Pro	0.505	0.780	0.570	0.338	0.479	0.369	0.480	0.726	0.535	0.389	0.592	0.408	0.521	0.677	0.555	0.483	0.726	0.536

Table 3: Performance comparison of different models on various benchmarks. P , R , and F1 represent Precision, Recall, and F1-score, respectively. LSA : Lab Safety Analysis, FKA : Foundational Knowledge Application, EMR : Experiment Mechanism Reasoning, RDA : Raw Data Extraction and Analysis, PAE : Performance & Application Exploration. The best results are in bold and the second best are underlined .

Figure 5: Comparison of model performance across the eight key polymer science sub-fields in PolyReal , evaluated by F1 score.

Method	Precision ( $P$ )	Recall ( $R$ )	F1-Score ( $F_{1}$ )
Gemini-2.5-pro (w Image)	0.41	0.79	0.51
w/o Image (Text-Only)	0.36	0.71	0.45
o3 (w Image)	0.56	0.74	0.60
w/o Image (Text-Only)	0.53	0.70	0.56

Table 4: Ablation Study on the Importance of Modality. Performance comparison between the full multimodal input (”With Image”) and the ”Text-Only” (w/o Image) version.

Method	Precision ( $P$ )	Recall ( $R$ )	F1-Score ( $F_{1}$ )
GPT-4o (Closed-Book)	0.58	0.30	0.35
w/ Search Tool	0.56	0.38	0.41
GPT-4o-mini (Closed-Book)	0.50	0.25	0.29
w/ Search Tool	0.48	0.32	0.34

Table 5: Ablation Study on the Role of External Knowledge.

Closed-source Models
	Hard			Medium			Easy			Overall
Model	P	R	F1	P	R	F1	P	R	F1	P	R	F1
O3	0.458	0.503	0.430	0.544	0.672	0.561	0.621	0.853	0.686	0.567	0.733	0.601
GPT-5	0.432	0.546	0.419	0.483	0.692	0.520	0.590	0.876	0.669	0.529	0.760	0.578
Claude-Sonnet-4.5-Thinking	0.392	0.432	0.365	0.480	0.638	0.513	0.559	0.820	0.632	0.503	0.692	0.546
Gemini-2.5-Pro	0.416	0.455	0.378	0.465	0.679	0.512	0.519	0.853	0.608	0.483	0.726	0.536

Table 6: Performance comparison of different models across three difficulty levels ( Hard , Medium , Easy ) and the Overall weighted average. P , R , and F1 represent Precision, Recall, and F1-score, respectively. The best results are in bold and the second best are underlined .

Figure 6: Performance comparison across difficulty levels. F1-scores for representative models are reported across Easy, Medium, and Hard subsets. A consistent performance degradation is observed as task complexity increases, with the ”Hard” subset effectively differentiating robust state-of-the-art models (e.g., o3) from lightweight counterparts (e.g., GPT-4o-mini) which experience steeper declines.

Panel of Figure 7 — Figure 7: The standardized system prompt used for zero-shot inference across all MLLMs, enforcing a structured Chain-of-Thought reasoning process.

Panel of Figure 8 — Figure 8: The full prompt template for the Precision Evaluator. We explicitly categorize distinct types of noise (highlighted in red ) to rigorously penalize hallucinations and ensuring the Precision metric reflects true information density.

Panel of Figure 9 — Figure 9: Full prompt for the Precision Evaluator. The explicit categorization of False Positives (highlighted in red) ensures that verbose or hallucinatory content is accurately penalized.

Figure 10: Qualitative analysis of a failure case. Specific chemical substances are highlighted in blue, and identified safety hazards are highlighted in red.

Model	Precision (Avg.)	Recall (Avg.)	F1 (Avg.)
LLaVA-v1.6-Vicuna-13B	0.110	0.058	0.053
Qwen3-VL-2B-Thinking	0.213	0.196	0.166
InternVL3_5-8B	0.415	0.366	0.337
Qwen3-VL-8B-Instruct	0.543	0.345	0.366
Qwen3-VL-8B-Thinking	0.442	0.442	0.395

Table 7: Performance of additional small models on PolyReal.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery
Aug 7, 2026Taolin Han, Yuchen Zhang, Jinghang Wang +22Scientific Reasoning Scientific Discovery
1Alibaba Group · University of Chinese Academy of Sciences · Taolin Han1,3* +28
Can LLMs Predict Polymer Physics Just by Reading Synthesis and Processing Prose?
May 7, 2026Yuchu Liu, Rui Zhu, Jingwei Xiong +1Materials Property Prediction Polymer Property Prediction
Yale University · UC Davis · Indiana University Bloomington
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models
May 19, 2026Yiyang Gu, Junwei Yang, Junyu Luo +15AI for Science LLM Evaluation
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5

Aug 7, 2026J/K move · Enter open · S save

Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery

Taolin Han, Yuchen Zhang, Jinghang Wang, Yun Wu+21

1Alibaba Group · University of Chinese Academy of Sciences · Taolin Han1,3*+28

Scientific Reasoning Scientific Discovery Scientific Reasoning in Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly involved in scientific discovery, yet it remains unclear whether they can support complex real laboratory science. Here we introduce Science Edge Evaluation (SEE), a multimodal benchmark of expert-curated questions grounded in peer-reviewed literature and experimental practice in chemistry, biology, and materials science. Evaluation of 19 multimodal large language models (MLLMs) shows that even the best-performing model reaches only 48.7% accuracy. Moreover, general-purpose models outperform science-specialized models on average. In the visual-agent evaluation, the use of tools increases the best accuracy to 52.7%. Tool use can expand the information available to models, but more information does not necessarily lead to reliable scientific reasoning. The key challenge is whether models can manage tool-derived information within the boundaries of the original experimental evidence. Together, these findings reveal that current MLLMs still cannot reliably make justified and evidence-bounded inferences from experimental results, which is an essential capability in real scientific discovery. Bridging this gap requires MLLMs to transition from explaining established scientific concepts to deriving novel and evidence-based insights from experimental data.

Abstract

Multimodal Model Evaluation

Explore similar work

Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery

Can LLMs Predict Polymer Physics Just by Reading Synthesis and Processing Prose?

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models