The limits of bio-molecular modeling with large language models : a cross-scale evaluation

Yaxin Xu; Yue Zhou; Tianyu Zhao; Fengwei An; Zhixiang Ren

cs.LGApr 3, 2026

The limits of bio-molecular modeling with large language models : a cross-scale evaluation

Authors: Yaxin Xu, Yue Zhou, Tianyu Zhao, Fengwei An, Zhixiang Ren

Organizations: Southern University of Science and Technology, Shenzhen, 518055, China. · Pengcheng Laboratory, Shenzhen, 518055, China. · Institute of Mechanics, Chinese Academy of Sciences, Beijing, 100190, China.

Abstract

The modeling of bio-molecular system across molecular scales remains a central challenge in scientific research. Large language models (LLMs) are increasingly applied to bio-molecular discovery, yet systematic evaluation across multi-scale biological problems and rigorous assessment of their tool-augmented capabilities remain limited. We reveal a systematic gap between LLM performance and mechanistic understanding through the proposed cross-scale bio-molecular benchmark: BioMol-LLM-Bench, a unified framework comprising 26 downstream tasks that covers 4 distinct difficulty levels, and computational tools are integrated for a more comprehensive evaluation. Evaluation on 13 representative models reveals 4 main findings: chain-of-thought data provides limited benefit and may even reduce performance on biological tasks; hybrid mamba-attention architectures are more effective for long bio-molecular sequences; supervised fine-tuning improves specialization at the cost of generalization; and current LLMs perform well on classification tasks but remain weak on challenging regression tasks. Together, these findings provide practical guidance for future LLM-based modeling of molecular systems.

Figures & tables

Figure 1: Overview of BioMol-LLM-Bench. The framework addresses limitations in existing benchmarks by incorporating specialized bio-chemistry tools and enabling cross-scale evaluation with 26 tasks across 4 levels. The evaluation pipeline integrates automatic answer extraction and metrics computation. 13 models are compared under different experiment settings.

Level	Task (abbr.)	Inputs	#Num	Source
L0	General_Text	Text	226	GPQA_Diamond Rein et al. () , MMLU_Pro Wang et al. (2024) , AI2ARC Olea et al. (2024)
L1	MOL_Solubility	Text, SMILES	109	AqSolDB Sorkun et al. (2019) , esol Delaney (2004)
	MOL_Freesolv	Text, SMILES	43	freesolv Mobley and Guthrie (2014)
	MOL_Syn	Text, SMILES	176	USPTO-50K Maziarz (2024)
	MOL_Resyn	Text, SMILES	176	USPTO-50K Maziarz (2024)
	MOL_Excited	Text, SMILES	200	QM8 Ramakrishnan et al. (2015)

Table 1: Summary of benchmark tasks. The table presents 26 tasks categorized into 4 hierarchical levels (L0-L3) based on input complexity. For each task, the table specifies the task abbreviation, input modalities, number of test samples (#Num), and the original data sources. SMILES and AA-Seq. represent sequence strings of molecule and protein, respectively.

Model	#Params	Open	Domain-FT	Architecture
DeepSeek-v3.1 DeepSeek-AI (2025)	685B	✓	✗	MoE
Qwen3-235b-a22b-2507 Bai et al. (2023)	235B	✓	✗	MoE
GPT-5-mini OpenAI (2025)	/	✗	✗	MoE
NatureLM-8x7B Xia et al. (2025)	56B	✓	✓	MoE
TxGemma-Chat-9B Wang et al. (2025)	9B	✓	✓	D.T.
Phi-4-14B Abdin et al. (2024)	14B	✓	✗	D.T.

Table 2: Overview of evaluated models. The table lists 13 language models evaluated, models vary in the number of parameters (#Params), open-source availability (Open), whether the model has undergone domain-specific fine-tuning (Domain-FT), and the underlying architectural type (Architecture). MoE denotes Mixture-of-Experts architecture, while D.T. refers to Dense Transformer. Parameter counts marked with ”/” indicate undisclosed model sizes.

Tasks	A	B	C	D	E	F	G	H	I	J	K	L	M
General_Text	3	11	8	5	7	10	9	1	12	13	4	6	2
PROT_EC	12	13	8	1	3	4	5	7	10	11	6	9	2
PROT_Location	11	13	7	1	5	2	4	10	6	12	3	9	8
PPI_Type	1	12	9	2	3	4	8	10	5	13	7	11	6
PPI_Binary	1	13	6	8	4	2	11	10	7	12	5	9	3
MOL_TOX	9	12	8	3	1	10	7	11	2	13	5	4	6

Table 3: Model rankings on each benchmark tasks. The overall rank is obtained by calculating the ranking of all models based on the average of each model’s rankings across all tasks. A: Phi-4-14B, B: GPT-oss-20B, C: Gemma-3-12B, D: Qwen-3-14B, E: NVIDIA-Nemotron-Nano-9B-v2, F: Mistral-Nemo-Instruct-2407, G: Llama-3.1-8B-Instruct, H: DeepSeek-R1-Distill-Qwen-14B, I: TxGemma-Chat-9B, J: NatureLM-8x7B, K: Qwen3-235b-a22b-2507, L: GPT-5-mini, M: DeepSeek-v3.1.

Figure 2: Performance of 13 LLMs among 8 classification tasks. The prediction accuracy are displayed with bars based on the left axis. The markers represent model output validity based on the right axis. Larger-parameter models rank among the top performers in overall accuracy.

Figure 3: The distributions of prediction results on eleven regression tasks. By calculating the difference between each model’s predictions and the ground-truth labels across different samples for each task, the mean and variance are computed, from which the boxplot in each subplot is generated.

Figure 4: Model performance on generation tasks. Higher values indicate better model performance. a). For MOL_Syn and MOL_Resyn tasks, fingerprint similarities of the prediction results were calculated using three methods: MACCS, MORGAN, and RDKIT. These values are displayed as bars corresponding to the left y-axis. The markers that represent the validity of the output results correspond to right y-axis. b). The results for PROT_GO and DDI_Interact were evaluated with Levenshtein similarity and Meteor similarity metric, respectively.

Figure 5: LLM performance with and without tool integration across 4 representative tasks. (left) For regression-based tasks (MOL_Solubility and MOL_Freesolv), performance is evaluated using RMSE, where lower value bars indicate better performance. The dashed bars indicate values exceeding the coordinate range, and the actual value is displayed with white numerical text. (right) For classification tasks, performance is measured using accuracy. The markers represent model output validity.

Figure 6: Model behavior under different input prompt complexity and think mode. Performance is evaluated with accuracy. Detailed prompt with more domain knowledge guidance generally exhibits better performance, and model performance under think mode varies depending on the task. NVIDIA-Nemotron-Nano-9B-v2 shows higher accuracy and sensitivity to different experiment settings.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Level	Task (abbr.)	Description
L0	General_Text	Domain knowledge understanding with text-input only.
L1	MOL_Solubility	The solubility of a molecule refers to its ability to dissolve in a solvent (typically water) to form a homogeneous solution.
	MOL_Freesolv	The solvation free energy of a molecule is a thermodynamic measure of how favorably a molecule dissolves in a solvent (typically water).
	MOL_Syn	Predict the forward synthesis production of two molecules.
	MOL_Resyn	Given production, predict the reactant molecules.
	MOL_Excited	E1 property of a moleculer refers to the energy of lowest singlet excited state relative to the ground state. Predict the energy of lowest singlet excited state relative to ground state of molecule in eV units.

Appendix

Table 4: *

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering
Sep 24, 2026Shun Ye, Vinny Chandran Suja, Chenlong Li +60LLM Evaluation Scientific Reasoning in Language Models
Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design
Apr 17, 2026Shriram Chennakesavalu, Kirill Shmilovich, Hayley Weir +5LLM Evaluation Molecular Property Prediction
Prescient Design, Genentech, South San Francisco, CA, USA
Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models
Sep 4, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation LLM Reliability
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3

Sep 24, 2026J/K move · Enter open · S save

BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering

Shun Ye, Vinny Chandran Suja, Chenlong Li, Chongming Jiang+59

LLM Evaluation Scientific Reasoning in Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing LLM benchmarking emphasizes factual recall, offering limited insight into model performance on frontier and multimodal tasks. We assembled BioEVAL (BioEngineering Validation of AI and LLMs), a global, multi-institutional initiative designed to assess experimental reasoning capability across bioengineering (BE) subfields. BioEVAL spans 11 major BE subfields plus a set of uncategorized items, bringing together 22 research groups to create a PhD-level benchmark comprising 608 evaluation items: 1) 380 multiple-choice questions (MCQs, 359 retained after audit), 2) 218 literature synthesis tasks, and 3) 10 multimodal problems with experimental image interpretation. Benchmark items underwent authoring-group expert review and centralized quality control before evaluation. Following evaluation, a blinded cross-group consensus audit of the highest- and lowest-accuracy MCQ items flagged 21 questions for revision or removal; these were withheld, and all reported MCQ results are computed on the 359 retained items. We evaluated diverse cloud-scale foundation/multimodal models (e.g., ChatGPT, Gemini, and Grok) and locally deployable models suitable for inference on consumer-grade GPUs. Models achieved the highest accuracy of up to 90% on MCQs, similarity score of 0.72 on literature synthesis, and accuracy of 80% on a small sample of multimodal reasoning questions, with substantial performance variation across subfields. Leaderboard rankings characterize current capabilities, limitations, and development priorities across the evaluated BE task categories. BioEVAL is maintained as an extensible benchmark with standardized protocols for continuing expert item contribution and model evaluation.

Abstract

LLM Evaluation

Explore similar work

BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering

Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design

Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models