Are LLMs Ready for Neural-integrated Mechanistic Modeling? A Benchmark and Agentic Framework

Zihan Guan; Rituparna Datta; Mengxuan Hu; Shunshun Liu; Aiying Zhang; Prasanna Balachandran; Sheng Li; Anil Vullikanti

cs.LGFeb 20, 2026

Are LLMs Ready for Neural-integrated Mechanistic Modeling? A Benchmark and Agentic Framework

Authors: Zihan Guan, Rituparna Datta, Mengxuan Hu, Shunshun Liu, Aiying Zhang, Prasanna Balachandran, Sheng Li, Anil Vullikanti

Organizations: University of Virginia

Abstract

Large language models (LLMs) have shown promise in constructing mechanistic models from data. However, existing evaluations largely focus on simplified settings and fail to capture the complexity of real-world scientific modeling. In practice, such modeling often involves neural-integrated formulations, where a mechanistic model component and a neural network component are jointly constructed, leading to a significantly more complex search space. Motivated by this gap, we introduce the Neural-Integrated Mechanistic Modeling (NIMM) benchmark, which evaluates LLM-generated neural-integrated mechanistic models across three scientific domains. Experiments on NIMM reveal that existing LLM-based approaches struggle to effectively explore this complex space, resulting in limited search stability and solution quality. To address this challenge, we propose NIMMGen, a tree-guided agentic framework that enables diversified exploration via branch-level search and improves solutions through atomic model refinement. Extensive experiments demonstrate that NIMMGen achieves state-of-the-art performance on NIMM, significantly improving search stability and solution quality.

Figures & tables

Figure 1: Overview of the NIMM benchmark across three domains: (a) Public Health, (b) Clinical Health, and (c) Materials Science. The benchmark includes epidemic forecasting tasks, lung cancer modeling tasks, and alloy yield strength prediction. NIMM evaluates whether LLMs can construct effective neural-integrated mechanistic models across diversified scientific tasks.

Figure 2: Overview of the NIMMGen framework. The initialized skeleton model serves as the root node, and the model is iteratively refined through a search tree. For each valid node, NIMMGen uses the assembled contextual information to perform three steps: model generation by a model agent, model evaluation in the NIMM environment, and result reflection by a reflection agent. For buggy nodes whose execution fails, the resulting error message is stored to update the contextual memory.

Dataset $\downarrow$	Mode	LSTM	Transformer	Grad-Metapopulation	Zero-shot		HDTwinGen		Ours
Dataset $\downarrow$	Mode	RMSE $\downarrow$	RMSE $\downarrow$	RMSE $\downarrow$	RMSE $\downarrow$	ESR # $\uparrow$	RMSE $\downarrow$	ESR $\uparrow$	RMSE $\downarrow$	ESR $\uparrow$
COVID-Bogota	Mechanistic	$633.88_{\pm 158.06}$	$901.96_{\pm 284.58}$	$1362.89_{\pm 165.26}$	$1102.62_{\pm 476.82}$	$0.80$	$339.13_{\pm 86.53}$	$0.69_{\pm 0.18}$	$\textbf{325.22}_{\pm 82.52}$	$\textbf{0.98}_{\pm 0.01}$
COVID-Bogota	Hybrid	$633.88_{\pm 158.06}$	$901.96_{\pm 284.58}$	$1362.89_{\pm 165.26}$	$1358.24_{\pm 266.25}$	$0.40$	$765.27_{\pm 64.58}$	$0.57_{\pm 0.13}$	$\textbf{398.62}_{\pm 131.57}$	$\textbf{0.98}_{\pm 0.03}$
COVID-Medellin	Mechanistic	$586.88_{\pm 16.96}$	$552.39_{\pm 46.89}$	$743.25_{\pm 25.79}$	$908.19_{\pm 357.17}$	$0.80$	$652.72_{\pm 102.79}$	$0.62_{\pm 0.10}$	${\textbf{480.51}}_{\pm 172.47}$	$\textbf{0.85}_{\pm 0.09}$
COVID-Medellin	Hybrid	$586.88_{\pm 16.96}$	$552.39_{\pm 46.89}$	$743.25_{\pm 25.79}$	$1026.25_{\pm 262.64}$	$0.75$	$675.82_{\pm 190.89}$	$0.80_{\pm 0.13}$	$\textbf{529.65}_{\pm 207.86}$	$\textbf{0.84}_{\pm 0.11}$
Influenza-USA	Mechanistic	$86.69_{\pm 1.73}$	$30.72_{\pm 0.27}$	$117.90_{\pm 10.93}$	$20.92_{\pm 5.38}$	$0.80$	$8.78_{\pm 1.49}$	$0.89_{\pm 0.11}$	$\textbf{7.57}_{\pm 2.05}$	$\textbf{0.98}_{\pm 0.01}$

Table 1: Comparison of the effectiveness and executability in the NIMM benchmark.

Figure 3: Loss curve over iterations.

Model Type	Model Name	Influenza-USA	MRSA-Virginia
Proprietary LLM	GPT-4.1	$7.57_{\pm 2.05}$	$33.74_{\pm 0.46}$
	GPT-4o	$7.20_{\pm 1.56}$	$38.15_{\pm 5.25}$
	GPT-5-mini	$10.20_{\pm 0.37}$	$39.12_{\pm 1.07}$
Open-source LLM	Qwen2.5-32B-Coder	$24.26_{\pm 3.12}$	$57.45_{\pm 12.98}$
Open-source LLM	Qwen3-30B-Coder	$20.24_{\pm 10.08}$	$58.25_{\pm 16.91}$

Table 2: Performance comparison under different LLMs.

Figure 4: Observed and simulated epidemic trajectories under social distancing interventions.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset Name	Data Source	Observation Period	Spatial Granularity	# Locations	Temporal Granularity	# Timestamps	Disease	# Features	Target
COVID-Bogota	Colombia NIH	2020-04-12 - 2021-01-30	City-level	1	Daily	343 (d)	COVID	11	Infection Counts
COVID-Medellin	Colombia NIH	2020-04-12 - 2021-01-30	City-level	1	Daily	343 (d)	COVID	11	Infection Counts
Influenza-USA	CDC	2017-W34 - 2021-W15 1 1 1 This follows the CDC/MMWR epidemiological week	State-level	51	Weekly	191 (w)	Influenza	15	Infection Counts
MRSA-Virginia	Virginia APCD	2016-W01 - 2021-W04	Facility-level	644	Weekly	244 (w)	MRSA	20	Infection Counts

Appendix

Table 3: Statistical Description of the datasets used in the NIMM benchmark (Public Health).

Figure 5: Experimental setup. (a) Mechanistic mode. (b) Hybrid Mode.

f^{0}\sim\mathrm{LLM}_{\mathrm{init}}\!\left(C,S\right)

Appendix

Algorithm 1 Tree-Guided Search in NIMMGen

\mathcal{X}^{g}=\Phi\!\left(C,\;\mathcal{S}(v_{p}^{g}),\;\mathcal{M}^{g-1}\right)

Appendix

Algorithm 2

\mathrm{ExpandAndEvaluate}(C,v_{p}^{g},\mathcal{T}^{g-1},\mathcal{M}^{g-1})

	10	20	40	80
Influenza-USA	$13.16_{\pm 3.81}$	$11.13_{\pm 1.26}$	$7.57_{\pm 2.05}$	$6.25_{\pm 1.78}$

Appendix

Table 4: Performance Comparison under Different Iterations with the Influenza-USA dataset in mechanistic mode.

Figure 7: Performance (measured by RMSE) of NIMMGen on the Influenza-USA dataset (mechanistic mode) with different choices of

D_{max}

and

C_{max}

. The performance becomes stably good with

C_{max}\geq 3

and

D_{max}\geq 3

Depth	Parent model	Best refinement explored at this stage	Test. loss	Val loss
1	Seed models	Best initial seed: seird baseline with latent exposure, recovery, and mortality	11.79	17.03
2	seird	Add external importation into the exposed compartment $E$ to model sporadic introductions	11.63	8.95
3	seird + importation	Add a cumulative incidence compartment $C$ to track total new infections over time	8.25	7.68
4	seird + patch-specific importation	Add a hospitalization compartment $H$ to capture severe-case progression	9.06	8.89
5	seirhd + patch-specific importation	Make hospital mortality patch-specific to capture spatial heterogeneity in severe outcomes	8.27	6.23
6	Best depth-5 model	Further refinements explored: time-varying transmission, patch-specific hospitalization, direct out-of-hospital mortality, and patch-specific waning immunity; none improved over depth 5	7.97	7.54

Appendix

Table 5: Evolution of Mechanistic Model Design and Validation Loss for the Influenza-USA dataset

Figure 8: An example of the expansion tree on the Influenza-USA dataset. The number in each node denotes the validation loss. The best node is highlighted in green, and the path from the root node to this node is also highlighted.

Large Language Model-Guided Scientific Discovery

Large language model-guided scientific discovery uses AI to propose, test, and refine hypotheses, models, experiments, and designs.

See all 274 papers

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists
Jul 13, 2026Chuhan Shi, Xiaoquan Ren, Sicheng Song +3AI for Science LLM Evaluation
Southeast University · East China Normal University · Hong Kong University of Science and Technology
MolQuest: A Benchmark for Agentic Evaluation of Abductive Reasoning in Chemical Structure Elucidation
Mar 26, 2026Taolin Han, Shuang Wu, Jinghang Wang +4Abductive Reasoning in Language Models Scientific Reasoning in Language Models
Alibaba Group, Hangzhou, China
A Probabilistic Framework for LLM-Based Model Discovery
Feb 20, 2026Stefan Wahl, Raphaela Schenk, Ali Farnoud +2Large Language Model-Guided Scientific Discovery
Machine Learning in Science, University of Tübingen, Tübingen, Germany · Tübingen AI Center, Tübingen, Germany · Boehringer Ingelheim, Biberach, Germany +1

Jul 13, 2026J/K move · Enter open · S save

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

Chuhan Shi, Xiaoquan Ren, Sicheng Song, Haobo Li+2

Southeast University · East China Normal University · Hong Kong University of Science and Technology

AI for Science LLM Evaluation Scientific Reasoning

Open paper AskPDF arXiv

Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics). SDABench comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, constructed through an automated pipeline. Evaluating 15 representative LLMs, we find that models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. SDABench further provides a five-stage error analysis framework that locates where LLMs fail: more advanced models more reliably identify the relevant scope and variables, but still struggle to select appropriate analytical procedures, model variable relationships, and draw valid conclusions.

Abstract

Large Language Model-Guided Scientific Discovery

Explore similar work

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

MolQuest: A Benchmark for Agentic Evaluation of Abductive Reasoning in Chemical Structure Elucidation

A Probabilistic Framework for LLM-Based Model Discovery