Chem4DLLM: 4D Multimodal LLMs for Chemical Dynamics Understanding

Xinyu Li; Zhen Zhang; Qi Chen; Anton van den Hengel; Lina Yao; Javen Qinfeng Shi

cs.LGMar 12, 2026

Chem4DLLM: 4D Multimodal LLMs for Chemical Dynamics Understanding

Authors: Xinyu Li, Zhen Zhang, Qi Chen, Anton van den Hengel, Lina Yao, Javen Qinfeng Shi

Organizations: Australian Institute for Machine Learning, Adelaide University, Adelaide, Australia · CSIRO’s Data61, Sydney, Australia

Abstract

Existing chemical understanding tasks primarily rely on static molecular representations, limiting their ability to model inherently dynamic phenomena such as bond breaking or conformational changes, which are essential for a chemist to understand chemical reactions. To address this gap, we introduce Chemical Dynamics Understanding (ChemDU), a new task that translates 4D molecular trajectories into interpretable natural-language explanations. ChemDU focuses on fundamental dynamic scenarios, including gas-phase and catalytic reactions, and requires models to reason about key events along molecular trajectories, such as bond formation and dissociation, and to generate coherent, mechanistically grounded narratives. To benchmark this capability, we construct Chem4DBench, the first dataset pairing 4D molecular trajectories with expert-authored explanations across these settings. We further propose Chem4DLLM, a unified model that integrates an equivariant graph encoder with a pretrained large language model to explicitly capture molecular geometry and rotational dynamics. We hope that ChemDU, together with Chem4DBench and Chem4DLLM, will stimulate further research in dynamic chemical understanding and multimodal scientific reasoning.

Figures & tables

Figure 1: Comparison between static 3D molecular captioning and our proposed 4D molecular understanding task. (Top) Previous 3D methods utilize a static point cloud input $N\times 3$ to identify the molecule, such as Cyclohex-2-enone. (Bottom) Our 4D approach processes a temporal sequence of point clouds $T\times N\times 3$ , enabling the model to describe dynamic chemical events. In this example, the model correctly identifies the trajectory of a C-O bond breaking within the Cyclohex-2-enone molecule, noting that the process initiates at $t=3$ and completes by $t=5$ .

Figure 2: Overview of the Chem4D benchmark. This benchmark is a suit which encompasses three distinct categories: (1) Reaction Product Prediction, involving the analysis of bond breaking/forming events and reaction barriers (derived from Transition1x and RGD1); (2) Catalytic Reaction, covering complex surface interactions such as desorption (derived from OC20). For each category, the figure illustrates the workflow from the user query and 4D input, through the identification of key physical events, to the generation of the final scientific narrative.

Figure 3: The Chem4DLLM model architecture. (1) A 4D equivariant graph encoder (UMA) processes each 3D frame $\mathcal{X}_{t}$ into a graph embedding; (2) A projector transforms the graph embeddings into vectors that are additively fused with the embeddings of the corresponding special <graph> tokens; (3) The language model (Qwen3-8B) takes the resulting embedding sequence $\mathbf{E}$ as a prefix and autoregressively generates the output.

Model	BLEU $\uparrow$	EXACT $\uparrow$	LEVENSHTEIN $\downarrow$	MACCS $\uparrow$	RDK $\uparrow$	MORGAN $\uparrow$	VALIDITY $\uparrow$	MAE Barrier $\downarrow$	MAE Enthalpy $\downarrow$
Transition1x In Distribution
3D-MoLM	0.216	0.075	10.385	0.296	0.258	0.168	0.514	3.157	3.618
3D-MolT5	0.431	0.070	4.566	0.514	0.400	0.240	0.992	0.902	1.392
Chem3D LLM	0.059	0.000	8.347	0.206	0.124	0.053	0.954	1.262	2.064
4D-MolT5	0.480	0.131	4.381	0.568	0.477	0.308	0.990	0.900	1.265
4D Text-based	0.051	0.000	8.318	0.200	0.119	0.049	0.957	1.552	2.204

Table 1: Reaction product prediction results on Transition1x and RDG1. We evaluate models on the task of predicting product SMILES from reaction trajectories. BLEU measures sequence-level similarity between predicted and ground-truth SMILES. EXACT and Levenshtein distance assess string-level accuracy, while MACCS, RDK, and Morgan report fingerprint-based structural similarity. VALIDITY measures the fraction of chemically valid predictions, and MAE of reaction barriers and enthalpies evaluates energetic consistency. Results are reported for in-distribution (ID) and multiple out-of-distribution (OOD) settings with unseen reactants, products, or both. Overall, Chem4DLLM achieves strong performance across string, structural, and energetic metrics, particularly in challenging OOD regimes.

Model	Type Acc $\uparrow$	Ads Exact $\uparrow$	Ads Morgan $\uparrow$	Prod Exact $\uparrow$	Prod Morgan $\uparrow$	TS Step MAE $\downarrow$	Barrier MAE $\downarrow$	Enthalpy MAE $\downarrow$
Chem3D	0.517	0.355	0.392	0.054	0.064	2.437	0.985	1.069
4D text-based	0.535	0.373	0.410	0.068	0.084	1.947	1.019	1.102
Chem4DLLM	0.774	0.762	0.776	0.432	0.454	1.348	0.848	1.019

Table 2: Evaluation results on the catalytic reaction subset. Metrics cover reaction type classification (Type Acc), inital adsorbate and reaction product SMILES prediction using exact match and Morgan fingerprint similarity, transition-state timestep prediction (TS Step MAE), and MAEs for the predicted reaction barriers and enthalpies. Chem4DLLM consistently outperforms both 4D text-based method and Chem3DLLM baselines across both structural and energetic metrics, highlighting the importance of explicit spatio-temporal modeling for reaction dynamics.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Statistical distribution of Reaction Product Prediction in the Chem4D benchmark. (a) The distribution of the number of atoms. (b) The distribution of the reaction barrier (eV). (c) The distribution of the reaction enthalpy (eV).

Figure 5: Statistical distribution of Catalytic Reaction Understanding in the Chem4D benchmark. (a) The distribution of the number of atoms. (b) The distribution of the reaction barrier (eV). (c) The distribution of the reaction enthalpy (eV).

Chem3DLLLM-style (3D Input)

4D-Text-based (Trajectory Input)

Given the reactant and transition state geometries, predict the product SMILES, reaction barrier and reaction enthalpy. The geometry is as follows: The system is non-periodic. The atom and its position are as follows: C1 -1.6169 0.5542 0.9918 C2 -2.2864 -0.7707 1.3265 C3 -1.4256 0.7574 -0.4801 ... H21 1.0702 -0.4440 -2.0669 The distance between atoms is as follows: atom1_index atom2_index distance(A) 1 7 1.0930 1 6 1.0964 ... 21 19 2.3363

Given the reactant and transition state geometries, predict the product SMILES, reaction barrier and reaction enthalpy. The geometry is as follows: Frame 1: The system is non-periodic. The atom and its position are as follows: C1 -1.62 0.55 0.99 C2 -2.29 -0.77 1.33 ... The distance between atoms is as follows: atom1_index atom2_index distance(A) 1 7 1.09 1 6 1.10 ... 21 19 2.34 Frame 2: The atom with big movements are listed as follows: C1 0.77 1.22 -0.69 C2 2.26 1.20 -0.37 ... H21 -1.48 -1.01 1.37 New edge distances: atom1_index atom2_index distance(A) 1 7 1.09 1 6 1.09 ... 21 19 2.25

Appendix

Table 3: Example inputs for Chem3D-style (3D) and 4D-text-based models. Both settings aim to predict the product SMILES, reaction barrier, and reaction enthalpy, but differ in how geometric information is provided.

Figure 6: Sample of Reaction Product Prediction

Figure 7: Sample of Catalytic Reaction Understanding

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning
Oct 12, 2026Aotian Chen, Frazier N. Baker, Reza Averly +5Scientific Reasoning Explainability Evaluation
The Ohio State University · Adelaide University
ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding
Apr 8, 2026Xuanle Zhao, Xinyuan Cai, Xiang Cheng +2Vision-Language Models VLM Reasoning
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
RxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature
Oct 23, 2026Jiaxing Sun, Junyuan Gao, Ruikang Zhang +16Multimodal Dataset Document Information Extraction
Shanghai Artificial Intelligence Laboratory · Peking University · South China Normal University +7

Oct 12, 2026J/K move · Enter open · S save

ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning

Aotian Chen, Frazier N. Baker, Reza Averly, Shan Zhang+4

The Ohio State University · Adelaide University

Scientific Reasoning Explainability Evaluation Scientific QA Multimodal Reasoning

Open paper AskPDF arXiv

Current chemistry benchmarks primarily evaluate outcome prediction or reaction pathway generation, but rarely test whether a model can explain why an observed chemical effect occurs. We introduce ChemExplain, a benchmark of 1,180 multiple-choice chemistry questions drawn from 383 peer-reviewed papers. Each question pairs a molecular or reaction image with an observed chemical effect and four candidate explanations, each associated with a different structural feature visible in the image. Answering requires the model to perceive the depicted structure, apply chemical knowledge, and identify which feature most directly explains the effect, a capability we term explanatory attribution. The correct feature-mechanism relation is derived from supporting evidence in the source paper rather than invented by the question-generation LLM. We construct the benchmark automatically with a multi-agent framework that applies progressive quality control at three stages and validate it with eight chemistry domain experts. The strongest evaluated model reaches 78.6% accuracy, leaving considerable room for improvement. On a 600-question expert-reviewed subset, reasoning-trace analysis classifies 71.4% of the strongest model's errors as attribution failures. Attribution is also the largest error category for the open-source and chemistry-specific models included in our analysis.

Abstract

Multimodal Large Language Models

Explore similar work

ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning

ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding

RxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature