MOF-VERIFY: A Failure-Aware Agentic Harness for MOF Hypothesis Verification

Donghyun Lee; Taehoon Lee; Geonhee Ahn; Jieun Kim; Jihyun Park; Suyeon Cho; Yoona Kim; Chaerim Shin

cs.AIOct 2, 2026

MOF-VERIFY: A Failure-Aware Agentic Harness for MOF Hypothesis Verification

Authors: Donghyun Lee, Taehoon Lee, Geonhee Ahn, Jieun Kim, Jihyun Park, Suyeon Cho, Yoona Kim, Chaerim Shin,

Organizations: Institute for Multiscale Matters and System (IMMS), Ewha Womans University, Seoul, Republic of Korea · Lymeric, Seoul, Republic of Korea

Abstract

Large language models are increasingly used as reasoning components in AI-driven materials Co-Scientists, yet the reliability of the resulting verification pipeline remains unclear. Metal-organic frameworks (MOFs) provide a particularly challenging setting because structures may appear under different identifiers, synthesis outcomes depend strongly on experimental conditions, evidence is distributed across heterogeneous sources, and some hypotheses require computation rather than literature alone. We introduce a diagnostic benchmark with four task families covering structural grounding, synthesis-condition verification, evidence-sufficiency verification, and MLIP-based computational verification. T-MOF-1-3 are evaluated under closed-book, retrieval-enabled, and oracle-evidence settings to localize failures in knowledge access, evidence acquisition, and reasoning, while T-MOF-4 separately evaluates computational verification. Guided by these diagnosed failure modes, we develop MOF-Verify, a failure-aware agentic harness that targets structural, literature, evidence-sufficiency, and computational bottlenecks before producing a final verdict. Across multiple backbone LLMs, MOF-Verify substantially improves hypothesis-verification performance over direct inference and retrieval-based baselines. Benchmark datasets are released at https://github.com/IMMS-Ewha/MOF-Verify-Benchmark.

Figures & tables

Task	Items	Controls	Abstention gold	Evidence anchor
T-MoF-1	276	130 (47%)	6	CSD refcode
T-MoF-2	869	425 (49%)	53	DOI + verbatim span
T-MoF-3	117	74 (63%)	18	DOI + decisive condition
T-MoF-4	8	0 (0%)	5	MLIP calc. spec. hash
Total	1,270	629 (50%)	82

Table 1: Composition of the T-MOF-1–4 diagnostic benchmark. We report the number of items, adversarial control cases, abstention-labeled examples, and evidence anchors for each task family.

Family	Model	Dataset	Closed	RAG	Oracle	$\boldsymbol{\Delta}_{\textbf{access}}$	$\boldsymbol{\Delta}_{\textbf{retrieval}}$	Ours
OpenAI	gpt-5.4	T-MOF-1	11.09	55.99	82.86	44.89	26.87	78.31
		T-MOF-2	24.47	51.76	60.96	27.29	9.20	50.00
		T-MOF-3	58.09	43.19	64.52	-14.90	21.33	54.40
		T-MOF-4	16.67	–	–	–	–	56.97
	gpt-5.6-terra	T-MOF-1	19.39	62.39	80.64	43.00	18.25	78.31
		T-MOF-2	39.70	53.19	62.39	13.50	9.20	50.78

Table 2: Macro-F1 across T-MOF-1–3 under Closed-book, Retrieval-enabled, and Oracle-evidence settings, and T-MOF-4 under the Closed-book setting, with MOF-Verify (Ours) for comparison. (Note: T-MOF-1 and T-MOF-4 in MOF-Verify are LLM-independent rule-based tasks, yielding identical Macro-F1 across models ( 78.31 and 56.97 , respectively), which we report for completeness. The best result among Closed, RAG, Oracle, and Ours is shown in bold .)

Family	Model	Dataset	Metric	Closed	RAG	Oracle	Average
OpenAI	gpt-5.4	T-MOF-1	$R_{\mathrm{struct}}$	98.18	98.99	92.31	96.49
		T-MOF-2	$R_{\mathrm{synth}}$	99.26	93.18	91.03	94.49
		T-MOF-3	$R_{\mathrm{evidence}}$	60.00	57.69	57.14	58.28
		T-MOF-3	$R_{\mathrm{over}}$	88.89	72.22	88.24	83.12
	gpt-5.6-terra	T-MOF-1	$R_{\mathrm{struct}}$	97.47	100.00	76.92	91.46
		T-MOF-2	$R_{\mathrm{synth}}$	97.64	91.42	88.78	92.61

Table 3: Task-specific failure rates among incorrect predictions (unit: %). Average denotes the unweighted macro-average across Closed, RAG, and Oracle. Cases quarantined because of unresolved annotation or gold-integrity issues are excluded from the corresponding denominators.

Figure 1: Workflow of MOF-Verify . An input hypothesis is processed through IR, SE, LE, ES, and CE modules, whose provenance-aware outputs are integrated by a deterministic verdict router to produce the final output verdict or abstention.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Model	Dataset	Closed			RAG			Oracle
			Yes	No	Uncertain	Yes	No	Uncertain	Yes	No	Uncertain
OpenAI	gpt-5.4	T-MOF-1	0.00	31.62	1.67	66.67	85.40	15.89	90.70	91.23	66.67
		T-MOF-2	34.98	25.10	13.33	61.85	74.20	19.23	73.78	81.84	27.27
		T-MOF-3	76.92	80.67	16.67	33.33	69.57	26.67	88.31	86.21	19.05
	gpt-5.6-terra	T-MOF-1	0.00	58.18	0.00	78.79	87.88	20.51	96.68	95.24	50.00
		T-MOF-2	52.55	46.33	20.22	69.02	76.28	14.29	78.98	84.49	23.70

Appendix

Table 4: Class-wise F1 across T-MOF-1–3 under Closed-book, Retrieval-enabled, and Oracle-evidence settings.

Family	Model	Dataset	MOF-Verify (Ours)
			Yes	No	Uncertain
OpenAI	gpt-5.4	T-MOF-2	59.65	69.74	20.62
	gpt-5.4	T-MOF-3	62.07	63.04	38.10
	gpt-5.6-terra	T-MOF-2	58.35	72.60	21.39
	gpt-5.6-terra	T-MOF-3	70.97	63.74	39.51
	gpt-5.6-sol	T-MOF-2	63.57	75.95	24.36

Appendix

Table 5: Class-wise F1 of MOF-Verify (Ours) on T-MOF-2–3.

Hypothesis (T-MOF-2). “In Cryst. Growth Des. 2015 (DOI 10.1021/acs.cgd.5b00140), HKUST-1 coating B was prepared. Is the reported route: a Cu anode + benzene-1,3,5-tricarboxylic acid in ethanol/H 2 O, heated at 99 ∘ C?”
Stage	Input	Output (typed, with provenance)
LE	the cited DOI	document digest verified, then one temperature-zero extraction over body and SI: a schema-validated record reading 49 ∘ C , ethanol/H 2 O, with the sentence it came from
compare	claimed vs. extracted	temperature contradicted ( $\|99-49\|=50\,^{\circ}\mathrm{C}$ , against a $\pm 5\,^{\circ}\mathrm{C}$ tolerance); solvent agrees
VR	the evidence bundle	No , citing the paper and the quoted sentence

Appendix

Figure 2: One hypothesis traced through the harness. When a stage cannot ground its step—for example, because of an unrecognized material, unverifiable file, missing literature record, or insufficient evidence—it emits a typed failure rather than silently passing unsupported information forward. VR then returns the corresponding verdict instead of forcing an unsupported conclusion.

Diagnosed failure	MOF consequence	Component
Identity resolution	wrong material identity	IR
Structure acquisition	missing or untrusted deposited structure	SE
Literature acquisition	missing or misread synthesis conditions	LE
Evidence sufficiency	debatable or conditionally correct claim	ES
Verdict routing	unsupported or over-confident verdict	VR (compare)
Missing computational evidence	unresolved computational claim	CE

Appendix

Table 6: Failure classes and the harness component that addresses each.

Scientific Reasoning in Language Models

Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.

See all 132 papers

Chemically Meaningful Textualization Enables Explainable Validation of Metal-Organic Frameworks by Large Language Models
Aug 11, 2026Guobin Zhao, Xiao-Yan LiLLM Evaluation LLM Interpretability
Department of Chemistry, National University of Singapore, 3 Science Drive 3, Singapore 117543, Singapore
MOF-Sleuth: Tool-Grounded Reward Alignment for Explainable Fine-Grained MOF CIF Auditing
Jul 22, 2026Yu Liu, Zhiwei Yang, Diandian Guo +7LLM Auditing Materials Science
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China +2
Interpretable Inverse Design of Metal-Organic Frameworks with Large Language Model Agents
Jun 28, 2026Kyungmin Nam, Seunghee Han, Jihan KimLarge Language Model-Guided Optimization Large Language Model-Guided Scientific Discovery
Department of Chemical and Biomolecular Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea.

Aug 11, 2026J/K move · Enter open · S save

Chemically Meaningful Textualization Enables Explainable Validation of Metal-Organic Frameworks by Large Language Models

Guobin Zhao, Xiao-Yan Li

Department of Chemistry, National University of Singapore, 3 Science Drive 3, Singapore 117543, Singapore

LLM Evaluation LLM Interpretability Materials Science Molecular Representation Learning

Open paper AskPDF arXiv

Computation-ready metal-organic framework (MOF) databases are essential for high-throughput screening, yet many reported crystal structures remain chemically unreasonable or disordered, compromising simulation fidelity. Existing validation approaches can identify non-computation-ready structures, but they often rely on heuristic rules, license requirement, or offer limited interpretability. Here, we show that large language models (LLMs) can serve as interpretable validators of MOF structures when crystallographic information is transformed into chemically meaningful text. By benchmarking nine descriptors, we find that successful LLM-based validation depends not on the amount of structural information alone, but on whether local coordination, framework connectivity, and chemical context are organized into a linguistically learnable representation. Fine-tuned LLMs using specialized descriptors (mof2text) achieve performance comparable to graph-based models in identifying unreasonable MOFs. Importantly, these models extend beyond black-box classification by generating diagnostic rationales for likely error sources, including abnormal bonding, connectivity, and charge states, as well as error-category predictions for annotated datasets. This work establishes chemically informed textualization as the key step that transforms LLMs from generic text models into practical and explainable tools for curating MOF databases.

Abstract

Scientific Reasoning in Language Models

Explore similar work

Chemically Meaningful Textualization Enables Explainable Validation of Metal-Organic Frameworks by Large Language Models

MOF-Sleuth: Tool-Grounded Reward Alignment for Explainable Fine-Grained MOF CIF Auditing

Interpretable Inverse Design of Metal-Organic Frameworks with Large Language Model Agents