cs.CLNov 8, 2026

Building and Auditing Chemical Datasets with LLMs: A General LLM-as-a-Judge Framework Applied to Homogeneous Catalytic PET Depolymerisation

Authors: Muhammad bin Javaid, Karim K. Ben Hicham, Enkhnyam Battulga, Alexander Mitsos, Martin Grohe

Organizations: RWTH Aachen University, Chair of Logic and Theory of Discrete Systems, Aachen, Germany. · RWTH Aachen University, Process Systems Engineering (AVT.SVT), Aachen, Germany. · JARA-CSD, Aachen, Germany. · Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems ICE-1: Energy Systems Engineering, Jülich, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models can rapidly extract structured multivariate experimental data from the chemistry literature, yet verifying thousands of generated records without ground truth remains difficult. We present a general, citation-grounded extraction and verification methodology centered on an independent "LLM-as-a-judge", applied to the literature on homogeneous catalytic poly(ethylene terephthalate) (PET) depolymerisation. In this pipeline, an extractor LLM generates schema-enforced records from research papers with citations to document chunks tagged with universally unique identifiers (UUIDs); an independent LLM judge then reads the full paper alongside each record, flags unsupported fields, and proposes cited corrections. Prompt ablations show that requiring chunk-level citations not only provides provenance down to specific paragraphs and tables but also improves extraction accuracy. Before deploying the judge at scale, we benchmarked it against manual evaluation on extractions from a hand-curated reference set of 295 ionic-liquid-catalysed PET glycolysis experiments. The judge agreed with human evaluation on 85.7% of reviewed records and was significantly more effective than a heuristic string-matching baseline at catching errors across related experiments. Deployed across 3,942 extracted records from 561 papers, the framework accepted 67.5% of them as written. Its proposed corrections concentrated on quantitative fields, where the extractor often conflated related experiments. We release the extracted dataset with the complete LLM judge audit logs, as well as the Chemistry Data Extractor Toolkit, an open-source application implementing this end-to-end pipeline in a code-free, browser-based environment for document parsing, customizable schema and prompt design, model-agnostic extraction and auditing, interactive provenance-highlighted review, and reproducible dataset bundling.

Figures & tables

Fig. 1: Overview of the large language model (LLM) extraction and auditing pipeline for homogeneous catalytic poly(ethylene terephthalate) (PET) depolymerisation. Articles identified through Scopus are retrieved as XML or PDF files, converted to Markdown and segmented into text and table chunks with universally unique identifiers (UUIDs), preserving table structure. The extraction model (GPT-5.6 Luna) receives the chunked and UUID-tagged paper, a predefined schema, domain-specific guidelines, and one labelled worked example from an open-licensed glycolysis paper, excluding the target article. Schema-constrained extraction produces JSON records containing catalyst and solvent identities, reaction conditions, reactant amounts, and reaction outcomes. Each record includes a source chunk UUID field linking the extracted information to supporting source chunks. An independent LLM judge (GPT-OSS-120B) evaluates each record against the full chunked paper, issues an accept/reject verdict and proposes corrections supported by cited text evidence and a reasoning trace. Judge validation uses an ionic liquid benchmark comprising 295 curated experiments from 24 papers, with manual assessment of 112 extracted records and comparison against a heuristic grader. The resulting dataset covers PET glycolysis, methanolysis and hydrolysis, retaining raw extractions alongside audit logs and proposed corrections.
Fig. 2: Extraction performance and automated grader validation. ( a ) Precision ( PP ), recall ( RR ), and F1F_{1} scores across candidate extraction models evaluated against the curated benchmark dataset. ( b ) Grader consensus across 296 benchmark records. The 81.8% slice represents mutual agreement on evaluable records (242/259, 93.4%); total agreement reaches 82.8% when accounting for three uncurated records mutually rejected by both graders. ( c ) Manual evaluation outcomes across the 51 informative disagreement pairs, where 43 cases favored the judge and only 8 favored the heuristic. ( d ) Human evaluation of judge-proposed corrections by edit type across 120 reviewed interventions, showing that 97% were either demonstrably correct (49%) or represented equally valid reporting variants (“both right”, 48%), with only 4 proposals (3%) rejected as erroneous.
Fig. 3: Dataset composition, field completeness, and LLM judge interventions. ( a ) Distribution of extracted experiments per paper. ( b ) Breakdown across depolymerisation routes, led by glycolysis (48%) and hydrolysis (25%). ( c ) Completeness by schema field. ( d ) Most frequent catalyst entries, dominated by uncatalysed baselines, zinc acetate, and simple alkaline catalysts. ( e ) Field-level correction counts for nine of the eleven fields (excl. selectivity, pressure) proposed by the LLM judge, concentrated heavily on catalyst and solvent amounts (restricted to the 3,942 homogeneous catalysis subset). ( f ) Solvent identities for the 922 unrouted records from ( b ), accounting for unconventional reaction media and omitted solvent names.
Fig. 4: Reported reaction conditions statistics and chemical consistency checks. ( a – g ) Kernel density estimation (KDE) plots of key experimental parameters broken down by depolymerisation route (blue: glycolysis; red: hydrolysis; yellow: methanolysis). Panels (a–g) cover the 4,641 route-assigned records. Each label gives how many report the field, preceded where relevant by the percentage falling inside the plotted range; this is not field completeness, which Fig. 3c provides. ( a ) Reaction temperature, exhibiting a sharp glycolysis peak at 190–200 °C corresponding to the atmospheric reflux of ethylene glycol. ( b ) Reaction time, clustered at standard bench protocol intervals. ( c ) Product yield and ( d ) PET conversion, both skewed toward 100%. ( e ) Catalyst mass, ( f ) PET feedstock mass, and ( g ) solvent mass, concentrated at standard laboratory scales. ( h ) Physical consistency check between yield and conversion against the 1:1 parity line (dashed; yield≤conversion\text{yield}\leq\text{conversion} ), with only 28 of 1,423 records violating the physical bound. ( i ) Bivariate density contour comparison of temperature versus yield between this work (solid blue) and the ionic liquid depolymerisation dataset by Gao et al. 9 (hatched) for the 19 articles common to both datasets (extended field comparisons in Fig. S1 †{\dagger} , ESI).
Selection stageArticles
Candidates identified via Scopus searches2,777
Excluded: no polymer named−550-550{}
Excluded: no depolymerisation route named−381-381{}
Excluded: off-topic subjects or reviews−271-271{}
Articles passing title and abstract filter1,575 (56.7%)
Obtained: Elsevier full-text XML447
Table 1: From search to corpus.
MetricValue
Articles processed1,026
Articles yielding records664
Articles yielding no records362
Unparseable outputs0
Extraction cost$8.63
Total extracted experimental records5,563
Table 2: Mass extraction and curation summary.
Verdict categoryRecordsShare
Accepted as written2,66167.5%
Rejected, with proposed correction1,20230.5%
Rejected outright792.0%
Total3,942100.0%
Table 3: The judge’s verdicts on the released dataset.
Fig. 5: Screenshot of the interactive Chemistry Data Extractor Toolkit. The local browser-based interface enables end-to-end dataset curation and auditing without code modification. Users configure target schemas, extraction prompts, and judge rubrics directly in the settings. The dual-pane review interface links extracted records to the source paper, highlighting cited UUID chunks upon selection and allowing one-click inspection of the judge’s proposed corrections and reasoning.
More in this topic

LLM Information Extraction

Large language model information extraction converts unstructured documents into structured entities, relations, events, and records.

See all 245 papers

Explore similar work

CardsList
Feb 16, 2026

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.

Sönke Tenckhoff, Mario Koddenbrock, Erik Rodner

KI-Werkstatt / FB2, University of Applied Sciences Berlin

LLM Information ExtractionDocument Parsing BenchmarksStructured Output GenerationLLM EvaluationLLM Prompting
AskPDF
Oct 28, 2025

LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature

Wide access to advanced experimental methods in materials science has given rise to an abundance of procedural knowledge, which is scattered across decades of scientific literature and recorded in unstructured formats that are challenging to analyze systematically. In this work, we present LeMat-Synth Parser, a modular, open-source, and multi-modal extraction toolbox that utilizes large language models (LLMs) and vision language models (VLMs) to automatically structure synthesis protocols and performance metrics extracted from both text and figures of publications. Applying LeMat-Synth Parser to 81K open-access publications, we curate LeMat-Synth, an extensive dataset of 58K synthesis procedures and to our knowledge the largest and most diverse structured inorganic materials synthesis dataset to date, covering 35 synthesis methods and 16 material classes based on a domain-specific ontology. We validate extraction quality against annotations by domain experts and a scalable LLM-as-a-judge framework, and benchmark a suite of models to identify optimal configurations and characterize cross-model biases. To demonstrate the extensibility of LeMat-Synth Parser, we apply it to two distinct domains. First, we link synthesis protocols and catalyst identity to thermocatalytic performance across a corpus of ammonia-decomposition publications. Second, we cross-validate text- and figure-reported critical transition temperatures across 1,384 superconductivity papers, then use the validated pipeline to recover the critical transition temperature for every composition in a sample series. We release LeMat-Synth Parser and the LeMat-Synth dataset openly on GitHub and Hugging Face

Magdalena Lederbauer, Siddharth Betala, Valerie Gentzke +22

Entalpic · ENSA Lyon UMR-MAP Aria · Independent Researcher +8

Document Information ExtractionMaterials ScienceLLM Information Extraction
AskPDF
Apr 8, 2026

From Papers to Property Tables: A Priority-Based LLM Workflow for Materials Data Extraction

Scientific data are widely dispersed across research articles and are often reported inconsistently across text, tables, and figures, making manual data extraction and aggregation slow and error-prone. We present a prompt-driven, hierarchical workflow that uses a large language model (LLM) to automatically extract and reconstruct structured, shot-level shock-physics experimental records by integrating information distributed across text, tables, figures, and physics-based derivations from full-text published research articles, using alloy spall strength as a representative case study. The pipeline targeted 37 experimentally relevant fields per shot and applied a three-level priority strategy: (T1) direct extraction from text/tables, (T2) physics-based derivation using verified governing relations, and (T3) digitization from figures when necessary. Extracted values were normalized to canonical units, tagged by priority for traceability, and validated with physics-based consistency and plausibility checks. Evaluated on a benchmark of 30 published research articles comprising 11,967 evaluated data points, the workflow achieved high overall accuracy, with priority-wise accuracies of 94.93% (T1), 92.04% (T2), and 83.49% (T3), and an overall weighted accuracy of 94.69%. Cross-model testing further indicated strong agreement for text/table and equation-derived fields, with lower agreement for figure-based extraction. Implementation through an API interface demonstrated the scalability of the approach, achieving consistent extraction performance and, in a subset of test cases, matching or exceeding chat-based accuracy. This workflow demonstrates a practical approach for converting unstructured technical literature into traceable, analysis-ready datasets without task-specific fine-tuning, enabling scalable database construction in materials science.

Koushik Rameshbabu, Jing Luo, Ali Shargh +2

Department of Applied Mathematics and Statistics, Johns Hopkins University · Department of Mechanical Engineering, Johns Hopkins University

Document Information ExtractionLLM Information ExtractionMaterials Science
AskPDF