Building and Auditing Chemical Datasets with LLMs: A General LLM-as-a-Judge Framework Applied to Homogeneous Catalytic PET Depolymerisation

Muhammad bin Javaid; Karim K. Ben Hicham; Enkhnyam Battulga; Alexander Mitsos; Martin Grohe

cs.CLNov 8, 2026

Building and Auditing Chemical Datasets with LLMs: A General LLM-as-a-Judge Framework Applied to Homogeneous Catalytic PET Depolymerisation

Authors: Muhammad bin Javaid, Karim K. Ben Hicham, Enkhnyam Battulga, Alexander Mitsos, Martin Grohe

Organizations: RWTH Aachen University, Chair of Logic and Theory of Discrete Systems, Aachen, Germany. · RWTH Aachen University, Process Systems Engineering (AVT.SVT), Aachen, Germany. · JARA-CSD, Aachen, Germany. · Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems ICE-1: Energy Systems Engineering, Jülich, Germany

Abstract

Large language models can rapidly extract structured multivariate experimental data from the chemistry literature, yet verifying thousands of generated records without ground truth remains difficult. We present a general, citation-grounded extraction and verification methodology centered on an independent "LLM-as-a-judge", applied to the literature on homogeneous catalytic poly(ethylene terephthalate) (PET) depolymerisation. In this pipeline, an extractor LLM generates schema-enforced records from research papers with citations to document chunks tagged with universally unique identifiers (UUIDs); an independent LLM judge then reads the full paper alongside each record, flags unsupported fields, and proposes cited corrections. Prompt ablations show that requiring chunk-level citations not only provides provenance down to specific paragraphs and tables but also improves extraction accuracy. Before deploying the judge at scale, we benchmarked it against manual evaluation on extractions from a hand-curated reference set of 295 ionic-liquid-catalysed PET glycolysis experiments. The judge agreed with human evaluation on 85.7% of reviewed records and was significantly more effective than a heuristic string-matching baseline at catching errors across related experiments. Deployed across 3,942 extracted records from 561 papers, the framework accepted 67.5% of them as written. Its proposed corrections concentrated on quantitative fields, where the extractor often conflated related experiments. We release the extracted dataset with the complete LLM judge audit logs, as well as the Chemistry Data Extractor Toolkit, an open-source application implementing this end-to-end pipeline in a code-free, browser-based environment for document parsing, customizable schema and prompt design, model-agnostic extraction and auditing, interactive provenance-highlighted review, and reproducible dataset bundling.

Figures & tables

Fig. 1: Overview of the large language model (LLM) extraction and auditing pipeline for homogeneous catalytic poly(ethylene terephthalate) (PET) depolymerisation. Articles identified through Scopus are retrieved as XML or PDF files, converted to Markdown and segmented into text and table chunks with universally unique identifiers (UUIDs), preserving table structure. The extraction model (GPT-5.6 Luna) receives the chunked and UUID-tagged paper, a predefined schema, domain-specific guidelines, and one labelled worked example from an open-licensed glycolysis paper, excluding the target article. Schema-constrained extraction produces JSON records containing catalyst and solvent identities, reaction conditions, reactant amounts, and reaction outcomes. Each record includes a source chunk UUID field linking the extracted information to supporting source chunks. An independent LLM judge (GPT-OSS-120B) evaluates each record against the full chunked paper, issues an accept/reject verdict and proposes corrections supported by cited text evidence and a reasoning trace. Judge validation uses an ionic liquid benchmark comprising 295 curated experiments from 24 papers, with manual assessment of 112 extracted records and comparison against a heuristic grader. The resulting dataset covers PET glycolysis, methanolysis and hydrolysis, retaining raw extractions alongside audit logs and proposed corrections.

Fig. 2: Extraction performance and automated grader validation. ( a ) Precision (

P

), recall (

R

), and

F_{1}

scores across candidate extraction models evaluated against the curated benchmark dataset. ( b ) Grader consensus across 296 benchmark records. The 81.8% slice represents mutual agreement on evaluable records (242/259, 93.4%); total agreement reaches 82.8% when accounting for three uncurated records mutually rejected by both graders. ( c ) Manual evaluation outcomes across the 51 informative disagreement pairs, where 43 cases favored the judge and only 8 favored the heuristic. ( d ) Human evaluation of judge-proposed corrections by edit type across 120 reviewed interventions, showing that 97% were either demonstrably correct (49%) or represented equally valid reporting variants (“both right”, 48%), with only 4 proposals (3%) rejected as erroneous.

Fig. 3: Dataset composition, field completeness, and LLM judge interventions. ( a ) Distribution of extracted experiments per paper. ( b ) Breakdown across depolymerisation routes, led by glycolysis (48%) and hydrolysis (25%). ( c ) Completeness by schema field. ( d ) Most frequent catalyst entries, dominated by uncatalysed baselines, zinc acetate, and simple alkaline catalysts. ( e ) Field-level correction counts for nine of the eleven fields (excl. selectivity, pressure) proposed by the LLM judge, concentrated heavily on catalyst and solvent amounts (restricted to the 3,942 homogeneous catalysis subset). ( f ) Solvent identities for the 922 unrouted records from ( b ), accounting for unconventional reaction media and omitted solvent names.

Fig. 4: Reported reaction conditions statistics and chemical consistency checks. ( a – g ) Kernel density estimation (KDE) plots of key experimental parameters broken down by depolymerisation route (blue: glycolysis; red: hydrolysis; yellow: methanolysis). Panels (a–g) cover the 4,641 route-assigned records. Each label gives how many report the field, preceded where relevant by the percentage falling inside the plotted range; this is not field completeness, which Fig. 3c provides. ( a ) Reaction temperature, exhibiting a sharp glycolysis peak at 190–200 °C corresponding to the atmospheric reflux of ethylene glycol. ( b ) Reaction time, clustered at standard bench protocol intervals. ( c ) Product yield and ( d ) PET conversion, both skewed toward 100%. ( e ) Catalyst mass, ( f ) PET feedstock mass, and ( g ) solvent mass, concentrated at standard laboratory scales. ( h ) Physical consistency check between yield and conversion against the 1:1 parity line (dashed;

\text{yield}\leq\text{conversion}

), with only 28 of 1,423 records violating the physical bound. ( i ) Bivariate density contour comparison of temperature versus yield between this work (solid blue) and the ionic liquid depolymerisation dataset by Gao et al. 9 (hatched) for the 19 articles common to both datasets (extended field comparisons in Fig. S1

{\dagger}

, ESI).

Selection stage	Articles
Candidates identified via Scopus searches	2,777
Excluded: no polymer named	$-550{}$
Excluded: no depolymerisation route named	$-381{}$
Excluded: off-topic subjects or reviews	$-271{}$
Articles passing title and abstract filter	1,575 (56.7%)
Obtained: Elsevier full-text XML	447

Table 1: From search to corpus.

Metric	Value
Articles processed	1,026
Articles yielding records	664
Articles yielding no records	362
Unparseable outputs	0
Extraction cost	$8.63
Total extracted experimental records	5,563

Table 2: Mass extraction and curation summary.

Verdict category	Records	Share
Accepted as written	2,661	67.5%
Rejected, with proposed correction	1,202	30.5%
Rejected outright	79	2.0%
Total	3,942	100.0%

Table 3: The judge’s verdicts on the released dataset.

Fig. 5: Screenshot of the interactive Chemistry Data Extractor Toolkit. The local browser-based interface enables end-to-end dataset curation and auditing without code modification. Users configure target schemas, extraction prompts, and judge rubrics directly in the settings. The dual-pane review interface links extracted records to the source paper, highlighting cited UUID chunks upon selection and allowing one-click inspection of the judge’s proposed corrections and reasoning.

LLM Information Extraction

Large language model information extraction converts unstructured documents into structured entities, relations, events, and records.

See all 245 papers

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction
Feb 16, 2026Sönke Tenckhoff, Mario Koddenbrock, Erik RodnerLLM Information Extraction Document Parsing Benchmarks
KI-Werkstatt / FB2, University of Applied Sciences Berlin
LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature
Oct 28, 2025Magdalena Lederbauer, Siddharth Betala, Valerie Gentzke +22Document Information Extraction Materials Science
Entalpic · ENSA Lyon UMR-MAP Aria · Independent Researcher +8
From Papers to Property Tables: A Priority-Based LLM Workflow for Materials Data Extraction
Apr 8, 2026Koushik Rameshbabu, Jing Luo, Ali Shargh +2Document Information Extraction LLM Information Extraction
Department of Applied Mathematics and Statistics, Johns Hopkins University · Department of Mechanical Engineering, Johns Hopkins University

Feb 16, 2026J/K move · Enter open · S save

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

Sönke Tenckhoff, Mario Koddenbrock, Erik Rodner

KI-Werkstatt / FB2, University of Applied Sciences Berlin

LLM Information Extraction Document Parsing Benchmarks Structured Output Generation LLM Evaluation LLM Prompting

Open paper AskPDF arXiv

We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.

Abstract

LLM Information Extraction

Explore similar work

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature

From Papers to Property Tables: A Priority-Based LLM Workflow for Materials Data Extraction