RegExtract: A Benchmark for PFAS Regulatory Information Extraction with LLMs
Authors: Defne Circi, Krystelle Lionti, Sarathkrishna Swaminathan, Daniel P. Sanders, L. Catherine Brinson, Bhuwan Dhingra, Basel Shbita, Mohab Elkaref, Sara Capponi, Tim Erdmann, Brandi Ransom
Organizations: Duke University, USA · IBM Research, USA · IBM Research, UK
Extracting structured information from regulatory documents is a critical but challenging task. Relevant facts are scattered across long, legally dense texts, and traceability of extracted facts to their source is essential for verification. With increasing awareness of the environmental and health impacts of per- and polyfluoroalkyl substances (PFAS), regulatory documents have expanded significantly, varying in scope and product relevance, making interpretation challenging for manufacturers and policymakers. We introduce RegExtract, a benchmark for structured information extraction from PFAS regulatory documents, comprising 488 human-annotated field instances across 20 US state documents and 46 regulations. Each field instance includes a value and a provenance entry, yielding 976 entries in total. We define a typed, multi-field schema targeting key elements such as material definitions, permitted uses, and product exemptions, and propose provenance-aware evaluation metrics tailored to regulatory text. Through experiments with open- and closed-source LLMs across prompting strategies, document condensing methods, and schema partitioning approaches, we find that document identification fields remain challenging, and that field-specific few-shot examples improve extraction. This work lays the groundwork for AI tools for hazardous substance identification, question answering, and compliance monitoring. The code and data are available at https://github.com/IBM/PFAS-regulations.
Figures & tables
Figure 1: An example document where the highlighted text should be extracted for each field.
Figure 2: Prompting strategies for regulation extraction with LLMs.
Figure 3: These heatmaps represent the recall rougeL scores calculated on the provenance of each field outlined in the schema, in comparison to the scores on the full text. Higher scores (more blue) indicate better provenance retention. The scores averaged across all documents for each text after condensed by the top performing models from the LLM protocol with chunk size of 512 tokens (top rows) and RAG protocol with chunk size of 256 tokens and 10 chunks (bottom rows).
Figure 4: GPT-4o extraction performance on full documents under different prompting strategies. (a) Average value scores for eight regulation fields and their overall average. (b) Average value and provenance scores for six document-level fields. Scores are averaged across valid documents for each strategy; higher scores indicate better performance.
Prompt Strategy
#Valid
t (sec)
Reg. P
Reg. R
Reg. F1
Avg. Val. Score
Avg. Prov. Score
Joint Extraction (baseline)
19
27.48
1.00
0.83
0.88
0.71
0.65
Sequential Extraction
18
28.14
1.00
0.91
0.94
0.67
0.64
Separated (Regulation)
16
27.24
0.94
0.96
0.93
0.71
0.61
One-Shot
20
39.66
0.98
0.89
0.91
0.73
0.59
Few-Shot Structured
17
32.14
0.97
0.85
0.88
0.74
0.64
Separated (Document-level)
20
8.59
Table 1: Summary of evaluated prompts with gpt-4o, total valid document outputs, and average processing time (s) per configuration. Metrics include regulation-level Precision (P), Recall (R), and F1 , along with averaged value and provenance scores for regulation fields. Scores are averaged across valid documents. Field-by-field and Separated (Document-level) are not applied within regulations.
Model
#Valid
Title
Loc.
Ref.
Status
Date
PFAS Def.
llama-3-3-70b-instruct
17
0.63 / 0.68
0.94 / 0.63
0.53 / 0.64
0.65 / 0.36
0.65 / 0.71
0.88 / 0.85
llama-4-maverick
20
0.70 / 0.67
0.95 / 0.68
0.40 / 0.62
0.60 / 0.19
0.60 / 0.53
0.95 / 0.94
gpt-4o
17
0.70 / 0.74
0.88 / 0.63
0.53 / 0.65
0.65 / 0.40
0.71 / 0.55
0.94 / 0.87
granite-3.3-8b-instruct
20
0.74 / 0.53
0.60 / 0.53
0.60 / 0.31
0.40 / 0.17
0.55 / 0.50
0.30 / 0.77
mixtral-8x22B-instruct-v0.1
16
0.51 / 0.54
0.88 / 0.74
0.44 / 0.59
0.75 / 0.43
0.56 / 0.56
0.94 / 0.85
granite-4.0-h-micro
20
0.62 / 0.24
0.75 / 0.27
0.40 / 0.59
0.40 / 0.20
0.55 / 0.39
0.75 / 0.57
Table 2: Performance metrics using few-shot structured prompt for various LLMs. Each document-level field shows the value and provenance score.
Model
#Valid
Reg. ( P , R , F1 )
Reg. Value / Prov.
llama-3-3-70b-instruct
17
1.00 / 0.82 / 0.88
0.70 / 0.55
llama-4-maverick-17b-128e-instruct-fp8
20
1.00 / 0.86 / 0.91
0.68 / 0.57
gpt-4o
17
0.97 / 0.85 / 0.88
0.74 / 0.64
granite-3.3-8b-instruct
20
0.89 / 0.83 / 0.81
0.64 / 0.30
mixtral-8x22B-instruct-v0.1
16
0.97 / 0.86 / 0.89
0.69 / 0.54
granite-4.0-h-micro
20
1.00 / 0.71 / 0.78
0.62 / 0.16
Table 3: Performance of different large language models on regulation-level extraction tasks using the few-shot structured prompting approach. The table reports regulation matching precision, recall, and F1 scores, along with value and provenance evaluation metrics. Scores are averaged across valid documents. The best-performing results for each metric are highlighted in bold.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Products per regulation
Figure 6: Regulation type distribution
Figure 7: Top 5 products
Figure 8: Top 5 exceptions
Figure 9: Regulations per file
Figure 10: The length of each document using AutoTokenizer.from_pretrained("bert-base-uncased") .
Figure 11: These data points represent the difference in the text length after the text is condensed by a RAG or LLM protocol. Each box plot represents all of the documents for a single model, using various chunk size and numbers for each of the condensing protocols. The left column shows all of the RAG variations which were tested, the right column shows all of the LLM variations which are tested.
Figure 12: Regulation field evals after condensing with RAG
Figure 13: Document level field evals after condensing with RAG
Prompt Strategy
#Valid
Reg. (P,R, F1 )
Reg. Value / Prov.
Joint Extraction (baseline)
18
1.00 ,0.84,0.88
0.71 ,0.61
Separated (Regulation)
16
0.97, 0.89 ,0.90
0.68,0.61
Sequential Extraction
18
1.00 ,0.86,0.91
0.69, 0.62
One-Shot
19
0.97,0.84,0.87
0.70,0.56
Few-Shot Structured
19
1.00 ,0.88, 0.92
0.70,0.60
Appendix
Table 4: Comparison of GPT-4o prompting strategies on the RAG condensed text for PFAS regulation extraction. Metrics include regulation-level Precision (P), Recall (R), and F1 , along with averaged value and provenance scores for regulation fields.
Figure 14: Regulation field evals after condensing with LLM
Figure 15: Document level field evals after condensing with LLM
Prompt Strategy
#Valid
Reg. (P,R, F1 )
Reg. Value / Prov.
Joint Extraction (baseline)
20
1.00 ,0.78,0.85
0.67,0.61
Separated (Regulation)
16
0.97, 0.93 , 0.93
0.73 , 0.62
Sequential Extraction
17
1.00 ,0.89, 0.93
0.66,0.62
One-Shot
20
0.98,0.83,0.87
0.70,0.58
Few-Shot Structured
19
1.00 ,0.83,0.88
0.71,0.58
Appendix
Table 5: Comparison of GPT-4o prompting strategies on the LLM condensed text for PFAS regulation extraction. Metrics include regulation-level Precision (P), Recall (R), and F1 , along with averaged value and provenance scores for regulation fields.
Figure 16: These data points represent the difference in the text length after the text is condensed by a RAG or LLM protocol. Each box plot represents all of the documents for a single model, using various chunk size and numbers for each of the condensing protocols. The left column shows all of the RAG variations which were tested, the right column shows all of the LLM variations which are tested.
Figure 17: rovenance ROUGE-L recall scores relative to the full-text baseline after RAG condensing. Scores are averaged across fields and documents included in each panel. The left panel includes all 20 documents; the right includes only documents condensed to less than 95% of their original length. Cell labels indicate document counts.
Figure 18: Provenance ROUGE-L recall scores relative to the full-text baseline after LLM condensing. Scores are averaged across fields and documents included in each panel. The left panel includes all 20 documents; the right includes only documents condensed to less than 95% of their original length. Cell labels indicate document counts.
Figure 19: For each of the top performing models from the LLM protocol with chunk size of 512 tokens and RAG protocol with chunk size of 256 tokens and 10 chunks, each document length is represented by a point. The point is the percent of the document which is retained after the condensing protocol.